tonym/NastyNAS
1
0
Fork 0
Code Issues 70 Pull Requests Actions Packages Projects Releases Wiki Activity

Configure LocalAI API Server #33

New Issue
Open
opened 2025-09-14 08:10:18 +00:00 by tonym · 0 comments
tonym commented 2025-09-14 08:10:18 +00:00
Owner
Copy Link

LocalAI API Server Configuration for Unraid 7.0.1

Container Installation

  • From Community Applications: Search "LocalAI"
  • Container Selection: localai/localai
  • Repository: quay.io/go-skynet/local-ai:latest

Container Configuration

Basic Settings:

  • Container Port: 8080
  • Host Port: 8080
  • Network Mode: bridge
  • Repository Path: quay.io/go-skynet/local-ai:latest

Volume Mappings:

  • Models: /mnt/user/appdata/localai/models:/models
  • Config: /mnt/user/appdata/localai/config:/config
  • Data: /mnt/user/appdata/localai/data:/tmp

Environment Variables:

  • PUID: 99
  • PGID: 100
  • TZ: America/Chicago
  • UMASK: 002
  • MODELS_PATH: /models
  • DEBUG: true
  • THREADS: 4
  • CONTEXT_SIZE: 2048
  • F16: true

Initial Configuration

  1. First Access: http://[SERVER_IP]:8080
  2. Model Installation:
    • Access model gallery
    • Download recommended models
    • Verify model loading
  3. API Testing: Test OpenAI-compatible endpoints

Service Configuration

Model Management:

  • Model Gallery: Built-in model repository
  • Custom Models: Place in /models directory
  • Model Formats: GGUF, GGML supported
  • Auto-loading: Configure model preloading

Performance Settings:

  • Thread Count: Match CPU cores (4-8)
  • Context Size: 2048-4096 tokens
  • Memory Usage: Monitor RAM consumption
  • GPU Support: Enable if available

API Configuration:

  • OpenAI Compatibility: Full API compatibility
  • Authentication: Optional API key
  • Rate Limiting: Configure as needed
  • CORS: Enable for web applications

Model Recommendations

Text Generation:

  • llama2-7b-chat: General purpose chat
  • codellama-7b: Code generation
  • mistral-7b: Efficient performance

Embeddings:

  • all-MiniLM-L6-v2: Fast, good quality
  • gte-base: Higher quality embeddings

Image Generation:

  • stable-diffusion: Image generation
  • controlnet: Guided generation

Integration Points

AnythingLLM Connection:

Other AI Applications:

  • OpenAI API: Drop-in replacement
  • LangChain: Native support
  • Custom Applications: RESTful API

Monitoring & Notifications

Uptime Kuma Integration:

Performance Monitoring:

  • GPU Usage: Monitor if GPU enabled
  • Memory Usage: Track RAM consumption
  • Response Times: Monitor API latency
  • Model Loading: Track model initialization

Backup Configuration

Kopia Integration:

  • Backup Paths:
    • /mnt/user/appdata/localai/config (configurations)
    • /mnt/user/appdata/localai/models (downloaded models)
  • Schedule: Weekly (models are large)
  • Retention: 4 weekly, 6 monthly

Critical Files:

  • Model Files: Downloaded model binaries
  • Configuration: YAML config files
  • Custom Models: User-uploaded models

Testing Procedures

Health Checks:

  1. Container Status: docker ps | grep localai
  2. API Accessibility: curl http://[SERVER_IP]:8080/v1/models
  3. Model Loading: Check available models

Functionality Tests:

  1. Text Generation: Test chat completions endpoint
  2. Embeddings: Test embedding generation
  3. Model Management: Test model loading/unloading
  4. Performance: Benchmark response times

Integration Tests:

  1. OpenAI Compatibility: Test with OpenAI clients
  2. AnythingLLM: Verify integration works
  3. API Endpoints: Test all supported endpoints

Troubleshooting

Common Issues:

  1. Model Loading Failed:

    • Check model file integrity
    • Verify file permissions
    • Monitor memory usage
    • Review container logs

  2. High Memory Usage:

    • Reduce context size
    • Limit concurrent requests
    • Use smaller models
    • Enable model unloading

  3. Slow Response Times:

    • Increase thread count
    • Enable GPU acceleration
    • Optimize model selection
    • Check system resources

Permission Fixes:

sudo chown -R 99:100 /mnt/user/appdata/localai
sudo chmod -R 755 /mnt/user/appdata/localai

Log Locations:

  • Container Logs: docker logs localai
  • Application Logs: /mnt/user/appdata/localai/logs
  • System Logs: /var/log/syslog

Maintenance

Regular Tasks:

  1. Model Updates: Check for model updates monthly
  2. Storage Cleanup: Remove unused models
  3. Performance Tuning: Optimize settings
  4. Security Updates: Update container regularly

Performance Optimization:

  1. Memory Management: Monitor and adjust limits
  2. Model Selection: Use appropriate model sizes
  3. Threading: Optimize thread count
  4. Caching: Enable response caching

Storage Management:

  1. Model Storage: Monitor disk usage
  2. Cache Cleanup: Clear temporary files
  3. Log Rotation: Manage log file sizes
  4. Backup Strategy: Balance backup size vs importance
# LocalAI API Server Configuration for Unraid 7.0.1 ## Container Installation - **From Community Applications:** Search "LocalAI" - **Container Selection:** localai/localai - **Repository:** quay.io/go-skynet/local-ai:latest ## Container Configuration **Basic Settings:** - **Container Port:** 8080 - **Host Port:** 8080 - **Network Mode:** bridge - **Repository Path:** quay.io/go-skynet/local-ai:latest **Volume Mappings:** - **Models:** /mnt/user/appdata/localai/models:/models - **Config:** /mnt/user/appdata/localai/config:/config - **Data:** /mnt/user/appdata/localai/data:/tmp **Environment Variables:** - **PUID:** 99 - **PGID:** 100 - **TZ:** America/Chicago - **UMASK:** 002 - **MODELS_PATH:** /models - **DEBUG:** true - **THREADS:** 4 - **CONTEXT_SIZE:** 2048 - **F16:** true ## Initial Configuration 1. **First Access:** http://[SERVER_IP]:8080 2. **Model Installation:** - Access model gallery - Download recommended models - Verify model loading 3. **API Testing:** Test OpenAI-compatible endpoints ## Service Configuration **Model Management:** - **Model Gallery:** Built-in model repository - **Custom Models:** Place in /models directory - **Model Formats:** GGUF, GGML supported - **Auto-loading:** Configure model preloading **Performance Settings:** - **Thread Count:** Match CPU cores (4-8) - **Context Size:** 2048-4096 tokens - **Memory Usage:** Monitor RAM consumption - **GPU Support:** Enable if available **API Configuration:** - **OpenAI Compatibility:** Full API compatibility - **Authentication:** Optional API key - **Rate Limiting:** Configure as needed - **CORS:** Enable for web applications ## Model Recommendations **Text Generation:** - **llama2-7b-chat:** General purpose chat - **codellama-7b:** Code generation - **mistral-7b:** Efficient performance **Embeddings:** - **all-MiniLM-L6-v2:** Fast, good quality - **gte-base:** Higher quality embeddings **Image Generation:** - **stable-diffusion:** Image generation - **controlnet:** Guided generation ## Integration Points **AnythingLLM Connection:** - **API URL:** http://[SERVER_IP]:8080/v1 - **Model Selection:** Configure in AnythingLLM - **API Key:** Set if authentication enabled **Other AI Applications:** - **OpenAI API:** Drop-in replacement - **LangChain:** Native support - **Custom Applications:** RESTful API ## Monitoring & Notifications **Uptime Kuma Integration:** - **Monitor Type:** HTTP - **URL:** http://[SERVER_IP]:8080/v1/models - **Check Interval:** 60 seconds - **Alert Threshold:** 3 failures **Performance Monitoring:** - **GPU Usage:** Monitor if GPU enabled - **Memory Usage:** Track RAM consumption - **Response Times:** Monitor API latency - **Model Loading:** Track model initialization ## Backup Configuration **Kopia Integration:** - **Backup Paths:** - /mnt/user/appdata/localai/config (configurations) - /mnt/user/appdata/localai/models (downloaded models) - **Schedule:** Weekly (models are large) - **Retention:** 4 weekly, 6 monthly **Critical Files:** - **Model Files:** Downloaded model binaries - **Configuration:** YAML config files - **Custom Models:** User-uploaded models ## Testing Procedures **Health Checks:** 1. **Container Status:** docker ps | grep localai 2. **API Accessibility:** curl http://[SERVER_IP]:8080/v1/models 3. **Model Loading:** Check available models **Functionality Tests:** 1. **Text Generation:** Test chat completions endpoint 2. **Embeddings:** Test embedding generation 3. **Model Management:** Test model loading/unloading 4. **Performance:** Benchmark response times **Integration Tests:** 1. **OpenAI Compatibility:** Test with OpenAI clients 2. **AnythingLLM:** Verify integration works 3. **API Endpoints:** Test all supported endpoints ## Troubleshooting **Common Issues:** 1. **Model Loading Failed:** - Check model file integrity - Verify file permissions - Monitor memory usage - Review container logs 2. **High Memory Usage:** - Reduce context size - Limit concurrent requests - Use smaller models - Enable model unloading 3. **Slow Response Times:** - Increase thread count - Enable GPU acceleration - Optimize model selection - Check system resources **Permission Fixes:** ```bash sudo chown -R 99:100 /mnt/user/appdata/localai sudo chmod -R 755 /mnt/user/appdata/localai ``` **Log Locations:** - **Container Logs:** docker logs localai - **Application Logs:** /mnt/user/appdata/localai/logs - **System Logs:** /var/log/syslog ## Maintenance **Regular Tasks:** 1. **Model Updates:** Check for model updates monthly 2. **Storage Cleanup:** Remove unused models 3. **Performance Tuning:** Optimize settings 4. **Security Updates:** Update container regularly **Performance Optimization:** 1. **Memory Management:** Monitor and adjust limits 2. **Model Selection:** Use appropriate model sizes 3. **Threading:** Optimize thread count 4. **Caching:** Enable response caching **Storage Management:** 1. **Model Storage:** Monitor disk usage 2. **Cache Cleanup:** Clear temporary files 3. **Log Rotation:** Manage log file sizes 4. **Backup Strategy:** Balance backup size vs importance
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
tonym
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: tonym/NastyNAS#33
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?