Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Trade-offs between Hosted APIs and self-hosting.
• Patterns for on-premises, edge, and air-gapped setups.
• Managing single models versus model portfolios.
• Comparing cost per decision.

Hardware Sizing and Capacity Planning

• Memory footprint analysis by parameter count.
• Comparing CPU-only performance versus GPU acceleration.
• Planning for concurrency and throughput.
• Allowing headroom for increasing task length.

Serving Stacks

• Options including vLLM and SGLang.
• Execution via llama.cpp and GGUF.
• Using Ollama for local development workflows.
• Selecting stacks based on decision workload characteristics.

Quantization Without Compromising Calibration

• Understanding precision formats and memory savings.
• Assessing quality and calibration degradation.
• Exploring per-tensor and importance-aware quantization variants.
• Conducting acceptance testing post-quantization.

Latency Engineering

• Distinguishing median versus tail latency.
• Managing latency growth as context length increases.
• Utilizing warm starts and model caching.
• Batching multiple decision queries.

Edge and Constrained Environments

• Running small decision models on CPUs.
• Deployment on Apple Silicon and embedded systems.
• Strategies for offline operation and updates.
• Distributing weights in air-gapped environments.

Containerization and Scaling

• Designing images for model serving.
• Managing volumes and caches.
• Implementing horizontal scaling and routing.
• Setting resource limits and GPU sharing policies.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Tracking decision outcomes and escalations.
• Setting alerts for calibration drift.

Versioning and Rollout

• Versioning models and adapters.
• Implementing shadow and canary deployments.
• Pinning model files for audit purposes.
• Establishing rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Managing network isolation and egress control.
• Audit logging of decisions.