Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Examining the risks associated with cloud LLMs, including data retention, input training, and jurisdictional issues.
- Exploring Ollama’s architecture, comprising the model server, registry, and OpenAI-compatible API.
- Benchmarking Ollama against alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Reviewing licensing terms for models like Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux systems with support for CUDA and ROCm.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2 instructions.
- Managing Docker deployments and setting up persistent volume mappings.
- Configuring multi-GPU environments and defining VRAM allocation strategies.
Model Lifecycle Management
- Fetching models from the Ollama registry using commands like ollama pull llama3.
- Importing GGUF format models from HuggingFace and TheBloke.
- Assessing the trade-offs of quantization levels such as Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent loading limitations.
Creating Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Fine-tuning parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce specific role-based behaviors.
- Building and publishing proprietary models to a local registry.
API Integration Strategies
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON output modes.
- Connecting with frameworks like LangChain, LlamaIndex, and bespoke applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Tuning and Optimization
- Managing context window sizes and KV cache efficiency.
- Optimizing batch inference and parallel request processing.
- Allocating CPU threads effectively with NUMA awareness.
- Monitoring GPU utilization levels and memory pressure.
Security Protocols and Compliance
- Establishing network isolation for model serving endpoints.
- Implementing pipelines for input filtering and output moderation.
- Maintaining comprehensive audit logs for all prompts and completions.
- Verifying model provenance through hash validation.
Requirements
- Proficiency in intermediate Linux administration and container management.
- A high-level grasp of machine learning concepts and transformer architectures.
- Working knowledge of RESTful APIs and JSON data structures.
Target Audience
- AI engineers and developers seeking to migrate away from cloud-based LLM APIs.
- Organizations constrained by data privacy regulations that prohibit cloud model usage.
- Government and defense entities necessitating air-gapped, isolated language model deployments.
