Course Outline
Module 1 — How AI Apps Break
Lab: none — architecture walkthrough & discussion
A builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flows: system, developer, user, and tool messages
- Points where untrusted data enters (or re-enters) the model
- Trust boundaries owned versus inherited by the developer
- Why AI attacks are semantic rather than syntactic
- Mapping the OWASP LLM Top 10 to written code
Key insight: Every point where untrusted text reaches the model—or where model output reaches your code—represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection moment” for AI — but complete avoidance is not possible.
Topics:
- Direct versus indirect prompt injection
- Hidden instructions within documents, web pages, and tool outputs
- Jailbreaks and role-confusion techniques
- The importance of separating instructions from data
- Defensive prompt design (using delimiters, structure, and minimal authority)
- Why prevention is partial — designing for containment
Hands-on:
- Attack your own chatbot
- Bypass a naive filter
- Restructure prompts to shrink the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The bug class developers underestimate most.
Topics:
- Treating model output as untrusted input for the rest of the application
- Insecure output handling (LLM02): downstream XSS, SSRF, command/SQL injection
- Avoid evaluating/executing/rendering raw model output
- Using structured outputs and schema validation
- Output encoding and allowlisting strategies
- Safe rendering in web/UI contexts
Hands-on:
- Identify and fix an insecure-output-handling vulnerability
- Enforce JSON schema validation on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the largest new attack surfaces — and you are responsible for building it.
Topics:
- Vector database and retrieval threats
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval mechanisms
Hands-on: Poison a RAG pipeline with a malicious document, then add ingestion sanitization and retrieval scoping to defend it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where a bug transforms into an action.
Topics:
- Excessive agency (LLM06) and tool abuse
- Implementing least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop processes
- Sandboxing tool execution environments
- Using scoped, short-lived credentials for agents
- Limits on autonomous loops and chaining capabilities
Hands-on:
- Secure an over-permissioned agent
- Add an allowlist plus approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
Operational mistakes that cause the fastest damage.
Topics:
- API key and secret management (never store in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval systems
- Denial-of-wallet: mitigating unbounded token/cost consumption
- Implementing rate limits, token budgets, and timeouts
- Logging securely without leaking secrets or PII
Hands-on:
- Remove secrets from the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Evaluating buy versus build for input/output safety.
Topics:
- What guardrail frameworks do (and what they don’t)
- Input guardrails: classifiers for injection/PII/topics
- Output guardrails: validation, filtering, and grounding checks
- When to use a guardrail versus a deterministic check
- Layering guardrails with controls from earlier modules
- Performance impacts, false positives, and failure modes
Hands-on:
- Add an input/output guardrail layer to an AI feature
- Analyze what it catches and what it misses
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Ship with the assumption that an attacker already has access.
Topics:
- Building abuse/test suites for AI features
- Automated prompt-injection and jailbreak tests
- Regression testing guardrails and policies
- Integrating AI security checks into CI pipelines
- Model and dependency supply chain management (provenance, pinning)
- A pre-ship security checklist for AI features
Hands-on:
- Write automated red-team tests for an AI feature
- Wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: none — scoring exercise (uses the Capstone app)
Transform your accumulated knowledge into a repeatable score.
Topics:
- The AI Security Hexagon: six guiding questions instead of “is it secure?”
- Six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Using pre/post hardening scores as metrics
Hands-on:
- Score the Capstone app on the 100-point scale
- Identify the single change that most significantly raises the score
Key insight: The three highest-weighted categories map to the trust boundaries a developer owns — meaning the score measures exactly what this course taught.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app contains:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets embedded in the prompt path
- No cost limits
Students apply course concepts to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Move secrets out and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened app plus a short OWASP LLM Top 10 self-assessment.
Module - Lab map
Labs run in sequential order, following module order. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither has its own lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone app.
Requirements
- Skill Level: Intermediate.
- Students should be comfortable with: building and consuming REST APIs, scripting languages (labs utilize Python), basic application authentication, git, and the command-line interface (CLI).
- No machine learning background is required. This is an application security course for developers who build with LLMs, not those who train them.
Audience
- Software and backend engineers developing LLM features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers responsible for AI feature ownership
Testimonials (2)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
