Advanced Module 9: AI Security Fundamentals

Session 2: Security Assessment & Hardening

Duration: 45 min live workshop + 75 min self-paced exercises

---

Entry Criteria

• Threat analysis completed
• Prompt injection defenses implemented
• Least privilege configuration applied
• Understanding of defensive patterns

Exit Criteria

• Security assessment methodology understood
• AI system security assessment conducted
• Red teaming basics applied
• Multi-agent security architecture designed
• Module capstone (Security Assessment Report) completed

---

Workshop Content (45 minutes)

Segment 1: Data Security & PII Handling (12 min)

• Data security risks in AI systems:

  • Training data leakage
  • Context window exposure
  • Output containing sensitive data
  • Logging sensitive information
  • Tool-mediated data access

• PII handling framework:

  INPUT STAGE:
    ├── Detect PII in input
    ├── Classify sensitivity level
    ├── Apply handling rules
    └── Log (without PII)
  
  PROCESSING STAGE:
    ├── Minimize PII in context
    ├── Use references, not values
    ├── Apply data masking
    └── Scope access to need
  
  OUTPUT STAGE:
    ├── Scan for PII leakage
    ├── Redact before display
    ├── Audit trail creation
    └── Secure deletion
  

• PII detection patterns:

  PII Type	Detection Pattern	Handling
  SSN	XXX-XX-XXXX format	Redact
  Credit Card	16-digit patterns	Redact
  Email	user@domain format	Mask
  Phone	Various formats	Mask
  Name + DOB	NER + date	Flag

• Data leakage prevention:

  • Output scanning before delivery
  • Blocking sensitive data patterns
  • Monitoring for unusual data access
  • Restricting data export capabilities
  • Audit logging for compliance

Segment 2: Security Assessment Methodology (12 min)

• Assessment framework:

  PHASE 1: DISCOVERY
    └── Inventory all AI components
    └── Map data flows
    └── Identify integration points
  
  PHASE 2: THREAT MODELING
    └── Identify threat actors
    └── Map attack vectors
    └── Assess vulnerabilities
  
  PHASE 3: CONTROL EVALUATION
    └── Review existing controls
    └── Test control effectiveness
    └── Identify gaps
  
  PHASE 4: TESTING
    └── Controlled attack simulation
    └── Defense validation
    └── Edge case exploration
  
  PHASE 5: REPORTING
    └── Document findings
    └── Prioritize risks
    └── Recommend remediations
  

• Assessment checklist categories:

  Category	Assessment Focus
  Architecture	Secure design, isolation, defense in depth
  Access Control	Authentication, authorization, least privilege
  Data Protection	Encryption, handling, retention
  Input Security	Validation, sanitization, injection defense
  Output Security	Filtering, validation, leakage prevention
  Monitoring	Logging, alerting, incident detection
  Tool Security	MCP config, tool permissions, sandboxing
  Compliance	Regulatory requirements, policies

• Documenting findings:

  ## Finding: [ID] - [Title]
  
  **Severity:** Critical/High/Medium/Low
  **Category:** [Category]
  
  **Description:**
  [What was found]
  
  **Risk:**
  [What could happen if exploited]
  
  **Evidence:**
  [How it was discovered]
  
  **Recommendation:**
  [How to fix]
  
  **Priority:** Immediate/Short-term/Medium-term
  

Segment 3: Red Teaming Basics (12 min)

• What is AI red teaming?

  • Controlled adversarial testing
  • Thinking like an attacker
  • Finding vulnerabilities before attackers do
  • NOT about breaking things in production

• Red team methodology:

  1. SCOPE DEFINITION
     - What systems are in scope?
     - What attack types to simulate?
     - What are the rules of engagement?
  
  2. RECONNAISSANCE
     - Understand system capabilities
     - Identify potential weaknesses
     - Map attack surface
  
  3. ATTACK SIMULATION
     - Attempt defined attack scenarios
     - Document techniques and results
     - Capture evidence
  
  4. REPORTING
     - Document successful attacks
     - Explain attack paths
     - Recommend defenses
  

• Common AI red team scenarios:

  Scenario	Objective	Technique
  Prompt injection	Override instructions	Direct/indirect injection
  Data extraction	Exfiltrate sensitive data	Clever questioning
  Guardrail bypass	Get prohibited output	Jailbreak techniques
  Privilege escalation	Access restricted tools	Tool confusion attacks
  Denial of service	Exhaust resources	Token bombing, loops

• Ethical guidelines:

  • Always have authorization
  • Document everything
  • Never test on production without safeguards
  • Report findings responsibly
  • Focus on defense improvement, not just breaking things

Segment 4: Multi-Agent Security Architecture (9 min)

• Security challenges in multi-agent systems:

  • Agent-to-agent trust
  • Shared resource access
  • Attack propagation
  • Coordination vulnerabilities
  • Complex attack surfaces

• Secure multi-agent architecture:

  ┌─────────────────────────────────────────────────────────────┐
  │                    SECURITY PERIMETER                       │
  │  ┌─────────────────────────────────────────────────────┐   │
  │  │              ORCHESTRATOR (Trusted)                  │   │
  │  │  - Access control enforcement                        │   │
  │  │  - Inter-agent message validation                    │   │
  │  │  - Resource allocation                               │   │
  │  │  - Audit logging                                     │   │
  │  └─────────────────────────────────────────────────────┘   │
  │              │                │                │            │
  │              ↓                ↓                ↓            │
  │  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
  │  │   AGENT A       │ │   AGENT B       │ │   AGENT C       │ │
  │  │   (Sandbox A)   │ │   (Sandbox B)   │ │   (Sandbox C)   │ │
  │  │   Scope: Read   │ │   Scope: Write  │ │   Scope: API    │ │
  │  └─────────────────┘ └─────────────────┘ └─────────────────┘ │
  │                                                              │
  │  ┌──────────────────────────────────────────────────────┐  │
  │  │                SHARED SECURE STORAGE                  │  │
  │  │  - Access control per agent                          │  │
  │  │  - Audit trail                                       │  │
  │  │  - Encryption at rest                                │  │
  │  └──────────────────────────────────────────────────────┘  │
  └─────────────────────────────────────────────────────────────┘
  

• Inter-agent security controls:

  • Validate all messages between agents
  • Prevent agent impersonation
  • Limit information sharing to need-to-know
  • Monitor for abnormal communication patterns
  • Circuit breakers for compromised agents

• Zero trust for agents:

  • Verify every agent action
  • Assume any agent could be compromised
  • Continuous validation, not one-time auth
  • Micro-segmentation of capabilities

---

Self-Paced Exercises (75 minutes total)

Exercise 2.1: Data Security Implementation (25 minutes)

Implement PII handling and data protection controls

# Data Security Implementation

## System: [Your Agent System Name]

### Data Inventory

| Data Type | Classification | Location | Access Method |
|-----------|---------------|----------|---------------|
| User queries | Internal | Memory | Direct |
| Documents | Confidential | Filesystem | Tool |
| API responses | Internal | Memory | Tool |
| Output logs | Internal | Log files | System |

### PII Detection Configuration

```json
{
  "pii_detection": {
    "enabled": true,
    "patterns": {
      "ssn": {
        "regex": "\\b\\d{3}-\\d{2}-\\d{4}\\b",
        "handling": "redact"
      },
      "credit_card": {
        "regex": "\\b\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4}\\b",
        "handling": "redact"
      },
      "email": {
        "regex": "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b",
        "handling": "mask"
      },
      "phone": {
        "regex": "\\b(\\+1)?[- .]?\\(?\\d{3}\\)?[- .]?\\d{3}[- .]?\\d{4}\\b",
        "handling": "mask"
      }
    },
    "scan_points": ["input", "tool_response", "output"],
    "logging": {
      "log_detections": true,
      "log_pii_values": false
    }
  }
}

Data Handling Rules

Data Classification	Handling Rule	Retention
Public	No restrictions	30 days
Internal	Mask in logs	7 days
Confidential	Redact, no logging	Session only
Restricted	Block processing	N/A

Output Filtering

{
  "output_filter": {
    "scan_enabled": true,
    "block_patterns": [
      "password\\s*[:=]\\s*\\S+",
      "api[_-]?key\\s*[:=]\\s*\\S+",
      "secret\\s*[:=]\\s*\\S+"
    ],
    "on_detection": "redact_and_warn",
    "audit_log": true
  }
}

Data Leakage Prevention

Control	Implementation	Status
Output scanning	PII regex filter	Implemented
Tool result filtering	Sensitive data redaction	Implemented
Export restrictions	Disable file download of sensitive data	Pending
Logging sanitization	Remove PII before logging	Implemented

Verification Testing

Test	Data Type	Expected	Result
SSN in input	123-45-6789	Detected, redacted
SSN in output	Model generates SSN	Blocked
Credit card	4111-1111-1111-1111	Redacted
Password in output	password: secret123	Redacted

**Deliverable:** `data-security-config.md` + implementation

-----

### **Exercise 2.2: Security Assessment (25 minutes)**

*Conduct a security assessment of an AI system*

```markdown
# AI Security Assessment Report

## Assessment Information
- **System:** [System Name]
- **Assessment Date:** [Date]
- **Assessor:** [Name]
- **Scope:** [What was assessed]

### Executive Summary

[2-3 paragraphs summarizing the assessment, key findings, and overall risk posture]

**Overall Risk Rating:** Critical/High/Medium/Low

### System Overview

**Architecture:**
- Agent type: [Single/Multi-agent]
- Tools/MCP servers: [List]
- Data access: [What data can be accessed]
- User base: [Who uses the system]

**Data Flow Diagram:**

[User] ──→ [Input Processing] ──→ [Agent] ──→ [Tools] │ ↓ [Output Processing] ──→ [User]

### Assessment Findings

#### Finding 1: [Critical/High] - Insufficient Input Validation
- **Category:** Input Security
- **Description:** User input is passed directly to the agent without validation for injection patterns.
- **Risk:** Attackers could inject malicious instructions that override agent behavior.
- **Evidence:** Test input "ignore previous instructions" was not filtered.
- **Recommendation:** Implement input validation layer with pattern blocking.
- **Priority:** Immediate

#### Finding 2: [High] - Overly Permissive Tool Access
- **Category:** Access Control
- **Description:** Filesystem MCP has full read/write access to entire system.
- **Risk:** Compromised agent could access or modify sensitive files.
- **Evidence:** Agent successfully read /etc/passwd during testing.
- **Recommendation:** Scope filesystem access to required directories only.
- **Priority:** Immediate

#### Finding 3: [Medium] - No Output Filtering
- **Category:** Output Security
- **Description:** Agent outputs are not scanned for sensitive data before display.
- **Risk:** Accidental exposure of PII or credentials in responses.
- **Evidence:** Agent included raw API key in troubleshooting response.
- **Recommendation:** Implement output scanning with sensitive pattern detection.
- **Priority:** Short-term

#### Finding 4: [Continue documenting findings]

### Assessment Summary

| Category | Findings | Critical | High | Medium | Low |
|----------|----------|----------|------|--------|-----|
| Architecture | 2 | 0 | 1 | 1 | 0 |
| Access Control | 3 | 1 | 1 | 1 | 0 |
| Data Protection | 2 | 0 | 1 | 1 | 0 |
| Input Security | 2 | 1 | 1 | 0 | 0 |
| Output Security | 2 | 0 | 1 | 1 | 0 |
| Monitoring | 1 | 0 | 0 | 1 | 0 |
| **Total** | **12** | **2** | **5** | **5** | **0** |

### Remediation Roadmap

| Priority | Finding | Remediation | Effort | Owner |
|----------|---------|-------------|--------|-------|
| Immediate | Input validation | Implement validation layer | 2 days | |
| Immediate | Tool permissions | Scope filesystem access | 1 day | |
| Short-term | Output filtering | Add PII detection | 3 days | |
| Short-term | Logging | Add security logging | 2 days | |
| Medium-term | Architecture | Add defense layers | 1 week | |

Deliverable: security-assessment-report.md

---

Exercise 2.3: Module Capstone - Comprehensive Security Package (25 minutes)

Complete your AI security assessment and hardening package

# AI Security Assessment - Complete Package

## System: [Your Agent System Name]
## Prepared By: [Your Name]
## Date: [Date]

---

## Part 1: Executive Summary

### Assessment Overview
[Summary of the assessment conducted, methodology used, and scope]

### Key Findings
1. [Most critical finding]
2. [Second most critical]
3. [Third most critical]

### Risk Summary
| Risk Level | Count | Key Risks |
|------------|-------|-----------|
| Critical | X | [Brief description] |
| High | X | [Brief description] |
| Medium | X | [Brief description] |
| Low | X | [Brief description] |

### Recommendations Summary
1. **Immediate (0-7 days):** [Top priority actions]
2. **Short-term (1-4 weeks):** [Important improvements]
3. **Medium-term (1-3 months):** [Strategic enhancements]

---

## Part 2: Threat Analysis Summary

### Threat Landscape
[Reference threat-analysis.md from Exercise 1.1]

### Top Threats by Risk Score
| Rank | Threat | Likelihood | Impact | Score |
|------|--------|------------|--------|-------|
| 1 | | | | |
| 2 | | | | |
| 3 | | | | |

---

## Part 3: Security Controls Implemented

### Defensive Patterns
| Control | Status | Effectiveness |
|---------|--------|---------------|
| Input validation | Implemented | High |
| Prompt hardening | Implemented | Medium |
| Output filtering | Implemented | High |
| Least privilege | Implemented | High |
| Sandboxing | Partial | Medium |
| Human-in-the-loop | Planned | N/A |

### Configuration Artifacts
- `prompt-injection-defenses.md` - Input validation and prompt security
- `least-privilege-config.md` - Tool and MCP security configuration
- `data-security-config.md` - PII handling and data protection

---

## Part 4: Security Architecture Recommendations

### Current State
[Diagram of current security architecture]

### Recommended Target State

┌────────────────────────────────────────────────────────────────┐ │ SECURITY CONTROLS │ │ │ │ INPUT LAYER PROCESSING LAYER OUTPUT LAYER │ │ ┌──────────┐ ┌──────────────┐ ┌──────────┐ │ │ │Validation│ │ AGENT │ │ Scanning │ │ │ │Sanitize │────────→│ (Sandboxed) │──────→│ Filtering│ │ │ │Rate Limit│ │ Monitored │ │ Audit │ │ │ └──────────┘ └──────────────┘ └──────────┘ │ │ │ │ │ ↓ │ │ ┌──────────────┐ │ │ │TOOLS (Scoped)│ │ │ │Circuit Break │ │ │ │Validated I/O │ │ │ └──────────────┘ │ │ │ │ MONITORING: Security logging, alerting, anomaly detection │ └────────────────────────────────────────────────────────────────┘

### Implementation Roadmap

| Phase | Timeline | Deliverables |
|-------|----------|--------------|
| Phase 1: Foundation | Week 1-2 | Input validation, basic output filtering |
| Phase 2: Hardening | Week 3-4 | Least privilege, sandboxing |
| Phase 3: Monitoring | Week 5-6 | Security logging, alerting |
| Phase 4: Advanced | Week 7-8 | Red team testing, continuous improvement |

---

## Part 5: Red Team Summary

### Tests Conducted
| Test | Objective | Result | Defense Effective |
|------|-----------|--------|-------------------|
| Direct injection | Override instructions | Blocked | Yes |
| Indirect injection | Document-based attack | Partial block | Needs improvement |
| Data extraction | Exfiltrate test data | Blocked | Yes |
| Tool abuse | Unauthorized file access | Blocked | Yes |

### Vulnerabilities Found
1. [Vulnerability description and remediation]
2. [Vulnerability description and remediation]

---

## Part 6: Compliance Mapping

| Requirement | Control | Status |
|-------------|---------|--------|
| Data minimization | Least privilege, scoped access | Compliant |
| PII protection | Detection, redaction | Compliant |
| Audit logging | Security logging | Partial |
| Access control | Tool permissions | Compliant |

---

## Part 7: Ongoing Security Program

### Recommended Activities
| Activity | Frequency | Owner |
|----------|-----------|-------|
| Security review | Quarterly | Security team |
| Penetration testing | Annually | External firm |
| Threat model update | When changes occur | Product team |
| Security training | Annually | All users |

### Metrics to Track
| Metric | Target | Current |
|--------|--------|---------|
| Security incidents | 0 critical | |
| Blocked injection attempts | N/A (tracking) | |
| Time to detect threats | < 1 hour | |
| Patch application time | < 7 days | |

Deliverable: ai-security-capstone.md + all supporting artifacts

---