Topic 9: AI Security & Guardrails interview questions

AI guardrails are controls that constrain, monitor, or verify model behavior and the actions around it. They can include input screening, access control, prompt and output validation, tool permissions, human approval, rate limits, and audit logs. They are needed because models are probabilistic and can be manipulated, hallucinate, leak sensitive information, or invoke tools incorrectly. Good guardrails are layered and tied to concrete risks; a system prompt alone is not a dependable security boundary.

I would map assets, users, model providers, tools, data stores, trust boundaries, and actions that can affect the outside world. Then I would enumerate threats such as prompt injection, sensitive-data leakage, excessive agency, insecure plugins, supply-chain compromise, denial of service, model theft, and tenant isolation failures. For each threat, I would identify preventive controls, detection signals, recovery steps, and tests. I would prioritize risks by impact and likelihood, especially irreversible or privacy-sensitive actions.

Prompt injection occurs when untrusted text attempts to override instructions or manipulate a model into revealing data or taking an unintended action. It can appear in user input, retrieved documents, web pages, emails, images, or tool results. Mitigation is layered: separate instructions from data, minimize exposed context, use least-privilege tools, validate arguments in code, require approval for sensitive actions, and treat model output as untrusted. Testing should include direct, indirect, encoded, and multilingual attacks.

First, I classify the data and decide whether the task needs the sensitive fields at all. I minimize, redact, tokenize, or mask values before sending prompts, and enforce tenant-aware authorization before retrieval. I verify provider retention and training settings, encrypt data in transit and at rest, restrict logs, and define deletion periods. Outputs also need scanning because a model may reproduce secrets. Access should be auditable, and sensitive workflows should have a tested incident-response process.

Least privilege means an agent receives only the identity, data, tools, permissions, and time it needs for its current task. A read-only research agent should not have email, payment, shell, or database-write access. Permissions should be scoped by user, tenant, object, operation, and environment, with short-lived credentials where possible. I would enforce policy in the tool or service itself, not only in the agent prompt, and require explicit confirmation for high-impact actions.

Tool calls should pass through deterministic validation before execution. I would use typed schemas, allowlists, length and range limits, canonicalization, authentication, authorization, timeouts, and bounded output sizes. The server should validate again even if the client already did so. Destructive, external, or financial operations need user confirmation bound to the exact arguments. Tools should be narrow and purpose-specific rather than exposing arbitrary shell commands, unrestricted SQL, or generic HTTP requests.

I limit the agent’s action space, iteration count, budget, runtime, and accessible resources. Read and write capabilities are separated, and irreversible actions require a human checkpoint. The agent should explain the proposed action and target before execution, while the application verifies policy independently. I also add loop detection, idempotency keys, rate limits, cancellation, and audit events. The design goal is bounded assistance: the agent can complete useful steps without silently expanding its own authority.

Defense in depth means no single control is trusted to handle every failure. For an AI workflow, layers might include identity and tenant authorization, input limits, retrieval filtering, isolated tool execution, output validation, human approval, monitoring, and rapid credential revocation. If a prompt filter misses an attack, the tool policy should still block it; if output validation fails, the user should receive a safe error. Layered controls reduce blast radius when any probabilistic or implementation-level defense fails.

I would first classify the output by consequence. For low-risk drafting, I can label uncertainty and let the user edit it. For factual or high-impact tasks, I would ground responses in approved sources, require citations or evidence, validate structured fields, and route ambiguous cases to review. The system should avoid presenting guesses as facts and should have a safe fallback when confidence or evidence is insufficient. Evaluation must measure both ordinary accuracy and harmful confident errors.

I enforce document-level authorization before retrieval and filter by tenant, user, and data classification. Documents are treated as untrusted data, so embedded instructions cannot change application policy. I use trusted source metadata, bounded chunks, relevance checks, citation tracking, and output validation. Retrieval logs should record identifiers without unnecessarily storing sensitive text. I also test poisoned documents, conflicting sources, stale data, prompt injection, and requests for information the user is not allowed to access.

Risks include malicious or tampered model weights, unsafe adapters, poisoned training data, compromised packages, vulnerable serving runtimes, and unclear licensing or provenance. I would pin and verify dependencies, obtain models from trusted sources, record hashes and licenses, scan artifacts, isolate loading and execution, and test behavior before promotion. Model provenance should be part of release metadata. Updates need regression, safety, and performance evaluations because a new artifact can change both capability and risk.

Secrets belong in a managed secret store or protected runtime configuration, never in prompts, repositories, model context, client-side code, or ordinary logs. Credentials should be scoped to one service and operation, rotated, monitored, and revoked quickly. I would prevent tools from reading arbitrary environment variables or filesystem paths. Logs and traces need redaction tests. If a model outputs a possible secret, the application should block or sanitize it and trigger an appropriate security signal.

Output validation checks that a model response conforms to the application’s required type, schema, constraints, and safety rules before it is displayed or used. For example, an invoice extractor should validate dates, currencies, required fields, and confidence handling; an agent action should validate its target and allowed operation. Structured output helps but does not guarantee correctness. Invalid or suspicious responses should be rejected, repaired under strict limits, or sent to a human rather than executed blindly.

Rate limits reduce abuse, runaway costs, denial of service, credential testing, and accidental agent loops. I would apply limits at several dimensions: user, tenant, IP, API key, model, tool, tokens, concurrency, and spend. Expensive operations need stricter budgets than simple classification. Responses should distinguish temporary overload from permanent denial and provide safe retry guidance. Limits must be observable and tested under legitimate bursts so security controls do not unnecessarily block normal users.

I would log structured security events such as authentication failures, denied tool calls, policy violations, unusual retrieval volume, prompt-injection detections, large exports, repeated validation failures, and unexpected destinations. Metrics should include latency, token usage, error rates, refusal rates, and spend anomalies, with sensitive content minimized or redacted. Alerts need thresholds and context to avoid noise. I would preserve request correlation identifiers so an investigation can connect user intent, model output, tool decisions, and side effects.

The plan should define how to detect, triage, contain, investigate, recover, and learn from incidents. It should include contacts, severity levels, credential revocation, tool or model disablement, tenant isolation, evidence preservation, user communication, and a degraded safe mode. Runbooks should cover leaked data, malicious tool use, model compromise, runaway spend, and provider outage. After containment, I would identify the control failure, add regression tests, rotate affected secrets, and update the threat model.

I make controls proportional to risk and explain them in user-facing language. Low-risk reads can be fast, while irreversible or sensitive actions show the exact target, effect, and approval request. I avoid vague blocks by giving a safe alternative or a clear recovery path. Progressive permission, sensible defaults, caching, and good error messages reduce friction. I measure completion, abandonment, false positives, and security outcomes together, because an unusable control will eventually be bypassed.

Approval should happen after the system has validated the action and before the side effect. The reviewer should see the tool, target, arguments, expected consequence, relevant evidence, and any uncertainty. Approval must bind to that exact request, expire quickly, and not be reusable for different parameters. I would record who approved, when, and what executed. For high-volume low-risk tasks, policy-based approval can be appropriate, but exceptions and irreversible operations still need stronger review.

I combine unit tests for policy functions, integration tests at tool and retrieval boundaries, adversarial evaluations, dependency scanning, and operational drills. Cases should include prompt injection, data exfiltration, cross-tenant access, malformed schemas, path traversal, SSRF, oversized inputs, tool loops, leaked credentials, and denied approvals. I measure false positives as well as blocked attacks. Tests should run against the actual model, prompts, runtime, and configuration used for release, because small changes can alter behavior.

I verify data classification, retention, provider terms, authentication, tenant authorization, secret handling, input and output limits, tool scopes, approval paths, logging redaction, rate limits, dependency and model provenance, abuse monitoring, and incident runbooks. Then I run representative quality and adversarial evaluations, test failure recovery, and confirm that disabling the model or a tool leaves a safe degraded experience. Finally, I document known limitations and assign owners for alerts, policy changes, and future reviews.