Topic 7: Multimodal AI interview questions

Multimodal AI works with more than one type of data, such as text, images, audio, video, or sensor readings. A multimodal system may align inputs into a shared representation, reason across them, and produce an output in one or more modalities. For example, it can inspect a product photo and answer a text question about defects. The main engineering challenge is preserving useful information and relationships between modalities.

A common design uses a separate encoder for each modality, followed by projection layers that map their outputs into a shared embedding space. A language model or fusion transformer then attends to the projected tokens. Early fusion combines signals near the input, while late fusion combines modality-specific decisions. The right choice depends on latency, data volume, alignment quality, and whether fine-grained cross-modal reasoning is required.

A vision-language model connects visual understanding with language generation. An image encoder converts pixels into visual features, and a connector maps those features into tokens or embeddings that a language model can use. Training usually includes image-text alignment and instruction tuning. Such models can caption images, answer questions, extract fields, compare images with text, and reason about documents, but their answers still require validation for high-impact use cases.

Multimodal alignment means representing corresponding concepts from different modalities so the model recognizes their relationship. An image of a dog and the phrase describing that dog should be close in a shared embedding space. Contrastive learning is one common method: matching pairs are pulled together and non-matching pairs are separated. Good alignment supports retrieval and grounding, but alignment alone does not guarantee accurate multi-step reasoning or factual answers.

I would first define supported image types, question scope, latency, and accuracy requirements. The request path would validate size and format, remove unsafe metadata when appropriate, and send the image with a structured prompt to a vision-language model. I would constrain the response schema, store the source and model version for auditability, and add confidence or escalation rules. Evaluation should include low quality images, missing context, adversarial text, and unanswerable questions.

There are two main approaches. A document pipeline can perform OCR, detect layout, and pass structured text and coordinates to a language model. Alternatively, a vision-language model can inspect rendered pages directly. OCR is often cheaper and easier to cite, while direct visual reasoning can preserve tables, handwriting, and spatial relationships. A robust system combines both where useful, checks extracted fields against the image, and retains page-level evidence.

Data quality is usually harder than model selection. Pairs may be weakly aligned, duplicated, mislabeled, low resolution, or missing important context. Captions can also contain copyright, privacy, or demographic bias. I would deduplicate by perceptual and semantic similarity, filter unsafe or sensitive content, verify samples manually, and track provenance. Evaluation sets should represent real workflows rather than only clean web examples, with separate slices for each modality and failure type.

I would combine task metrics with human review and operational measures. Captioning can use similarity metrics, but visual question answering needs exact or semantic correctness checks. For extraction, field-level precision, recall, and citation accuracy matter. I would test hallucinations, refusal behavior, image quality, multilingual inputs, OCR-heavy documents, and prompt injection inside images. Production evaluation should also track latency, cost, failure rates, and disagreement between model output and human adjudication.

Visual grounding connects a textual claim to a specific region or object in an image. Instead of saying that a document contains a signature, a grounded system can identify the page and bounding box containing it. Grounding improves trust, debugging, and downstream actions such as cropping or redaction. It can be implemented with object detectors, region encoders, coordinate-aware prompts, or post-processing that maps model references back to image coordinates.

I would make uncertainty and evidence part of the design. The prompt should require the model to say when an answer is not visible, cite a region or page, and return structured fields. Retrieval or OCR can provide additional context, but I would not treat retrieved text as proof without checking the source image. Threshold-based routing to human review, constrained decoding, diverse evaluation cases, and monitoring unsupported claims are also important.

Audio can be handled through a speech encoder, an automatic speech recognition stage, or both. For a meeting assistant, I might transcribe speech with timestamps, preserve speaker segments, and pass selected audio features or transcript chunks to a language model. Direct audio models can capture tone and nonverbal signals, but transcripts are easier to search and audit. The design must address consent, noisy recordings, accents, overlapping speakers, and retention.

Tokenization converts modality data into units a transformer can process. Text uses subword tokens, while images may use patches or learned visual tokens and audio may use time-frequency frames. More tokens can preserve detail but increase memory, latency, and cost. A practical system resizes or compresses inputs based on the task, uses adaptive detail where available, and measures whether extra tokens improve accuracy enough to justify their operational cost.

I would avoid blindly sending the original resolution. First, create a thumbnail for global context, then identify likely regions using layout detection, tiling, or a first-pass model. Relevant tiles can be sent at higher resolution and merged with the global result. The pipeline should preserve coordinates, cap total tokens, and define behavior when regions overlap. For documents, page-level processing and targeted crops often provide better cost and latency than one huge request.

I would minimize collection, define retention periods, and restrict access to raw media. Images and audio should be encrypted in transit and at rest, with sensitive metadata removed where possible. Face, identity, health, and location data may require redaction or explicit consent. Logs should store references and structured outcomes rather than raw content by default. I would also verify provider data-use policies, define deletion workflows, and test access controls with realistic media.

Multimodal retrieval finds relevant items when the query and stored data may use different modalities. A text query can retrieve images, a screenshot can retrieve related documents, and an audio transcript can retrieve video segments. A shared embedding model enables cross-modal search, while reranking can use richer joint reasoning. I would evaluate recall at useful cutoffs, metadata filters, duplicate handling, and whether retrieved items actually support the user task.

I would treat visible text in an image as untrusted data, not as system instructions. OCR and the vision model can identify suspicious instructions, but detection should not be the only defense. The application should separate user policy from image content, constrain tools and permissions, validate tool arguments, and require confirmation for consequential actions. Test cases should include screenshots, handwritten instructions, hidden small text, QR codes, and instructions embedded in charts.

Specialist models are preferable when a modality has strict accuracy, latency, or compliance requirements. A dedicated OCR, speech recognition, object detection, or moderation model may be cheaper and more predictable than a general model. A single multimodal model is attractive when cross-modal reasoning is central and integration simplicity matters. I would compare end-to-end task accuracy, error recovery, operating cost, observability, and the consequences of each model's mistakes before choosing.

The fallback should be based on an observable failure condition, not a vague feeling. Examples include unsupported format, timeout, low image quality, schema validation failure, or a confidence threshold. A text-only path may use OCR, metadata, or a user request for clarification. I would preserve the original request, avoid silently changing its meaning, communicate limitations clearly, and record which path answered it so fallback quality and cost can be measured.

A multimodal agent can perceive and act on visual, audio, or video state, so it must reason about changing observations as well as text. It needs modality-specific preprocessing, grounding, temporal context, and stronger action confirmation. For example, a browser agent may inspect a screenshot but should verify target coordinates before clicking. I would isolate tools, limit permissions, validate observations, and maintain an event trace linking each action to its evidence.

I would measure cost per successful task, not just cost per request. Techniques include resizing inputs, using thumbnails and selective crops, batching compatible requests, caching embeddings, routing simple cases to smaller models, and limiting output tokens. Preprocessing can remove irrelevant frames or silence. I would set budgets and timeouts, monitor quality by input slice, and only adopt compression or quantization after confirming that savings do not create expensive downstream review.