Building Multimodal Chatbots
Combine uploads, vision, speech, and text into a useful conversational experience.
Define the conversation contract
Tell users what the assistant accepts, what it can inspect, how long processing may take, and which results require review. A multimodal chatbot should make the current assets visible and let users remove or replace them. Clear boundaries prevent the interface from implying that every file, language, or task is supported equally well.
Upload handling
Validate file type, size, dimensions, duration, page count, and decompression cost before invoking a model. Scan files for malware and store them with opaque identifiers rather than user-controlled paths. Generate safe previews and preserve originals only as long as necessary. Errors should identify the corrective action without exposing internal storage or provider details.
message = {
"role": "user",
"content": [
{"type": "text", "text": "Describe this receipt."},
{"type": "image_url", "image_url": {"url": "receipt.jpg"}},
],
}
print(message)Routing by modality
Route images, documents, audio, and text to the capability that fits the task. A document may need OCR plus layout extraction, while a voice message may need transcription before summarization. Keep a shared conversation ID but preserve each asset’s metadata and processing state. Routing rules should be observable so a wrong answer can be traced to the selected path.
Conversation state
Store user messages, asset references, model outputs, and validation results separately. Do not place full files or unbounded history into every prompt. Summarize older turns, retrieve relevant assets, and enforce a context budget. Support retries without duplicating side effects, and make it clear when a response is based on a previous upload that may no longer be available.
Safety and prompt injection
Text embedded in an image or document is data, not automatically an instruction. Keep system policy and tool permissions outside untrusted content, and validate any action proposed by the model. Add moderation for inputs and outputs, rate limits, abuse monitoring, and human escalation for sensitive categories. A chatbot’s friendly tone must not weaken server-side authorization.
Human-centered feedback
Show progress for slow operations, partial transcripts when useful, and uncertainty when extraction may be wrong. Let users correct fields, retry a failed modality, report harmful output, and inspect sources. Feedback should be tied to the model, prompt, asset, and pipeline version so it improves the system. Avoid collecting more conversation data than the feedback purpose requires.
Release and evaluation
Test complete journeys: upload, preview, processing, response, correction, retry, deletion, and failure recovery. Include large files, unsupported formats, network interruption, ambiguous images, noisy audio, and adversarial documents. Measure task completion, correction effort, latency, cost, privacy incidents, and unsafe outputs. Launch with a narrow promise and expand only when the evidence supports it.