Frontend Beauty
ESC

Type to search the knowledge base.

Multimodal LLM Apps

Images, audio, and PDFs are tokens too. How to send them, what the model actually sees, and the UI problems of slow vision calls.

intermediate6 min read
  • multimodal
  • vision
  • audio

A user pastes a screenshot of an invoice. The multimodal LLM reads the table, invents a total that is not on the page, and your UI shows it in a green “extracted” chip. Vision is not a spreadsheet. It is a lossy look at pixels, with the same overconfidence problem as text, plus OCR mistakes you cannot see in the transcript.

Images, audio clips, and PDF pages all become tokens. They blow the context window, they move time-to-first-token, and they still do not give you an audit trail for a number. Offer a text fallback, and verify anything that will hit a ledger.

Key takeaways

  • A multimodal LLM consumes pixels or waveforms as tokens. It does not “open Excel.”
  • Resize and pick a detail level on purpose. High-detail images are a context and latency tax.
  • Never trust extracted amounts, SKUs, or totals without a deterministic check or a human.
  • Prefill on vision calls is slow. Stream captions, keep Stop wired, and show a still of what you sent.
  • PDFs are layout. Parse text when you can; use vision for scans, then RAG the resulting text.

What a multimodal LLM actually sees

Text in, tokens out is the usual story. For vision, the provider tiles or downsamples the image, embeds those patches, and the decoder attends over them with your question. Low detail is a cheap thumbnail. High detail is more tiles, more tokens, more bill, still not pixel-perfect OCR.

Audio is usually transcribed or encoded into a sequence the text model can attend to. A cough, a second speaker, and a clipped upload all become “the model heard.” You will not get a waveform back in the citation.

PDFs are worse: native text, columns, headers, stamps, and scanned pages in one file. Dumping the binary into a vision call is how you pay vision prices to misread a footer. Prefer text extraction first (RAG on PDFs and HTML). Use vision on pages that have no text layer.

The model has no access to your filesystem. If you pass a URL, the provider fetches it (or fails). If you pass base64, you already fetched it. Either way, you chose the bytes.

How images enter the request

Keep the payload schematic. Field names move between Chat Completions and Responses APIs; the idea does not.

type ImagePart = {
  type: 'image_url';
  image_url: {
    url: string; // https://... or data:image/jpeg;base64,...
    detail?: 'low' | 'high' | 'auto';
  };
};

type UserTurn = {
  role: 'user';
  content: Array<{ type: 'text'; text: string } | ImagePart>;
};

export async function visionTurn(file: File, question: string): Promise<UserTurn> {
  if (!file.type.startsWith('image/')) {
    throw new Error('not an image');
  }

  const resized = await downscale(file, { maxEdge: 1600 });
  const buf = Buffer.from(await resized.arrayBuffer());

  return {
    role: 'user',
    content: [
      {
        type: 'text',
        text: `${question}\nIf a number is unreadable, say so. Do not guess totals.`,
      },
      {
        type: 'image_url',
        image_url: {
          url: `data:${resized.type};base64,${buf.toString('base64')}`,
          detail: 'low',
        },
      },
    ],
  };
}

// Failure: sending a 12MP PNG at detail: 'high' for "is this a cat?"
// Prefill dominates, the answer does not get better.

Rules of thumb:

  • Downscale on the server (or a worker). Do not ship a 20MB photo through your chat route.
  • Strip EXIF if you do not want GPS in the provider’s logs.
  • One question per image set. “Summarize, extract every line item, and rewrite the email” is three jobs and a confused decoder.
  • Count the image toward the context window the same way you count a retrieved chapter.

Gemini and OpenAI both accept inline bytes or uploaded file ids. Prefer file ids when the same image will be reused across turns, so you are not re-uploading the base64 on every message.

Numbers, PDFs, and a text fallback

Extraction is a schema plus arithmetic, not a caption.

import { z } from 'zod';

const InvoiceGuess = z.object({
  currency: z.string(),
  lineItems: z.array(
    z.object({
      label: z.string(),
      cents: z.number().int(),
      readable: z.boolean(),
    }),
  ),
  totalCents: z.number().int(),
  notes: z.string(),
});

export function acceptInvoice(raw: unknown) {
  const inv = InvoiceGuess.parse(raw);
  const readable = inv.lineItems.filter((i) => i.readable);
  const sum = readable.reduce((s, i) => s + i.cents, 0);

  if (readable.length !== inv.lineItems.length) {
    return { ok: false as const, reason: 'unreadable-line', inv };
  }
  if (sum !== inv.totalCents) {
    return { ok: false as const, reason: 'total-mismatch', inv, sum };
  }
  return { ok: true as const, inv };
}

If ok: false, send the user to a form with the image on the side. Do not “fix” the total in the prompt. That is how you launder a hallucination into a payment.

For documents that already have a text layer, skip vision. Chunk the text, retrieve, cite. Vision is the fallback for scans and screenshots, not the default pipeline.

Audio: show the transcript as a first-class object. Let the user edit it before you run tools. A wrong “cancel my account” in a noisy recording is a tool-call incident.

Latency and the UI

Vision prefill is heavier than a short chat prompt. Users feel the blank bubble. Treat it like a slow upload, not like a snappy text completion.

  1. As soon as the file is selected, show a thumbnail of exactly the bytes you will send (after resize).
  2. Disable Send while the upload is in flight. One in-flight vision call per conversation is enough.
  3. Stream tokens. A 200ms first byte is rare here; a spinner with the thumbnail is honest.
  4. Keep Stop hooked to abort. A cancelled vision call should not retry on its own (retries still apply to 5xx, not to user abort).
  5. Do not JSON.parse a partial object. If you need structured extraction, wait for the schema to close, or stream a caption and extract in a second, cheaper text call.

The chat list still has to stay responsive. That is a streaming chat UI problem: do not re-render the whole transcript on every token while a 2MB data URL sits in state. Store images by id, not inline in every message object you pass through React.

Footguns

  • Screenshot of a screenshot. Compression artifacts become “letters.” Ask for a CSV.
  • Tiny text in a huge UI chrome. Crop to the table before you send it.
  • Multiple images with no labels. “The second one” is ambiguous in the packed token stream. Caption each part ([image: receipt], [image: packing slip]).
  • Trusting a URL the user typed. SSRF is your problem if your server fetches it. If the provider fetches it, you still need to know what you allowed.
  • Putting the image in the system prompt forever. Every later turn pays for it. Drop it from history once you have a verified extraction.
  • Accessibility. Alt text and a transcript are not optional because the model “can see.” Screen reader users still need the text you claim to have extracted.

Interview answer (out loud)

“Multimodal means the model attends over image or audio tokens as well as text. I resize, pick a detail level, and count those tokens against the window. I treat vision as lossy OCR: structured extract, then verify sums and ids in code, and fall back to a form when they disagree. PDFs with a text layer get parsed and retrieved, not screenshotted. The UI shows the exact thumbnail we sent, streams, and aborts. I would not post a guessed total to a ledger.”

Further reading (primary sources)

Related guides