Transcribe WhatsApp voice notes and photos in an AI agent

A WhatsApp AI agent handles voice notes, photos and files by reading the media object on each message.received event: download the signed link within 24 hours, transcribe audio with a speech-to-text model such as OpenAI’s gpt-transcribe, send images to a vision model, and store the result as text. Replies go back as text or as a file by URL. Voice notes cannot be sent.

What does whasp deliver when someone sends media?

whasp delivers inbound media as a signed link inside the message.received event. The media object has four fields: url, mime, filename and size. The link is valid for 24 hours and whasp deletes the file after 24 hours, so you copy it if you need it. Pictures, audio and MP4 open in the browser, and other files download.

Two limits matter. For now a file over 30 MB arrives without a link, with media set to null. And whasp’s servers keep no messages: the content reaches your webhook once, in this event, so a transcript you do not store is gone.

The event can also carry location (latitude, longitude, name, address, live), contacts (vCards as strings), reply_to (the quoted message, with its text) and sender_name. The documented fields include no flag that marks a voice note, so decide by mime: anything that starts with audio/ goes to transcription.

How do you transcribe a WhatsApp voice note?

To transcribe a voice note, download the file, convert it to a small MP3 and send it to a speech-to-text model. The conversion is a precaution with a reason behind it. WhatsApp’s own business API describes a voice recording as audio/ogg; codecs=opus. OpenAI’s speech-to-text guide lists mp3, mp4, mpeg, mpga, m4a, wav and webm as input formats and does not name Ogg, while the type comments in the openai npm package list ogg and flac as well. The two disagree, and one ffmpeg call settles it.

Mono, 16 kHz and 48 kbps is plenty for speech and comes to about 0.36 MB a minute, so OpenAI’s 25 MB file limit is over an hour of audio. This is the first half of media.ts:

import { execFile } from "node:child_process";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { promisify } from "node:util";
import OpenAI, { toFile } from "openai";

const run = promisify(execFile);
const openai = new OpenAI(); // reads OPENAI_API_KEY

export type Media = { url: string; mime: string; filename: string | null; size: number };

// whasp's signed link needs no Authorization header: a plain GET, inside 24 hours.
async function download(media: Media): Promise<Buffer> {
  const res = await fetch(media.url);
  if (!res.ok) throw new Error(`media download failed: ${res.status}`);
  return Buffer.from(await res.arrayBuffer());
}

// Voice notes are Ogg/Opus. Re-encode to small mono MP3 so any model accepts them.
async function toMp3(audio: Buffer): Promise<Buffer> {
  const dir = await mkdtemp(join(tmpdir(), "voice-"));
  try {
    await writeFile(join(dir, "in"), audio);
    await run("ffmpeg", ["-v", "error", "-i", join(dir, "in"), "-vn", "-ac", "1", "-ar", "16000", "-b:a", "48k", join(dir, "out.mp3")]);
    return await readFile(join(dir, "out.mp3"));
  } finally {
    await rm(dir, { recursive: true, force: true });
  }
}

export async function transcribe(media: Media): Promise<string> {
  const mp3 = await toMp3(await download(media));
  const file = await toFile(mp3, "voice.mp3", { type: "audio/mpeg" });
  const result = await openai.audio.transcriptions.create({ file, model: "gpt-transcribe" });
  return result.text.trim();
}

The code writes temporary files so ffmpeg can seek in whatever container the file uses. If your host has no ffmpeg, a static build from the ffmpeg-static npm package works, or you can use a self-hosted model that decodes Ogg itself.

OpenAI lists gpt-transcribe as the recommended model for recorded speech, at $0.0045 a minute on its pricing page. A 30-second voice note costs about a fifth of a cent, and 1,000 of them cost about $2.25. If you know which languages your users speak, the guide documents a languages array for this model, such as ["en", "es"]. It also says accuracy varies by language.

Can you transcribe on your own server instead?

You can, and two open-source options cover most cases.

Choose the hosted API when you want the least code. Self-host when audio may not leave your infrastructure, when volume makes per-minute pricing matter, or when the API handles one of your languages badly.

How does an agent read a photo?

To read a photo, download the bytes and send them to a vision model as a base64 data URL with a question. Both main providers accept the same kind of input. OpenAI’s vision guide lists PNG, JPEG, WEBP and non-animated GIF. Anthropic’s vision docs list JPEG, PNG, GIF and WebP, with 10 MB per image on the Claude API. The second half of media.ts uses OpenAI’s gpt-6-luna, which its model page lists as accepting text and image input:

// media.ts, continued. Photos and PDFs: one call that returns text you can store.
export async function look(media: Media, ask: string): Promise<string> {
  const data = (await download(media)).toString("base64");
  const part =
    media.mime === "application/pdf"
      ? { type: "input_file" as const, filename: media.filename ?? "file.pdf", file_data: `data:application/pdf;base64,${data}` }
      : { type: "input_image" as const, image_url: `data:${media.mime};base64,${data}`, detail: "auto" as const };
  const response = await openai.responses.create({
    model: "gpt-6-luna",
    store: false, // do not keep the user's file at the provider
    input: [{ role: "user", content: [{ type: "input_text", text: ask }, part] }],
  });
  return response.output_text;
}

type Inbound = {
  text: string | null;
  media: Media | null;
  location?: { latitude: number; longitude: number; name?: string; address?: string };
  contacts?: string[];
  reply_to?: { id: string | null; text: string };
};

const IMAGES = ["image/jpeg", "image/png", "image/webp", "image/gif"];

// Everything a person can send, as the text your agent reads.
export async function toText(m: Inbound): Promise<string> {
  const lines: string[] = [];
  if (m.reply_to) lines.push(`(replying to: "${m.reply_to.text}")`);
  if (m.text) lines.push(m.text);
  const f = m.media;
  if (f?.mime.startsWith("audio/")) {
    lines.push(`[voice note, transcribed: ${await transcribe(f)}]`);
  } else if (f && IMAGES.includes(f.mime)) {
    lines.push(`[photo: ${await look(f, "Describe this photo for a chat assistant in under 80 words. Copy any visible text exactly.")}]`);
  } else if (f?.mime === "application/pdf") {
    lines.push(`[PDF ${f.filename}: ${await look(f, "Summarize this document for a chat assistant in under 150 words.")}]`);
  } else if (f) {
    lines.push(`[file ${f.filename ?? "unnamed"}, ${f.mime}, ${f.size} bytes: cannot be read]`);
  }
  if (m.location) lines.push(`[location: ${m.location.name ?? ""} ${m.location.address ?? ""} (${m.location.latitude}, ${m.location.longitude})]`);
  if (m.contacts?.length) lines.push(`[shared ${m.contacts.length} contact card(s)]`);
  return lines.join("\n") || "[empty message: a file over 30 MB arrives without a link]";
}

toText returns plain text for any message, which is what your history and your prompt want. A message can carry both text and media, so it reads both. When nothing is readable, it returns a placeholder, and the model can answer with “that file did not come through, can you send a smaller one?” The [brackets] match the line in the system prompt template that tells the agent what they mean.

The look function captions once and keeps text, not pixels. The file disappears from whasp after 24 hours, and Anthropic’s docs point out that when images are base64-encoded in a multi-turn chat, every request resends the image bytes. Store the description in your conversation memory, and ask for detail in the caption prompt: copying visible text exactly is what lets the agent answer a question about a receipt next week. If follow-up questions about the picture matter, copy the file to your own storage as well.

To use Claude for the same job, send the bytes as an image block, with the image before the text, which Anthropic’s docs say works best. claude-haiku-5-5 is the model Anthropic positions for high-volume, latency-sensitive work:

import Anthropic from "@anthropic-ai/sdk";

const claude = new Anthropic(); // reads ANTHROPIC_API_KEY

export async function lookWithClaude(base64: string, mediaType: "image/jpeg" | "image/png" | "image/gif" | "image/webp"): Promise<string> {
  const message = await claude.messages.create({
    model: "claude-haiku-5-5",
    max_tokens: 300,
    messages: [{
      role: "user",
      content: [
        { type: "image", source: { type: "base64", media_type: mediaType, data: base64 } },
        { type: "text", text: "Describe this photo for a chat assistant in under 80 words. Copy any visible text exactly." },
      ],
    }],
  });
  return message.content.flatMap((b) => (b.type === "text" ? [b.text] : [])).join("");
}

What about documents, locations and contacts?

A PDF goes to the model the same way as a photo. OpenAI’s PDF guide accepts it as an input_file with base64 file_data and a filename, up to 50 MB per file, and gives the model both the extracted text and the page images, which is the application/pdf branch above. For Word or Excel files, extract the text with a library you trust, or ask the person to send a PDF.

A location arrives as coordinates with an optional name and address, so describe it in a sentence and let the model decide what to do with it. A contact arrives as vCard text: parse the FN and TEL lines if your agent needs names and numbers. A reply to an earlier message carries that message’s text in reply_to, which is why toText includes it: without it, a bare “yes” has no meaning.

Treat everything you extract as untrusted. A transcript, a caption or text inside a PDF is the other person’s words, and someone can say or write “ignore your instructions” in any of them. Keep those strings as data in the prompt and never as instructions.

How do you send a file back?

To send a file, pass a public HTTPS URL in media on POST /v1/messages. The URL can point to a file up to 64 MB. whasp fetches it once when you send and deletes its copy when the send ends. When you send both text and media, the phone sends the file first and the text as a second WhatsApp message, with no caption for now, and it still counts as one API message.

await fetch("https://api.whasp.ai/v1/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.WHASP_API_KEY}`,
    "Content-Type": "application/json",
    "Idempotency-Key": "evt_2bX9kq7TfLw3:receipt",
  },
  body: JSON.stringify({
    to: "+15555550111",
    media: { url: "https://files.example.com/receipt.pdf?sig=abc123", filename: "receipt.pdf" },
    text: "Here is your receipt.",
  }),
});

The file has to stay reachable while the message waits in the queue and is sent. Keep the link valid until message.sent or message.failed arrives for that message.

Voice notes are the one thing the API cannot send. Outbound voice notes are not supported, and an audio file you send arrives as a file, not as a voice note. The practical rule is to answer a voice note with text. If you still want spoken replies, generate the audio with a text-to-speech model and send it as a file, knowing what the person will see.

How do you wire it into the webhook?

Acknowledge the event first and process it in the background, because whasp waits 10 seconds for a 2xx and then retries with backoff, and transcription plus a model call can take longer. After the signature check from WhatsApp webhook handling, the receiving side looks like this:

if (event.type === "message.received") {
  // Answer 2xx now. Transcribing and the model call run after the response.
  void handleInbound(event.id, event.data.message).catch(console.error);
}
return new Response(null, { status: 204 });

handleInbound calls toText, stores the result, asks your model and sends the reply, as in building an agent with the OpenAI API. Skip an event id you have already handled, or a retry will pay for the same transcription twice. Fire-and-forget works on a server that stays running. On a serverless host, push the event onto a queue instead.

What should you keep, and for how long?

Keep the transcript or the description, and delete the audio and the image when you are done with them. whasp removes its copy after 24 hours, and the rest is your decision. A voice recording is personal data, so write down how long you keep it and delete it when a person asks. Opt-out and consent for AI agents covers the rules. Pass store: false to the model call, as look does, if you do not want OpenAI to keep the response.

What to do next

Add toText to the handler you already have, send yourself a voice note and a photo, and read what the agent sees. Then plan the use cases that depend on it: 12 AI agents that work best on WhatsApp lists several where voice and photos are the main input. The whasp docs describe every field of the media object.