How we run Whisper locally on Apple Silicon

Historical note — published 2025. Minuta has since moved its transcription pipeline to the cloud (Deepgram primary, OpenAI/Groq fallback). See ADR 0002 and ADR 0005 for the current architecture. This post remains for engineering context, not product positioning.

There are two obvious ways to transcribe meeting audio in a Mac app. The first is to ship the PCM up to OpenAI's Whisper API — three lines of code, low latency, $0.006 per minute. The second is to compile whisper.cpp and run everything on the user's chip. We picked the second. This post is about why and — more importantly — what it costs.

Why local

Three reasons, in order of weight:

1. Speech in a customer call is sensitive data. Discovery calls in legal,

healthcare, and financial verticals — those prospects bail the second they see a bot join the call. "100% local, nothing leaves your Mac" is a promise the Whisper API simply cannot make. 2. Cost scales badly. $0.006/min × 8 hours/day × 22 days = $63/month per heavy Pro user. That eats the margin on a $9/month subscription. Local takes that to zero. 3. Network latency disappears. With whisper.cpp on Metal, transcribing 15 seconds of audio takes ~600ms on an M2 Pro. Cloud, with TLS handshake and server queue, lands between 1.5 and 3 seconds. For a copilot that has to suggest a reply mid-call, that difference changes the UX.

The stack

AudioCapture (Swift, ScreenCaptureKit)
       │ Float32 PCM @ 16 kHz mono via stdout
       ▼
Whisper (TS) ── HTTP ──▶ whisper-server (vendor/whisper, Metal)
       │
       └─ frame-level VAD (30ms RMS) ──▶ buffers up to 600ms of silence or 15s

whisper-server is the HTTP binary from whisper.cpp. It keeps the model (ggml-medium.en.bin, ~547 MB) loaded in Metal GPU memory between requests. Spinning it up takes ~3 seconds on first start; after that, each chunk is just a multipart POST with the 15-second WAV. The model never leaves the Neural Engine.

Spawning it and keeping it alive is the Electron main process's job — we have a health-check and auto-respawn if the process dies. The TypeScript client talks to it on localhost:49220.

VAD: the part nobody talks about

Whisper, like any encoder-decoder seq2seq model, hallucinates on silence. Feed it 15 seconds of audio where nobody is speaking and it will reply with a random phrase from some training dataset (the classic one is "Thank you for watching" from YouTube transcripts). To avoid that, the pipeline runs frame-by-frame VAD (Voice Activity Detection) before calling Whisper.

Cheap, simple implementation: RMS over a 30ms window. Above a threshold (~-40 dBFS in practice), it's speech; below it, silence. When it sees 600ms of continuous silence after at least 1 second of speech, it closes the segment and ships it to whisper-server. Hard cap of 15 seconds so a long monologue doesn't take forever to surface on screen.

// src/main/whisper.ts (simplified)
const SILENCE_MS = 600;
const MAX_SEGMENT_MS = 15_000;

if (rms(frame) > THRESHOLD) {
  buffer.push(frame);
  silenceMs = 0;
} else {
  silenceMs += FRAME_MS;
  if (buffer.length > 0 && silenceMs >= SILENCE_MS) {
    flush();
  }
}
if (segmentDurationMs() >= MAX_SEGMENT_MS) {
  flush();
}

Even with VAD, Whisper still hallucinates in some edge cases — a long pause before a short word, for example. For those we shipped a regex list of "known phrases that pop up out of nowhere" (HALLUCINATION_PATTERNS in pipeline.ts) that we strip before showing the text to the user. Not elegant; just what works.

How much RAM

The medium.en model in FP16 sits at ~1.2 GB of unified VRAM when loaded. Add whisper-server itself, llama-server (Qwen 2.5 7B Q4_K_M, ~4.5 GB) for minutes and copilot, Electron, the Swift AudioCapture, and the Vite renderer, and the app's steady-state during a meeting is ~7 GB. On a 16 GB Mac that's noticeable but tolerable; from 24 GB up it doesn't even register in Activity Monitor.

The choice of medium.en was deliberate. large-v3 is more accurate but it's ~3 GB and twice the per-chunk latency. For workplace-call speech (technical but not academic vocabulary), medium.en is ~3% worse and runs 2× faster. The trade-off is worth it.

Pitfalls that cost a day

Absolute rpath. whisper.cpp (and llama.cpp) is built with CMake, which by default bakes the build path into the binary as an absolute rpath. Move the repo to a different folder — or have the user install the app to a path different from the build path — and the dynamic linker breaks with dyld: Library not loaded: @rpath/libwhisper.1.dylib. The fix is to rebuild in-place with a relative rpath (@executable_path/) or to ship the whole folder via extraResources in electron-builder. We landed it in CI as npm run build:whisper.

Apple Silicon vs. Intel. whisper.cpp on Metal does not run on Intel Macs; those fall back to the AVX2 CPU backend, which is ~5× slower. In Minuta that means a real-world Intel Mac can't keep up with real-time transcription and accumulates backlog. The decision: ship a dual arm64 + x64 build, but the Intel UI shows an explicit notice that copilot and diarization are disabled (you can still transcribe, just not in real time).

Gatekeeper on first launch. Since we don't notarize yet (shipping with 1.0), the first launch of the .app downloaded from the web shows "damaged and can't be opened". User workaround: xattr -dr com.apple.quarantine /Applications/Minuta.app. It's documented on the download page, but it's real friction that lost us beta testers.

What we did NOT do

A few things that looked like good ideas and didn't survive contact:

• Chunk-by-chunk streaming to Whisper. whisper.cpp supports it, but accuracy

drops a lot when the model doesn't see a whole sentence. We kept VAD-driven 5–15s chunks; the UX feels nearly as real-time, with 30% fewer errors. - Live diarization. pyannote.audio is good, but it runs on Python over PyTorch with no decent Metal acceleration (still true in 2026). We diarize the whole WAV after stop() and back-fill speaker_id by temporal overlap. Less fancy, but deterministic. - Our own server in the cloud. We already had a Go API on ECS for auth and billing; the temptation to ship audio there for diarization was real. We killed it for the same reason as the start of this post — it breaks the "nothing leaves your Mac" promise and opens the door to DPA conversations with enterprise.

So what?

The short version: running Whisper locally on Apple Silicon is viable in 2026, costs ~7 GB of RAM for the whole app, ~600ms of latency per 15s chunk, and removes entire privacy conversations that would kill the product. In exchange you inherit packaging problems OpenAI hides from you — rpath, xattr, dual-arch builds, Metal vs. AVX2.

If you're building anything that processes customer speech and the margin matters, it's worth looking at this path before shipping everything to the cloud.

If you want to see the result in practice, Minuta is Mac-only for now — and what you just read is what runs on your Mac when you record a call.