WebCodecs for low-level video frame processing

// @types/dom-webcodecs isn't installed in this repo, so VideoDecoder/VideoFrame/
// EncodedVideoChunk aren't typed here — cast to `any` rather than pretend otherwise.
export const decodeFrames = (chunks: any[], onFrame: (frame: any) => void): Promise<void> => {
  const Decoder = (globalThis as any).VideoDecoder;
  const decoder = new Decoder({
    output: (frame: any) => {
      onFrame(frame);
      frame.close();
    },
    error: (err: unknown) => console.error("video decode failed", err),
  });

  decoder.configure({ codec: "vp09.00.10.08", codedWidth: 1280, codedHeight: 720 });
  for (const chunk of chunks) decoder.decode(chunk);
  return decoder.flush();
};

What

WebCodecs exposes the browser's own (often hardware-accelerated) video encoders and decoders directly as low-level primitives — VideoDecoder, VideoEncoder, VideoFrame — instead of only through the black box of <video> playback or MediaRecorder.

Why it matters

Before WebCodecs, touching individual decoded video frames meant one of two bad options: play a <video> element and grab frames off it by polling drawImage() onto a canvas (tied to real-time playback speed, no access to encoded-domain data, awkward for anything faster or slower than 1x), or ship a WASM decoder like ffmpeg.wasm to reimplement in JS what the browser's native decoder already does, frequently in hardware. WebCodecs hands you the same decoder <video> itself uses, but as an explicit queue: feed it EncodedVideoChunks, get VideoFrame objects out the other side — each one real pixel data with a timestamp — at whatever rate you can produce and consume them, decoupled entirely from real-time playback. That decoupling is what makes browser-side video editing, frame-accurate thumbnail extraction, and real-time frame effects (piped through WebRTC Insertable Streams) practical without round-tripping through a server.

How it works

decodeFrames wires up a VideoDecoder and runs a batch of encoded chunks through it:

• The VideoDecoder constructor takes exactly two callbacks: output, called once per decoded frame, and error, called if anything in the decode queue fails. There's no other way to observe either — decode() itself never rejects or throws for a bad chunk; failures always surface asynchronously through error.
• configure() sets the codec before any chunk can be decoded. The codec string is exact and codec-specific — "vp09.00.10.08" is a full VP9 profile string, not a shorthand for "VP9" — and codedWidth/codedHeight must match the actual encoded stream's dimensions.
• The loop calls decoder.decode(chunk) for each chunk without awaiting anything per-chunk — decoding is pipelined internally; output fires for each frame as it becomes ready, not necessarily in lockstep with the loop.
• decoder.flush() returns a promise that resolves once every queued chunk has been decoded and its output callback has fired — the function returns that promise so callers know when all frames have actually arrived, not just when all chunks were handed off.
• Each frame is closed (frame.close()) immediately after being handed to onFrame. A VideoFrame holds real backing memory (GPU or CPU) until closed explicitly — this isn't garbage-collected on your schedule.

Gotchas

• The very first chunk fed to a freshly configured (or just-flushed) decoder must be a key frame (chunk.type === "key"). Feeding a delta frame first throws inside the decoder's internal queue, surfaced via the error callback, not synchronously.
• If your onFrame callback needs to hold onto a frame past its own return (queue it for later drawing, buffer a few for reordering), don't close it in the loop the way this snippet does — close it only once you're actually done with it. At 30-60fps, holding every frame without closing any of them exhausts available frame memory fast.
• Codec strings are unforgiving. Call the static VideoDecoder.isConfigSupported() before configure() in production code to check a given codec/resolution combination is actually decodable on the current browser, rather than discovering it via the error callback at runtime.
• This repo's TS lib doesn't include WebCodecs types (they live in the separate @types/dom-webcodecs package, not TypeScript's bundled DOM lib) — the snippet uses any for the decoder and its callbacks to typecheck here; install that package in a real project for full types.

Browser support: Chrome/Edge 94+ (2021). Firefox 130+, desktop only — no Android Firefox support yet. Safari shipped a partial, video-only implementation from Safari 16.4, and reached full parity (including audio and image decoding) in Safari 26 in 2026 — the first point all four engines have WebCodecs, but Safari's audio path is the newest piece. Feature-detect narrowly ("AudioDecoder" in window) if audio decoding specifically matters for Safari users who haven't updated.

Related

• MediaRecorder / <video> playback — the higher-level APIs this doesn't replace; reach for WebCodecs only when you need per-frame access WebCodecs' encoded-domain queue provides and they don't.
• WebRTC Insertable Streams — pipes VideoFrame/EncodedVideoChunk objects through a live peer connection, for real-time effects processing mid-call.