Use requestVideoFrameCallback for smooth animation timing

type RvfcVideo = HTMLVideoElement & {
  requestVideoFrameCallback: (cb: (now: number, meta: { mediaTime: number }) => void) => number;
  cancelVideoFrameCallback: (handle: number) => void;
};

export const trackVideoFrames = (video: HTMLVideoElement, onFrame: (mediaTime: number) => void) => {
  const rvfcVideo = video as RvfcVideo;
  let handle: number;
  const tick = (_now: number, meta: { mediaTime: number }) => {
    onFrame(meta.mediaTime);
    handle = rvfcVideo.requestVideoFrameCallback(tick);
  };
  handle = rvfcVideo.requestVideoFrameCallback(tick);
  return () => rvfcVideo.cancelVideoFrameCallback(handle);
};

What

video.requestVideoFrameCallback(callback) runs callback exactly when a new decoded video frame is presented to the compositor, passing the frame's actual media timestamp — unlike requestAnimationFrame, which fires on the display's refresh cycle regardless of whether a new video frame actually arrived.

Why it matters

Syncing anything to video playback — drawing a frame to a <canvas> for a filter effect, running a captioning overlay, feeding frames to a WebCodecs pipeline — with requestAnimationFrame means firing once per display refresh (typically 60Hz) even when the video itself is 24 or 30fps, or has stalled buffering. You end up redrawing stale frames, or checking video.currentTime defensively to guess whether the frame actually changed. requestVideoFrameCallback fires once per decoded frame, with metadata (mediaTime, presentedFrames, expectedDisplayTime) that tells you exactly which frame you're looking at and when it was scheduled to appear — no guessing, no redundant work on repeated frames.

How it works

• RvfcVideo is a narrow intersection type that adds requestVideoFrameCallback/cancelVideoFrameCallback on top of HTMLVideoElement — TypeScript's bundled DOM types don't include this API (see Gotchas), so video as RvfcVideo is the cast that unlocks it without resorting to any.
• trackVideoFrames sets up a self-rescheduling loop: tick calls onFrame with the frame's mediaTime, then immediately re-registers itself as the next callback — the API fires once and stops, so re-registration inside the callback is what turns it into a per-frame loop, exactly like the same pattern with requestAnimationFrame.
• handle is declared with let before the closure so cancelVideoFrameCallback can be called from outside with whichever handle is currently pending — the returned cleanup function closes over the mutable handle, always referencing the latest one.

Gotchas

• TypeScript's DOM lib doesn't know this API exists. It's a long-standing gap in lib.dom.d.ts (open since 2020) — the cast in this snippet isn't a workaround for a bug in your code, it's required regardless of TypeScript version.
• The callback fires at most once per displayed frame, not once per animation frame — a 24fps video on a 120Hz display fires roughly 5x less often than requestAnimationFrame would on the same screen.
• Support is now broad but arrived at different times: Chrome/Edge since version 83 (2020), Safari since 15.4 (2022), and Firefox only since version 132 (October 2024) — a project supporting older Firefox needs a requestAnimationFrame fallback that reads video.currentTime instead.
• meta.mediaTime can repeat across consecutive callbacks near the end of a video or during a stall — check whether it actually advanced before doing per-frame work that assumes forward progress.

Related

• requestAnimationFrame — the display-driven equivalent; still the right choice for animating anything that isn't gated on a specific video frame arriving.
• WebCodecs VideoDecoder — for lower-level frame access (decoding without an attached <video> element at all), when you need frames the DOM video pipeline doesn't expose.