Video Editing Without a Codec: Lossless MP4 Trimming and the Scrambled Frames Bug

The conventional route to editing video in a web application is compiling a full multimedia framework like ffmpeg to WebAssembly. However, shipping a 30 MB WebAssembly payload introduces significant network weight, patent licensing questions, and slow CPU-intensive re-encoding.

When we investigated video operations in docs/VIDEO_SPIKE.md, the engineering team confirmed a crucial principle: three of the four operations people need most require no codec at all: - Trimming: Slicing a range of video requires selecting a subset of existing compressed frames and updating the container index. - Muting: Removing audio simply means excluding the audio track and its sample tables from the output file. - Audio Extraction: Pulling audio requires writing a new container holding only the audio track's compressed packets. - GIF Conversion: Converting video to GIF is the only common task that genuinely requires decoding video frames and re-encoding with an LZW palette engine.

Because trimming, muting, and audio extraction are container operations rather than transcoding tasks, our tool at /video/trim executes them in fractions of a second without re-encoding. In automated test suites, the first frame of a trimmed MP4 file decodes to bytes that are bit-for-bit identical to the source video at that timestamp (matching length and CRC32 checksums).

During the development of the MP4 writer (lib/tools/video/writer.ts), the first working implementation produced files that played back with severely scrambled, jittering video frames when tested in independent media players.

The cause was a subtle discrepancy between decode time and display time: - In modern video encoding (such as H.264/AVC), video frames are not stored in the order they are shown. Bi-directional predictive frames (B-frames) depend on both preceding and subsequent anchor frames. Consequently, a B-frame must be decoded *after* the future frame it references, but displayed *before* it. - In the ISO base media file format (MP4), the stts atom records the Decode Time-to-Sample (DTS), while the ctts (Composition Time to Sample) atom records the offset between DTS and Presentation Time (PTS).

Our initial writer omitted the ctts atom when assembling the trimmed sample tables. Without ctts, media players played back the frames in raw decode order rather than presentation order, causing video to stutter back and forth rapidly.

Critically, the audio-only extraction path passed every test during this time because audio packets do not utilize B-frame reordering. Had we only verified the simpler audio extraction feature, we would have shipped a completely broken video editor. Incorporating ctts support with both version 0 (unsigned) and version 1 (signed offsets) restored proper presentation synchronization.

Manipulating video without re-encoding imposes physical constraints that the user interface discloses plainly: - Keyframe Boundary Snapping: In compressed video, inter-frames (P-frames and B-frames) store only pixel differences relative to preceding reference frames. A video cut cannot begin on an arbitrary inter-frame; it must begin on an instantaneous decoder refresh keyframe (IDR frame). If a user requests a cut at 1.5 seconds in a file with keyframes spaced every 1.0 second, the trimmer snaps the start point to 1.0 second. The tool page reports both the requested timestamp and the actual keyframe cut point, clearly displaying the time delta. - Supported Containers: The trimmer operates on ISO BMFF containers: MP4 and MOV only. Non-compliant formats like WebM and Matroska (MKV) are refused by name. - No GIF Export: Because animated GIF creation requires full video frame decoding and palette quantization, /video/trim does not offer GIF conversion rather than providing a poor approximation. - Verbatim Codec Descriptions: The tool copies the stsd (Sample Description) atom verbatim, ensuring that codec initialization parameters (avcC for H.264 and esds for AAC) remain completely unaltered.