The macOS ZIP UTF-8 Flag Bug: When Archiver Flags Lie and Bytes Tell the Truth

In the PKWARE ZIP format specification, General Purpose Bit Flag bit 11 (0x0800) declares the Language Encoding Flag (EFS): - When bit 11 is set (1): The filename and comment fields must be encoded in UTF-8. - When bit 11 is cleared (0): The filename must be encoded using historical IBM Code Page 437 (the standard MS-DOS character set). For decades, older Windows archiving tools adhered to CP437. But modern operating systems operate in a multilingual Unicode world. When you create a ZIP archive in macOS (using the native Finder Archive utility or the built-in /usr/bin/zip command-line utility), macOS encodes all filenames as UTF-8. However, macOS frequently does not set bit 11. It leaves the flag as zero. If an extraction library blindly trusts the specification flag, it reads the cleared bit, assumes CP437, and maps multibyte UTF-8 byte sequences through the CP437 codepage table. The result is catastrophic mojibake: Hindi, Devanagari, Japanese, Cyrillic, and accented Latin filenames become a chaotic mess of box-drawing characters and math symbols.

This issue reflects a universal principle of file handling: declared metadata is only a claim, but raw bytes are empirical evidence. Consider the identical failure mode in subtitle formats: an SRT file might claim to be ANSI or Latin-1 in an email handoff, but inspecting the byte order mark and testing UTF-8 validity reveals the true representation. In lib/tools/archive/zip-reader.ts, our filename decoder does not trust bit 11 blindly: ``typescript function decodeName( bytes: Uint8Array, flaggedUtf8: boolean, ): { name: string; encoding: NameEncoding } { try { return { name: new TextDecoder('utf-8', { fatal: true }).decode(bytes), encoding: 'utf-8', }; } catch { return { name: decodeCp437(bytes), encoding: 'cp437' }; } } ` The architectural strategy: 1. Attempt decoding using new TextDecoder('utf-8', { fatal: true })`. 2. UTF-8 has strict multibyte structural rules. Invalid sequences throw immediately. 3. If the bytes form valid UTF-8, accept the string as UTF-8—even if bit 11 was cleared! 4. Only if the bytes violate UTF-8 grammar do we fall back to CP437. 5. Report the actual encoding used rather than silently guessing.

In our test suite (e2e/archive-toolkit.spec.ts), we verify this behavior against authentic fixtures. The test fixture simple.zip was generated on macOS without the UTF-8 flag set. Inside the archive is an entry named: ``text notes/हिंदी.txt ` When opened in our [ZIP Archive Toolkit](/file/archive), the filename renders cleanly as notes/हिंदी.txt. Under a naive bit-11 parser, those same bytes would render as notes/हिंदी.txt` or CP437 line-drawing glyphs. Testing bytes against mathematical validity ensures robust character preservation regardless of the operating system that packaged the archive.