The macOS ZIP Bug Where Filename Flags Lie

In the PKWARE ZIP format specification, General Purpose Bit Flag bit 11 (0x0800) declares the Language Encoding Flag (EFS): - When bit 11 is set (1): The filename and comment fields must be encoded in UTF-8. - When bit 11 is cleared (0): The filename must be encoded using historical IBM Code Page 437 (the standard MS-DOS character set). For decades, older Windows archiving tools adhered to CP437. But modern operating systems operate in a multilingual Unicode world. When you create a ZIP archive in macOS (using the native Finder Archive utility or the built-in /usr/bin/zip command-line utility), macOS encodes all filenames as UTF-8. However, macOS frequently does not set bit 11. It leaves the flag as zero. If an extraction library blindly trusts the specification flag, it reads the cleared bit, assumes CP437, and maps multibyte UTF-8 byte sequences through the CP437 codepage table. The result is catastrophic mojibake: Hindi, Devanagari, Japanese, Cyrillic, and accented Latin filenames become a chaotic mess of box-drawing characters and math symbols.

This issue reflects a universal principle of file handling: declared metadata is only a claim, but raw bytes are empirical evidence. Consider the identical failure mode in subtitle formats: an SRT file might claim to be ANSI or Latin-1 in an email handoff, but inspecting the byte order mark and testing UTF-8 validity reveals the true representation. In lib/tools/archive/zip-reader.ts, our filename decoder does not trust bit 11 blindly: ``typescript function decodeName( bytes: Uint8Array, flaggedUtf8: boolean, ): { name: string; encoding: NameEncoding } { try { return { name: new TextDecoder('utf-8', { fatal: true }).decode(bytes), encoding: 'utf-8', }; } catch { return { name: decodeCp437(bytes), encoding: 'cp437' }; } } ` The architectural strategy: 1. Attempt decoding using new TextDecoder('utf-8', { fatal: true })`. 2. UTF-8 has strict multibyte structural rules. Invalid sequences throw immediately. 3. If the bytes form valid UTF-8, accept the string as UTF-8—even if bit 11 was cleared! 4. Only if the bytes violate UTF-8 grammar do we fall back to CP437. 5. Report the actual encoding used rather than silently guessing.

In our test suite (e2e/archive-toolkit.spec.ts), we verify this behavior against authentic fixtures. The test fixture simple.zip was generated on macOS without the UTF-8 flag set. Inside the archive is an entry named: ``text notes/हिंदी.txt ` When opened in our [ZIP Archive Toolkit](/file/archive), the filename renders cleanly as notes/हिंदी.txt. Under a naive bit-11 parser, those same bytes would render as notes/हिंदी.txt` or CP437 line-drawing glyphs. Testing bytes against mathematical validity ensures robust character preservation regardless of the operating system that packaged the archive.

If the declared flag cannot be trusted, the encoding has to be inferred, and there is a reliable way to do it that does not involve guessing.

UTF-8 is a self-validating encoding. Its multi-byte sequences follow a strict grammar: a lead byte in the range C2–DF must be followed by exactly one continuation byte in 80–BF; a lead byte in E0–EF by two; F0–F4 by three. Arbitrary bytes from a legacy codepage almost never satisfy that grammar, because the probability of a random byte sequence forming valid multi-byte UTF-8 falls away sharply with length.

That gives a decision procedure with no heuristics in it:

function decodeName(bytes, utf8FlagSet) {
  if (utf8FlagSet) return new TextDecoder('utf-8').decode(bytes);
  try {
    return new TextDecoder('utf-8', { fatal: true }).decode(bytes);
  } catch {
    return new TextDecoder('cp437').decode(bytes);
  }
}

The fatal: true flag is the whole mechanism. Without it TextDecoder replaces invalid sequences with U+FFFD and never signals failure, so the fallback branch is unreachable and every legacy file decodes to a string full of replacement characters.

Two caveats worth stating. Pure ASCII is valid UTF-8 and valid CP437 simultaneously, which is fine because both decode identically. And a short CP437 name can occasionally form accidentally-valid UTF-8 — a two-byte sequence has a non-trivial chance of doing so — which is why the check is strongest on longer names and why some tools additionally score the result for plausibility.

Filename encoding is one instance of a general pattern: ZIP metadata is written by the producer and believed by the consumer, and producers disagree.

Sizes appear twice. The local file header and the central directory both carry compressed and uncompressed sizes. For streamed archives the local header may contain zeros with the real values deferred to a data descriptor after the payload. A reader trusting the local header on a streamed archive reads nothing. The central directory is authoritative — it is written last, when the sizes are known.

Timestamps are ambiguous. The classic MS-DOS timestamp has two-second resolution and no timezone, so it means local time on whichever machine created the file. Extra fields carrying UTC timestamps exist and are not always present. A ZIP that appears to shift by hours between machines is displaying this, not a bug.

Paths may escape. Nothing in the format prevents ../../etc/passwd as an entry name. Extracting without normalising and rejecting paths that climb out of the target directory is the Zip Slip vulnerability, which has affected a long list of libraries. Reject, do not sanitise silently — a name attempting to escape is evidence of something wrong.

Entry order is not authoritative. Directories may appear after the files inside them, or not at all, and a reader that assumes ordering breaks on archives from perfectly ordinary tools.