Classification
Whitespace & typographic normalization for comparison
Origin
Two copies of one text from different sources — PDF vs Word, editor vs email
Target
Git diff, online text comparers, redline and review tools

The problem

A diff tool compares characters, not meaning. It has no idea that a curly quote and a straight quote mean the same thing to you, or that a trailing space is invisible. So when you compare two versions of the same passage — one copied out of a PDF, the other out of Word — the tool faithfully reports every byte that differs, including the ones your eyes can’t see.

What the diff tool actually sees

These two lines are identical to a human. To a diff tool they are different, because of what the pills mark:

Before · two versions, same words
Version A (from a PDF)
The clientnbspshall pay within 30 days.·

Version B (from Word)
The client shall pay within 30 days.CRLF
— “days” uses curly ” not straight
flagged as changed · reads identical

Version A has a non-breaking space after “client” and a trailing space at the end of the line. Version B uses Windows line endings and a curly closing quote. Every one of those is a byte-level difference the diff will highlight.

The consequence

The comparison fills with false positives. A review that should show one real edit instead flags a dozen lines that look untouched, and the person reading the redline has to squint at each one to decide whether anything actually changed. Lawyers reviewing a contract, writers checking an edit, and developers reading a diff all lose the one thing the tool is supposed to give them: confidence that a highlighted line really changed.

The fix: standardize both sides first

Run both versions through the same sequence before you compare them, so any difference that remains is a real one:

  • Straighten Quotes (key 5) — curly quotes, dashes, and ellipses become standard characters on both sides.
  • Remove Hidden Characters (key 8) — non-breaking and exotic spaces normalize to ordinary spaces; zero-width marks disappear.
  • Trim Each Line (key 3) and Remove Extra Spaces (key 2) — trailing spaces and doubled spaces are flattened.
After · both versions normalized, then compared
  The client shall pay within 30 days.
- The client shall pay within 30 days.
+ The client shall pay within 45 days.
one real change, clearly shown

With both texts reduced to the same canonical form, the diff collapses to the single edit that matters — 30 days became 45 — instead of burying it under invisible noise.

The clean fix

Whenever two copies of a text come from different places, paste each into cleanplaintext.com, run the same actions on both, and only then feed them to your diff tool. Deterministic input in, meaningful diff out.