fix: expand typographic ligatures instead of dropping them (#172)

[viwe-monai]

What

Expand Unicode typographic ligatures (U+FB00–U+FB06) to their constituent ASCII letters in clean_text(), instead of dropping them entirely.

Code point	Glyph	Expansion
U+FB00	ff	ff
U+FB01	fi	fi
U+FB02	fl	fl
U+FB03	ffi	ffi
U+FB04	ffl	ffl
U+FB05/06	ſt/st	st

Why

Fixes #172 — PDFs using typographic ligatures silently corrupt words in the output: classifies → classi es, affected → a ected.

How it was verified

• Added ligatures_expanded test covering all six code points
• Existing layout_invisibles_stripped and join_controls_preserved tests pass (no changes to their behavior)

Follow-ups deliberately left out

• A fix in pdf_inspector's ToUnicode mapping would prevent the characters from being dropped at the parsing level. This anydoc-level fix is a safety net regardless of the upstream behavior.

---

Summary by cubic

Expands Unicode typographic ligatures (U+FB00–U+FB06) to their ASCII letters instead of dropping them, fixing silent word corruption in PDF extraction (classifies → classi es, affected → a ected).

• Applies expansion in both clean_text() and the PDF extraction path (expand_ligatures() in src/formats/pdf.rs), since PDF output bypasses the shared normalizer.
• Adds a regression test covering all six code points; layout_invisibles_stripped and join_controls_preserved behavior is unchanged.
• A pdf_inspector-level ToUnicode fix is deliberately out of scope; this anydoc-side change is a safety net regardless of upstream behavior.

^{Written for commit 312b6c6. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 1 file

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[viwe-monai]

Thanks for catching both issues! Fixed in 312b6c6:

P2 (PDF path bypasses clean_text): You are right — to_markdown_bytes routes Format::Pdf directly to formats::pdf::to_markdown, which returns pdf_inspector's markdown without any shared normalization. Added a dedicated expand_ligatures() in src/formats/pdf.rs and applied it to the extracted markdown before returning. The clean_text() expansion remains as defense-in-depth for the document-model path.

P3 (missing U+FB05/FB06 coverage): Added fa\u{fb05} → "fast" and ju\u{fb06} → "just" to ligatures_expanded.