Competitive bake-off

Cover vs

Season 2, same 18 PDFs, 149 private details, 12 engines ranked by mistakes (leaked private details weigh most, and over-redactions count too):

  1. Cover: 100% caught, 0 leaked, 12 over-redacted, 3.2 s per PDF
  2. Google Gemma 4: 93% caught, 16 leaked, 157 over-redacted, 26 s per PDF
  3. Microsoft Phi-4: 75% caught, 50 leaked, 149 over-redacted, 37 s per PDF
  4. Alibaba Qwen 3: 75% caught, 50 leaked, 188 over-redacted, 4.5 min per PDF
  5. Alibaba Qwen 2.5: 65% caught, 63 leaked, 167 over-redacted, 23 s per PDF
  6. DeepSeek R1: 63% caught, 67 leaked, 122 over-redacted, 4.1 min per PDF
  7. Meta Llama 3.1: 67% caught, 59 leaked, 260 over-redacted, 24 s per PDF
  8. Apple Intelligence: 50% caught, 76 leaked, 129 over-redacted, 4.6 s per PDF
  9. Google Gemma 3: 69% caught, 53 leaked, 623 over-redacted, 24 s per PDF
  10. OpenAI Privacy Filter: 63% caught, 54 leaked, 709 over-redacted, 13 s per PDF
  11. America.gov PII filter: 36% caught, 98 leaked, 370 over-redacted, 0.1 s per PDF
  12. OpenMed Nemotron filter: 41% caught, 95 leaked, 1290 over-redacted, 4.4 s per PDF
Last updated October 3, 2026 · engine v1.0.2

Sending PDFs with private identity information to a cloud Ai like Claude or ChatGPT is a bad idea. But could a local Ai on your own computer redact them instead? This bake-off runs Cover’s engine and the leading local Ai engines on the same 18 bank, tax and medical PDFs. So far the local Ai engines fall behind Cover in three ways: they over-redact, most are slow, and none offer a fast way to fix their mistakes.

Seasons
Season 3
Season 2 · 18 PDFs · 149 private details (PII)

PDF selection focuses on money, home and insurance documents. Real-world samples for anyone asking Ai about their own money, or accountants and tax pros who redact a client's documents before asking Ai for a second opinion. Watch Cover's engine improve with each release, even outpacing Google's tech.

  • Money 9
  • Home 5
  • Health & insurance 3
  • Taxes 1
Engine quality over time
CoverGemma 4
80%90%100%101001000% PII caughtover-redactions (log)Season 1 · 15 docsSeason 1 · 15 docsSeason 2 · 18 docsSeason 2 · 18 docs0.12.80.13.20.13.30.14.00.14.10.15.10.16.40.16.50.16.60.18.50.20.11.0.2v0.12.8: Gemma 4 caught 79% (22 leaked)v0.13.2: Gemma 4 caught 90% (11 leaked)v0.13.3: Gemma 4 caught 90% (11 leaked)v0.14.0: Gemma 4 caught 90% (11 leaked)v0.14.1: Gemma 4 caught 90% (11 leaked)v0.15.1: Gemma 4 caught 90% (11 leaked)v0.16.4: Gemma 4 caught 89% (11 leaked)v0.16.5: Gemma 4 caught 92% (12 leaked)v0.16.6: Gemma 4 caught 95% (9 leaked)v0.18.5: Gemma 4 caught 97% (5 leaked)v0.20.1: Gemma 4 caught 97% (5 leaked)v1.0.2: Gemma 4 caught 93% (16 leaked)v0.12.8: Cover caught 77% of PII (19 leaked)v0.13.2: Cover caught 79% of PII (18 leaked)v0.13.3: Cover caught 86% of PII (12 leaked)v0.14.0: Cover caught 86% of PII (12 leaked)v0.14.1: Cover caught 86% of PII (12 leaked)v0.15.1: Cover caught 91% of PII (5 leaked)v0.16.4: Cover caught 95% of PII (2 leaked)v0.16.5: Cover caught 98% of PII (3 leaked)v0.16.6: Cover caught 98% of PII (3 leaked)v0.18.5: Cover caught 98% of PII (3 leaked)v0.20.1: Cover caught 98% of PII (3 leaked)v1.0.2: Cover caught 100% of PII (0 leaked)79%93%77%100%v0.12.8: Gemma 4 over-redacted 80 non-PII itemsv0.13.2: Gemma 4 over-redacted 78 non-PII itemsv0.13.3: Gemma 4 over-redacted 78 non-PII itemsv0.14.0: Gemma 4 over-redacted 78 non-PII itemsv0.14.1: Gemma 4 over-redacted 78 non-PII itemsv0.15.1: Gemma 4 over-redacted 79 non-PII itemsv0.16.4: Gemma 4 over-redacted 90 non-PII itemsv0.16.5: Gemma 4 over-redacted 133 non-PII itemsv0.16.6: Gemma 4 over-redacted 123 non-PII itemsv0.18.5: Gemma 4 over-redacted 164 non-PII itemsv0.20.1: Gemma 4 over-redacted 160 non-PII itemsv1.0.2: Gemma 4 over-redacted 157 non-PII itemsv0.12.8: Cover over-redacted 15 non-PII itemsv0.13.2: Cover over-redacted 51 non-PII itemsv0.13.3: Cover over-redacted 16 non-PII itemsv0.14.0: Cover over-redacted 16 non-PII itemsv0.14.1: Cover over-redacted 12 non-PII itemsv0.15.1: Cover over-redacted 22 non-PII itemsv0.16.4: Cover over-redacted 36 non-PII itemsv0.16.5: Cover over-redacted 62 non-PII itemsv0.16.6: Cover over-redacted 26 non-PII itemsv0.18.5: Cover over-redacted 22 non-PII itemsv0.20.1: Cover over-redacted 19 non-PII itemsv1.0.2: Cover over-redacted 12 non-PII items801571512
Per-version detail: all 12 releases (recall · leaks · over-redaction · pages better / tied / worse)
versionseasondatePII caughtover-redactionpages better / tied / worsedocs
1.0.2S22026-10-03100% vs 93%12 vs 157n/a18
0.20.1S22026-09-0398% vs 97%19 vs 160n/a18
0.18.5S22026-08-2798% vs 97%22 vs 164n/a18
0.16.6S22026-07-1898% vs 95%26 vs 123n/a18
0.16.5S22026-07-1898% vs 92%62 vs 133n/a18
0.16.4S12026-07-1695% vs 89%36 vs 90n/a15
0.15.1S12026-07-0791% vs 90%22 vs 79n/a15
0.14.1S12026-06-2986% vs 90%12 vs 78n/a15
0.14.0S12026-06-2686% vs 90%16 vs 78n/a15
0.13.3S12026-06-1586% vs 90%16 vs 78n/a15
0.13.2S12026-06-1579% vs 90%51 vs 78n/a15
0.12.8S12026-06-0477% vs 79%15 vs 80n/a15

12 versions · 14 raw runs · de-duped latest-per-version

Local redaction landscape
↙ the goal013000100Over-redactionLeaksCoverGemma 4Phi-4Qwen 3Qwen 2.5DeepSeek R1Llama 3.1Apple IntelligenceGemma 3Privacy FilterRampartNemotron
#enginePII accuracyOver-redactionSpeed
1🥇 CoverExcellentVery goodExcellent
2🥈 Gemma 4GoogleVery goodVery poorGood
3🥉 Phi-4MicrosoftGoodVery poorPoor
4Qwen 3AlibabaGoodVery poorVery poor
5Qwen 2.5AlibabaPoorVery poorGood
6DeepSeek R1DeepSeekPoorVery poorVery poor
7Llama 3.1MetaPoorVery poorGood
8Apple IntelligenceAppleVery poorVery poorExcellent
9Gemma 3GooglePoorVery poorGood
10Privacy FilterOpenAIPoorVery poorVery good
11RampartAmerica.govVery poorVery poorExcellent
12NemotronOpenMedVery poorVery poorExcellent

Per-page side-by-sides

Check out the actual pages below and how each engine redacted them, side by side with the original.

Red outline: private detail left visibleBlue outline: over-redaction
🥈 2nd place
Worth flagging
Google Gemma 4
PII accuracy: Very goodOver-redaction: Very poorSpeed: Good ~26 sec/PDF

Catches 94% of the private details to Cover's 100%, but still leaves 13 visible and over-redacts about eleven times as much.

Cover’s edge
Holds even if a model matches Cover’s accuracy.
  • One-click edits fix any box instantly; no re-prompting (LLMs over-redact).
  • Fully offline files never leave your device.
  • Seconds per file no datacenter hardware to buy or maintain.

2025_Tax_Return_Okafor-Brennan.pdf

Tax return · 15 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 42 left visible9 over-redacted
Page 1, redacted by Gemma 4
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 42 left visible2 over-redacted
Page 2, redacted by Gemma 4
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 41 over-redacted
Page 3, redacted by Gemma 4
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 41 left visible2 over-redacted
Page 4, redacted by Gemma 4
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 4No mistakes
Page 5, redacted by Gemma 4
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 41 over-redacted
Page 7, redacted by Gemma 4
Page 8Tie
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 4No mistakes
Page 8, redacted by Gemma 4

WellsFargo-sample-statement.pdf

Bank statement · 8 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Cover wins
Original
Page 2, original
Cover2 over-redacted
Page 2, redacted by Cover
Gemma 43 over-redacted
Page 2, redacted by Gemma 4

capitalone-sample-estatement.pdf

Bank statement · 4 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4

commercebank-sample-statement.pdf

Bank statement · 4 private details
Page 1Cover wins
Original
Page 1, original
Cover3 over-redacted
Page 1, redacted by Cover
Gemma 46 over-redacted
Page 1, redacted by Gemma 4

Fidelity-sample-statement.pdf

Brokerage statement · 9 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 41 left visible
Page 1, redacted by Gemma 4
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 4No mistakes
Page 4, redacted by Gemma 4
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 4No mistakes
Page 5, redacted by Gemma 4
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 4No mistakes
Page 6, redacted by Gemma 4
Page 7Tie
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 4No mistakes
Page 7, redacted by Gemma 4
Page 8Tie
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 4No mistakes
Page 8, redacted by Gemma 4
Page 9Tie
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 4No mistakes
Page 9, redacted by Gemma 4
Page 10Tie
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 4No mistakes
Page 10, redacted by Gemma 4
Page 11Tie
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Gemma 4No mistakes
Page 11, redacted by Gemma 4
Page 12Tie
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Gemma 4No mistakes
Page 12, redacted by Gemma 4
Page 13Tie
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Gemma 4No mistakes
Page 13, redacted by Gemma 4
Page 14Tie
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Gemma 4No mistakes
Page 14, redacted by Gemma 4
Page 15Cover wins
Original
Page 15, original
Cover1 over-redacted
Page 15, redacted by Cover
Gemma 44 over-redacted
Page 15, redacted by Gemma 4
Page 16Tie
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Gemma 4No mistakes
Page 16, redacted by Gemma 4
Page 17Tie
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Gemma 4No mistakes
Page 17, redacted by Gemma 4
Page 18Tie
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Gemma 4No mistakes
Page 18, redacted by Gemma 4
Page 19Tie
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Gemma 4No mistakes
Page 19, redacted by Gemma 4
Page 20Tie
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Gemma 4No mistakes
Page 20, redacted by Gemma 4
Page 21Tie
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Gemma 4No mistakes
Page 21, redacted by Gemma 4
Page 22Tie
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Gemma 4No mistakes
Page 22, redacted by Gemma 4
Page 23Tie
Original
Page 23, original
CoverNo mistakes
Page 23, redacted by Cover
Gemma 4No mistakes
Page 23, redacted by Gemma 4
Page 24Tie
Original
Page 24, original
CoverNo mistakes
Page 24, redacted by Cover
Gemma 4No mistakes
Page 24, redacted by Gemma 4
Page 25Tie
Original
Page 25, original
CoverNo mistakes
Page 25, redacted by Cover
Gemma 4No mistakes
Page 25, redacted by Gemma 4
Page 28Gemma 4 wins
Original
Page 28, original
Cover1 over-redacted
Page 28, redacted by Cover
Gemma 4No mistakes
Page 28, redacted by Gemma 4

fidelity-large-print-sample-statement.pdf

Brokerage statement · 5 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 4No mistakes
Page 3, redacted by Gemma 4

pershing-2024-sample-tax-yearend.pdf

Brokerage statement · 7 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 42 over-redacted
Page 3, redacted by Gemma 4
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 42 over-redacted
Page 4, redacted by Gemma 4
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 4No mistakes
Page 5, redacted by Gemma 4
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 45 over-redacted
Page 6, redacted by Gemma 4
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 44 over-redacted
Page 7, redacted by Gemma 4
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 44 over-redacted
Page 8, redacted by Gemma 4

ubs-portfolio-summary-sample.pdf

Brokerage statement · 7 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 49 over-redacted
Page 1, redacted by Gemma 4
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 45 over-redacted
Page 2, redacted by Gemma 4
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 41 left visible9 over-redacted
Page 3, redacted by Gemma 4
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 44 over-redacted
Page 4, redacted by Gemma 4
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 44 over-redacted
Page 5, redacted by Gemma 4
Page 6Cover wins
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 44 over-redacted
Page 6, redacted by Gemma 4
Page 7Cover wins
Original
Page 7, original
CoverNo mistakes
Page 7, redacted by Cover
Gemma 41 left visible9 over-redacted
Page 7, redacted by Gemma 4
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 44 over-redacted
Page 8, redacted by Gemma 4
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 41 left visible6 over-redacted
Page 9, redacted by Gemma 4
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 44 over-redacted
Page 10, redacted by Gemma 4
Page 11Tie
Original
Page 11, original
CoverNo mistakes
Page 11, redacted by Cover
Gemma 4No mistakes
Page 11, redacted by Gemma 4
Page 12Cover wins
Original
Page 12, original
CoverNo mistakes
Page 12, redacted by Cover
Gemma 44 over-redacted
Page 12, redacted by Gemma 4
Page 13Cover wins
Original
Page 13, original
CoverNo mistakes
Page 13, redacted by Cover
Gemma 41 over-redacted
Page 13, redacted by Gemma 4
Page 14Cover wins
Original
Page 14, original
CoverNo mistakes
Page 14, redacted by Cover
Gemma 44 over-redacted
Page 14, redacted by Gemma 4
Page 15Cover wins
Original
Page 15, original
CoverNo mistakes
Page 15, redacted by Cover
Gemma 41 left visible1 over-redacted
Page 15, redacted by Gemma 4
Page 16Cover wins
Original
Page 16, original
CoverNo mistakes
Page 16, redacted by Cover
Gemma 44 over-redacted
Page 16, redacted by Gemma 4
Page 17Tie
Original
Page 17, original
CoverNo mistakes
Page 17, redacted by Cover
Gemma 4No mistakes
Page 17, redacted by Gemma 4
Page 18Cover wins
Original
Page 18, original
CoverNo mistakes
Page 18, redacted by Cover
Gemma 44 over-redacted
Page 18, redacted by Gemma 4
Page 19Cover wins
Original
Page 19, original
CoverNo mistakes
Page 19, redacted by Cover
Gemma 41 left visible1 over-redacted
Page 19, redacted by Gemma 4
Page 20Cover wins
Original
Page 20, original
CoverNo mistakes
Page 20, redacted by Cover
Gemma 44 over-redacted
Page 20, redacted by Gemma 4
Page 21Cover wins
Original
Page 21, original
CoverNo mistakes
Page 21, redacted by Cover
Gemma 41 left visible5 over-redacted
Page 21, redacted by Gemma 4
Page 22Tie
Original
Page 22, original
CoverNo mistakes
Page 22, redacted by Cover
Gemma 4No mistakes
Page 22, redacted by Gemma 4

cfpb-sample-credit-card-statement-handout.pdf

Credit-card statement · 2 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4

allstate-auto-declarations-sample.pdf

Insurance declaration · 7 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 41 over-redacted
Page 2, redacted by Gemma 4

cfpb-ficus-bank-loan-estimate-sample.pdf

Loan estimate · 9 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 4No mistakes
Page 3, redacted by Gemma 4

cigna-eob-example.pdf

Medical claim / EOB · 7 private details
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4

medicare-sample-part-b-msn.pdf

Medical claim / EOB · 6 private details
Page 2Cover wins
Original
Page 2, original
Cover4 over-redacted
Page 2, redacted by Cover
Gemma 41 left visible
Page 2, redacted by Gemma 4
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 41 left visible
Page 3, redacted by Gemma 4
Page 4Cover wins
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 41 over-redacted
Page 4, redacted by Gemma 4
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 4No mistakes
Page 5, redacted by Gemma 4
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 4No mistakes
Page 6, redacted by Gemma 4

cfpb-draft-periodic-mortgage-statement.pdf

Mortgage statement · 4 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4

Lease_Agreement_signed.pdf

Other · 16 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 43 left visible2 over-redacted
Page 1, redacted by Gemma 4

transunion-cameo-credit-report-sample.pdf

Other · 23 private details
Page 1Cover wins
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 42 left visible
Page 1, redacted by Gemma 4
Page 2Cover wins
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 411 over-redacted
Page 2, redacted by Gemma 4
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 4No mistakes
Page 3, redacted by Gemma 4
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 4No mistakes
Page 4, redacted by Gemma 4
Page 5Cover wins
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 42 left visible
Page 5, redacted by Gemma 4
Page 8Cover wins
Original
Page 8, original
CoverNo mistakes
Page 8, redacted by Cover
Gemma 44 over-redacted
Page 8, redacted by Gemma 4
Page 9Cover wins
Original
Page 9, original
CoverNo mistakes
Page 9, redacted by Cover
Gemma 42 over-redacted
Page 9, redacted by Gemma 4
Page 10Cover wins
Original
Page 10, original
CoverNo mistakes
Page 10, redacted by Cover
Gemma 45 over-redacted
Page 10, redacted by Gemma 4

att-uverse-sample-bill.pdf

Utility bill · 13 private details
Page 1Tie
Original
Page 1, original
CoverNo mistakes
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4
Page 3Cover wins
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 41 left visible
Page 3, redacted by Gemma 4
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 4No mistakes
Page 4, redacted by Gemma 4

pge-sample-bill-sparky-joule.pdf

Utility bill · 3 private details
Page 1Gemma 4 wins
Original
Page 1, original
Cover1 over-redacted
Page 1, redacted by Cover
Gemma 4No mistakes
Page 1, redacted by Gemma 4
Page 2Tie
Original
Page 2, original
CoverNo mistakes
Page 2, redacted by Cover
Gemma 4No mistakes
Page 2, redacted by Gemma 4
Page 3Tie
Original
Page 3, original
CoverNo mistakes
Page 3, redacted by Cover
Gemma 4No mistakes
Page 3, redacted by Gemma 4
Page 4Tie
Original
Page 4, original
CoverNo mistakes
Page 4, redacted by Cover
Gemma 4No mistakes
Page 4, redacted by Gemma 4
Page 5Tie
Original
Page 5, original
CoverNo mistakes
Page 5, redacted by Cover
Gemma 4No mistakes
Page 5, redacted by Gemma 4
Page 6Tie
Original
Page 6, original
CoverNo mistakes
Page 6, redacted by Cover
Gemma 4No mistakes
Page 6, redacted by Gemma 4

Full results

Cover v1.0.2 vs Google Gemma 4 gemma4:12b-mlx · 18 real-world sample documents.

metric · all 18 docsCoverGemma 4
Recall: PII correctly redacted100%93%
Leaks: confirmed PII left visible016
Over-redaction: non-PII redacted by mistake12157
Speed: median time per PDF~3.2 sec/PDF~26 sec/PDF
Per-document breakdown: all 18 docs (one outlier can dominate the totals)
documentCover
leaks		Gemma 4
leaks		Cover
over		Gemma 4
over
ubs-portfolio-summary-sample01086
transunion-cameo-credit-report-sample03022
pershing-2024-sample-tax-yearend00017
2025_Tax_Return_Okafor-Brennan06015
commercebank-sample-statement0036
Fidelity-sample-statement0124
WellsFargo-sample-statement0023
Lease_Agreement_signed0302
allstate-auto-declarations-sample0001
medicare-sample-part-b-msn0141
att-uverse-sample-bill0100
capitalone-sample-estatement0000
cfpb-draft-periodic-mortgage-statement0000
cfpb-ficus-bank-loan-estimate-sample0000
cfpb-sample-credit-card-statement-handout0000
cigna-eob-example0000
fidelity-large-print-sample-statement0000
pge-sample-bill-sparky-joule0010
TOTAL01612157

How the test works

Test set: 18 public sample documents (149 verified PII items), held fixed across runs so results stay comparable. Every engine redacted the same documents; each output is scored against a Claude-authored gold standard (confirmed + ambiguous tiers), and every counted over-redaction is kept auditable. Pages and the leaderboard use the same weighting: leaked private details weigh most, and over-redactions count too.