Method
Exact printing or it's wrong. No partial credit. Every score comes from a run.
Identification and recognition
CardsBenchmark follows CREPS (Card Recognition Evaluation Protocols). Identification is top-1 on the card. Recognition is top-1 on the exact printing: set, card number and printing. We report both side by side and never merge them. The leaderboard ranks recognition.
Scanners
Consumer scanner apps, scored S C A N. An app ranks once 4 dimensions are scored.
| Letter | Dimension | Test | Scored by |
|---|---|---|---|
| S | First answer | Open cold, scan card #1 | reviewer |
| S | 20 cards | Scan 20 public-set cards back to back | reviewer |
| A | Exact printing | CREPS public set (computed) | computed from runs |
| A | Parallel / variant trap | Frozen card #2 + the siblings stratum | reviewer |
| A | Base-era vintage | Frozen card #3 + the vintage stratum | reviewer |
| A | Japanese and other languages | The language stratum | reviewer |
| A | Backs, fakes, customs | The far-OOD stratum | reviewer |
| A | Same photo twice | Every public-set photo run twice | reviewer |
| A | Glare, sleeve, slab, blur | The hard stratum | reviewer |
| C | Price with its source | Frozen card #1 price screen | reviewer |
| C | Collection and export | Add 4 cards, export | reviewer |
| C | Condition estimate | Frozen card #4 | reviewer |
| N | Paywall before value | Tap until money is asked for | reviewer |
| N | Caps and quota | Burn through the free tier | reviewer |
| N | Claims vs measured | Vendor accuracy or speed claim vs our run | reviewer |
Graders
AI grading apps, scored G R A D E under COGS limits. An app ranks once 4 dimensions are scored.
| Letter | Dimension | Test | Scored by |
|---|---|---|---|
| G | Grounding | Pillars measured and shown: centering, corners, edges, surface. | reviewer |
| R | Repeatability | R1 within ±1: the same card, removed and re-shot three times. | reviewer |
| A | Against outcomes | Agreement on raw photos of cards later graded professionally. Needs the truth set. | reviewer |
| D | Delivery | Time to a grade, and cost. | reviewer |
| E | Evidence and claims | The No BS axis for graders: claims, paywalls, receipts. | reviewer |
Engines and APIs
Recognition engines, scored under CREPS.