Classify the document.
01 / CLASSIFYJev
139ms
40 / 40 correct
GPT-5.6 Luna
794ms
40 / 40 correct
5.73× Luna / Jev median time · equal accuracy here
DocJev / real-document benchmark / September 2026
100 tasks completedClassify in 139 ms.
Split in 210 ms.
Jev median decision time · OCR excluded
Real PDFs + your category rules → document labels and page ranges.
Document + rules40 PDFs / 8 packets
Shared LiteParse OCRsame page text for both engines
Jev / Luna decisiononly this stage is timed below
Label / page rangescompared with frozen truth
Find the boundaries.
02 / SPLITJev
210ms
7 / 8 exact packets
GPT-5.6 Luna
1,352ms
8 / 8 exact packets
6.45× Luna / Jev median time · Jev made one extra cut
40real PDFs
8assembled packets
116original pages
6.8¢total estimated API cost
One pass · small curated sample · agent-reviewed labels · no human reviewJev 1.13.0 / GPT-5.6 Luna
02 / Splitting
See exactly where they cut.
Both found every true boundary and labeled every page correctly. Jev made one extra cut. Pick a packet to compare the complete page ranges.
Select a segment to inspect its first page. Page controls show every original page.
The four adjacent same-category boundaries were recovered by both engines. Exact-packet scoring requires every category and page range to match; one extra cut makes the whole packet incorrect.
The interesting mistake / p001
The attachment
that got away.
The Federal Reserve statement says “Attachment.” Its next page begins with a new implementation heading. Jev inserted a boundary at that heading.
Frozen rule: keep a publication’s supporting material together. Pages 7–10 should be one segment. Jev returned 7–8 and 9–10; Luna kept them together. The original labels stand.
Your turn
From PDF to prediction.
Start with the real documents in the repo, then swap in your own document and category rules.
Requires Python 3.11+, uv, and a TypeSafe API key. Set TYPESAFE_API_KEY in your shell before running a decision. These commands use local LiteParse OCR and make a paid Jev call.
One-time setup
git clone https://github.com/jerryjliu/docjev.git
cd docjev
uv sync
uv run docjev doctor --smokeThe browser report is a replay of recorded results. Run these commands locally to make a new prediction.
The paperwork
Real pages. Real variety.
Tax forms, fund statements, releases, notices, and public handouts. Complete originals, right down to the dense tables and attachments.
Original English public-sector PDFs, preserved in full. The same 40 sources appear once each in eight constructed packets. Source reuse and shared templates make these observations dependent. Source rights and attribution are linked below.
The stopwatch & the tab
A little time. A little change.
JevGPT-5.6 Luna
Decision time
Recorded medians, replayed at 1× time. This animation makes no API calls. Lower is faster. Provider caches and network conditions were uncontrolled.
The entire experiment
100 / 100 complete6.8¢
Estimated API cost for both engines,
including all four warmups.
| Jev · 48 measured tasks | |
| Luna · 48 measured tasks | |
| Both engines · 4 warmups | |
| LiteParse · local OCR | $0 API fees |
Usage × recorded list prices. All usage recorded. Local compute is not priced; the $2 local guard is not a provider billing cap.
01 / Classification
Forty for forty. Twice.
Both engines picked the right category for every document. Select any tile to see its source, prediction, and recorded decision time.
Jev correctLuna correct
A correct category can still carry a review flag. “Other” is a valid category; confidence is not claimed to be calibrated.
How we kept score
Small study. Open notebook.
40 originals+ 8 constructed packets
→LiteParse onceidentical text for both engines
→Jev + Luna96 tasks + 4 warmups
→Every outcomefrozen scoring denominators
What we measured
One pass, one task at a time, with no retries. Median comparisons use the same 40 classification inputs and eight split inputs per engine. All 100 tasks completed. Local preparation took 56.6 seconds; the paid stage took 58.1 seconds. Two warmup inputs used cached OCR. Decision latency excludes preparation. End-to-end per-document latency was not measured in this run; the headline ratios do not describe the complete OCR pipeline.
What these numbers can say
This is a curated sample of short English public-sector PDFs, with labels proposed and separately reviewed by agents. Human review was not performed. Shared sources and layouts limit generalization. Eight packets cannot establish production accuracy. Scans, Office files, handwriting, repeatability, and broad confidence intervals were outside this run. All eight split inputs fit in one Jev request, so this run does not test window seams. The next small benchmark targets harder cases.
An independent open-source integration by LlamaIndex. Jev is hosted by TypeSafe; the baseline is GPT-5.6 Luna. LiteParse runs locally. This implementation does not call hosted Classify or Split APIs. No source agency endorses the project. Model probabilities are not calibrated accuracy estimates.