10
Choosing the right model: GPU benchmarks
Tested open-source OCR and vision models against a managed AI service on the same documents, then matched each document type to the model that earned it.
- Role
- Full Stack Developer · Mavericks Tech Solutions
- When
- Sep 2025 – present
- Relevant for
- AI & AgentsTesting & QACloud & Infrastructure
My part
I led the evaluation: rented GPU instances, deployed each model, tunnelled ports to test from my machine, and measured field-level accuracy against cost per document.
Problem
The managed AI service was accurate, but it was billed per call, so the cost grew with every document.
Approach
Deployed 5+ open-source OCR and vision-language models on Akamai Cloud and Google Cloud GPUs (NVIDIA L4, RTX 6000 Pro), ran the same labelled documents through each, and scored every field by hand. Checkboxes got their own benchmark because they were the hardest part. Also evaluated AWS Bedrock Knowledge Bases for document Q&A; made-up answers on missing checkbox marks were the blocking issue.
Outcome
A hybrid setup chosen by measured accuracy and cost: NuExtract 3 and Docling for dense forms, PaddleOCR then Kimi for tables and general PDFs.
What I did
Rented GPUs
GPU instances on Akamai Cloud and Google Cloud with NVIDIA L4 and RTX 6000 Pro.
Akamai CloudGoogle Cloud
Tested from my machine
Deployed each model and tunnelled ports to test it locally.
DockerSSH tunnels
Same documents for every model
5+ open-source OCR and vision-language models run on the same labelled set.
NuExtract 3PaddleOCRPaddleOCR-VLDoclingGOT-OCR
Scored by hand
Every field scored; checkboxes got their own benchmark because they were the hardest part.
Accuracy against cost
Field-level accuracy measured against cost per document.
Bedrock ruled out
AWS Bedrock Knowledge Bases tried for document Q&A; made-up answers on missing checkbox marks were the blocking issue.
AWS Bedrock
Stack
- Akamai Cloud
- Google Cloud
- NVIDIA L4
- RTX 6000 Pro
- Docker
- SSH tunnels
- Python
- AWS Bedrock