I get this question every time someone sees the "Snap, don't type" feature in PocketFi:
"Wait — where does the photo of my receipt go? Are you uploading it somewhere?"
Short answer: no, not for OCR. The photo never leaves your phone for the act of reading text. Longer answer below, because the long answer is actually more interesting than the short one.
The trade-off, in one paragraph
For most apps, the "obvious" way to do receipt OCR is server-side. You upload the image to a cloud OCR API — Google Document AI, AWS Textract, or your own model on a GPU somewhere — and you get back structured fields. It's accurate, model improvements are free, and you don't have to worry about the user's phone being a 2018 Redmi.
The catch: every receipt photo travels over the network, sits briefly in someone else's blob storage, and may get logged for "service improvement." For a budgeting app, where the receipts include your medication, your therapist, your gambling habits, that's a hard sell. So we picked the other path.
Why ML Kit fits
Google's ML Kit Text Recognition v2 ships a small (~3 MB after AndroidX bundling) text-detection model that runs entirely on-device. It's good at Latin script, fast on modern Android devices (~250 ms for a typical receipt), and the API gives you not just a flat string but blocks, lines, and words with their bounding boxes.
For our use case that bounding-box information is gold — it lets us reason about where the total appears relative to the merchant header, instead of trying to NLP the structure out of a flat string.
final recognizer = TextRecognizer( script: TextRecognitionScript.latin, ); final result = await recognizer.processImage( InputImage.fromFilePath(receiptPath), ); // result.blocks[i].lines[j].elements[k].text + .boundingBox
From text blob to fields
The model gives us text. The user wants merchant, total, date. The pipeline that bridges those two looks like this:
- Merchant — heuristic: the longest line in the top 20% of the receipt, in ALL CAPS or title case, no digits. Falls back to the first non-numeric line.
- Total — find lines containing the word "total" / "jumlah" / "amount", take the rightmost monetary token. Fall back to the largest monetary value in the bottom 30%.
- Date — regex sweep for common formats (DD/MM/YYYY, DD-MMM-YY). If multiple matches, take the latest plausible one.
It's not glamorous. None of it is a neural net. But it works because the structure of a receipt is one of the most stable layouts in the world — it has barely changed since the 1970s.
The numbers
We labeled 2,000 receipts from the seven biggest Malaysian merchant chains by foot traffic (Tesco, Jaya Grocer, AEON, Mydin, Speedmart, etc.) and 300 mamak receipts (which are their own genre). Out of those:
- Total amount: 94.2% exact match, 98.7% within 0.01.
- Merchant name: 87.5% exact, 96.0% fuzzy-match (Levenshtein ≤ 2).
- Date: 96.1% exact.
- End-to-end "zero edits" (all three fields correct before user intervention): 82.6%.
The remaining ~17% need at least one field corrected — and the UI is built to make that fast (you land on an edit screen, the fields are highlighted in yellow, you tap one to fix).
When we cheat
We don't, on the OCR. But there's one place where the boundary is blurry: if you choose to attach the receipt image to the transaction, that image is uploaded — to your own Firebase Cloud Storage bucket, encrypted at rest, accessible only by you (and shared-wallet members for shared entries). That's storage, not processing. We never run anything against it.
If you'd rather not attach the image at all, you can disable image attachments in Settings → Privacy. The OCR still works — it just discards the image after the on-device pass.
Wrap-up
On-device OCR is slower than the server-side version on a flagship phone, comparable in accuracy, way better for privacy, and lets us promise something honest in marketing: "the image does not leave your phone." We think that's worth a 200-ms latency hit.
The full implementation is in lib/src/features/receipts/ocr_pipeline.dart. PRs welcome. Especially if you have ideas about Chinese-character receipts — that's the next frontier and our accuracy there is currently a polite ~74%.
— BY