Blog

Polishing a Tesseract

Tesseract is the OCR engine people reach for when they want text out of an image with no strings attached. On Omni Parse Bench, Datalab's 2,937-page document benchmark, it passes 17.6% of the 8,499 text tests it can attempt; Datalab's Indra OCR passes 40.3% of the same tests. Oskar's summary when he asked for this post: "A bit of a slight, but then it is a bit of a turd."

He still wanted it on the GPU, and better. Tesseract reads about 160 languages and scripts, its code and models are Apache 2.0, and it is slow in a way a GPU can fix: it reads one text line at a time on one CPU core. This post covers Tesseract's line reader, the LSTM network that turns an image of one line into text, running on a GPU in a browser tab and on a server with Tesseract's exact output, and then two changes that make the same model read better: a tighter crop, and fine-tuned weights that load in stock Tesseract. Finding the lines on a page is still Tesseract's CPU code.

One network layout serves all 320 Tesseract models

Tesseract 5 scales each text line to a fixed height (36 pixels for English) and runs a small network over it: a 3x3 convolution and max-pooling, one LSTM that runs down each column of pixels, and three that run along the line (forward, backward, forward). A softmax scores each character code at each step, and a beam search turns the scores into text. The 320 models in Google's tessdata_best (float weights) and tessdata_fast (8-bit) all use this layout and differ only in layer sizes, so one set of GPU kernels covers every language.

The GPU version reproduces Tesseract at every step

I built Tesseract 5.3.4 from source in a sandbox, linked a small harness against it, and dumped the network's output for each line. A NumPy reference written from Tesseract's source matches it to within 3.3e-5 for the float models and 1.3e-6 for the 8-bit ones, with the same best character code at every step of 686 lines in English, French and Ancient Greek. The browser and server engines are each checked against that reference, and a port of Tesseract's beam search returns its exact characters on all 1,372 readings.

Three details decide whether a port matches. Tesseract fills the convolution's border with random numbers rather than zeros, as in training, and 5.3.x draws them from std::minstd_rand where the current master uses a 64-bit generator, so the first or last character of a line can depend on the Tesseract version. The 8-bit models need float32 arithmetic step by step, down to a rounding that adds 0.5F. And taking the most likely code at each step, the usual shortcut, gives Tesseract's text on only 76 to 85% of lines.

A laptop GPU reads about a thousand lines a second

The WebGPU engine packs a batch of lines longest first and advances every line still running through each LSTM step in one GPU dispatch. On Oskar's M5 Max MacBook Pro in Chrome, 343 benchmark lines took 176 milliseconds of GPU time with tessdata_best English and 339 milliseconds end to end, about a thousand lines a second; Tesseract's beam search adds about half a millisecond a line. A server engine in PyTorch reads 1,143 lines a second on a Kaggle Tesla T4, and the T4 is mostly waiting for the CPU to prepare lines.

Tesseract's own recognizer, run in-process on one core of a 2.1 GHz Xeon with the model loaded once, takes 60 milliseconds a line with tessdata_best and 17 with tessdata_fast, about 17 lines a second per core with best; matching the laptop GPU would take about 60 such cores. Tesseract reads a page's lines one after another, so a 40-line page costs 2.4 seconds of recognition, against under a tenth of a second on the laptop GPU (a batch of 64 lines took 92 milliseconds). The rest of the page still runs on the CPU: finding the lines takes 0.1 to 0.4 seconds, and the line clean-up described below about 11 milliseconds a line in the browser. A whole page is about three times faster than on one core, not fifty.

The weights pack to 8 bits without losing accuracy

Rounding tessdata_best's weights to 16-bit floats gives identical text. Rounding them to 8-bit integers with one scale per row changes the character error rate by at most 0.1 point in English, French and Greek and shrinks the English model to 1.5 MB from the 15.4 MB file Google ships. Four bits break it: English goes from 1.26% to 2.40% errors, French to 12% and Greek to 20%. Most of the work is in the last LSTM, 73% of the English model's multiply-adds.

A tighter crop reads better than Tesseract

Every line is scaled to the network's height, so a line cut out with a generous margin reaches the network with smaller text. Cropping each line to its ink first, and dividing out uneven lighting, makes the same English model read clean benchmark lines at 0.28% character errors, against 0.31% for stock Tesseract, and shaded lines at 0.52% against 4.86%. Tesseract's trouble with shading is in its page layout, which works on a black-and-white copy of the page: on 13 of the 300 shaded lines it found no text at all. The recognizer itself reads greyscale.

On 800 real scanned lines from forms (FUNSD) and receipts (SROIE), the gain is smaller, 14.2% to 13.0%.

Six minutes of fine-tuning cut errors on real scans by 30%

Google trained tessdata_best on rendered text, and real documents bring things it rarely saw: receipt printers, typewriters, faded photocopies, form rules running under the words. To train on them, the network was rebuilt as a PyTorch module that matches Tesseract's to within 4e-5. Trained weights are written back into the .traineddata file in place, so an untrained export is byte-identical to Google's.

The training set was 30,000 generated lines and 7,800 lines from CORD, a public set of receipt photos. The generated ones were sentences, prices, dates, phone numbers, invoice numbers and form labels, rendered in typefaces from serif to typewriter and dot-matrix, with random damage: printer dots, ink bleed, blur, fax-style thresholding, fading, rules under the text, a neighbouring line cut off at the edge. Training took 5,000 steps, about six minutes on a Kaggle T4.

Real scans went from 13.0% to 9.0% character errors. FUNSD forms, scanned office documents unlike anything in the training data, went from 14.0% to 9.8%. Synthetic English improved too, from 0.38% to 0.27%, so the model lost nothing on clean text. The file works in stock Tesseract 5.3.4 unchanged, where it reads the same scans at 9.8% against 12.9%.

When to use it

On the same real crops, PaddlePaddle's PP-OCRv6 small recognizer (21 MB, Apache 2.0) reads at 5.0% character errors, against 9.0% for the fine-tuned model and 13% for Google's. For English or Chinese documents it is still the more accurate choice. Tesseract's case is the rest: PP-OCRv6 covers about 50 languages and no accented Greek, Tesseract about 160, and anything built around Tesseract can take the fine-tuned file without other changes. Only English has been fine-tuned so far; the generator, trainer and Kaggle job work for any of the 320 models.

The demo page reads a page image in the browser with any of the 320 models or the fine-tuned English one, and nothing is uploaded. The fine-tuned model (15 MB float, 5 MB 8-bit), the engines, the checks and every number here are in the experiments repo.