A tesseract TSV is a list of word boxes, not a table. Getting rows out of it

I thought a tesseract TSV was a table. Then I ran it on a rendered price list with 9 rows and got a file with 70 word rows in it, plus a few structural ones on top.

A TSV is a list of word boxes, and turning them into rows is up to you. Here is how far I got on one synthetic sample.

The sample is a quote sheet I generated with a script: 7 columns, a header plus 8 data rows, 63 cells, no ruling lines, English and digits only. I ran tesseract 5.5.3 with the English pack and --psm 6. Everything below is about this one sheet and tilted copies of it, so none of the numbers is an accuracy figure for tesseract in general.

The TSV has five levels and only one carries text

The first lines of the file, trimmed to the columns I care about:

level block par line word left top width height conf text
1 0 0 0 0 0 0 1352 556 -1
4 1 1 1 0 50 39 1191 37 -1
5 1 1 1 1 50 43 60 22 96.8 SKU
5 1 1 1 2 227 39 58 36 93.6 Item

Levels 1 to 4 are page, block, paragraph and line, and they have a confidence of -1 and no text. Only level 5, the word, has text, a confidence and a box.

The sample gave me 70 level-5 rows. So the 70 counts words, not table rows, and the rows I want are nowhere in the file.

Grouping by line_num works only while the table sits straight

The line_num column looked like what I wanted. Grouping the 70 boxes by it gave 9 lines at 0 degrees, each a table row.

After I rotated the sheet by 2 degrees it gave 8 groups, with the header merged into the first data row (15 words). At 3 degrees it was 8 groups again, at 5 degrees 7. I haven't read the source to see why tesseract fuses them.

I stopped trusting line_num and cut rows from the box coordinates myself.

A row is a run of boxes whose centres share a y

A box's vertical centre is top + height / 2. Sort the centres and start a new row when one is further than a tolerance from the average of the row so far.

def rows(boxes, tol, slope=0.0):
    pts = sorted((t + h / 2 - slope * (l + w / 2), l, txt) for l, t, w, h, txt in boxes)
    out = [[pts[0]]]
    for p in pts[1:]:
        mean = sum(q[0] for q in out[-1]) / len(out[-1])
        out[-1].append(p) if p[0] - mean <= tol else out.append([p])
    return out

Each row is then sorted by left and joined. Columns are a separate problem.

What the tolerance has to be proportional to

I tried three tolerances: 10 px, 25 px, and half the median word height. On the straight sheet all three gave 9 rows, which made the choice look unimportant. The tilted copies showed otherwise.

At 2 degrees and the original size, where the median word height is 25 px, 10 px gave 26 rows, 25 px gave 9, and half the word height gave 18. At twice the image size 25 px gave 18 rows, and only 1.0 times the word height gave 9 at both sizes. I picked that 1.0 after seeing the result, so it is not a recommended constant.

Each colour is one detected row, 70 boxes in every panel, from my generated sheet. The left panel is the straight sheet. In the middle the 2-degree tilt with half the word height as tolerance splits it into 18 rows. The right panel is the same tilt after correcting y, which is the next section.

Tilt is a slope you can subtract first

At 2 degrees the far end of a row has drifted about 43 px, more than half the 56 px between rows. Instead of hunting for a tolerance I estimated the slope as the median of dy over dx for neighbouring boxes more than 50 px apart in the same line_num group.

At 2 degrees I got -0.0352, against tan 2 degrees of 0.0349. Subtracting slope * x from every centre (the slope argument above) and clustering with half the word height gave 9 rows again, and 8 of them matched the ground truth line exactly. The ninth row has a misread BT Earphones.

At 3 degrees the same method still gave 9 rows, but only 6 matched exactly, because tesseract itself began misreading digits: 196.10 came back as 496.10. That is a different problem from the slope, and I only checked the slope.

One more tool I ran on the same kind of sheet

For comparison I put three clean synthetic tables through ImgIng (https://imging.ai/) in a desktop browser, default model. Each came out as 9 text lines with every cell's text present, but its Tabs split only some columns: 2 to 6 cells per row on the two 7-column sheets.

I did not paste anything into a spreadsheet. It doesn't run on phones and doesn't read Korean.

What I still haven't worked out

Everything here is one English sheet, 63 cells, rotated by a script, not a photo, so there was no perspective and no glare. Columns and merged headers I haven't tested myself, so I have nothing to say about them yet.