What Machines Still Cannot Translate

AAutomatic translation is now good enough that its remaining errors are widely regarded as a residue: rare, awkward cases that a larger training set will eventually absorb. The assumption is comforting and, on the evidence, mistaken. When the failures of current systems are sorted rather than merely counted, they turn out to fall into a small number of recurring kinds, nearly all of which share one property, namely that the information needed to translate correctly is not in the sentence being translated. A pronoun whose reference lies three sentences back, a technical term that must keep the same rendering across forty pages, a form of address that depends on a relationship between two people mentioned only in a covering letter — none of these can be settled by a machine that reads one sentence at a time, however many sentences it has read before. The difficulty is structural. Adding data makes fluent output more fluent without giving a system access to what it does not have.

BThe most systematic account of this comes from Miriam Bellucci, whose group at the Loewen Centre for Applied Linguistics classified 6,300 errors drawn from professional revisers' corrections between 2018 and 2021. Bellucci maintains that the usual error counts are misleading because they are compiled sentence by sentence, so that a mistranslation which is defensible in isolation and wrong in context is simply scored as correct. On her classification, 38 per cent of the corrections her revisers made could not be justified by anything inside the sentence they appeared in. Systems built to read a whole document rather than a single line do reduce the problem: in a controlled comparison the same errors fell from 24 per cent of segments to 9. The improvement, however, was confined to documents longer than about 800 words, since shorter texts rarely contain the earlier mention that resolves the ambiguity.

CKarl Ostrowski, a computational linguist at Brandhoek University, is sceptical of the claim that the amount of text a system reads is the heart of the matter. He argues that widening the window merely postpones the difficulty, because the decisive information is often not written down anywhere in the document: whether a container is a bottle or a jar, whether a room is being entered or left, whether a speaker can see the person addressed. Human translators settle such questions by imagining the situation; a system trained only on text has no situation to imagine. Ostrowski's demonstration is elegant and small. He assembled 240 sentences whose correct rendering depends on a photograph supplied alongside them, tested two language pairs, and found that accuracy rose by 31 percentage points once the image was made available to a system able to use it. Critics reply that translators are rarely given photographs either. Ostrowski answers that a translator who has never seen the object can nonetheless picture it, and that the photograph merely supplies to a machine what the human mind produces unaided.

DFor languages with little parallel text the standard remedy is to manufacture more of it. A system translates monolingual material backwards into the source language, and the resulting artificial pairs are added to the training data, a procedure that raises quality substantially provided that at least ten million sentences of ordinary text exist in the language concerned. Below that threshold the technique tends to amplify a system's own habits rather than correct them. What makes the practice risky is a mismatch that has been measured repeatedly: fluency improves faster than accuracy. In a review of eleven low-resource systems published in 2022, output rated natural by native speakers was judged faithful to the original in only 62 per cent of cases, whereas the same speakers, shown the earlier and clumsier versions, had caught nearly every distortion because the wording itself announced that something had gone wrong.

EThat mismatch shapes how the tools are used where mistakes are expensive. Hospitals in the Averill region introduced machine translation for appointment letters in 2017 and for consent forms in 2019, then withdrew it from the second use within a year; a review found that the letters had been rendered accurately in almost every case, while the consent forms contained, on average, one alteration of meaning per document, most often in a phrase expressing degree of risk. Professional practice has since settled into a division that neither camp predicted. Routine documentation is drafted by machine and repaired by a reviser, work that is paid at lower rates than translation and reported, in surveys of the profession, as markedly less satisfying, while anything whose wording is itself the point remains largely untouched by the technology.

FNeither Bellucci nor Ostrowski expects the remaining problems to survive indefinitely, and both concede that predictions in this field have a poor record. Their disagreement is about what would count as a solution. If Bellucci is right, the path runs through longer inputs and better memory, and the engineering is already under way; if Ostrowski is right, no quantity of text will supply what only experience of the world can, and progress waits on a different kind of system altogether. The fairest reading of the evidence is that both are describing real limits, with the context problem the more tractable of the two and the grounding problem the more fundamental. Meanwhile the practical rule remains unglamorous. Where a mistranslation would merely be noticed, the machine is now the sensible starting point; where it would not be noticed, it is not.