When Forecasts Stopped Solving Equations

AFew public services are judged as harshly on single occasions as the weather forecast. A household remembers the afternoon it was promised sunshine and was rained on, while the office that issued the advice keeps score over tens of thousands of cases and reports that its five-day forecast is now about as reliable as its three-day forecast was in the early 1990s. That gain was won by one method. The atmosphere is divided into a grid, the equations governing air, heat and moisture are solved forward in short steps, and the whole calculation is repeated some fifty times from slightly different starting points so that the spread of possible outcomes can be read off. Each decade of that work added roughly a day of usable lead time. Since 2022, however, systems that have learned from archives of past weather rather than from any equation have matched those scores in seconds on a single processor, and the field has been forced to ask what such a machine is actually doing.

BMarta Beloch, an atmospheric modeller at the Kestrel Centre for Numerical Prediction, argues that what it is doing is interpolation so quick and so well informed that it can pass for understanding. Between 2019 and 2024 her group scored 8,400 paired forecasts, one from a learned system and one from a physics ensemble, against the observations that followed. Both members of a pair were given identical starting conditions, and no case was entered twice. The learned system's median error at day five was slightly the lower of the two on the standard upper-air measure, a margin wide enough to look like a change of era. What did not improve was its treatment of the heaviest rainfall, which it under-predicted as often at the end of the period as at the start, and the fields it produced remained smoother than any state the atmosphere is ever observed to take. Its output was also coarser in resolution than the ensemble's, a difference Beloch does not dispute. Averaging, she maintains, hides precisely the failure that matters.

CThe account makes a prediction that can be tested, and Beloch has tested it. If the skill comes from resemblance to earlier cases the system has been shown, then events with no close counterpart there should defeat the learned system even when the physical situation is plain, and the same reasoning should hold wherever a forecast has to describe an intensity the archive rarely contains. In 27 episodes of rapid tropical-cyclone intensification, her learned system reproduced the observed deepening in 21 per cent of attempts, against roughly 74 per cent for the physics ensemble run at the same resolution. Success was confined to the storms for which a closely similar case could be found in the training archive. The asymmetry is instructive, since the same system beats the ensemble comfortably on ordinary days. Beloch is nonetheless careful to present the result as a limit on one way of building such models rather than a verdict on the approach as a whole.

DJonas Weyer, a machine-learning researcher at the Ravensholm Data Institute, rejects the inference, on the ground that a failure by a system trained on a thin record measures the record at least as much as the method. His networks were trained on an archive enlarged with high-resolution simulations of storms of a kind the observing system has seldom caught, and under those conditions 68 per cent of an equivalent set of intensification episodes were reproduced. Training of that sort is now standard in his group, although it was distinctly unusual when the work began. Weyer insists that a test which denies a model the examples its design exists to exploit is measuring the archive rather than the method. He concedes two things, however. His set ran to 19 storms from a single ocean basin, and skill was scored against reanalysis rather than against direct measurement, because no continuous record of core pressure over the open ocean exists.

EThe dispute has moved well beyond the seminar room. Since 2023 several national services have issued some public warnings from arrangements in which a learned model supplies the first guess, and insurers have begun pricing storm risk from the same output. Beloch points out, uncomfortably for her own position, that nothing in her analysis bears on whether such a model is cheap enough to be worth running anyway, since low cost and physical fidelity are not alternatives. Weyer replies that the two questions cannot be kept apart in practice, because agencies read a low average error as evidence of reliability at the extremes, and a single missed storm as evidence against the method entirely. Both agree that the habit of publishing individual striking forecasts, rather than counts of what a system does in ordinary weeks, has served the subject badly.

FA settlement of sorts is emerging in which the two accounts are assigned to different tasks rather than ranked against each other. Where tomorrow's atmosphere resembles states the record holds many times over, the learned systems appear to reach an answer with almost no physical reasoning, which would explain both their speed and their indifference to the coarseness of their own grids; where the state is genuinely rare, the equations still do work that nothing else has been shown to do. The unresolved question is whether the record these systems learn from will go on describing the atmosphere they are asked to forecast, and no method now available can settle it. What both researchers reject is the assumption that produced the argument in the first place, namely that an unfamiliar way of making a forecast must be either a poor imitation of the equations or a replacement for them.