An analysis output from our quant desk was sitting on the wrong scale. Not wrong in the sense of a crash or a stack trace, or anything that would have interrupted a run. The values it produced came out orders of magnitude larger than anything in the inputs could account for, and they moved through the whole analysis layer without a single stage objecting. Nothing in a pipeline has an opinion about whether a value follows from what went into it.
I found it reviewing the backtest. There were several results I could not account for, and working out where they came from meant going back and comparing the data until the calibration step was the only thing left that explained them. That review is the first point in our process where anyone asks whether a result follows from its inputs, and it is close to the last place you want to be discovering a scale error, because by then every analysis upstream has already been computed on top of it.
So I fixed the math, revalidated that what came out landed on the scale it was defined on, recomputed the analyses, and ran the backtest again. Two days, almost all of it in those last two steps.
The bug itself was findable and fixable. What made it expensive was where in the process it surfaced.
The error was upstream. I found it at the far end.
Everything before the backtest review is machine talking to machine. An analysis takes inputs, computes, hands its result to the next stage, and that stage does the same. At no point does anything ask whether the value it just received follows from what produced it, because that is not a question a type system or a range check can answer. Those can tell you a number is finite and within bounds. They cannot tell you it is a number this method could have generated.
An uncalibrated value is finite, in range and wrong at the same time. That is the worst combination available, because every automated check you have will pass it.
Off its own scale, a value stops being evidence
This is the part that took me longer to accept than the bug did.
A value carries information about whether a computation is sane only while it stays on the scale it is defined on. Once it comes off that scale the magnitude stops being evidence. It is just a large number, and large is not a claim you can argue with.
Which is what a review is actually doing. You are checking whether you can account for a result from the inputs and the method, and that check is only available while the value is still expressed in the units the method works in. Break that link and the number goes quiet. It stops being able to tell you it is wrong.
In my case it cost a data comparison to establish something the value would have made obvious on its own scale.
Fixing the math was one problem. Everything computed on top of it was another.
What I had to redo was not the part that was wrong. It was everything that had consumed that output and produced something of its own.
That is the real cost structure of a scale error, and it is why finding one late hurts out of all proportion to the fix. A single wrong value does not stay a single wrong value. It becomes an input, then the thing it fed becomes an input, and by the time anyone notices, the question has stopped being what do I fix and become what can I still trust. In my case the answer was nothing downstream of it. Both days went into redoing work that had already been done once, carefully, on top of a wrong number.
Calibrate at the boundary
The change is small and it is not clever. Values get calibrated where they are produced, not where they eventually get displayed.
The ordering is the whole point. Calibrating at display time gives you a correctly scaled number after every decision has already been taken on the unscaled one. Calibrating at the boundary means every stage afterwards holds a value whose magnitude is still evidence, so a result that could not have come from its inputs is arguable at the stage that produced it, rather than four stages later in a backtest review by someone comparing data to work out where it came from.
The other half is revalidation. After any change to the math, outputs get checked against the scale they are defined on before anything downstream is allowed to run on them. The intermediates get the same treatment, so a value that could not have come from its inputs is arguable at the point it appears.
None of that would have stopped me making the error. It would have surfaced it at the stage that produced it instead of at the end of a backtest, and the distance between those two is the two days.