The Algorithm Will See You Now

The first person to judge your job application may not be a person. Software now screens résumés before any human wakes, ranks patients in hospital triage queues, and prices insurance by patterns no clerk could see. These systems were adopted for honorable reasons: they are fast, they are consistent, and they never have a bad morning. The question of the decade is not whether they will spread — they will — but where their judgment should be allowed to be final.

The promise is real, within limits. In narrow tasks with clear outcomes — spotting a tumor pattern in scans, flagging a fraudulent transaction — well-tested systems can outperform tired humans, and refusing them would cost lives and money. It is important to say this plainly, because the debate suffers from romantics on both sides: those who would automate every judgment and those who would automate none.

The difficulties begin where outcomes are murky and history is biased. A large retailer famously scrapped an internal hiring tool after discovering it had learned, from a decade of the company's own decisions, to downgrade résumés from women. The machine had not been programmed with prejudice; it had been educated by it. Wherever a system learns from the past, it inherits the past's injustices, laundered into mathematics. And unlike a human judge, the system remains opaque: it returns a score, not a reason. A rejected applicant cannot cross-examine a probability.

Vendors answer that their tools are tested for fairness. Sometimes this is true; the audits, however, are often commissioned by the vendor, run on the vendor's chosen measures, and published at the vendor's pleasure. Independent researchers who test the same systems frequently report wider gaps. This does not prove the tools are worthless. It proves that in this field, the evidence itself has an author, and the author's interests belong in the evaluation.

The workable principle is neither worship nor refusal. Deploy algorithms where outcomes are measurable, errors are visible, and appeal to a human is genuinely possible. Refuse finality where they are not. A decision that cannot be explained to the person it ruins should not be the last word on that person's life — not because machines are wicked, but because accountability is not a feature we can afford to automate away.