Accountability Is Not a Reason to Keep Humans in the Process

I want to make an argument that I expect most readers to reject, at least initially. I want to make it precisely because the initial rejection is instructive — it reveals an assumption so deeply embedded in how organisations think about process design that it is rarely examined.

The argument is this: for well-specified, high-volume, rule-based processes, human involvement does not produce accountability. It produces a bottleneck, and occasionally a point of failure that a well-specified automated system would not have. The instinct to keep humans in the loop for these processes is not caution. It is the institutionalisation of distrust in systems that were never designed well enough to deserve trust — and it has been, for most of the history of organisational life, the correct response to that distrust. But it is no longer the only possible response, because we now have the capability to build systems that deserve trust.

The queue backs up at exactly the point the process was supposed to be trusted.

1. What this argument rests on, and what would break it

What this paper argues from. The structure of two worked examples — expense-report review and NGO constitution review against a statutory checklist — reasoned through rather than measured directly. Published research on human error in sustained, repetitive checking tasks (the vigilance-decrement literature) and on how fraud is actually detected in practice (the ACFE's annual global survey). A well-known clinical-systems study on what happens when a checklist, not an automated system, replaces ad hoc human judgment. And public-administration theory on what accountability actually is, which turns out to already contain the move this paper is making for automated systems.

What this paper does not argue from. It does not argue from a controlled, head-to-head comparison of human-reviewed versus automated expense approval or constitution checking; no study measuring that specific comparison was found, and none is cited as if one existed. It does not argue from an audit of PBOMaster's or ProcureBee's own error rates against a matched set of human reviewers; Flux has not run and this paper does not cite such a study.

What this paper does not claim. It does not claim humans should be removed from processes involving genuine judgment, interpretation, or exception-handling — §6 is explicit that this is precisely where human involvement remains the substantive input, not a formality. It does not claim current automated systems are flawless; it claims a documented, auditable error is more correctable than an undocumented one, which is a narrower and more defensible claim. And it does not claim "keeping a human in the loop" for oversight of an automated system reliably produces real oversight — §8 engages this directly as the paper's own sharpest objection to itself.

What would refute it. Given in full, in decreasing order of damage, in §9. The sharpest version: if well-specified, rule-based automated systems are shown, at scale, to produce higher error rates — not merely different error distributions, but worse outcomes — than the trained human reviewers they would replace, the central claim of this paper is simply wrong, and the right response is to withdraw it, not to narrow the category of processes it applies to.

2. The Accountability Argument Examined

The standard argument for keeping humans in the loop is that accountability requires a human. When a human makes a decision — approves a transaction, signs a document, reviews a compliance output — there is a named person responsible for the quality of that decision. When software makes the decision, there is no named person. Accountability diffuses into the architecture of the system, the quality of the training data, the decisions of the engineers who built it.

This argument is correct in the cases where it applies. When the decision involves genuine judgment — the kind that depends on context, precedent, and interpretive skill that resists specification — a named human accountable for the outcome is not merely a governance requirement. It is a substantive input to the quality of the decision. A judge, a doctor, a senior manager approving a strategic exception — these are not bureaucratic checkpoints. They are nodes in the process where genuine expertise is applied, and where accountability is meaningful because it tracks a judgment that was actually made.

But the accountability argument is applied to a much broader category of decisions than the one it describes accurately. It is used to justify human involvement in processes where the decisions are not matters of judgment — where the policy is clear, the rules are specified, and the human's role is to check the request against rules that could be encoded. In these cases, the named human is not making a judgment. They are executing a process. And when they make errors — which they do, with the regularity that human attention and memory permit — those errors are not failures of judgment. They are failures of execution that a well-specified automated system would not produce.

3. The Error Rate Problem

Organisations that keep humans in the loop for rule-based processes rarely measure the error rate of the humans performing the rules check. This is a significant omission, because the error rate is often the clearest evidence that human involvement is not producing the accountability it is supposed to provide.

A finance officer reviewing expense reports against policy is, in practice, a human performing a pattern-matching task under time pressure with incomplete attention to each individual case. The approval rate is high because most requests are compliant. But attention to a repetitive checking task does not hold steady over time: a meta-analysis of vigilance research spanning decades of laboratory and applied studies found a consistent sensitivity decrement — a decline in the ability to correctly detect the thing being checked for — across the duration of a sustained monitoring task, more pronounced the higher the event rate and the heavier the memory load the task demands.¹ Reviewing a growing stack of expense claims against a policy, each one slightly different, at the end of a reporting period, is close to the applied conditions under which this literature finds decrements to be largest.

A compliance officer reviewing NGO constitutions against a statutory checklist is performing a task for which the checklist is known and the document is available. The error rate — provisions that are missing but not flagged, provisions that are present but incorrectly assessed — is a function of the reviewer's familiarity with the specific requirements, the quality of their attention over the course of a long document, and their experience with the specific language patterns that do and do not satisfy each requirement. The same vigilance-decrement pattern applies directly: a long, text-dense, low-variance document is exactly the kind of sustained-attention task in which detection accuracy is documented to decline over the course of the review, not merely vary randomly around a fixed rate.¹

In both cases, the named human is providing accountability in the formal sense: there is a person who can be asked "did you review this?" and who will say yes. The accountability is real in the process record. It is not always real in the substance. The human reviewed it — and the human may have missed things that a systematic process would not have missed, for reasons the vigilance literature describes as a property of sustained attention itself, not a personal failing of any individual reviewer.

4. What the fraud-detection evidence actually shows

Laboratory attention research alone would make this argument too easily. The best available field evidence on how rule violations are actually caught in practice belongs here too, including the parts that complicate this paper's own case.

The Association of Certified Fraud Examiners' 2024 global study, drawing on 1,921 real occupational-fraud cases investigated by certified examiners across 138 countries, found that tips — from employees, customers, and vendors — were by far the most common way fraud was initially detected, at 43% of cases.² Internal audit accounted for 14%, management review for 13%, document examination for 6%, account reconciliation for 5%, and external audit — the review mechanism organisations spend the most on — for just 3%.² The median loss across cases was $145,000, with a median duration of twelve months before detection.² If "a human reviewed it" were a strong guarantee against rule violations slipping through, formal review mechanisms — internal audit, management review, external audit combined — would not be outperformed by informal tips from people who were never assigned a review function at all.

But the same report complicates a simple version of this paper's skepticism. Where a management-review control was in place, the median loss was $100,000, against $250,000 where it was not — a 60% reduction.² That is not nothing. A human review process that exists is measurably better than no review process at all; the claim this paper is making is narrower and comparative — that for well-specified, rule-based checks specifically, a documented automated process is a better mechanism than the human review it would replace, not that human review currently provides zero value where it exists. I return to this directly in §8, because it is the strongest data-based objection available to this paper's own argument.

5. What Systematic Accountability Looks Like

I want to distinguish between two kinds of accountability.

The first is accountability as responsibility: a named person is responsible for the outcome. If something goes wrong, that person is accountable. This is the kind of accountability that the "keep humans in the loop" argument invokes. It maps closely onto what the public-administration theorist Mark Bovens calls accountability in the narrow sense: "a relationship between an actor and a forum, in which the actor has an obligation to explain and to justify his or her conduct, the forum can pose questions and pass judgment, and the actor may face consequences."³ That is a real and useful definition, and nothing in this paper argues it is wrong. What this paper argues is that the actor named in that relationship does not have to be an individual human for the relationship to hold.

Bovens's own work on this point is more useful here than a casual citation would suggest. He names what he calls the "problem of many hands": in any organisation of real size, policies pass through many desks before they take effect, and isolating which individual is actually responsible for a given outcome is often genuinely difficult, not merely inconvenient.³ His answer is that organisations have, in practice, developed several standing strategies for this — and the first one he names is "corporate accountability," in which the forum holds the organisation itself accountable as a unitary actor, rather than trying to trace blame to one named employee.³ That is, public-administration theory already contains the move this paper is making for software: when individual responsibility is genuinely diffuse or hard to trace, institutions do not generally conclude that accountability has become impossible. They relocate it to the process or the organisation as the accountable unit. A documented, auditable automated system is a specific, well-defined version of exactly that relocation.

The second kind of accountability is accountability as reliability: the process that produced the outcome is specified, documented, and auditable. If something goes wrong, the reason can be identified in the process record. The failure can be diagnosed and the process corrected. For well-specified, rule-based processes, this second kind of accountability is superior to the first. Not because named responsibility is unimportant, but because named responsibility for a rule-based process is a weak accountability mechanism: it tracks whether a person was present in the process, not whether the process produced the correct outcome. An automated process that fails is diagnosable and correctable. A human process that fails is explainable but harder to fix: "the reviewer missed it" is an account of what happened, not a solution to the structural problem that the reviewer could miss it again.

There is direct evidence, outside AI entirely, that specification and documentation beat ad hoc human judgment even when humans remain the ones carrying the process out. The WHO Surgical Safety Checklist study, conducted across eight hospitals on four continents, found that introducing a simple, specified, nineteen-item checklist into surgical teams' existing workflow reduced inpatient deaths by more than 40% and major complications by more than a third.⁴ Nothing about that checklist was automated; the surgeons, nurses, and anaesthetists did exactly what they had always done, just against a specified, documented sequence instead of unaided memory and habit. This is evidence that the mechanism this paper is arguing for — specification and documentation over unaided individual judgment — works even without automation. It is not, on its own, evidence that automating the check specifically outperforms a human following the same specified checklist. That gap reappears in §9 as a falsification condition.

PBOMaster's constitution analyser provides systematic accountability in the sense this section describes. Every check it runs is documented — the specific provision checked, the text of the constitution assessed against it, the determination reached. If the analyser reaches a wrong determination, the error is visible in the audit trail and can be corrected. The correction applies to every future analysis. There is no named human accountable for the analysis. There is a documented process accountable for every step of it.

This is better accountability for this task than the alternative of paying a lawyer sixty thousand shillings to do a review that produces a written opinion but not a systematic record of every check performed.

6. The Right Scope for Human Oversight

I am not arguing that humans should be removed from every process. The argument is about scope.

Humans should be in the loop for: decisions that involve genuine judgment, interpretive skill, and context-sensitivity that cannot be specified in advance. The exception cases — the situations that fall outside the parameters of a well-specified rule set. The policy design process itself, which determines what the automated system will enforce. The governance oversight function that determines whether the automated system is performing correctly over time.

Humans should not be in the loop for: the individual instances of a well-specified, high-volume rule-checking process. The routine expense approvals, constitution checks, data validation passes, document routing decisions that together constitute the administrative bulk of most organisations' operations. Not because these decisions don't matter, but because the mechanism for ensuring they are made correctly is better served by systematic automation than by individual human attention applied at high volume.

The organisations that understand this distinction will design their AI deployments differently from the ones that don't. They will not ask "how can AI assist our people in doing these reviews?" They will ask "what is the most reliable mechanism for ensuring that these rules are applied correctly?" For many processes, the answer is: build a system that applies the rules, audit the system's performance, and free your people for the work that actually requires them.

7. The Implication for Trust

There is a deeper issue behind the accountability argument that I want to name. The insistence on human oversight for well-specified processes is, at its root, an expression of distrust in software — and historically, it has been the correct expression. Most of the software that organisations have used to manage compliance, approvals, and process enforcement has been genuinely unreliable: misconfigured, poorly maintained, opaque in its operation, and difficult to audit when something goes wrong.

The correct response to unreliable software is to keep humans in the loop. Not because humans are more reliable than software in principle, but because a human who is wrong can be questioned and corrected, while software that is wrong often cannot be diagnosed without specialist intervention.

But the basis of this distrust has changed. The software that now exists for well-specified compliance and process tasks is more auditable, more configurable, and more transparent in its operation than the enterprise software that preceded it. When the system is wrong, the error is visible and correctable. The appropriate response to this change is to calibrate human oversight to the actual reliability of the current system — not to maintain the same level of oversight that was justified by the unreliability of previous systems.

8. Objections I take seriously

"Flux has a commercial interest in this conclusion." Flux builds and sells PBOMaster, ProcureBee, AnswerTab, and comparable automation products, and this essay's argument is, among other things, the sales case for removing human reviewers from exactly the processes those products automate. That is a direct financial stake. The honest mitigation is narrow: §§3-5's evidence (vigilance decrement, the ACFE's fraud-detection data, the surgical checklist study, Bovens's accountability theory) comes from independent research literatures that would be true whether or not Flux existed. Whether Flux's own products achieve the documented, auditable, correctable process this paper describes in practice is Flux's own claim to make, and this paper has not independently audited it.

"The ACFE's own data shows management review reduces losses by 60% where it exists — this paper's skepticism about human review proves too much." §4 reports this directly rather than omitting it, and it is a real complication. My answer is that the comparison the ACFE data supports is "review versus no review," not "human review versus well-specified automated review" — the study was not designed to test the second comparison, and no study I found was. This paper's claim survives that complication only if automated review turns out, when actually tested, to perform at least as well as the human review whose value the ACFE data documents — which is exactly what §9's second falsification condition asks for.

"Keeping a human in the loop for oversight of an automated system, which §6 recommends, does not reliably produce real oversight." This is the objection I take most seriously, because the evidence against my own recommendation is clear rather than merely plausible. A major review of the automation-complacency and automation-bias literatures found that complacency — reduced monitoring of an automated system's performance — occurs in both novice and expert operators and is not overcome by simple practice, and that automation bias, which produces both errors of omission and errors of commission when the automation is imperfect, occurs in both naive and expert participants and cannot be prevented by training or instructions.⁵ If that holds for the specific case of a human nominally supervising an automated rule-checking system, §6's "humans for governance oversight" recommendation is weaker than this paper has stated it. I do not have a fix for this beyond naming it: the honest position is that oversight of automated systems is itself a design problem this paper has not solved, not a problem the mere presence of a human reviewer solves by default.

"A specified checklist followed by humans, not an automated system, might capture most of the available gain — §5's checklist evidence doesn't actually support automation specifically." This is correct, and I said so in §5 rather than waiting for an objection to say it. The surgical-checklist evidence is about specification and documentation, not about automation. This paper's additional claim — that automating the check is better still than a human following the same specification — rests on the vigilance-decrement and error-rate arguments in §§2-3, not on the checklist study, and a reader who accepts §5's evidence but not §§2-3's could reasonably conclude that specified human review, not automation, is the more defensible recommendation.

9. What would falsify this

Stated as commitments, in decreasing order of how much damage each would do to this paper's claims.

Well-specified, rule-based automated systems are shown, at scale, to produce materially higher error rates than the human reviewers they would replace. Not a different error distribution — a worse one. If this were established, the central claim of this paper would be wrong outright, and the right response would be to withdraw it.

A direct, controlled comparison finds humans following a specified checklist perform as well as, or better than, an automated system checking the same rules. This would not refute §§2-3's account of vigilance decrement and error rates in unstructured human review, but it would falsify this paper's specific claim that automating the check is the better response, as distinct from simply specifying and documenting what the human checks — §8's fourth objection names this gap directly.

Automation complacency and bias are shown to degrade a human's nominal "oversight" role so severely that the governance function §6 assigns to humans stops functioning in practice. This would not falsify §§2-5's case for automating the individual rule-checks, but it would falsify §6's specific recommendation that humans remain a reliable check on the automated system's overall performance over time.

A large-sample study finds that formal review mechanisms (internal audit, management review, external audit) detect rule violations at a rate comparable to or exceeding informal detection (tips, accidents), contrary to the pattern in the ACFE data cited in §4. This would weaken, though not eliminate, this paper's argument that named human review is a weaker accountability mechanism in practice than its formal status suggests.

10. What this paper hasn't shown

No controlled, head-to-head measurement of error rates between human-reviewed and automated expense approval or constitution checking is cited above, because none was found.

It does not claim PBOMaster's, ProcureBee's, or AnswerTab's own error rates have been independently audited against a matched set of human reviewers; §8's first objection states this directly.

It does not claim humans add no value in processes involving genuine judgment or exception-handling; §6 states the opposite.

It does not claim a human nominally assigned to oversee an automated system reliably provides real oversight; §8's third objection takes the automation-complacency and -bias literature seriously as an unresolved problem with this paper's own recommendation, not a solved one.

And it does not claim the surgical-checklist evidence in §5 is evidence for automation specifically, as distinct from specification and documentation more generally; §5 and §8's fourth objection both say so.

Appendix A — Evidence table

Appendix B — Citation register

CITATION-NEEDED — automated vs. human rule-checking, head-to-head. Appendix A row 7. No study directly comparing error rates between an automated, well-specified rule-checking system and trained human reviewers checking the same rules was found; this is this paper's single most consequential missing measurement, and §9's second falsification condition names the study that would settle it.

CITATION-NEEDED — Flux's own product error rates. Appendix A row 8. Flux has not published, and this paper does not cite, an independent audit of PBOMaster's, ProcureBee's, or AnswerTab's error rates against a matched comparison group of human reviewers.

CITATION-NEEDED — oversight-role effectiveness. Appendix A row 9. §8's third objection names the automation-complacency literature as a serious, unresolved problem for §6's "humans for oversight" recommendation specifically; no study measuring whether a nominal human overseer of a rule-checking automated system improves outcomes relative to no overseer was found.