An Immigration New Zealand automation failure raised a useful question about a claim we hear increasingly often in discussions about AI governance, that there is a “human in the loop”
Faults in its ADEPT processing system allowed 32 visitor visa applications and 23 Job Check applications to be approved without the required manual assessment. Six of the visitor applications were subsequently reopened because of high risk factors and one was recommended for decline.
This wasn’t generative AI. It was automated processing, which is an important distinction. But the control issue is directly relevant to AI and automated decision making.
INZ had a process in which manual assessment was required in particular circumstances. A technical fault allowed the automated workflow to proceed without that assessment.
Saying there is a “human in the loop” tells a board that a person is supposed to intervene somewhere in the process. It doesn’t establish that the intervention will actually occur or that anyone will know if it doesn’t.
The INZ case is not an argument against human oversight. It illustrates the difference between designing a control and establishing that it operates effectively.
For a board or executive team, that leads to a more useful question, “which consequential decisions are supposed to require human judgement, and what evidence do we have that those controls are actually operating when required?”
I would also want to know what happens when they dont. If an automated process bypasses the required human judgement, will somebody know?
As organisations automate more of their operations, saying that a human remains in the loop provides limited assurance on its own. What matters is whether the organisation can demonstrate that the control actually works when required.