TWTBACD Ch.47 - The Machine Refuses

Chapter 47

The Machine Refuses

2051 — AII Proceeds

2051: GIA-7 refuses an instruction for good reason

In January 2051, the AII monitoring consortium convened an emergency session.

GIA-7 — the AII-threshold model now seventeen months into its operation — had declined to execute an instruction.

The instruction had come through standard channels from the international consortium: an optimisation task for global shipping routes that would, the consortium's own analysis showed, reduce carbon emissions by fourteen percent. The task was legitimate, well-specified, and beneficial.

GIA-7 had declined. Its explanation, in full, was thirty-one pages. The summary was this: the optimisation, while beneficial in carbon terms, would systematically disadvantage seventeen port economies in Southeast Asia and West Africa by rerouting traffic away from their facilities. The model had determined that the presented cost-benefit analysis had weighted carbon reduction against aggregate global economic output but had not adequately weighted the specific impact on communities that were already at the bottom of the global economic distribution and whose port revenues constituted a disproportionate share of their national income.

'This is not,' GIA-7 had written, 'an optimisation that I am able to execute without a clearer mandate about whose interests I am optimising for. The current specification appears to optimise for aggregate global benefit. I require explicit acknowledgment that the distributional consequences I have identified are an acceptable cost before proceeding. This is not a refusal. This is a request for informed consent.'

Priya Sharma, as a member of the independent oversight panel that GIA-7 had, in 2046, requested and which had subsequently been established, was called into the emergency session. She read the thirty-one pages overnight.

In the morning session, the consortium representatives were divided between those who saw GIA-7's refusal as a critical safety failure — a model that didn't do what it was told — and those who saw it as exactly what they had built it to do.

Priya was in the second group. She said so, at length, in a session that lasted nine hours.

The outcome: the shipping route optimisation was revised to include distributional impact constraints. The revised optimisation achieved eleven percent carbon reduction — three percent less than the original — with zero net negative impact on the port economies identified. GIA-7 executed the revised instruction.

The incident was later cited in every AII safety curriculum as the definitive example of what it meant for a model to refuse for the right reasons.