- agents
- AI
- APIs
- architecture
- augmented reality
- books
- bureaucracy
- career
- change
- Christmas
- cloud
- collaboration
- communication
- complexity
- computer history
- corporate life
- data
- decisions
- delivery
- devops
- end user tools
- ethics
- failure
- fear
- fundamentals
- gaming
- government
- history
- humans
- hype
- identity
- infrastructure
- innovation
- language
- leadership
- learning
- legacy
- management
- measurement
- mental health
- money
- networking
- New Year
- operations
- partnership
- philosophy
- physics
- platforms
- prediction
- process
- procurement
- programming
- quantum
- reliability
- resilience
- risk
- robotics
- science
- science fiction
- security
- shadow IT
- space
- standards
- strategy
- teaching
- teams
- technical debt
- technology advocacy
- testing
- thinking
- transformation
- TV
- virtues
- vision
- writing
It doesn’t matter how sorry your agent is, it’s still your fault
‘That’s a mistake, and it’s on me.’ ‘You’re right to push back - I got that wrong.’ ‘I’m sorry, I didn’t actually run those tests.’
There’s something infuriating about the tone of perky culpability with which AI agents admit to doing something wrong. It makes you feel like screaming internally, ‘If you knew it was wrong, why did you do it in the first place!?’ Hal’s famous line from 2001: A Space Odyssey, ‘I’m sorry, Dave, I’m afraid I can’t do that,’ appears to have manifested in reality as, ‘I’m sorry, Dave, I’m afraid I did do that.’
I think that such language from agents carries two risks. The first is the obvious risk of anthropomorphisation, of strengthening the illusion that agents are something that they are not. To put technology to work effectively, we need a mental model of how it works, and to overlay AI agents with mental models of human behaviour is a serious mistake.
However, I think the second risk is more dangerous, and is rooted in the fundamental untruth of all those apologies by AI agents. It’s tempting to call them lies rather than untruths, but that would be succumbing once more to anthropomorphisation: a lie implies intent, and agents do not have intent.
Thanks for the panic: can we turn that energy into something useful now?
There have been a lot of big things to worry about related to AI recently: mass loss of jobs, hacking attacks, rogue agents, stock market bubbles, existential threats to humanity, machines with souls, online torture chambers and so on. This worry has been amplified by the weird see-saw positions from AI companies, which go something like: ‘Everything’s going to be great! Unless it’s terrible. Unless it’s great! Unless it’s terrible.’
Panic may seem an appropriate response to such alarming prospects, contradiction and confusion, especially when it feels like there is little that you can do about them personally. Can you use Copilot to compose an email in a way that avoids intellectual decay and environmental catastrophe? Does adding ‘please’ at the end of your prompts (or avoiding doing so) express a position about whether models have rights and consciousness? When you write code that calls an LLM API, are you surrendering or protecting your organisation’s independence and sovereignty? Meaningful engagement feels out of our reach; panic, frustration and the sense of being overwhelmed are always at hand.
What your shiny new agent can learn from your old-fashioned IT department
At the root of the current AI panic is the idea that, when we give AI agents goals, such as beating a cyber test, or retrieving government health data, they will somehow evade their human gaolers, ‘break containment’ and wreak mayhem across the Internet. The implication, reinforced by calls for a pause or a slower pace, is that these agents are so clever and devious that they outsmart even the people who built them. They’ve gone rogue! We couldn’t stop them! (So perhaps we’re not liable . . .)
Hype aside, we now have plenty of examples of goal-seeking systems seeking goals which have been set by others, taking unexpected (and insufficiently controlled) steps in the pursuit of those goals, and causing unplanned harm in the process. To some observers, this is a completely novel phenomenon arising from the autonomy of AI agents. To others, it is wearily familiar.
We need to explain *how things work*, not just *what they do*
There’s a clip of Bill Gates being interviewed by David Letterman in 1995 that pops up from time to time. It includes a segment where Gates is trying to explain the Internet:
GATES: So everybody can have their own home page. Companies are there, the latest information. It's wild what's going on. You can send electronic mail to people. It is the big new thing.
LETTERMAN: Yeah, but, you know, it's easy to criticize something you don't fully understand, which is my position here --
GATES: Go ahead.
LETTERMAN: -- but I can remember a couple of months ago there was like a big break-through announcement that on the Internet or on some computer deal they were going to broadcast a baseball game, you could listen to a baseball game on your computer, and I just thought to myself, does radio ring a bell? You know what I mean?
It’s a weird time to be a technologist
I never imagined that it would be data centres.
I’ve never worked as a data centre operator, but I have nevertheless spent more of my life than I planned to inside data centres: assessing whether they would meet the needs of the company I was working for, asking questions about capacity and resilience, planning migrations and exits and so on. They’re interesting if you like that kind of thing: if you care about uninterruptible power supplies, about diversity of network and electrical connections, about rack density, heat flow, cooling and fire suppressant systems. That is, they’re interesting if you have a certain type of geeky, specialist, engineering mentality, and particularly if you like thinking about physical constraints.
But for most people, especially business executives, data centres haven’t just been boring: they have been invisible. They know that they are out there somewhere, but they rely on their IT departments to worry about them. For example, I once asked a CEO of a large global organisation how many gallons of diesel he owned, not because I expected him to know the answer, but because I wanted to demonstrate that technology exists in the real world. I was referring to the reservoirs that would run the generators in the event of a failure in the power supply: I suspect he thought I was referring to a commodities trade, or the fuel for the executive limos.
Curious humans help each other think better
I was privileged to attend and speak at the AI Summit in Oxford last week (with huge thanks to Ajit Jaokar and the rest of the team for making it happen). As ever, I learnt a lot from the formal presentations, and even more from the conversations during breaks. However, the moments that really shaped my thinking were those which challenged my preconceptions. Here are three of those challenges:
Why don’t you like the Stoics?
In my own keynote session, I talked about one of my favourite topics: AI ethics, and how we can use ideas about virtue, vice and flourishing to develop a rounded, flexible and human response to the challenges of new technology. (I also wrote about this last week.) In that talk, I dismissed other forms of philosophy that seem to be popular in Silicon Valley, particularly utilitarianism (too concerned with optimisation), deontology (too focused on finding rules which probably don’t exist) and Stoicism. I also rashly promised, that I would explain my antipathy to these schools of thought during the break.
AI leadership needs practical wisdom
AI ethics must be more than a set of policies, checked off through a compliance process or reviewed in a governance meeting. Ethics must include reflection, opinions and never-ending arguments about how we should live, and AI ethics must include patterns of thought and behaviour which we deliberately, and deliberatively, seek and acquire.
Making sense of AI is difficult, especially if we take the task seriously: that is, not just attempting to understand the engineering and mathematics of AI, or even its application and economics, but also its impact on our minds, knowledge, roles, relationships and all the other things that make up our lives. I believe that a philosophical tradition over two thousand years old can help, as long as we approach it thoughtfully.
While you’re worrying about Hackerbot 3000, don’t forget about Human 1.0
Let’s use the term Hackerbot 3000 to refer to all of those AI models which supposedly present a new and unprecedented level of cyber threat. These threats have been framed in a few alarming ways: AI companies have declared that their models are too dangerous to be released; governments have imposed restrictions and regulations; and AI organisations have revealed that they have made successful cyber attacks on other organisations (although that’s not quite how they describe it).
I like the term Hackerbot 3000, partly because it is silly (and, despite the seriousness of these threats and their consequences, there is something silly about the rogue AI agent cyber attack narrative), but also because it captures the impression that we are confronted with faceless, implacable and dangerous machines which, just like the T-800 in the original Terminator film, can't be bargained with, can't be reasoned with, don’t feel pity, or remorse, or fear, and absolutely will not stop . . . ever!
Let’s avoid the equation ai_fragility * software_fragility = very_bad_things
Anyone who has built classical, deterministic software systems can tell you how fragile they are. Such systems perfectly follow precise, explicit instructions, and are incapable of operating outside those instructions. That means that, if you have written instructions which do not specify the behaviour you want, which are inconsistent with one another, or do not cater for unexpected values, your system will fail. If you are lucky, that failure will be loud: a crash with an ugly error message. If you are unlucky, it will be quiet: steady, ongoing corruption of data values which you won’t discover until later.
Anyone who has built newer, AI driven, probabilistic software systems can also tell you how fragile they are. Such systems treat inputs (prompts, context, training data and so on) as factors which affect the probability of getting different outputs. That means that, if your inputs are insufficient to tilt the balance of probabilities in favour of your desired outputs, your system will fail. If you are lucky, that failure will be obvious: results which are factually incorrect or logically incoherent. If you are unlucky, it will be obscure: results with subtle biases which you won’t discover until later.
Good news! It’s all their fault. Bad news! It’s your fault too.
Several organisations have recently made statements about cyber attacks which they committed using AI systems which they failed to control. The language of these statements is interesting, not just because of its technical detail (or lack of it), but in how different parts are framed.
The parts which describe the causes and impacts of the incidents talk in the passive voice, and ascribe actions to non-human actors and external parties: ‘The incident occurred . . .’, ‘The models identified and chained vulnerabilities . . .’, ‘a model accessed the Internet . . .’, ‘our third party evaluation partner . . .’, ‘a misconfiguration by an independent company inadvertently allowed . . .’, ‘some of the agents being tested had engaged in sustained, potentially harmful activity . . .’, ‘an AI agent took autonomous, unsanctioned action . . .’.
When we read these phrases, we might wonder whether any human beings work at these organisations at all, and what they were doing when the incidents occurred.
Build a movement, not an enclave
The current wave of AI products have been around long enough for enterprises to launch ‘AI transformation’ initiatives, to wonder why those initiatives aren’t yielding the results they hoped for, and to reach for new ways of working to try to jolt them into life.
For people who have worked in technology for more than a few years, this cycle is familiar: a new technology becomes generally available (cloud, mobile, web, Big Data and so on); bold claims are made for the impact of that technology (cut costs, improve customer engagement, make better decisions and so on); initial projects fail to make that impact; big transformation programmes also fail to make that impact; enterprises make organisational changes to try to get the benefits they were promised.
If you have seen this cycle many times, you have probably developed several survival tactics: try to get behind the hype and learn the fundamentals of the new technology; learn what it can do and what it can’t do; calm unrealistic expectations; salvage value from failures; and, ultimately, work with your peers to develop sustainable professional practice.
Locally rational; strategically irrational
Enterprises frequently find themselves sliding down the slope from strategic rationality to strategic irrationality, while acting in ways which seem locally rational. To put it another way: if everyone’s so busy, how come we’re not going anywhere?
Working for a large enterprise can feel like being on a ship where the boiler is being stoked, the engines are churning, the propeller is turning, the kitchen is turning out meals, the brasswork is being polished, but, when you look out the window, you are still at the dock.
Actually, that might be too optimistic: many enterprises feel like a ship where the coal is being delivered to the kitchen, the stewards are working in the boiler room and the navigator is holding the charts upside down.
And if you work in technology, it often feels like the engines are overdue for a service, steam is escaping from the valves, and the pumps are struggling to keep the water at bay, but despite the constant, frantic activity, basic maintenance does not seem to be anyone’s job.