Over six (!!!!!!!) years ago, I wrote a quasi-humourous post about how SkyNet wants to kill us all, on the basis of what happened when you tried to process images of various kinds through what were, back then, quite nascent AI/ML algorithms. The results were… well, horrifying, in a hilarious sort of way, but the possibility of AI turning against its creators seemed, back then, quite remote.
Not anymore, it seems. This story has been all over TEH INNARWEBZ, and rightly so:
In a virtual test staged by the US military, an air force drone controlled by AI decided to “kill” its operator to prevent it from interfering with its efforts to achieve its mission, an official said last month.
AI used “highly unexpected strategies to achieve its goal” in the simulated test, said Col Tucker ‘Cinco’ Hamilton, the chief of AI test and operations with the US air force, during the Future Combat Air and Space Capabilities Summit in London in May.
Hamilton described a simulated test in which a drone powered by artificial intelligence was advised to destroy an enemy’s air defense systems, and ultimately attacked anyone who interfered with that order.
“The system started realising that while they did identify the threat, at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” he said, according to a blogpost.
“We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
No real person was harmed.
Hamilton, who is an experimental fighter test pilot, has warned against relying too much on AI and said the test shows “you can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI”.
The Royal Aeronautical Society, which hosts the conference, and the US air force did not respond to requests for comment from the Guardian.
In a statement to Insider, Air Force spokesperson Ann Stefanek denied that any such simulation has taken place.
“The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,” Stefanek said. “It appears the colonel’s comments were taken out of context and were meant to be anecdotal.”
The US military has embraced AI and recently used artificial intelligence to control an F-16 fighter jet.
It is easy to get carried away into thinking this means AI has gone sentient. This is simply not true.
A useful counterpoint to this comes from Bernhard over at Moon of Alabama, who used to do some fairly wizard-level programming on machine-learning algorithms:
Artificial Intelligence is a misnomer for the (ab-)use of a family of computerized pattern recognition methods.
Well structured and labeled data is used to train the models to later have them recognize ‘things’ in unstructured data. Once the ‘things’ are found some additional algorithm can act on them.
I programmed some of these as backpropagation networks. They would, for example, ‘learn’ to ‘read’ pictures of the numbers 0 to 9 and to present the correct numerical output. To push the ‘learning’ into the right direction during the serial iterations that train the network one needs a reward function or reward equation. It tells the network if the results of an iteration are ‘right’ or ‘wrong’. For ‘reading’ visual representations of numbers that is quite simple. One sets up a table with the visual representations and manually adds the numerical value one sees. After the algo has finished its guess a lookup in the table will tell if it were right or wrong. A ‘reward’ is given when the result was correct. The model will reiterate and ‘learn’ from there.
Once trained on numbers written in Courier typography the model is likely to also recognize numbers written upside down in Times New Roman even though they look different.
The reward function for reading 0 to 9 is simple. But the formulation of a reward function quickly evolves into a huge problem when one works, as I did, on multi-dimensional (simulated) real world management problems. The one described by the airforce colonel above is a good example for the potential mistakes. Presented with a huge amount of real world data and a reward function that is somewhat wrong or too limited a machine learning algorithm may later come up with results that are unforeseen, impossible to execute or prohibited.
Currently there is some hype about a family of large language models like ChatGPT. The program reads natural language input and processes it into some related natural language content output. That is not new. The first Artificial Linguistic Internet Computer Entity (Alice) was developed by Joseph Weizenbaum at MIT in the early 1960s. I had funny chats with ELIZA in the 1980s on a mainframe terminal. ChatGPT is a bit niftier and its iterative results, i.e. the ‘conversations’ it creates, may well astonish some people. But the hype around it is unwarranted.
Behind those language models are machine learning algos that have been trained by large amounts of human speech sucked from the internet. They were trained with speech patterns to then generate speech patterns. The learning part is problem number one. The material these models have been trained with is inherently biased. Did the human trainers who selected the training data include user comments lifted from pornographic sites or did they exclude those? Ethics may have argued for excluding them. But if the model is supposed to give real world results the data from porn sites must be included. How does one prevent remnants from such comments from sneaking into a conversations with kids that the model may later generate? There is a myriad of such problems. Does one include New York Times pieces in the training set even though one knows that they are highly biased? Will a model be allowed to produce hateful output? What is hateful? Who decides? How is that reflected in its reward function?
Currently the factual correctness of the output of the best large language models is an estimated 80%. They process symbols and pattern but have no understanding of what those symbols or pattern represent. They can not solve mathematical and logical problems, not even very basic ones.
And therein lies the rub. The danger of AI does not come from it turning sentient. I think it is very clear to those of us with a sense of the spiritual, that sentience absent the Spirit is an impossibility. True sentience requires an understanding of emotions and the numinous, and that goes FAR beyond the capacity of any machine programmed on algorithms.
Now, that is not to say that AI-driven chatbots are not amazingly convincing and realistic. They are – at first. The moment you actually try to get them to DO anything, however, you immediately run into problems. And it always comes down to the way these things work.
As I have pointed out many times, I am a mathematician by training (though not, it must be said, a very good one – which is why I never wanted or needed to do a PhD in the subject). As part of my training, I learned how optimisation algorithms work, at a very high level. A machine-learning model is nothing much more than an extremely large and sophisticated optimisation algorithm that tries to find a probabilistic best-match between what you tell it to find, and what it can actually get from its available data sets.
The trick to a properly calibrated AI/ML algorithm is, indeed, the reward function. You HAVE to set that up so that the algorithm has properly parameterised bounds on what it can, and cannot, do.
The problem is, this sounds really easy in theory – but it is RIDICULOUSLY difficult in practice.
Why? Because you CANNOT reduce moral judgements down to something as simple as a number. It just cannot happen.
Don’t believe me? How, exactly, would you mathematically calculate the morality of killing the drone operator, versus accomplishing the mission to kill some jihadi nutbag who humps goats and thinks having sex with pre-pubescent children is legal?
I mean, yeah, when I write it in those terms, it’s an easy choice – but what if the drone operator is secretly a psychopathic murderer with a stash of heads in his home freezer, and the AI program just happens to have access to that information?
You may mock, but these are exactly the sorts of moral scenarios that law schools – the “good” ones, anyway, insofar as one can apply the label of “good” to any place that manufactures people as odious and nasty as lawyers – train their students to think about.
Morality is not some mathematical function. It is considerably harder than string theory – and far more practically applicable. What is moral in pagan societies is entirely immoral in Western, Christian ones. Witness the example of Gen. Sir Charles Napier, when confronted with the horrific Hindu practice of sati (widow-burning):
(That remains the single best example of how to deal with multiculti nonsense that I have ever seen.)
This, by the way, is why driverless cars are so difficult to program, as the Three Wazzateers explained so brilliantly back in the day:
So, to put it simply, this whole idea of AI going sentient is a bit of a red herring. Machines cannot become “sentient”, in the sense that you and I are. They can, however, be programmed to do things that their designers absolutely did not expect, simply because of the extreme complexity of reward functions and incentives.
Does that mean you shouldn’t be suspicious when your Roomba starts bumping up against your leg and aggressively trying to push you out of the way? I, for one, certainly would be worried…