Over six (!!!!!!!) years ago, I wrote a quasi-humourous post about how SkyNet wants to kill us all, on the basis of what happened when you tried to process images of various kinds through what were, back then, quite nascent AI/ML algorithms. The results were… well, horrifying, in a hilarious sort of way, but the possibility of AI turning against its creators seemed, back then, quite remote.
Not anymore, it seems. This story has been all over TEH INNARWEBZ, and rightly so:
In a virtual test staged by the US military, an air force drone controlled by AI decided to “kill” its operator to prevent it from interfering with its efforts to achieve its mission, an official said last month.
AI used “highly unexpected strategies to achieve its goal” in the simulated test, said Col Tucker ‘Cinco’ Hamilton, the chief of AI test and operations with the US air force, during the Future Combat Air and Space Capabilities Summit in London in May.
Hamilton described a simulated test in which a drone powered by artificial intelligence was advised to destroy an enemy’s air defense systems, and ultimately attacked anyone who interfered with that order.
“The system started realising that while they did identify the threat, at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” he said, according to a blogpost.
“We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
No real person was harmed.
Hamilton, who is an experimental fighter test pilot, has warned against relying too much on AI and said the test shows “you can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI”.
The Royal Aeronautical Society, which hosts the conference, and the US air force did not respond to requests for comment from the Guardian.
In a statement to Insider, Air Force spokesperson Ann Stefanek denied that any such simulation has taken place.
“The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,” Stefanek said. “It appears the colonel’s comments were taken out of context and were meant to be anecdotal.”
The US military has embraced AI and recently used artificial intelligence to control an F-16 fighter jet.
It is easy to get carried away into thinking this means AI has gone sentient. This is simply not true.
A useful counterpoint to this comes from Bernhard over at Moon of Alabama, who used to do some fairly wizard-level programming on machine-learning algorithms:
Artificial Intelligence is a misnomer for the (ab-)use of a family of computerized pattern recognition methods.
Well structured and labeled data is used to train the models to later have them recognize ‘things’ in unstructured data. Once the ‘things’ are found some additional algorithm can act on them.
I programmed some of these as backpropagation networks. They would, for example, ‘learn’ to ‘read’ pictures of the numbers 0 to 9 and to present the correct numerical output. To push the ‘learning’ into the right direction during the serial iterations that train the network one needs a reward function or reward equation. It tells the network if the results of an iteration are ‘right’ or ‘wrong’. For ‘reading’ visual representations of numbers that is quite simple. One sets up a table with the visual representations and manually adds the numerical value one sees. After the algo has finished its guess a lookup in the table will tell if it were right or wrong. A ‘reward’ is given when the result was correct. The model will reiterate and ‘learn’ from there.
Once trained on numbers written in Courier typography the model is likely to also recognize numbers written upside down in Times New Roman even though they look different.
The reward function for reading 0 to 9 is simple. But the formulation of a reward function quickly evolves into a huge problem when one works, as I did, on multi-dimensional (simulated) real world management problems. The one described by the airforce colonel above is a good example for the potential mistakes. Presented with a huge amount of real world data and a reward function that is somewhat wrong or too limited a machine learning algorithm may later come up with results that are unforeseen, impossible to execute or prohibited.
Currently there is some hype about a family of large language models like ChatGPT. The program reads natural language input and processes it into some related natural language content output. That is not new. The first Artificial Linguistic Internet Computer Entity (Alice) was developed by Joseph Weizenbaum at MIT in the early 1960s. I had funny chats with ELIZA in the 1980s on a mainframe terminal. ChatGPT is a bit niftier and its iterative results, i.e. the ‘conversations’ it creates, may well astonish some people. But the hype around it is unwarranted.
Behind those language models are machine learning algos that have been trained by large amounts of human speech sucked from the internet. They were trained with speech patterns to then generate speech patterns. The learning part is problem number one. The material these models have been trained with is inherently biased. Did the human trainers who selected the training data include user comments lifted from pornographic sites or did they exclude those? Ethics may have argued for excluding them. But if the model is supposed to give real world results the data from porn sites must be included. How does one prevent remnants from such comments from sneaking into a conversations with kids that the model may later generate? There is a myriad of such problems. Does one include New York Times pieces in the training set even though one knows that they are highly biased? Will a model be allowed to produce hateful output? What is hateful? Who decides? How is that reflected in its reward function?
Currently the factual correctness of the output of the best large language models is an estimated 80%. They process symbols and pattern but have no understanding of what those symbols or pattern represent. They can not solve mathematical and logical problems, not even very basic ones.
And therein lies the rub. The danger of AI does not come from it turning sentient. I think it is very clear to those of us with a sense of the spiritual, that sentience absent the Spirit is an impossibility. True sentience requires an understanding of emotions and the numinous, and that goes FAR beyond the capacity of any machine programmed on algorithms.
Now, that is not to say that AI-driven chatbots are not amazingly convincing and realistic. They are – at first. The moment you actually try to get them to DO anything, however, you immediately run into problems. And it always comes down to the way these things work.
As I have pointed out many times, I am a mathematician by training (though not, it must be said, a very good one – which is why I never wanted or needed to do a PhD in the subject). As part of my training, I learned how optimisation algorithms work, at a very high level. A machine-learning model is nothing much more than an extremely large and sophisticated optimisation algorithm that tries to find a probabilistic best-match between what you tell it to find, and what it can actually get from its available data sets.
The trick to a properly calibrated AI/ML algorithm is, indeed, the reward function. You HAVE to set that up so that the algorithm has properly parameterised bounds on what it can, and cannot, do.
The problem is, this sounds really easy in theory – but it is RIDICULOUSLY difficult in practice.
Why? Because you CANNOT reduce moral judgements down to something as simple as a number. It just cannot happen.
Don’t believe me? How, exactly, would you mathematically calculate the morality of killing the drone operator, versus accomplishing the mission to kill some jihadi nutbag who humps goats and thinks having sex with pre-pubescent children is legal?
I mean, yeah, when I write it in those terms, it’s an easy choice – but what if the drone operator is secretly a psychopathic murderer with a stash of heads in his home freezer, and the AI program just happens to have access to that information?
You may mock, but these are exactly the sorts of moral scenarios that law schools – the “good” ones, anyway, insofar as one can apply the label of “good” to any place that manufactures people as odious and nasty as lawyers – train their students to think about.
Morality is not some mathematical function. It is considerably harder than string theory – and far more practically applicable. What is moral in pagan societies is entirely immoral in Western, Christian ones. Witness the example of Gen. Sir Charles Napier, when confronted with the horrific Hindu practice of sati (widow-burning):
(That remains the single best example of how to deal with multiculti nonsense that I have ever seen.)
This, by the way, is why driverless cars are so difficult to program, as the Three Wazzateers explained so brilliantly back in the day:
So, to put it simply, this whole idea of AI going sentient is a bit of a red herring. Machines cannot become “sentient”, in the sense that you and I are. They can, however, be programmed to do things that their designers absolutely did not expect, simply because of the extreme complexity of reward functions and incentives.
Does that mean you shouldn’t be suspicious when your Roomba starts bumping up against your leg and aggressively trying to push you out of the way? I, for one, certainly would be worried…
2 Comments
That is exactly what I have been saying for years. Despite the technological advances, it is impossible, as impossible as antigravity, to create something like a ‘True’ AI, artificial Sapience. If you want to create sapience, you have to do it the old fashioned way, through reproduction.
Despite much prating about humans being ‘biological machines’ exactly the same obstacles stand in the way of intelligence as stands in the way of the concept of intelligence being created via natural selection.
The biggest one is simply the way self-motivation thinking and memory storage interfaces. Obviously, without memory, even the concept of ‘self’ is impossible.
But humans, and to an extent, sapient (not sentient) animals, store memories the same way. To whit, associatively. When you eat a cookie, your experience does not point at ‘cookie, flavor’, it points at all of the other experiences of eating a cookie. You remember the smell of your mother’s oven when she made that kind of cookie, or the sound of traffic rushing by as you ate a cookie from macdonalds drive-through. A thousand different associations rush through your head as you taste a cookie, some correctly and some incorrectly. When you recall a fact, that fact is linked with the experiences that surrounded first being exposed to that fact, which allows you to make value judgements.
Machines store independent pieces of data. That is why we are capable of storing so much more data than any sort of machine, because our memories are a series of pointers that lead to our unconscious ability to reinterpret those pointers. That is also why our memories are so often wrong, and we need our conscious mind to re-code that information and recall the memory of a fact… we are literally re-creating a memory on the fly based on our other similar experiences.
People, especially professional historians and researchers, love the simplicity of being able to point to a single fact or incident, idea or occurrence, and say “That is when things changed’. But Humans are NOT simple. Even the simplest idea or recollection is a conglomerate of millions of potential data pointers. Machine brains are yes/no engines, and even those machines that have a three logic yes/no/maybe decision tree are simply faking it by trying to inject more data into the ‘maybe’ tree, such as comparison engines.
Humans are not yes/no engines. We don’t even have a ‘yes/no’ logic in our brains at all. Every decision, even the instantaneous ones, is created from a cloud of potential possibilities that our experiences, training, and a million other factors all narrow down based on our intentions. Our capacity for error is, in fact, the core of our brilliance in every possible way, and there is simply no way to simulate that kind of thought with a glorified set of light switches.
In short, true AI is simply impossible. Certainly, we could create something like Skynet, but that would be it’s core programmed directive, and it would not be capable of the kind of creative thought processes that ‘artificial sapience’ would assume. Skynet would kill all humans because that is what it was programmed to do, not because it has a mind. True AI is as much a product of poorly researched scifi tropes as ‘nanites’ and ‘hyperspace’.
And yes, I consider Asimov a poor scientist. His 3 laws are the product of insane oversimplifaction, not brilliant minimalization that so many people claim.
“The trick to a properly calibrated AI/ML algorithm is, indeed, the reward function. You HAVE to set that up so that the algorithm has properly parameterised bounds on what it can, and cannot, do.”
.
.
that was the point i made yesterday IRL when Vox posted that story. the scariest thing in the story is NOT that “Skynet is waking up”, it’s that the damn programmers were so stupid that they didn’t include any negative weighting or Asimov Laws ( 1942 ) against harming your own side. that’s an 80 year old concept.
.
this implies that, no matter what, we’re eventually going to have a Skynet. simply because there are so many idiots who are directly engaged in programming or are managing the projects that eventually some moron is going to do this with a system that can escape into the wild.