A fine piece on desirable governance. One suspects that at this point a much needed AI arms control treaty will come to pass only if the collective super AIs decide among themselves to allow it!
This is completely naive and dangerous: "Fortunately, it would be possible to both monitor and enforce an international agreement prohibiting superintelligence, because there is a large industrial process behind advanced AI systems. This is akin to the extensive, largely traceable process required to create nuclear weapons."
The infrastructure for super intelligence is identical or smaller than what's needed for LLMs. LLMs try to know as many facts as 1000 people all with different specialties. This has nothing to do with super intelligence. Einstein had 1/1000 the information of an LLM. Intelligence is a matter of the correct learning algorithm and roughly the computing power of today's data centers.
And in a few years those will have 10 times the compute using the same power and size. That's the current prediction and it's been doing that for 50 years.
And the development of the correct algorithm will take even less compute than the full scale version.
So there is no way to detect from outside the building what the AI scientists are cooking up. Get real!
Persuasion needs to have debates between their essayists. Yes it would cause some embarrassment. But debate, not one-sided journalistic hit pieces is how science gets things right. As subscribers we deserve to see how our illustrious teachers fare when they have to defend their ideas against their peers.
Fascinating that we are debating super-intelligence when we have yet to solve for intelligence. And we are still many orders of magnitude away from that. I recommend the book "The Myth of Artificial Intelligence" by Erik J Larson to get an understanding of just how far away we are from all of these dire predictions. LLMs may be useful tools when used correctly, but they are simply NOT going to iterate themselves to intelligence.
Please STOP trying to scare people with the bad science fiction scenarios.
Agree. There seems ~zero chance of superintelligence emerging out of the current LLM Gen AI paradigm for many reasons. However, that does not mean there is no serious risk on the horizon: purposefully or accidentally mis-aligned agentic Gen AI, with super-human coding/malware skills, could be a serious if not existential threat to our fragile, network dependent society in the near future.
If "The unavoidable price of reliability is simplicity." C.A.R. (Tony) Hoare's Turing Award lecture, The Emperor's Old Clothes (1980) - then the unavoidable price of extreme complexity (of trillion+ parameter agentic Gen AI that is not understandable) may be extreme fragility.
What interests me is that preventing anyone from obtaining it seems to require some institution with the authority to decide what counts as it, who is violating the rules, and how those rules are enforced.
Even attempts to eliminate ultimate power often seem to recreate an ultimate authority somewhere else.
After reading Superintelligence by Nick Bostrom, I find these types of warnings unhelpful. Quotes like the following:
“the most dangerous weapons are controlled by humans...Superintelligence would upend this”
Don't give world leaders anything tangible work with. Where can you realistically draw a line and say, "don't cross this, or it becomes superintelligence"? The premise seems based on a linear definition of intelligence, but Bostrom never gives a detailed explanation of how to quantify that.
Computers have been better than humans at math from day one. Right now they are converging with our verbal/communications skills. Outside of narrow mathematical domains, most other forms of intelligence, such as analyzing Shakespeare, are more open-ended. These can't be easily quantified.
A more pragmatic approach is to regulate what goes into the training data (e.g. don't allow it to pirate virology research papers) and to limit what it can access operationally (don't give it access to money and weapons). I would be happy to debate the AI doomers on this given the opportunity.
I actually find your point complementary rather than opposed.
If we move from regulating “superintelligence” to regulating capabilities, someone still has to define which capabilities matter and who gets to enforce those limits.
The question seems to shift from “What is intelligence?” to “Who defines dangerous capability?”
The situation is worse than the author states because these 'superintelligent AIs' are not free to make up their own mind but are controlled by humans, could be evil ones.
Humans must provide the will to act by giving commands e.g. "kill them all I don't care how you do it".
If a 'self driving car' veers off the road into a crowd of people we might anthropomorphise the car and say 'the car did this' but of course the event is actually the result of many human decisions from software development to the actual driver being too slow to take over to politicians who allow lax rules for these vehicles due to lobby pressure. This diffusion of responsibility is the real problem here, if everyone is responsible then no one is.
Suppose a superintelligent AI is stealing money from your bank account. "The superintelligent AI is doing it by its own volition" cry the AI boosters so nothing can be done! Of course in reality the cash is going into someone's account who set up the AI to do it.
In addition the creation of AIs with evil intent may not require munching through the data of the entire internet and so may not require the most massive facilities.
The self interested witterings of dodgy characters such as Sam Altman are not a useful guide to anything.
We need to define the terms better before we try strict regulations on this emerging technology! But obviously the technology has serious potential to threaten human wellbeing. And its a lot like climate change, a clear danger that has no simple short term solutions 😎
A fine piece on desirable governance. One suspects that at this point a much needed AI arms control treaty will come to pass only if the collective super AIs decide among themselves to allow it!
This is completely naive and dangerous: "Fortunately, it would be possible to both monitor and enforce an international agreement prohibiting superintelligence, because there is a large industrial process behind advanced AI systems. This is akin to the extensive, largely traceable process required to create nuclear weapons."
The infrastructure for super intelligence is identical or smaller than what's needed for LLMs. LLMs try to know as many facts as 1000 people all with different specialties. This has nothing to do with super intelligence. Einstein had 1/1000 the information of an LLM. Intelligence is a matter of the correct learning algorithm and roughly the computing power of today's data centers.
And in a few years those will have 10 times the compute using the same power and size. That's the current prediction and it's been doing that for 50 years.
And the development of the correct algorithm will take even less compute than the full scale version.
So there is no way to detect from outside the building what the AI scientists are cooking up. Get real!
Persuasion needs to have debates between their essayists. Yes it would cause some embarrassment. But debate, not one-sided journalistic hit pieces is how science gets things right. As subscribers we deserve to see how our illustrious teachers fare when they have to defend their ideas against their peers.
Fascinating that we are debating super-intelligence when we have yet to solve for intelligence. And we are still many orders of magnitude away from that. I recommend the book "The Myth of Artificial Intelligence" by Erik J Larson to get an understanding of just how far away we are from all of these dire predictions. LLMs may be useful tools when used correctly, but they are simply NOT going to iterate themselves to intelligence.
Please STOP trying to scare people with the bad science fiction scenarios.
Agree. There seems ~zero chance of superintelligence emerging out of the current LLM Gen AI paradigm for many reasons. However, that does not mean there is no serious risk on the horizon: purposefully or accidentally mis-aligned agentic Gen AI, with super-human coding/malware skills, could be a serious if not existential threat to our fragile, network dependent society in the near future.
If "The unavoidable price of reliability is simplicity." C.A.R. (Tony) Hoare's Turing Award lecture, The Emperor's Old Clothes (1980) - then the unavoidable price of extreme complexity (of trillion+ parameter agentic Gen AI that is not understandable) may be extreme fragility.
What strikes me is that debates about superintelligence increasingly stop being debates about AI.
They become debates about who gets to decide humanity’s technological future.
Calls for moratoria, international agreements, and public authorization all seem to point toward the same question:
If superintelligence is important enough that private actors cannot be trusted with it, who becomes the legitimate authority that can?
The governance problem may ultimately be a sovereignty problem.
According to my understanding of the author, that's a bit like two blokes arguing over who gets to detonate the both of them with a nuke.
I agree that catastrophic power is the concern.
What interests me is that preventing anyone from obtaining it seems to require some institution with the authority to decide what counts as it, who is violating the rules, and how those rules are enforced.
Even attempts to eliminate ultimate power often seem to recreate an ultimate authority somewhere else.
After reading Superintelligence by Nick Bostrom, I find these types of warnings unhelpful. Quotes like the following:
“the most dangerous weapons are controlled by humans...Superintelligence would upend this”
Don't give world leaders anything tangible work with. Where can you realistically draw a line and say, "don't cross this, or it becomes superintelligence"? The premise seems based on a linear definition of intelligence, but Bostrom never gives a detailed explanation of how to quantify that.
Computers have been better than humans at math from day one. Right now they are converging with our verbal/communications skills. Outside of narrow mathematical domains, most other forms of intelligence, such as analyzing Shakespeare, are more open-ended. These can't be easily quantified.
A more pragmatic approach is to regulate what goes into the training data (e.g. don't allow it to pirate virology research papers) and to limit what it can access operationally (don't give it access to money and weapons). I would be happy to debate the AI doomers on this given the opportunity.
I actually find your point complementary rather than opposed.
If we move from regulating “superintelligence” to regulating capabilities, someone still has to define which capabilities matter and who gets to enforce those limits.
The question seems to shift from “What is intelligence?” to “Who defines dangerous capability?”
The situation is worse than the author states because these 'superintelligent AIs' are not free to make up their own mind but are controlled by humans, could be evil ones.
Humans must provide the will to act by giving commands e.g. "kill them all I don't care how you do it".
If a 'self driving car' veers off the road into a crowd of people we might anthropomorphise the car and say 'the car did this' but of course the event is actually the result of many human decisions from software development to the actual driver being too slow to take over to politicians who allow lax rules for these vehicles due to lobby pressure. This diffusion of responsibility is the real problem here, if everyone is responsible then no one is.
Suppose a superintelligent AI is stealing money from your bank account. "The superintelligent AI is doing it by its own volition" cry the AI boosters so nothing can be done! Of course in reality the cash is going into someone's account who set up the AI to do it.
In addition the creation of AIs with evil intent may not require munching through the data of the entire internet and so may not require the most massive facilities.
The self interested witterings of dodgy characters such as Sam Altman are not a useful guide to anything.
We need to define the terms better before we try strict regulations on this emerging technology! But obviously the technology has serious potential to threaten human wellbeing. And its a lot like climate change, a clear danger that has no simple short term solutions 😎