A fine piece on desirable governance. One suspects that at this point a much needed AI arms control treaty will come to pass only if the collective super AIs decide among themselves to allow it!
This is completely naive and dangerous: "Fortunately, it would be possible to both monitor and enforce an international agreement prohibiting superintelligence, because there is a large industrial process behind advanced AI systems. This is akin to the extensive, largely traceable process required to create nuclear weapons."
The infrastructure for super intelligence is identical or smaller than what's needed for LLMs. LLMs try to know as many facts as 1000 people all with different specialties. This has nothing to do with super intelligence. Einstein had 1/1000 the information of an LLM. Intelligence is a matter of the correct learning algorithm and roughly the computing power of today's data centers.
And in a few years those will have 10 times the compute using the same power and size. That's the current prediction and it's been doing that for 50 years.
And the development of the correct algorithm will take even less compute than the full scale version.
So there is no way to detect from outside the building what the AI scientists are cooking up. Get real!
Persuasion needs to have debates between their essayists. Yes it would cause some embarrassment. But debate, not one-sided journalistic hit pieces is how science gets things right. As subscribers we deserve to see how our illustrious teachers fare when they have to defend their ideas against their peers.
The situation is worse than the author states because these 'superintelligent AIs' are not free to make up their own mind but are controlled by humans, could be evil ones.
Humans must provide the will to act by giving commands e.g. "kill them all I don't care how you do it".
If a 'self driving car' veers off the road into a crowd of people we might anthropomorphise the car and say 'the car did this' but of course the event is actually the result of many human decisions from software development to the actual driver being too slow to take over to politicians who allow lax rules for these vehicles due to lobby pressure. This diffusion of responsibility is the real problem here, if everyone is responsible then no one is.
Suppose a superintelligent AI is stealing money from your bank account. "The superintelligent AI is doing it by its own volition" cry the AI boosters so nothing can be done! Of course in reality the cash is going into someone's account who set up the AI to do it.
In addition the creation of AIs with evil intent may not require munching through the data of the entire internet and so may not require the most massive facilities.
The self interested witterings of dodgy characters such as Sam Altman are not a useful guide to anything.
What interests me is that preventing anyone from obtaining it seems to require some institution with the authority to decide what counts as it, who is violating the rules, and how those rules are enforced.
Even attempts to eliminate ultimate power often seem to recreate an ultimate authority somewhere else.
A fine piece on desirable governance. One suspects that at this point a much needed AI arms control treaty will come to pass only if the collective super AIs decide among themselves to allow it!
This is completely naive and dangerous: "Fortunately, it would be possible to both monitor and enforce an international agreement prohibiting superintelligence, because there is a large industrial process behind advanced AI systems. This is akin to the extensive, largely traceable process required to create nuclear weapons."
The infrastructure for super intelligence is identical or smaller than what's needed for LLMs. LLMs try to know as many facts as 1000 people all with different specialties. This has nothing to do with super intelligence. Einstein had 1/1000 the information of an LLM. Intelligence is a matter of the correct learning algorithm and roughly the computing power of today's data centers.
And in a few years those will have 10 times the compute using the same power and size. That's the current prediction and it's been doing that for 50 years.
And the development of the correct algorithm will take even less compute than the full scale version.
So there is no way to detect from outside the building what the AI scientists are cooking up. Get real!
Persuasion needs to have debates between their essayists. Yes it would cause some embarrassment. But debate, not one-sided journalistic hit pieces is how science gets things right. As subscribers we deserve to see how our illustrious teachers fare when they have to defend their ideas against their peers.
The situation is worse than the author states because these 'superintelligent AIs' are not free to make up their own mind but are controlled by humans, could be evil ones.
Humans must provide the will to act by giving commands e.g. "kill them all I don't care how you do it".
If a 'self driving car' veers off the road into a crowd of people we might anthropomorphise the car and say 'the car did this' but of course the event is actually the result of many human decisions from software development to the actual driver being too slow to take over to politicians who allow lax rules for these vehicles due to lobby pressure. This diffusion of responsibility is the real problem here, if everyone is responsible then no one is.
Suppose a superintelligent AI is stealing money from your bank account. "The superintelligent AI is doing it by its own volition" cry the AI boosters so nothing can be done! Of course in reality the cash is going into someone's account who set up the AI to do it.
In addition the creation of AIs with evil intent may not require munching through the data of the entire internet and so may not require the most massive facilities.
The self interested witterings of dodgy characters such as Sam Altman are not a useful guide to anything.
What strikes me is that debates about superintelligence increasingly stop being debates about AI.
They become debates about who gets to decide humanity’s technological future.
Calls for moratoria, international agreements, and public authorization all seem to point toward the same question:
If superintelligence is important enough that private actors cannot be trusted with it, who becomes the legitimate authority that can?
The governance problem may ultimately be a sovereignty problem.
According to my understanding of the author, that's a bit like two blokes arguing over who gets to detonate the both of them with a nuke.
I agree that catastrophic power is the concern.
What interests me is that preventing anyone from obtaining it seems to require some institution with the authority to decide what counts as it, who is violating the rules, and how those rules are enforced.
Even attempts to eliminate ultimate power often seem to recreate an ultimate authority somewhere else.