AI armageddon? | Gary Marcus
The new book by Eliezer Yudkowsky and Nate Soares is everywhere and everything all at once. The physicist Max Tegmark claims it to be “the most important book of the decade”. The list of blurbs, from a range of experts, including computer scientists and air force generals, is impressive. Major media began contacting me for comment about it long before its release. And certainly the authors are convinced that no book could be more central to the future of humanity.
Their fundamental claim is concisely stated in their title: if anybody builds artificial general intelligence, the human species will go extinct. They lay out this thesis thoughtfully, entertainingly, earnestly, provocatively and doggedly. Yet their book is also deeply flawed. It deserves to be read with an immense amount of salt. This review will bring that salt. Even so, there is much in it that is instructive and worth heeding.
Yudkowsky and Soares have both been thinking about these matters for a long time. Yudkowsky was one of the first people to draw public awareness to the risks of runaway AI; Soares is the president of the Machine Intelligence Research Institute, a body committed to understanding and preparing for such risks. And much of what they say strikes me as 100 per cent correct and 100 per cent worthy of broad, global attention, starting with these eight claims:
- Rogue AI is a possibility that we should not ignore. We don’t know for sure what future AI will do and we cannot rule out the possibility that it will go rogue.
- We currently have no solution to the “alignment problem” of making sure that machines behave in human- compatible ways.
- Figuring out a solution to the alignment problem is really, really important.
- Figuring out a solution to the alignment problem is really, really hard.
- Superintelligence might come relatively soon, and we are not prepared for it.
- The public should be more concerned than it is.
- Governments should be more concerned than they are.
- The short-term benefits of AI (e.g., in terms of economics and productivity) may not be worth the long-term risks.