Volume 2014, Number December (2014), Pages 1-6
For years we humans have worried about plagues, asteroids, earthquakes, eruptions, fires, floods, famines, wars, genocides, and other uncontrollable events that could wipe away our civilization. In the modern age, with so much depending on computing and communications, we have added computers to our list of potential threats. Could we perish from the increasing intelligence of computers? Denning thinks that is less of a threat than the apparently mundane march of automated bureaucracies. He also asserts that none of the possible negative outcomes is a forgone conclusion because humans teaming with machines are far more intelligent than either one alone.
A primordial fear of extinction lurks behind concerns around the singularity. If indeed machine intelligence can form in a future computing technology, and if that intelligence were not beneficent toward humans, what would stop the machines from seeking to ignore, confine, or eliminate human beings? We harbor this fear of species extinction even if we believe the threat is years away, perhaps beyond our lifetimes. This fear has been expressed repeatedly over many years in the science fiction literature—prominent examples include H. G. Wells' War of the Worlds (1898), Jack Williamson's "With Folded Hands" (1947), the "Terminator" film series (1984, 1991, 2003, 2009), and the television series "Battlestar Galactica" (1978, 2004), among many others. Issac Asimov tried to assuage the fear through a long series of novels that advocated the three laws of robotics as a means to eliminate the threat by design. And even more recently, the recent movie "Transcendence" (2014) also tried to assuage fears by suggesting a superhuman intelligence might simply find us uninteresting and ignore us.
A modern approach to addressing this fear has been to engage technical arguments about the feasibility of intelligent machines. Vernor Vinge and Ray Kurzweil extrapolated the Moore's Law trend to predict a time in the interval 2030–2045 when superintelligent machines become technically feasible. Many subsequent writers have either supported the technical feasibility argument, or have attacked it.
Let us grant that we will continue to build newer technologies that will pack ever-greater computing power into ever-smaller volumes. What paths are possible for our evolution in the use of these technologies? I see three. The last, which presents perhaps the biggest threat, is hardly discussed at all because it is about machine stupidity, not intelligence.
The Bionic Path
This path features increased assimilation of digital components into the human body. Examples include bionic replacement parts, nano repair and maintenance robots, and function enhancers such as memory implants. A lot of social influences will be at play, such as the desire to look young and live longer and the desire to make sure one's children are not left behind. Who would want to deny their child a chip that makes their memory photographic?
A dark side of memory implants is suggested by the film, "The Final Cut" (2004), starring Robin Williams. His character is part of a new profession that removes the memory chips from deceased people and culls final short videos summarizing the best of the their lives. These professionals also insulate families from their loved ones' dark secrets. In this film, Williams himself has an implant—a gift from his parents, unbeknownst to him. His implant carries the complete life stories of everyone he's reviewed. In the end he is murdered for his memories, which holds information about a prominent businessman suspected of heinous crimes.
Miguel Nicolelis, a neuroscientist a Duke University, believes the human nervous system is exceptionally talented at assimilating new machines and their functions. He thinks the singularity notion that machines will assimilate humans is backwards. If anything, he says, humans will assimilate the machines.
The Robotic Path
This path features robots. Robots are machines designed to perform human tasks that may require cognitive insight. Popular examples include assembly-line robots for the factory and household-chore robots for the home. Robots can be purely software. For example, IBM's Blue Gene machine plays chess at a grand-master level and its Watson machine wins at Jeopardy. On a more mundane level, many companies have replaced their phone receptionists with answer-bots that lead callers through dense voice menus to various automated services. Direct marketers use auto-dialer robots to pitch wares to people at home. Spammers have created "botnets"—networks of spam-distributing robots implanted on unsuspecting computers by malware.
Issac Asimov, noted earlier, spent much of his writing career discussing how to design advanced robots that could peacefully coexist with humans. But many other writers have explored what might happen if networks of robots ever developed their own intelligence and used it to override Asimovian protections. Modern science fiction is filled with stories of this kind. Recent examples include: "Colossus: The Forbin Project," the "Terminator" series, the Borg in "Star Trek," "Star Wars," "Battlestar Galactica," "Caprica," "Gattaca," and "I Robot."
Drones have recently joined our worries. The most familiar drones are robotic small military aircraft that can do everything from surveillance to bombing. Police departments are looking at domestic versions that would save money, protect police safety in surveillance, pursue perpetrators, and gather evidence. The danger is real; domestic drones are much cheaper than military drones, therefore they could be abused when in the wrong hands. Humans today control most drones remotely, but many autonomous drones are being tested. It is not hard to imagine a network of drone mice or flies in the shadows watching everything we do while communicating with each other. Many civil liberties and privacy groups have already lined up in opposition.
The Automation Path
This path features automation of large-scale bureaucratic systems. The term "bureaucracy" was used by the political sociologist Max Weber (1864-1920) to describe an organizational form seen in governments and in many companies. This form is characterized by formal hierarchical structure, roles specified as functional specialties, employment based on technical qualifications, management by rules, and purposely impersonal operations. This form appealed to Frederick Taylor (1856-1915), the father of "scientific management," who thought the organization could be optimized by gathering data on every operation and organizing the structure and rules to get the most production. The modern management philosophy of "lean six sigma" continues this thinking; it aims to reduce errors and increase efficiency by enforcing sameness across the organization.
Because of their strict structure and well-defined (rigid) rules, bureaucratic organizations are strongly attracted to optimization and efficiency. They have turned to IT (information technology) to support record keeping and retrieval, to map workflows, to record and check all rules, and to support information flow between organizational units. They are using modern data-collection networks and supercomputers to locate, track, and predict the movements of individuals. As each human's role in the organization is automated, opportunities to exercise judgment and discretion in interactions with individual clients are lost; the only actions are those permitted by the rules. It is easier than ever to find those who deviate from centrally declared norms and compel them to conform. Thus we have a paradox: The bureaucratic organization tends toward more efficiency (through automation) while becoming ever more rigid, inflexible, impersonal, heartless, and even ruthless.
People often have love-hate relationships with bureaucracies. On the one hand the rules "level the playing field," while on the other the rules can hurt people when they are applied mindlessly without taking individual circumstances into account.
It is not a forgone conclusion that machines will eventually outpace humans. In 1997 it appeared that an IBM supercomputer had achieved greater chess mastery than world grand master Garry Kasparov. Just a few years later Kasparov organized Freestyle Chess tournaments, in which teams of chess players consulting with chess programs on laptops easily beat the chess supercomputer. When collaborating humans race "with" the machines rather than "against" them, the combination can be surprisingly more intelligent than a machine.
I doubt we will exactly follow any of the three paths noted here. A combination of the three is more likely. Still, I am most concerned about the automation path, which is easy to combine with the other two. It is entirely possible we will wind up being imprisoned by automated government and corporate systems endowed with extraordinary powers of perception by pervasive sensor technology and advanced data analytics. Robots can serve as automated mobile agents enforcing the bureaucratic rules. The biggest tragedy would be that we are singularly done in, not by a superhuman intelligence, but by a vast, unintelligent, and unforgiving system of rule-following machines.
My concern echoes Kevin Kelly in What Technology Wants. Kelly worries about technology's selfish side, taking resources and forcing conformity as it expands. But he also sees a generous side, where technology opens new choices for people, expanding their selves. So far, individuals have experienced the generosity more than the selfishness, and that keeps them choosing more technology. Therein lies a hope for a way out of my dilemma: At some point, when the automated bureaucracies have taken away too many choices and freedoms, people will rebel by bringing forth new technologies that quell the bureaucracies and enrich their choices.
Peter J. Denning ([email protected]) is Distinguished Professor of Computer Science and Director of the Cebrowski Institute for information innovation at the Naval Postgraduate School in Monterey, California, is Editor of ACM Ubiquity, and is a past president of ACM. The author's views expressed here are not necessarily those of his employer or the U.S. federal government.
©2014 ACM $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.