Planetary Optimization and Advanced General Intelligences: the 'intelligence stairway'

Jann Tallinn

The emergence of self-replicating physical systems 4 billion years ago marked the start of terrestrial evolution - which we view as a set of competing processes, optimising for the fitness of self-replicators. After evolution produced humans, however, a phase change occurred, because the optimisation power of humans, through technology, exceeded that of evolution. One can ask if future computers, smarter than their creators, could cause a similar phase transition. If so, then such AGIs (artificial general intelligences) will lead to a new AGI-driven "intelligence explosion" phase. Models indicate this may manifest itself as a sudden global ecological
catastrophe.
I discuss two ways of avoiding the catastrophe: either (i) collaboration (the "Friendly AI" approach), or (ii) artificial limitation of the AI-s to narrow domains. The latter approach is more pragmatic, and will require a "practical safety protocol" for AI developers to prevent accidental AGI's being produced.

NOTE: This is the second of two consecutive PITP events (see previous talk by Scott Aaronson). There will be a panel discussion, open to the public, immediately after this second talk (panel members to be announced).

Additional resources for this talk: slides and video.