It once occurred to me that modern civilization occupies an unstable state. I.J. Good’s hypothesized intelligence explosion describes a dynamically unstable system, like a pen precariously balanced on its tip. If the pen is exactly vertical, it may remain upright; but if the pen tilts even a little from the vertical, gravity pulls it farther in that direction, and the process accelerates. So too would smarter systems have an easier time making themselves smarter.
A dead planet, lifelessly orbiting its star, is also stable. Unlike an intelligence explosion, extinction is not a dynamic attractor – there is a large gap between almost extinct, and extinct. Even so, total extinction is stable.
The essay is called Artificial Intelligence as a Positive and Negative Factor in Global Risk and is certainly an insightful read. Download the PDF from here.
