I.

The cobra effect describes the unintended consequences of a system, caused by perverse incentives from ill-defined goals.

The story goes that during British colonial rule in India, the government wanted to cull the cobra populations in Delhi. They did this by paying out a bounty for every cobra killed.

Initially, this worked out fine — people killed the cobras that they found, and turned them over to the authorities. Eventually, people figured out that it was more profitable for to breed even more cobras, and kill those too.

When the government realized this, they discontinued this system, and people released the cobras that they had been breeding, making the problem even worse than before.

II.

The paperclip maximizer is a thought experiment in AI risk that shows the (hypothetical) adverse consequences of ill-defined goals at their most extreme.

The premise is as follows: suppose you program a (sufficiently advanced) robot to maximize the number of paperclips. It might start by (reasonably) collecting all the paperclips it could find, and bringing them to you.

But after it does that, it might also realize that it could convert other things into paperclips. Things like raw metal, other machines, and the atoms in the human body.

III.

The cobra killer says, "maximize the number of dead cobras at all costs."

The paperclip robot says, "maximize the number of paperclips at all costs."

The high school student says, "maximize my grade at all costs."

The corporation says, "maximize shareholder value at all costs."

The social network says, "maximize user engagement at all costs."

Are not all systems just different manifestations of cobra effects?