- Why do bureaucracies of good people do bad things?
- Why do apps with good intentions cause bad social outcomes?
- Why do helpful innovations — like cars, single-family homes, and smartphones — lead people into isolated lives that don’t serve them?
I’ve spent much of my life learning about values-misalignment, which covers everything above.
This is a quick overview of the problem and why common design approaches (including mechanism design, XD, design thinking, and red teaming) don’t address it well. Based on why those fail, I’ll outline of an alternative.
Values-misalignment
Values-misalignment is ubiquitous in human life. It’s there in a shallow dinner conversation among people who’d rather go deep; there when an organization’s promotions process makes honest people withhold information; there when an “egalitarian” team structure gives rise to hidden power dynamics; there when attempts to make the world “more open and connected” give rise to political polarization on an unprecedented scale.
**You can tell people want to make fewer values-misaligned things.**There are popular essays, like Meditations on Moloch. There are talking heads like Eric Weinstein and Daniel Schmachtenberger. And academic fields like social choice, game theory, and mechanism design purport to deal with values-alignment, with a vast literature on market and coordination failures.
To make the problem clearer, let’s divide it in two:
- The Hard Problem of Values-Misalignment is when there’s no agreement about values, or no resources to check designs against them. Examples: (1) an organization is run by sociopaths who claim certain values but pursue a different, secret agenda; (2) users and designers have different values; (3) cutthroat competition means there’s no time to think ahead.
- The Easy Problem of Values-Misalignment is when there’s rough agreement among stakeholders about what’s desired and what’d be bad, and there are resources to make something good.
Even with the Easy Problem, we’ve made very little progress. Usually, good people fail to anticipate the social consequences of what they build, even when they take time to think.
Existing Approaches
Imagine you’re working on something. Your team has rough agreement about what’d be good and bad, and can afford to design thoughtfully.
What kind of thinking should you do together?
I can’t cover all the existing approaches, but the most popular are plainly inadequate:
- The most common, I’ll call naive, ideological optimism. That’s when you latch on to an abstract ideological vision — like “decentralized”, “anonymous”, or “inclusive”, and hope that by building things that way, the right social outcomes will result. This approach seldom works: when you design around an ideology without considering specific human impacts (such as whether people can be vulnerable, creative, or take charge of things), your design is unlikely to work out regarding those specific human impacts (that you forgot to think about).
- Somewhat more sophisticated is red teaming, where to avoid bad outcomes, you split participants into “bad actors” and normal people, and guess how the bad actors will ruin things. Bad people do exist, and red teaming helps limit their influence. But, while most of us aren’t bad actors, we all struggle to live by our values online and at work. Many problems (like clickbait, political polarization, and office politics) aren’t mainly due to “bad actors”. Red teaming doesn’t help much to address them.
Next up are design subfields like mechanism design, experience design, and speculative futurism. Each is inadequate in its own way:
- Mechanism design tries to incentivize good performance, or drive an allocation of resources. Unfortunately, as I’ve covered elsewhere, mechanism design is based on a inaccurate model of human beings. For this reason, “incentivizing good things” usually creates a mess: badges, points, endless optimization, golden handcuffs, bureaucracies, and credit scores.