Overview

Our work at MAI is woven from 5 academic strands.

  1. A set of critiques of revealed preference as a basis for welfare metrics and social choice mechanisms, spelled out mainly by Nobel laureate, welfare econ and social choice pioneer Amartya Sen.
  2. An alternate basis for an alignment target / social welfare function / etc, based on a specific definition of human values and a way to reconcile/aggregate the values of groups.
  3. Related theories of human moral learning, ****which explain how a person’s values (or a machine’s) might evolve when faced with new situations of moral weight.
  4. Theories about the relationship between attention, meaning, and values which inform how we collect values, disambiguate them, etc.
  5. Finally, a broad theory of history where big advances in coordination come from new ways to say what matters to us (so we can coordinate around these new vocabularies of benefit).

1. Critiques of Revealed Preferences

From my “Values, Preferences, Meaningful Choice”

In his 1938 paper introducing revealed preference, @Samuelson1938 warned:

I should like to state my personal opinion that nothing said here... touches upon at any point the problem of welfare economics, except in the sense of revealing the confusion in the traditional theory of these distinct subjects.

Similar sentiments followed, in @Arrow1951, @Sen1977, @Anderson2001, etc. As an information basis for welfare, optimality, social choice, etc, revealed preference has been much critiqued.

A rich literature covers how revealed preferences---which, when summed up, are called engagement metrics---lead us astray. You can often get people to choose something without serving their real interests: you can misinform them, or leverage their misplaced hopes.

Or, you can make it so people need your thing for what once was possible without it: they need your car to get to work, your social media account to find a job, your dress to socialize with their friends, etc.

More broadly, you can manufacture social circumstances where people choose your thing to "keep up with the Joneses", to signal allegiance with their tribe, or because they've lost the ability to coordinate a real solution.[See discussions of the prisoner's dilemma in e.g., @Sen1973; @Anderson2001]

In some cases, the person will know their choice doesn't express their true interests---that they are bucking to external pressure, caught in the system, or setting aside their goals to conform to a social rule.[See @Sen1977; @Anderson1993 on 'commitment'] In other cases, options have been limited or biased behind our backs.

Specifically regarding social choice, economist Amartya Sen writes eloquently about how information on people’s preferences—even their stated preferences across a menu of options—isn’t enough for good social choices. Even richer information—about people's wellbeing or utility in different hypothetical states—isn’t good enough.

Preference-based systems like markets, voting, and recommenders don’t carry all the “demand” needed for good social choices.

Many of the failures of markets to provide for what we value (community, adventure, meaningful work, personal growth, ...) are because markets act on the preference supposedly revealed by our choices, rather than delivering what’s actually important to us;

Sen advocates for what he calls “information broadening”, collecting extra information to help systems like markets and recommenders make better choices:

Can it be said that one of the things we have learned from Arrow’s impossibility theorem and related results, and the discussion and scrutiny that followed this mathematical development, is the need to alter the formulation of social choice problems, perhaps through broadening its informational base? I would argue that the short answer to the question is: yes.” (Sen, 1970 - Collective Choice and Social Welfare)

Sen, together with philosopher Martha Nussbaum, developed his own information broadening approach, “the capability approach”. Unfortunately, it’s a fairly low-res account of what to honor in human life, insufficient to direct LLMs and recommenders.

<aside> 📄 Reading