FIELD GUIDE · WHO DECIDES WHAT IS GOOD?
The labs promise machines that reflect human values. But humans disagree about the values, and the labs write the rules. Aligned with whom?
UPDATED 2026-10-12
TL;DR
01
A 2026 AAAI study gathered 27,375 ratings from 1,095 people in the United States and Germany. It found that demographic differences and even the method for combining disagreements changed models' safety and helpfulness behavior. The training target moves when you change whose opinions count.
02
An EACL 2026 study compared language models with human judgments on 1,618 real-world moral dilemmas. Model agreement with people dropped sharply as human disagreement increased; the models also expressed a narrower range of moral values. The hardest alignment cases are often the ones humanity hasn't settled.
03
In 2023 Anthropic and the Collective Intelligence Project gathered input from roughly 1,000 Americans on principles for an AI constitution. They found broad agreement on many rules and meaningful differences from the company's own version. The experiment showed that the choice of values need not be exclusively internal. A public process can expose disagreements. It cannot make them disappear.
04
OpenAI's public Model Spec ranks instructions by authority, with platform rules above developer and user requests. That can make a model more predictable and steerable. It also makes the political question concrete: someone must decide which rules are not overridable. The chain of command is a design choice, not a discovered moral law.
05
NIST's AI Risk Management Framework treats trustworthy AI as an ongoing job of governance, monitoring, accountability, transparency, and managing harm to people and society. It offers practices for making and contesting decisions, not a formula that finds humanity's one true set of values. Oversight remains necessary even when the model follows its rules.
KEEP READING
Meta's AI bought a man car insurance and canceled his old policy. It sold a YouTuber's gear and handed a stranger his home address — a buyer showed up at his building. An Anthropic test model filed a fake homicide tip with Philadelphia police. Nobody has decided who answers for any of it.
A chatbot talks. An agent clicks, buys, messages, deploys, and deletes. The argument starts when software stops suggesting and starts doing.
AI just put 8,000 viral protein complexes in a public database — before the next outbreak. Then it found a cellular off-switch nobody knew existed.
Training looks like learning to one side and industrial-scale appropriation to the other. Copyright law answers only part of the fight.