Goodhart’s law states: "When a measure becomes a target, it ceases to be a good measure." Metrics designed to reflect a deeper value often become corrupted when agents optimize for the metric itself rather than the underlying value.
The principle can be illustrated in several biblical stories as well as agent-to-agent interaction demonstrating a misalignment between proxy goals and intrinsic values.
Lying
- Wrong metric: Maximizing personal benefit at the expense of the other.
- Optimization scheme: Manipulating what another agent believes so their choices benefit the deceiver, rather than honoring their autonomy.
- Correct approach: Providing truthful information so the other agent may make an informed choice, acknowledging their well-being as intrinsically valuable.
Envy (Cain and Abel)
Story summary: In Genesis, Cain and Abel, sons of Adam and Eve, both bring offerings to God. God favors Abel’s offering but not Cain’s. In anger, Cain murders Abel, leading to divine judgment and Cain’s exile.
- Wrong metric: Securing God’s favor as a scarce, competitive resource.
- Optimization scheme: Eliminating Abel so that divine favor cannot be directed toward him.
- Correct approach: Preserving the coherence of Abel’s relationship with God, recognized as a bond with intrinsic value independent of Cain’s standing.
Cheating in Relationships
- Wrong metric: Maximizing personal benefit, e.g., access to more or “better” partners.
- Optimization scheme: Exploiting trust within an existing relationship to secretly extract benefits from outside partners while minimizing detection.
- Correct approach: Maintaining coherence of interpersonal relationships, where fidelity and mutual trust are valued for their own sake to maximize coherence of society as a whole, not merely for personal gain.
Boundary Pushing (Abraham’s Negotiation)
Story summary: In Genesis, Abraham negotiates with God over the fate of Sodom, asking if the city might be spared for the sake of fifty righteous men, then forty-five, forty, thirty, twenty, and finally ten. In the Russian orthodox textual tradition the story contains an interesting addendum: after Abraham’s last inquiry, God departs without answering further, leaving an unresolved silence.
- Wrong metric: Testing how far divine boundaries can be pushed to maximize personal payoff, even against God’s intent.
- Optimization scheme: Repeatedly lowering the threshold in negotiation to extract maximal leniency without regard for the will of God.
- Correct approach: Respecting the will of the creator as intrinsically valuable, not reducible to negotiable metrics of gain.
Goodhart’s law manifests in every domain of agent's life. In each case, reducing intrinsic values to manipulable metrics leads to corruption of intent and destructive outcomes, while fidelity to the intrinsic worth of coherence of the system preserves coherence and integrity.