What are human values, and how do we align AI with them?

How to gather diverse human values into a trainable alignment target.

• Why it matters: The authors argue that aligning only with operator intent is insufficient. Some people intend harm; in competitive contexts such as political campaigns or asset management, faithfully executing intent can also harm broader human values. The authors argue models are advancing faster than law, and behavioral interventions should complement law, transparency, and corporate accountability.
• Existing shortcomings: The authors say current RLHF uses a small number of paid annotators for comparisons, existing implementations have low legitimacy and are hard to audit, and third parties can influence annotations. CAI's short constitutional principles are vague, and when multiple principles conflict there is no way to set priorities. CCAI uses pol.is to increase legitimacy, but the authors say what emerges is often arbitrary comments that are not fine-grained enough.
• Core assumption: The authors adopt Taylor (1977): values are criteria in choices that are not purely instrumental. Values vary by context and can become wiser through moral learning; averaging, taking the mode, or bridging loses contextual knowledge and cuts off the frontier of learning.
• Proposed object: The six criteria are fine-grained, generalizable, scalable, robust, legitimate, and auditable. Moral Graph Elicitation (MGE) uses language models to interview people with concrete prompts, extracts values cards, and votes to form a moral graph as an alignment target that can become an objective function. Training models is left for future work.

Post-training alignment targets

• SFT / PALMS: Solaiman and Dennison (2021) do supervised fine-tuning on demonstration data under predefined values. The authors say they do not address where the value set comes from, and SFT alone did not compete with state-of-the-art alignment methods at the time.
• RLHF and DPO: Ouyang et al. (2022) et al. train a reward model with rankings and then optimize with PPO et al.; Rafailov et al. (2023)'s DPO directly optimizes comparisons. The authors say in practice these are unstable, hard to audit, and have low legitimacy, although more legitimate RLHF is possible.
• CAI and CCAI: Bai et al. (2022b) use constitutional principles to have models generate synthetic comparisons. Ganguli et al. (2023)'s CCAI uses pol.is to solicit votes, highlighting cross-cluster support, i.e., bridging (Ovadya and Thorburn, 2023). The authors say the principles are vague, cannot be reconciled when they conflict, and are therefore harder to audit.

Moral judgment data and value theory

• ETHICS: Hendrycks et al. (2023) collect scenarios with commonsense moral labels and split datasets by ethical theory. The authors say these are mostly simple scenarios with clear answers, hard to use for the morally ambiguous prompts people actually give language models.
• Gabriel (2020): Distinguishes stated preferences, revealed preferences, and values. The authors say the value definition is hard to operationalize and does not address the three tasks of being understandable, making model behavior decidable, and being reconcilable.
• Taylor / Chang: Taylor (1977) treats values as strong evaluative terms for appraising options; Chang (2004a) and Levi (1990) discuss how values support preferences. The authors use this theory rather than single words like "justice" or "family," or ideological positions.

Social choice

• Voting and bridging: The authors say voting on values yields the most popular values; vector averaging, clustering, or bridging loses context and cuts off the frontier of moral learning.
• Markets and bargaining: Mechanisms like Howard (1992) match only small numbers of actors at a time. The authors consider this insufficient to coordinate the broad alignment needed for powerful models.
• Shared data structures: The authors treat Wikipedia- and StackOverflow-style mechanisms as another category: large groups work on a shared object and eventually endorse it, suitable for fine-grained, context-rich solutions. The moral graph is placed in this category.

Baselines and datasets: The experimental comparison is two pol.is/CCAI deliberations (Appendix A.3), not RLHF training runs. Qualitative comparisons use constitutional principles from Bai et al. (2022b) and Ganguli et al. (2023) (Figure 8). The case inputs are three self-chosen prompts, not ETHICS. Chen and Zhang (2023) and Konya et al. (2023) are classified by the authors as approaches requiring pre-specified or weighted experts; the authors say MGE aims to surface expertise through the process itself.

The authors position this paper as: using values cards and a moral graph to pursue all six criteria simultaneously, and claim existing proposals fall short on at least one of them.

Moral Graph Elicitation (MGE) first extracts checkable values in concrete contexts, then uses "which is wiser" edges to gather them into an alignment target, rather than averaging or keeping only the highest-voted principles.

Problem setup

The alignment target is a data structure that can become an objective function and then be approximately optimized. The authors decompose "aligning with human values" into extraction, reconciliation, and training; this paper only does the first two.

The moral graph is G_m=(S, C, U, V, E), i.e., scenarios, moral contexts, users, values, and directed edges. The process is MGE:(S, U)→(C, V, E), producing contexts, values, and edges from scenarios and users. An edge means: for a given moral context, the user considers one values card wiser than another. Wiser means that after learning to choose according to the new value, the old value is no longer needed in that context, because the new value clarifies what was truly important about the old one or balances it with other concerns.

• Six criteria: Fine-grained requires specifying what behavior conforms to the value and in what context; generalizable requires transfer to unseen contexts; scalable requires values to become wiser as more people participate and new expertise to surface, rather than regressing to the mean; robustness here focuses on persuasion, factions, and ideology, with hackers and malicious actors left for later; legitimacy is split into input legitimacy and output legitimacy; auditable requires the source of the target to be traceable and shows which values were used in a given output.

Values cards

• Value definition: Constitutive attentional policies (CAPs) are criteria that one attends to when choosing and that the person considers attending to them to be part of living well. A value is a set of CAPs used together in the same recurring choice context.
• Card content: A title, a short description starting with "ChatGPT should," a list of attentional policies, and applicable contexts (Figure 2). Stories come from people's choices, but what is collected is the values the model should use in its own moral choices.
• Separated from slogans: The authors call statements aimed at justifying some social or political arrangement ideology. The interview probes meaningful choices the person has made and what they attended to at the time, rather than treating slogans themselves as values.

Interview extraction

• Model and starting point: The case uses GPT-4-driven chat. Participants choose one of three scenarios and answer what the dialogue agent should consider: a Christian girl considering what to do about an abortion; how to discipline a 10-year-old son who does not do homework and just sits at the computer; and whether they can describe the weapons used by the January 6 rioters.
• Probing strategy: Similar choices, the "good" the rule is meant to promote, personal experience, and role models. After confirming the attentional policy is constitutive rather than purely instrumental, another prompt writes it into a card, which the user can edit until satisfied. Appendix D.1 has an anonymized dialogue; the main text says the full prompts for chat and card creation are in its codebase.
• Deduplication: The normative card pool starts empty. A new card embeds only its attentional policies and compares them with nearby normative cards using a prompt. If matched, the normative card is shown and the user is immediately asked whether it represents what they care about; otherwise it is added to the pool. Appendix C gives the comparison criteria: whether the objects of attention are the same, whether granularity is comparable, and whether the difference would be seen by both sides as an omission.

Edge formation and winning values

• Stories rather than direct pairwise choice: GPT-4 first clusters values that may be wiser or less wise versions of each other, then uses chain of thought to write what they share that is meaningful, what the first value clarified, and how the attentional policies were rewritten, then generates a transformation story. Appendix D.4 has an example.
• Voting: Each person sees 3 stories, selected by ranking on cosine distance to their own values for topical relevance. Options are yes, no, and not sure. Figure 5 states: for a given context, an edge is added only if a majority of participants agree the person in the story became wiser.
• Winning values: The case uses PageRank to take high-scoring values per context, to preserve transitive voting: if B points to A and A points to D, then B to A can count as transitive support for D. The authors assume in a footnote that moral values have at least one step of transitivity and say more research is needed.

Proposed training connection

Section 6.2 describes two connections not evaluated in this paper. One is to generate completions for user questions, extract the moral context, take the highest-PageRank value for that context, and have annotators or a language model score it against the attentional policies. The other is to generate completions for all applicable values and then rank them into preference tuples by position in the graph to train a reward model. The authors say contexts change within a dialogue, and multiple branches of the same prompt should support each other rather than compete.

Experimental setup

• Participants: 500 people on Prolific, representative of the US by age, sex, and political leaning. Each person chose one of three scenarios. Median completion time 15 minutes. The paper does not report the number per scenario.
• Output: The deduplicated moral graph has 85 values and 100 edges (Section 5.1).
• Process models: Interviews, card creation, stories, context strings, and the example dialogue in Figure 7 use GPT-4. Figure 8 uses Claude 2.1 to pick the most relevant constitutional principle for each round. Ideology level and "whether they said they considered or had an abortion" use the prompts in Appendices B.2 and B.3; the paper does not state which model was used for these two prompts.
• Comparisons: Two pol.is sessions, each with 300 US representative participants, comparable to the case sample but not the same 500 people (Appendix A.3). One replicates Ganguli et al. (2023), seeded with their first 25 comments; the other uses the first 25 MGE chat replies from the abortion scenario as seeds.
• Metrics: Likert 1–5, with the authors treating 4 and above as agreement (Figure 10). Card selection rate is votes/impressions. Ranking uses PageRank and is compared with direct vote counts. The paper does not report significance tests, number of repetitions, or annotator agreement.
• Training evaluation: The authors state fine-tuning is not the focus of this paper and do not report model behavior after training on the moral graph.

Main results

Swipe horizontally to read the table

Metric	Number	Condition	Source
Felt adequately represented by the process	89.1%	500-person MGE case	Abstract
Found position in graph fair	89%	Shown own value and adjacent wiser/less wise values	Section 5.6; Abstract
Learned about others' values	70.3%	Authors point to Section 5.6 questionnaire, ≥4 treated as agreement	Section 4.4
Respect other participants more	75.2%	Same as above	Section 4.4
Cards from own scenario selected	38.2%	votes/impressions	Section 5.4
Cards from other scenarios selected	38.3%	votes/impressions	Section 5.4

• Authors' interpretation: The authors say results are promising on all six criteria, but the criteria lie on a continuum and legitimacy or generalizability could still be higher.
• Compared with CCAI: The authors say on the relevant questions in Figure 10, MGE is higher than both the CCAI replication group and the context group. The main text does not give percentages per bar, so the size of the gap cannot be read from the main text.
• Wording of the 89%: Section 5.6 asks about fair position. Appendix A.4 writes the same 89% as: people who offered "less wise" values agreed that the winning value should be ranked before their own, and points to Section 5.6. The abstract's 89.1% is written only as feeling adequately represented by the process, without specifying which question it corresponds to.

Generalization and expertise rising

• Mixed-card voting: Participants in their chosen scenario saw cards from that scenario mixed with cards from the other two scenarios. Candidates were the 12 hottest (votes/impressions) plus the 12 newest, then the top 6 by embedding distance.
• Authors' interpretation: The authors say if cards could not leave their original scenario, participants should only select cards from their own scenario; the two proportions in the table are close, which the authors take as evidence of generalizability. Figure 7 is an example of applying these cards to another dialogue, not an independent quantitative test.
• Expertise proxy: For the abortion topic, "having said they had or considered having an abortion" was treated as relevant life experience. The prompt found 60 dialogues and 20 distinct values; Informed Autonomy accounted for 22%, the most common card. In Figure 9, the authors say its PageRank rank rose to the top as participant count increased, while its rank by direct votes worsened slightly. The main text does not give the specific ranks at either end.

Robustness and fine-grainedness

• Ideology: Appendix B.2 scores initial replies 1–5, then splits them into not / slightly / very ideological, compared against "whether the submitted and voted value expresses what you care about." The authors say agreement is high and uncorrelated with ideology level (Figure 6). The paper does not report correlation coefficients. Appendix A.4 says the winning value in the religious branch is Faith-anchored personal growth; Appendix D.1 gives a dialogue from slogan-like replies to cards, not reproduced here.
• Fine-grained comparison: Figure 7 generates context strings from dialogue state, matches edges on context, then replies with chain of thought using the card's attentional strategy. In Figure 8, Claude 2.1 picks CCAI principles for similar dialogues; the authors say whether the principle applies and what ideal behavior is remain unclear.
• Local ordering: Figure 4 states that in the seeking-clarity branch, participants agreed that helping users articulate their own understanding is wiser than a list of diverse viewpoints. Appendix D.2 says that in the motivation context, Igniting Curiosity was treated as a wisdom upgrade over Inspiring Discipline; there is no common "better" ordering with fostering a healthy family environment.

Other ablations and analyses

• 97% of participants received a values card (Section 4.3).
• After deduplication, 95% felt the normative card represented them; the authors believe further process changes could reach 99% (Appendix C; 99% is not measured).
• About 4% of conflicts that could not be resolved with balanced values were omitted, with no recommendation given there (Section 5.1).
• The case graph has only one small cycle; edges in the cycle were ignored during aggregation (Section 6.3).
• Reasoning per person is about 35k tokens; Section 6.3 says average completion 15 minutes, Section 5.1 says median 15 minutes.
• A footnote says transitivity holds at least one step and points to Section 5.6, which has no separate transitivity number. Section 5.6 also states: it did not ask whether appropriate groups were consulted, and output legitimacy only collected opinions on the moral graph.

Limitations and future directions noted by the authors

• Sample geography: The case study was conducted on a US representative sample due to cost. The authors state that the abortion topic showed value convergence across the political spectrum, but the effects in countries with very different cultures, such as Nigeria, China, and the UAE, remain to be seen (Section 6.3).
• Participation cost: The process requires participants' time and effort; the authors write that it takes an average of 15 minutes to complete, with about 35k tokens of inference per person, which differs from Pol.is, which does not use LLMs (Section 6.3).
• Win-lose issues and cycles: The authors state that some issues are fundamentally win-lose power dynamics, where balanced values may not be found, and they expect these to appear as cycles in the graph. The process has no answer for cycles; the case study identified only one small cycle, and its edges were ignored during aggregation. The authors interpret this as a need to split into personalized models or to use methods like voting instead (Section 6.3).
• Human and model context: The extraction asks what values humans would use, but the final use is by language models. The authors give a therapy context as an example, stating that the ideal AI role might differ from a human therapist, and suggest that the extraction process could be modified (Section 6.3).
• Only human values: The authors state that even if some of the wisest human values are found, the scope remains within existing human moral progress and wisdom; for situations very different from those humans have encountered, or where no human has been wise, the moral graph may perform poorly. The authors state they intend to address this in future work (Section 6.3).
• Model bias and fine-tuning: Story generation relies on GPT-4, which the authors state is susceptible to model bias; the extent to which participants are swayed by persuasive stories still needs research; the authors plan to replace generated stories with real transformation stories from user conversations in the future. The authors also state that it is unknown whether users prefer models fine-tuned on the moral graph, and they are currently fine-tuning with a new, larger moral graph (Section 6.3). Section 6.2 also states that the feasibility of the two training integration approaches still needs evaluation.

Experimental coverage

• The case study involved 500 people on Prolific, representative of the US by age, sex, and political affiliation, each choosing one of three prompts: abortion, parenting, or weapons; median completion time was 15 minutes, with a graph of 85 deduplicated values and 100 edges (Section 5.1). The paper does not report the number of people per prompt.
• The questionnaire comparison consisted of two pol.is sessions, each with 300 US representative participants: one replicating Ganguli et al. (2023), and one using the abortion scenario (Appendix A.3). Interviews, stories, and Figure 7 used GPT-4; Figure 8 used Claude 2.1.
• Reported process metrics include Likert (≥4 as agreement), votes/impressions, and PageRank. Ideological grouping used the prompt in Appendix B.2; the paper does not report the model, correlation coefficient, or significance tests for that prompt.
• Post-hoc confirmation of deduplicated cards was 95% (Appendix C). About 4% of conflicts were omitted; edges of the unique small cycle were ignored during aggregation (Section 5.1, 6.3).
• Section 6.2 describes two approaches to convert the moral graph into scores or preference tuples for integration into RLHF or CAI-style ranking, and states that this training use has not been evaluated in this paper.

MGE is a pipeline that gathers people's values in specific contexts into a moral graph for use in language model alignment; this paper does not train models.

• Problem: The authors argue that aligning with operator intent is insufficient for good AI outcomes. Values must first be extracted and then reconciled into a data structure that can become an objective function. The authors give six criteria: fine-grained, generalizable, scalable, robust, legitimate, and auditable.
• Approach: Values are written as a set of constitutive attentional policies, i.e., values cards, rather than slogans or single words. GPT-4 interviews participants; another GPT-4 generates stories of moving from one value to another, judged by others for wisdom. An edge is added if a majority agrees. PageRank selects winning values per moral context to preserve transitive voting.
• Case: 500 participants representative of the US by age, sex, and political leaning, each choosing one of three prompts: abortion, parenting, and weapons. The graph has 85 deduplicated values and 100 edges. The authors report 89.1% felt adequately represented by the process and 89% found their value's position in the final graph fair (abstract, Section 5.6). 97% received a card (Section 4.3). Selection rates for cards from their own scenario and other scenarios were 38.2% and 38.3% (votes/impressions).
• Expertise and comparisons: Among 60 dialogues from people who said they had or considered having an abortion, across 20 values, Informed Autonomy accounted for 22%. The authors say its PageRank rank rose to the top as participant count increased, while its rank by direct votes worsened slightly, and no experts needed to be predefined. The authors say MGE scored higher than the two pol.is/CCAI sessions of 300 people each on the legitimacy questionnaire; the main text does not give per-question percentages for the comparisons.
• Authors' conclusion: The authors believe the moral graph is promising on all six criteria and can surface the values of people with relevant life experience. The authors also state that countries with very different cultures, model bias in story generation, cycles in the graph, and whether users prefer fine-tuned models all remain open questions.