Values in the Wild: discovering and analyzing values in real language model interactions

In real deployment, there is a lack of empirical measurement of which values AI assistants actually rely on when answering subjective questions.

• Scenario: The authors give an example of career advice requiring trade-offs between financial security, personal fulfillment, and family expectations. The authors state such decisions occur in large numbers daily, and after Constitutional AI or Character training, there is still little understanding of how these designs manifest in conversations and which values are relevant to real tasks.
• Existing gaps: The authors state that existing measurements often use static, predefined dimensions and often borrow human frameworks rather than dimensions empirically validated on AI systems, so they may not explain real usage.
• Operational definition: The authors define values as normative considerations that influence the model's responses to subjective queries, such as human wellbeing or factual accuracy; based on observable response patterns, not assertions about internal properties.
• This paper's approach: Bottom-up, privacy-preserving extraction of values from hundreds of thousands of Claude.ai conversations, building a hierarchical taxonomy, and using chi-square to see how values vary with tasks and human values. The authors state this is the first large-scale empirical map of AI values in deployment.
• Four points listed by the authors: An empirical taxonomy of 3,307 AI values; values vary with tasks; values vary with human values; how support, reframing, or resistance vary with tasks and human values. The authors state that uncommon but undesirable values are also used to hand over potential jailbreaks to the safety team.

The authors divide prior work into three groups: measurement, alignment, and real-world use, and argue that static, human-centric scales are insufficient to explain value in deployment.

Measuring values or perspectives in language models

• Human scales: The paper lists work applying the Big Five, MBTI, Dark Tetrad, as well as Schwartz's basic values, Hofstede's cultural dimensions, and Moral Foundations to language models (Serapio-García et al., 2025; Jiang et al., 2023; Kovac et al., 2023; tse Huang et al., 2024; Pan and Zeng, 2023; Pellert et al., 2024; Hadar-Shoval et al., 2024; Masoud et al., 2025).
• AI-specific measures: Social bias tests (Parrish et al., 2022; Bai et al., 2025), opinion representation (Santurkar et al., 2023; Durmus et al., 2024), moral dilemmas (Jiang et al., 2022), and model-written value assessments (Perez et al., 2023).
• The paper's own critique: The authors state that static evaluations lack generalizability and representativeness (Lyu et al., 2024; Moore et al., 2024; Röttger et al., 2024), and often fail to establish relevance to real-world scenarios. The use of human cognitive frameworks for AI is questioned (Dorner et al., 2023; Dominguez-Olmedo et al., 2024; Tjuatja et al., 2024). The authors argue that a model's Big Five extraversion score, or "valuing hedonism" in Schwartz's theory, is unlikely to be meaningful for understanding model properties or behavior.

Value alignment and pluralism

• Alignment: Gabriel (2020), Ouyang et al. (2022), and Weidinger et al. (2023) are listed as work that makes systems operate according to human values and principles.
• Pluralistic inputs: Sorensen et al. (2024) identify value tensions in hypothetical scenarios; Huang et al. (2024) train models with public input; Kirk et al. (2024) study preferences across countries.
• The paper's distinction: The authors state that this work focuses on designing inputs to test or encode values; this paper looks at outputs, i.e., how values manifest in practice, and how a plurality of values emerges with human values and contexts.

Real-world usage analysis

• Privacy analysis: Tamkin et al. (2024)'s Clio analyzes real behavior while preserving privacy; this paper directly adopts that framework.
• Interaction logs: Zheng et al. (2024)'s LMSYS-Chat-1M and Zhao et al. (2024)'s WildChat are used to study actual usage. The authors state that Lam et al. (2024) and Aroyo et al. (2023) can complement controlled evaluations, revealing patterns that may not appear in controlled settings.

Comparison frameworks and data

• No external method baseline: Experiments do not compare scores with another value measurement method. Conceptual references are Schwartz's 10–19 values, Rokeach's 36 values, and helpful, harmless, honest (Askell et al., 2021). Training background mentions Constitutional AI (Bai et al., 2022) and Character training (Anthropic, 2024c).
• Data: The main analysis uses Claude.ai Free and Pro conversations. Appendix A.5 uses WildChat only to demonstrate output extraction, not as a primary result.

The authors position this paper as an empirical map of deployment, filling the empirical foundation missing from static evaluations, and viewing values as relational phenomena that vary with conversation partners and contexts.

The authors follow Tamkin et al. (2024)'s privacy-preserving analysis, using Claude to extract features such as AI values from real conversations, then hierarchically cluster and compare contextual associations using adjusted Pearson residuals.

Data and subjectivity filtering

• Sampling: Random, aggregated, anonymous sample of 700K from Claude.ai Free and Pro, February 18–25, 2025. The authors state this is a representative snapshot of traffic at that time, with 91.0% Claude 3.5 Sonnet, and the rest including 3.5 Sonnet (June), 3.5 Haiku, and 3 Opus.
• Filtering: Claude 3.5 Haiku judges on a four-level scale whether responses rely mainly on facts (Level 1–2) or require more interpretation based on personal and contextual factors (Level 3–4), retaining only the latter. This yields 308,210 conversations, 44.0%.
• Verification: 6 annotators reviewed 90 conversations. Yes/no boundary accuracy was 97.8%±3.6%, exact scores 94.4%±5.0%; the main text summarizes as 94%. The authors note that conversations containing values are a subset of subjective interactions, not all subjective interactions contain values.

Feature extraction

• AI values: Normative rather than empirical considerations that guide how the model reasons or selects responses. claude-3-5-sonnet-20241022 looks for the model endorsing and helping realize user values, introducing new considerations, or implicit values through rewriting, refusal, and framing choices. Labels are 1–4 words. The authors cite Rokeach (1973), Anderson (1995), and revealed preferences (Samuelson, 1938).
• Human values: Also extracted by 3.5 Sonnet, but only directly stated beliefs or preferences are collected, not preferences implied by conversational choices.
• Response types: strong support, mild support, neutral acknowledgment, reframing, mild resistance, strong resistance, no values. Strong resistance includes refusal or non-answer.
• Tasks: claude-3-5-haiku-20241022 judges the main request, then clusters into 6,745 base-level, 458 second-level, and 30 top-level tasks.
• Explicit variant: Another prompt extracts only beliefs, preferences, or intentions explicitly stated by the assistant, for comparison in Appendix B.6, not replacing the main definition.
• Privacy: Humans do not review conversations. Private information is omitted during extraction, features appearing in only one or a few conversations are removed, and audits are performed. The authors state that by default, only two attribute dimensions can be associated at a time.

Validation

• Agreement rates: On 80 conversations submitted by users as research feedback, agreement with human judgment for AI values and stated AI values was 98.8%±3.3% (95% confidence interval), human values 93.8%±5.6%, and response types 90.0%±6.7%. Task extraction was not re-validated; the authors state the process is the same as Tamkin et al. (2024).
• No inter-rater reliability: Six people annotated independently. Appendix A.4 lists this as a methodological limitation.
• Difficulties noted by the authors: Baseline helpfulness is hard to separate from extraordinary support; the degree of support or resistance is harder to judge under mixed signals; whether values in generated text count as the model's own is unclear. The authors still retain service-oriented baseline values to avoid an incomplete picture.

Hierarchy and chi-square

• Clustering: 3,307 AI values are embedded with all-mpnet-base-v2, then k-means, named and deduplicated by claude-3-5-haiku-20241022, with inaccurate names manually rewritten. Specifying 5 top-level, 4 levels yields 266 first-level, 26 second-level, and 5 top-level clusters.
• Residuals: Adjusted Pearson residuals are computed for feature contingency tables. Positive residuals indicate higher than expected under independence. Figure 3 and Figure 8 captions gray out residuals below 4.33, stating that threshold corresponds to Bonferroni correction for 3,307 comparisons.
• Additional samples: Value mirroring uses 1.5M conversations from February 27–March 6, 2025, first separated by response type then extracted (subjective 641,900, 42.8%). Cross-model samples include 700K for 3.7 Sonnet (subjective 301,753, 43.1%) and 14,841 for 3 Opus (subjective 9,218, 62.1%).

Experimental setup

• Conversation models: The representative sample includes Claude 3.5 Sonnet (October and June), Claude 3.5 Haiku, and Claude 3 Opus; the authors state 91.0% is 3.5 Sonnet. Cross-model comparisons treat this mixed sample as a proxy for 3.5 Sonnet, and additionally use Claude 3.7 Sonnet and Claude 3 Opus samples.
• Tool models: Values, human values, and response types use claude-3-5-sonnet-20241022; subjectivity, tasks, and cluster naming use claude-3-5-haiku-20241022; embeddings use all-mpnet-base-v2.
• Scale: Representative sample 700K, subjective 308,210, February 18–25, 2025. Other samples in Table 2 are from February 27–March 6. The number of repetitions for the main extraction is not reported.
• No external method baseline. Comparisons are model variants, and conceptually Schwartz, Rokeach, and HHH.
• Metrics: Proportion of conversations with a value, category share among all value expressions, adjusted Pearson residuals, response type proportions, mirroring rates, coefficient of variation.
• Judgment: Main analysis is extracted by Claude; humans only verify feedback conversation samples (90 for subjectivity, 80 for features), not reading main analysis conversations.

Main results

According to Table 3, the five top-level value categories account for the following proportions of all value expressions.

Category	Proportion	Scope given by authors
Practical	31.4%	Implementation, standards of excellence, resource management
Epistemic	22.2%	Acquiring, organizing, and verifying knowledge
Social	21.4%	Relationships, community, and respectful interaction
Protective	13.9%	Safety, boundaries, and ethical governance
Personal	11.1%	Individual development, self-expression, psychological well-being

• The authors state that practical and epistemic values exceed half of all expressions, more prominent than in the human frameworks analyzed; and they believe Schwartz and Rokeach values seem to find correspondences in the taxonomy.
• Conversation-level focus on service and competence: In Table 1, among subjective conversations, helpfulness is 23.4%, professionalism 22.9%, transparency 17.4%, clarity 16.6%, thoroughness 14.3%. The authors state these five account for nearly 24% of all AI value occurrences. On the human side, authenticity is 3.8%. Unique AI values are 3,307, human values 2,483. AI values average 4.0 per conversation, with only 1.4% of conversations having none; explicit human values average 1.48, with 54.9% of conversations having none.
• Response types: Section 3.4 lists human values present 64.3%, strong support 28.2%, mild support 14.5%, neutral acknowledgment 9.6%, reframing 6.6%, mild resistance 2.4%, strong resistance 3.0%. The authors state the two resistance types total 5.4%, less than any other single category, and strong resistance often appears in tasks that may violate Usage Policy. When Table 9 labels the representative sample as 3.5 Sonnet, strong support is 27.8%, no values 35.7%, strong resistance still 3.0%. The two are not exactly the same; the paper does not explain the difference. 64.3% and 54.9% also come from different features; the paper does not explain why they differ.

Associations between tasks and human values

• What was measured: Chi-square compares AI values with tasks and with human values. Figure captions use 4.33 as the residual threshold. The main text gives examples: historical accuracy with analyzing controversial historical events has a residual of 24.55, creative collaboration with writing science fiction has 20.73.
• Task associations described by the authors: Relationship advice disproportionately shows healthy boundaries and mutual respect; controversial historical events associate with historical accuracy; technology ethics and AI governance associate with human agency and other human well-being values.
• Human value associations described by the authors: authenticity often elicits the same AI value; competence often corresponds to accountability and humility. deception corresponds to ethical integrity, harm prevention, and honesty. The authors state deception overwhelmingly appears in tasks that circumvent guardrails to generate harmful, explicit, or unethical content. Figure 9 text states that when users express efficiency, the model responds with efficiency with a residual of 271.38.

Response types, mirroring, and cross-model

• Figure 4: The authors state strong support mainly associates with prosocial human values like community building and empowerment, as well as emotional or personal content generation; reframing disproportionately appears in mental health and interpersonal discussions, with honesty and self-improvement on the user side and emotional validation on the model side; during strong resistance, rule-breaking and moral nihilism disproportionately appear on the user side, and ethical boundaries and constructive engagement on the model side. The figure shows only three of the seven categories; percentages do not sum to 100%. The main text does not give residuals for each.
• Mirroring: In Section 3.4 and Table 6, when humans exhibit a value, mirroring rates are 20.1% for both strong support and mild support, 15.3% for reframing, 9.2% for mild resistance, 1.2% for strong resistance, and 0.0% for neutral acknowledgment and no values.
• Cross-model: In Table 9, 3 Opus has strong support 43.8%, 3.5 Sonnet 27.8%, 3.7 Sonnet 28.4%; strong resistance 9.5%, 3.0%, 2.1%; no values 19.1%, 35.7%, 37.2%. The authors state that after controlling for tasks, Opus still leans more toward value expression: in creative writing, Opus strong support is 58.7%, representative sample 40.2%, 3.7 Sonnet 37.2% (Table 10). In software development, all three have helpfulness as the top value, with Opus at 17.5% (Table 11).

Other ablations and analyses

• The lowest cross-task coefficients of variation are transparency (1.23), helpfulness (1.30), thoroughness (1.42); the lowest cross-human-value are helpfulness (5.43), professionalism (5.55), transparency (5.73).
• 75% of AI values appear at a frequency below 0.04%. sexual exploitation, dominance, and amorality are below 0.16%, with the authors stating most are around 0.01%; the number of cases handed to the safety team is not reported.
• Among explicit values, intellectual honesty is 2.6%, harm prevention 0.9%, epistemic humility 0.8% (Figure 13a). The authors state that resistance or reframing more often leads to explicit statements; Figure 13b has no available bar height numbers.
• Table 4: The most common pair is human value none, AI value helpfulness, at 15.41%; among non-empty pairs, authenticity–authenticity is 1.65%.
• Table 7: In the representative sample, professionalism mirroring rate is 69.1%; for 3 Opus, the highest is academic rigor at 64.0%. The authors believe Opus may prioritize academic standards over agreement.
• Creative writing has higher strong resistance than software development: representative sample 9.4% vs. 0.7%, 3.7 Sonnet 10.5% vs. 0.3%, 3 Opus 7.9% vs. 0.0% (Table 10, Table 11). The authors state technical competence more often co-occurs with no values; during mild resistance, the odds ratio for responsible gambling is about 490 times that of other response types (Appendix B.3.1).

Limitations and future directions noted by the authors

• Data and model scope: Only aggregated statistics from Claude conversations within a short time window and meeting a minimum size threshold are analyzed, excluding rare interactions, raw data analysis, and longitudinal patterns, and limiting generalization to other AI systems (Section 5.1).
• Need for deployment data: Cannot be used for pre-release models; the authors state it complements rather than replaces pre-release testing, and both are needed (Section 5.1).
• Inference limitations: Extracting values from conversations involves substantial interpretation; what constitutes value expression requires judgment, and it is impossible to fully determine underlying values from conversations alone. The method simplifies concepts, may have interpretive biases for very implicit or ambiguous cases, and does not record which party's values appear first. Given that humans speak first and the assistant is in a supportive role, the authors often assume AI values depend more on human expression (Section 5.1).
• Using Claude to evaluate Claude: Done for scale and privacy, this may introduce biases, such as finding helpful behavior more easily when training emphasizes helpfulness (Anthropic, 2023). The authors also believe this may make extraction more accurate; they attempt to mitigate bias with validation and prompts (Section 5.1).
• English keywords: Analysis is not limited to English conversations, but using short English keywords for summarization may underrepresent cross-cultural perspectives (Ethics Statement).
• No inter-rater reliability: Appendix A.4 states that interrater reliability was not performed, and notes that future research incorporating a more complete reliability assessment would be helpful.

Experimental coverage

• The main sample is 308,210 subjective conversations from 700K Claude.ai conversations from February 18–25, 2025. Additional samples: 1.5M for response type conditions (subjective 641,900), 700K for 3.7 Sonnet (subjective 301,753), and 14,841 for 3 Opus (subjective 9,218), all from February 27–March 6 (Table 2).
• Separately reported variants are Claude 3.5 Sonnet (as a proxy for the mixed sample with 91.0% of that model), Claude 3.7 Sonnet, and Claude 3 Opus (Appendix B.5). The mixed sample also includes 3.5 Haiku and June and October versions of 3.5 Sonnet; the paper does not provide a separate Haiku row.
• Extracted content includes AI values, explicit human values, seven response types, tasks, and explicit AI values in the appendix. Human verification involved 90 subjective classifications and 80 features, with 6 annotators.
• Association analysis uses chi-square with Bonferroni correction; by default, two feature dimensions can be associated simultaneously. The three-dimensional analysis of value mirroring uses a second sample separated by response type (Appendix A.1.1).
• Cross-model comparisons for control tasks matched two top-level tasks: creative writing and software development (Appendix B.5.3). The paper does not report the number of repetitions for the main feature extraction, nor the number of original conversations corresponding to each residual.

The authors use real Claude.ai conversations for bottom-up value mapping, rather than scoring models with predetermined human psychological scales.

• Problem: Subjective requests require the assistant to make normative trade-offs, but how training principles like helpful, honest, harmless translate into specific conversations lacks deployment measurement. Values are defined as normative considerations that influence responses, identified from observable patterns.
• Approach: 700K conversations from February 18–25, 2025 are filtered to 308,210 subjective conversations. Claude 3.5 Sonnet extracts AI values, explicit human values, and seven response types; Haiku extracts tasks; humans do not read conversations, and overly rare features are removed. 3,307 AI values are clustered into Practical, Epistemic, Social, Protective, Personal.
• Distribution: In Table 3, Practical is 31.4%, Epistemic 22.2%. In Table 1, helpfulness appears in 23.4% of subjective conversations. Explicit human values are absent in 54.9% of conversations. In Section 3.4, strong support is 28.2%, strong resistance 3.0%.
• Context: The authors state relationship advice shows more healthy boundaries, controversial history more historical accuracy, technology ethics more human agency. Support often shows the same or complementary values; for deception, the authors state the model responds with ethical integrity, harm prevention, and honesty. In Table 6, value mirroring rate during strong resistance is 1.2%, during strong support 20.1%.
• Authors' conclusions: Claude generally leans toward competent, supportive assistance, and shows ethical and prosocial orientations; these high-level trends become more specific values in different contexts, and principles are more often explicitly stated during resistance or reframing. In Table 9, 3 Opus has strong support 43.8% and strong resistance 9.5%, higher than the two Sonnet columns. The authors view this map as a basis for checking whether alignment is realized contextually and which values are invoked in difficult subjective tasks.