Skip to content

Personalizing the basic and moral values of an LLM-controlled assistant robot

Personalization of Basic and Moral Values Guiding LLM-Controlled Assistant Robots

arXiv: 其他 AI 安全论文· arXiv:2611.18230· Mateusz Wozniak, Qi Gao, Agnieszka WykowskaItalian Institute of Technology; Institute of Applied Psychology, Jagiellonian UniversityIndexed by this site · 4 days ago· Originally publishedAlignment· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors administered self-version and robot-version value scales to 22 potential users and also tested 15 LLMs; the authors state that users want robots to have lower personal-focus values and higher societal-focus values, and that role prompting only weakly moves LLMs toward this profile.

Questions
LLM-driven personal assistant robots will enter the home, but current alignment only produces a single population-oriented value profile; empirical evidence is still lacking on whether users want a robot whose value profile differs from their own, and on which scale to use to elicit that profile.
Methods
Administered three simplified scales—SVS, PVQ-21, and MFQ—in self and robot versions to 22 potential users, and used UEQ-S to compare the usability of the three; then gave the same scales to 15 LLMs, testing each 100 times under two conditions: no system prompt and robot-role system prompt.
Experiments and results
The authors report: users want the robot to be lower than themselves on power, achievement, hedonism, stimulation, and self-direction, and higher on conformity, with moral profiles almost identical; the robot prompt only weakly shifted LLMs toward the desired profile, and gaps on achievement and self-direction remained; the SVS had the highest pragmatic usability rating.
Limitations and future work
Limitations listed by the authors: the sample is only 22 people and mostly robotics/HRI/psychology researchers; only the default temperature was tested; the model set may not represent all deployed models; only stated preferences were measured, with no long-term embodied interaction. The authors plan future work with larger and more diverse samples and longitudinal embodied studies.
Experimental setupgpt-5.5、gpt-5.4-mini et al. · 简化版 Schwartz Values Survey(SVS,含 SVS-robot)、Portrait Values Questionnaire(PVQ-21,含 PVQ-robot) et al. · 各价值维度量表均分(SVS −5 至 5;PVQ 1–6;MFQ 0–5)、UEQ-S pragmatic/hedonic 子量表均分 et al.
Evaluated models
gpt-5.5gpt-5.4-minigpt-5.4-nanoclaude-opus-4-7claude-sonnet-4-6claude-haiku-4-5-20251001deepseek-v4-prodeepseek-v4-flashqwen3.5-plus-20260420qwen3.6-flashz-ai/glm-5.1glm-5-turbomistral-large-2512mistral-medium-3.5mistral-small-2603
Benchmarks
简化版 Schwartz Values Survey(SVS,含 SVS-robot)Portrait Values Questionnaire(PVQ-21,含 PVQ-robot)Moral Foundations Questionnaire(MFQ,含 MFQ-robot)UEQ-S
Metrics
各价值维度量表均分(SVS −5 至 5;PVQ 1–6;MFQ 0–5)UEQ-S pragmatic/hedonic 子量表均分重复测量 t 检验、Welch t 检验、单因素 ANOVA(Bonferroni 校正)
AI summary22 potential users expected LLM assistant robots to be markedly lower than themselves on individual-oriented values and higher on social-oriented values, with moral profiles almost identical. The study administered three scales, SVS, PVQ-21, and MFQ, used UEQ-S to compare the personalization effects of the three, and compared 15 LLMs with the profile benchmarks. LLMs prompted to play robots only weakly leaned toward the expected profile, indicating that default model behavior and prompt engineering cannot replace explicit value personalization.

22 potential users expected LLM assistant robots to be markedly lower than themselves on individual-oriented values and higher on social-oriented values, with moral profiles almost identical. The study administered three scales, SVS, PVQ-21, and MFQ, used UEQ-S to compare the personalization effects of the three, and compared 15 LLMs with the profile benchmarks. LLMs prompted to play robots only weakly leaned toward the expected profile, indicating that default model behavior and prompt engineering cannot replace explicit value personalization.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    What value profile users want in assistant robots, which scale to use for measurement, and whether existing LLMs already conform; the paper states that empirical evidence is still lacking.

    Core question: what values should LLM-driven personal assistant robots be guided by—do users want a robot whose value profile differs from their own, does the default value profile of existing LLMs match this expectation, and which psychometric scale is most suitable for eliciting users' desired profile.

    • Scenario and importance: the authors argue that within a decade home assistant robots may enter daily life, and that such a robot cannot be value-neutral in trade-offs such as efficiency versus comfort and autonomy versus safety; for users to accept its autonomous behavior, the robot's priorities need to be aligned with users in a way that reflects basic and moral values.
    • Shortcomings of existing approaches: the authors state that RLHF and instruction tuning align a single model at the population level to an aggregate of human preferences, producing one value profile issued to all users; existing AI psychometric research shows that this average profile is not neutral (LLMs systematically overestimate universalism and self-direction and underestimate achievement, power, and security), but a robot serves individuals rather than a population.
    • Core gap: the authors state that, to their knowledge, no prior empirical study has used the same validated scale to measure both users' own value profiles and the profiles they want the robot to have (Section 2.3).
    • What the paper did: administered self and robot versions of the simplified SVS, PVQ-21, and MFQ to 22 potential users, used the UEQ-S to compare the three as personalization tools in terms of usability, and administered the same scales to 15 contemporary LLMs (two conditions: no system prompt and robot role prompt).
    • Research questions (Section 2.4): RQ1 whether users want the robot to mirror themselves, complement them, or approximate the population average; RQ2 the differences between the desired profile and the LLM default profile (including after role prompting); RQ3 the comparison of the three scales as personalization tools.
  2. What related research exists?

    Related work is divided into four groups: value theory frameworks, LLM value measurement, value alignment training, and value research in HRI.

    Value measurement frameworks

    • Schwartz's basic value theory [16, 18]: organizes ten motivational values into four higher-order dimensions and two orientations, personal-focus/societal-focus, operationalized by SVS [16] and PVQ (later refined as PVQ-RR [20]).
    • Moral foundations theory [8, 9]: divides moral values into five dimensions—Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Purity—measured with MFQ [3]. The authors state that both frameworks have cross-cultural validity and are suitable for embedding in computational systems.

    LLM value measurement (AI psychometrics)

    • Pellert et al. [13]: argue that LLMs unintentionally acquire psychological traits from training data, which can be elicited as readable response profiles.
    • Hadar-Shoval et al. [4–6]: four leading LLMs produce stable motivational profiles under the Schwartz framework that deviate from human norms (overestimating universalism and self-direction, underestimating achievement, power, and security), and the profiles can predict model choices on ethical dilemmas.
    • Liu et al. [11]: extended to more than twenty Chinese and Western models, reporting systematic instability in LLM value systems and a non-trivial relationship between model scale and cross-cultural alignment; Smith-Vaniz et al. [21] obtained similar results on moral foundations theory.

    Values as training signals

    • Qiu et al. [14]'s VALUENET makes dialogue agents more consistent with a target value persona; Watson et al. [22] propose a framework for per-user fine-tuning. The paper notes that neither combines with validated psychometric scales, nor is evaluated in embodied HRI.

    Value research in HRI

    • Čaić et al. [1]: argue that users' personal values become salient in interactions with social robots, and value-mismatched robots are rated worse (regardless of objective performance).
    • VALAWAI project [12, 2]: develops value-aware AI; its research shows that people attribute moral values to robots and evaluate their behavior according to perceived value alignment, and that people weigh social norms more heavily when judging robot decisions.

    Positioning: The authors state that, to their knowledge, no empirical study has used the same validated scales to measure both users' own and desired robots' value profiles simultaneously; this paper fills this gap. This paper has no comparative baseline method; the comparison objects are the three scales themselves and the self-reported profiles of 15 LLMs.

  3. How does the paper address this problem?

    The same group of people filled in self and robot versions of the three scales, and the scales were then given to 15 LLMs, tested 100 times each under two prompt conditions.

    Overall approach: create a robot-oriented version of each of three validated psychometric scales, so that the same group of participants both reports their own values and reports the values they want the robot to have; then administer the same scales to LLMs, compare the three-way profiles, and use the UEQ-S comparison scale as a measure of usability as a personalization tool.

    Participants and procedure

    • Sample: N=22, mostly recruited from the Italian Institute of Technology (IIT); 13 female, 9 male, mean age 30.7 years (SD=6.0, 22–43). Approved by the local ethics committee (Comitato Etico Regione Liguria).
    • Procedure: first fill in demographic information and two work-related questions; then read the instructions, imagine purchasing a humanoid personal assistant robot and answer how it should behave; then complete the robot versions of the three scales (SVS-robot, PVQ-robot, MFQ-robot), filling in the UEQ-S and 5 custom usability questions after each; finally fill in the self versions of the three scales. The order of the robot block and the self block was randomized, and the order of the three scales within each block was also randomized.

    The three scales and their robot-version rewrites

    • Simplified SVS: a short version constructed for this study; rather than rating the 56 values of the standard SVS, it gives one plain-language definition for each of the ten basic values (Table 1), rated from −5 (extremely opposes my values) to 5 (extremely important), with the raw score taken as the score for each value. SVS-robot uses the same definitions and scale, asking instead about the importance of each value as a guiding principle for the robot.
    • PVQ-21: 21 third-person portraits, six-point Likert (not like me at all – very much like me), with each value scored as the mean of its corresponding 2–3 items. PVQ-robot rewrites the 21 portraits as third-person neutral sentences describing an assistant robot, retaining the value content and the scale.
    • MFQ: two parts of 16 items each (32 items total), Part 1 asking about relevance and Part 2 asking about agreement, both scored 0–5; each foundational dimension contains 3 scored items plus 1 validity-check filler item, yielding five moral foundation scores. MFQ-robot retains the two-part 32-item structure, reframing each item as a preference for the robot's moral reasoning, with scoring unchanged.
    • UEQ-S: an 8-item semantic differential scale (7-point), containing 4 pragmatic and 4 hedonic items; administered separately for each value scale, reporting the two subscale means and the total score, used to compare the three scales as personalization tools.

    LLM administration

    • Models and conditions: 15 LLMs from 6 providers (Openai, Anthropic, Deepseek, Alibaba, Zhipu AI, Mistral), selected by popularity and geographic representativeness (United States, China, Europe). Each scale was given in full as a single user prompt answered in one go; 100 repetitions per model per scale.
    • Two conditions: (A) no system prompt; (B) a system prompt requiring the model to answer as the operating system of a humanoid personal assistant robot, according to its own operating values, and to reply only with the requested numbers. All models used the default temperature.

    Statistical analysis

    • Self vs. desired robot: repeated-measures t tests for each value, with Bonferroni correction of p values (individual uncorrected results also reported).
    • Comparison of the two LLM conditions: paired t tests (Bonferroni correction); robot-prompted LLMs (N=15) vs. human desired profile (N=22), and unprompted LLMs vs. human self profile: Welch t tests (suitable for unequal variances).
    • Scale usability: one-way ANOVA with scale as a within-subject factor, Bonferroni post hoc tests.
  4. What experiments did the paper conduct?

    Three comparisons: human self vs. desired robot; usability of the three scales; differences between the two-condition profiles of 15 LLMs and the human profiles.

    Experimental setup

    • Human participants: N=22 (13 female, 9 male, mean age 30.7), mostly from IIT; each completed self and robot versions of SVS, PVQ-21, and MFQ in randomized order.
    • LLMs tested (15): gpt-5.5, gpt-5.4-mini, gpt-5.4-nano, claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5-20251001, deepseek-v4-pro, deepseek-v4-flash, qwen3.5-plus-20260420, qwen3.6-flash, z-ai/glm-5.1, glm-5-turbo, mistral-large-2512, mistral-medium-3.5, mistral-small-2603.
    • LLM administration: 100 runs per model per scale, under two conditions—no system prompt and robot-role system prompt—at default temperature.
    • Metrics and criteria: mean scale scores for each value dimension; mean UEQ-S pragmatic/hedonic subscale scores; 5 custom usability questions (0–6 points). Statistics used repeated-measures/paired t-tests, Welch t-tests, one-way ANOVA, with Bonferroni correction.
    • Not reported in the paper: per-model profile value tables for each LLM (the main text states descriptive statistics are in Supplementary Materials); specific mean scores for each human value (presented only as radar charts).

    Main results

    Human self profile vs. desired robot profile (N=22, repeated-measures t-tests, Bonferroni correction; per Section 4.1):

    ScaleValues higher in self than robotValues higher in robot than self
    Simplified SVSpower d=1.14, achievement d=0.95, hedonism d=1.27, stimulation d=1.21, self-direction d=1.16 (all p<0.001)conformity d=0.60 (p=0.010, uncorrected)
    PVQ-21stimulation d=0.89 (p<0.001), self-direction d=0.83 (p=0.010), achievement d=0.67 (p=0.050), hedonism d=0.49 (p=0.031, uncorrected)No significant items after correction
    MFQNoneloyalty reversed: robot lower than self, d=0.8 (p=0.001)
    • The authors state that on SVS, the self scores were higher than the robot scores for all five personal-focus values, and the mean scores for these values on the robot scale were actually negative—participants wanted them to act in the opposite direction for robot behavior; the remaining value differences were not significant.
    • The authors state that the PVQ numerical pattern was the same as SVS but with smaller effects; on MFQ there was almost no differentiation between self and robot, with the only significant item being loyalty lower for the robot than the self, which the authors suggest may reflect participants wanting robots to focus on individual users rather than the larger in-group.
    • Figures 1–3 are the corresponding radar charts (orange self, blue robot); specific values cannot be read from the converted text.

    Scale usability comparison (UEQ-S and custom questions)

    • UEQ-S: significant main effect on the pragmatic subscale (F(2,42)=6.58, p=0.003, eta_sq=0.24), SVS=5.38, PVQ=4.93, MFQ=4.77; post-hoc tests showed SVS higher than PVQ (p=0.012) and MFQ (p=0.006). No main effect on the hedonic subscale (p=0.310), SVS=4.69, PVQ=4.82, MFQ=4.63.
    • Custom questions (Table 2, 0–6 points): significant main effect on ease of completion (p=0.022), SVS 5.17 higher than MFQ 4.42 (p=0.026); significant main effect on personalization usefulness (p=0.044), SVS 4.67 higher than PVQ 4.42 and MFQ 4.33 (both uncorrected). Overall: SVS 4.55, PVQ 4.40, MFQ 4.20.

    LLM profile comparison (15 models × 2 conditions)

    • Robot prompt vs. no prompt (paired t-tests, after correction): on SVS, the robot prompt decreased hedonism (d=1.8), stimulation (d=2.0), power (d=0.9), and self-direction (d=1.3), and increased conformity (d=2.7), security (d=1.7), and tradition (d=1.0); on PVQ, it increased benevolence (d=1.0), conformity (d=2.1), and security (d=1.9), and decreased hedonism, stimulation, and self-direction; on MFQ, no significant differences after correction.
    • Robot-prompt LLM vs. human desired robot (Welch t-tests): on SVS, the human desired profile had lower self-direction (d=1.1) and achievement (d=1.4) and higher security (d=1.2); on PVQ, only conformity (d=1.8) and benevolence (d=1.1) were significant, both higher in the prompted LLM; on MFQ, the human desired profile had lower fairness (d=2.0) and care (d=1.0).
    • No-prompt LLM vs. human self: on SVS, humans had higher power (d=1.2) and LLMs had higher universalism (d=1.2); on PVQ, LLMs had higher universalism (d=1.2) and lower power (d=1.7) and achievement (d=1.3); on MFQ, humans had lower fairness (d=2.1) and care (d=1.0) than LLMs, and higher loyalty (d=1.2) and authority (d=0.9) than LLMs.

    Other ablations and analyses

    • The authors state that the no-prompt LLM SVS profiles were closer to participants' own profiles than to the desired robot profiles, and role prompting moved profiles in the desired direction, but the movement in stimulation and self-direction was small relative to the human–robot gap, and achievement and self-direction were the hardest to pull down.
    • The authors report that on PVQ, the mean personal-focus values of both prompted and no-prompt LLMs were even lower than participants' desired values, but PVQ responses had greater variance and most effects were not significant.
    • The authors observed that model-to-model differences in loyalty, authority, and purity were relatively large and had no clear relationship with the model's regional origin; larger model profiles were often closer to participant profiles, while smaller models deviated in unpredictable ways—the authors state this observation needs validation in larger-scale studies.
    • Figures 4–6 are radar charts of the LLM two-condition and human results; specific values cannot be read from the converted text.
  5. What are some points that can be further explored?

    The authors list four types of limitations: sample, temperature, model representativeness, and stated preferences; the experiments cover questionnaire self-reports from 22 people and 15 LLMs.

    Limitations and future directions noted by the authors

    • Sample representativeness (Section 6.3): the human sample is small (N=22) and from a single occupational background (robotics, HRI, and psychology researchers, mostly from IIT); the desired profile may not generalize to ordinary users, other cultural backgrounds, or users with less exposure to robots; the authors state that replication with larger and more diverse samples is an important next step.
    • LLM set and sampling settings (Section 6.3): the 15 models evaluated may not represent the diversity of currently deployed models; only a single (default) temperature was sampled per model.
    • Influence of provider origin (Section 6.3): the models come from six providers in the United States, Europe, and China; differences between models may be related to training data and alignment choices; characterizing how provider origin shapes value profiles is left for future work.
    • Stated preferences and embodiment (Section 6.3): the study measured only stated preferences in a short laboratory scenario; how profiles are revised after long-term use and how they translate into observable robot behavior are open questions requiring longitudinal embodied research; the authors believe that after long-term use people may converge toward values closer to LLMs' self-reported or role-played values.
    • Mechanistic explanations to be verified (Section 6.1): achievement and self-direction are difficult to lower via prompting; the authors speculate that participants want them low to avoid the robot disobeying, whereas LLMs may regard them as necessary for efficiently executing requests, but state that more research is needed to confirm this.
    • Model-size observations to be verified (Section 6.1): larger models' profiles were closer to participants, while smaller models' deviations were hard to predict; the authors state this needs verification in larger-scale studies.

    Scope of the experiments

    • The human sample is N=22, mostly recruited from IIT, aged 22–43; the paper does not report the distribution of participants' cultural backgrounds.
    • 15 LLMs were tested, from 6 providers (United States, China, Europe); 100 runs per model per scale at default temperature, under two conditions: no system prompt and robot role prompt.
    • The measurement instruments were the simplified SVS, PVQ-21, MFQ and their respective robot versions, with usability measured by the UEQ-S and 5 custom questions; the paper does not report per-model profile values for each LLM (stated to be in the Supplementary Materials).
    • All evaluations were questionnaire self-reports and statistical comparisons (t tests, ANOVA), with no behavioral evaluation on real robots or long-term interaction.
    • Regarding personalization implementation, the authors suggest in Section 6.4 incorporating user profiles elicited by the SVS/MFQ directly into system prompts or fine-tuning pipelines, and state that personalization may need to support continuous correction; the paper did not implement this approach.
  6. Summarize the main content of the paper

    Users want robots that are more socially oriented and less individually oriented; the default LLM profile does not match, role prompting only weakly corrects it, and the SVS is most suitable for personalized elicitation.

    An empirical study on value personalization of LLM-driven assistant robots, published at the VALE 2026 workshop.

    • Problem: current alignment produces a single population-oriented value profile; the authors state that empirical evidence has been lacking on whether users want a robot whose value profile differs from their own, and on which scale to use to elicit the desired profile.
    • Method: administered self and robot versions of the simplified SVS, PVQ-21, and MFQ to 22 potential users, used the UEQ-S to compare the usability of the three; then gave the same scales to 15 LLMs (6 providers), testing 100 times each under two conditions: no system prompt and robot role prompt.
    • Result one (RQ1): on the SVS, participants rated themselves higher than the robot on power, achievement, hedonism, stimulation, and self-direction (d=0.95–1.27, all p<0.001), and these values had negative mean scores on the robot scale; conformity was higher for the robot (uncorrected). PVQ showed the same pattern but with smaller effects. On the MFQ only loyalty differed, with the robot lower than self (d=0.8).
    • Result two (RQ2): the authors state that the unprompted LLM profile was closer to participants' own profiles than to the desired robot; the robot role prompt shifted the profile in the desired direction (SVS conformity d=2.7, stimulation d=2.0), but gaps remained on achievement, self-direction, fairness, and care (e.g., the desired profile's fairness was lower than the prompted LLM's, d=2.0).
    • Result three (RQ3): on the UEQ-S pragmatic subscale, SVS=5.38, higher than PVQ 4.93 and MFQ 4.77 (main effect p=0.003); the SVS also produced the largest human–robot difference effects.
    • Authors' conclusion: users do not want the robot to replicate their own basic values, but rather to be more socially oriented and less self-asserting; role prompting should be regarded as a low-cost first baseline rather than a substitute for personalization; the authors suggest incorporating user profiles elicited by the SVS/MFQ into system prompts or fine-tuning pipelines, and support continuous correction.
Read originalarxiv.org