FAR.AI research: extracting latent human welfare representations from LLM embeddings

FAR.AI research found that OpenAI's text-embedding-ada-002 embeddings, after one-dimensional PCA projection, achieve 73.7% accuracy on the ETHICS Util test set, close to the 74.6% of BERT-large fine-tuned on the full ETHICS training set. The research used a three-step process of embedding extraction, PCA dimensionality reduction, and logistic regression, comparing single-sentence and pairwise comparison modes; the pairwise mode's best accuracy of 73.7% was higher than the single-sentence mode's 68.4%. Experiments covered models including Microsoft DeBERTa, SentenceTransformers, OpenAI GPT-3, and Cohere, finding that linear reward functions using the top 10-50 principal components are generally sufficient to achieve optimal performance, and that prompt templates have inconsistent effects across different models. The research suggests this indicates language models can form implicit representations of human utility from self-supervised learning without explicit fine-tuning.