EventPast event

Exploring the use of unsupervised methods to recover the refusal direction in large language models

1 reports1 reporting sourcesLatest reportOct 22 10:01

Start here

AI overview

On October 22, 2026, the author posted on LessWrong, applying PCA, Isomap, and diff-in-means to recover the refusal direction of large language models, and tested on Llama 3.1 8B Instruct and Qwen2.5-7B-Instruct. The author states that the direction recovered by PCA is close to the direction obtained by diff-in-means in cosine similarity and causal ablation effect, and that the required sample size is not large; Isomap consistently performed worse than PCA, and the author therefore concludes that there is no evidence that the activation geometry of refusal is nonlinear.

Generated by AI from reports · Updated 21 days ago

Latest updateOct 22 10:01

Exploring the use of unsupervised methods to recover the refusal direction in large language models

Related research

Papers whose work the reports mention

The reports mention JailbreakBenchJailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
“We also amalgamate a held out harmful dataset from many of the same sources Arditi et al. procured its harmful dataset from, including AdvBench, JailbreakBench, HarmBench, and MaliciousInstruct.”
Exploring the use of unsupervised methods to recover the refusal direction in large language models
Proposes the jailbreak evaluation benchmark JailbreakBench, unifying evaluation criteria for attacks and defenses
arXiv · 2024-03-28 · Deep reading
The reports mention HarmBenchHarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
“We also amalgamate a held out harmful dataset from many of the same sources Arditi et al. procured its harmful dataset from, including AdvBench, JailbreakBench, HarmBench, and MaliciousInstruct.”
Exploring the use of unsupervised methods to recover the refusal direction in large language models
Propose the red-teaming evaluation framework HarmBench, compare multiple attacks and models, and propose the adversarial training defense R2D2
arXiv · 2024-02-06 · Deep reading

Progress timeline

Oct 22

10:01
LessWrong
Exploring the use of unsupervised methods to recover the refusal direction in large language models
The authors apply principal component analysis (PCA), Isomap and diff-in-means to recover the refusal direction of large language models, and test on Llama 8b 3.1 Instruct and Qwen2.5-7B-Instruct. The authors state that the direction recovered by PCA is close to the direction obtained by diff-in-means in cosine similarity and causal ablation effect, and that the required sample size is not large. Isomap consistently performs worse than PCA, and the authors accordingly conclude that there is no evidence that the activation geometry of refusal is nonlinear. The authors also use a projection threshold to evaluate classification performance on held-out harmful and harmless prompts, and the code has been released.