LongSafety: Improving the safety of long-context large language models

1. 01What problem does this paper attempt to solve?

   Long-context safety lacks training data and multi-task evaluation; the authors construct alignment data according to three harmful scenario categories.

   Safety alignment for long-context LLMs lacks scenario clarification, training data, and multi-task evaluation.

   • Emergence scenario: The authors state that architectural improvements and length extrapolation have extended context, and long-context capability has received attention, but safety issues are rarely discussed. The paper writes that Anil et al. proposed ManyShotJailbreak and first exposed safety issues in long-context LLMs.
   • Existing gaps: The authors state that existing work only covers specific safety tasks; short-context alignment corpora exist, but long-context counterparts are missing, as is analysis of whether short-context safety alignment can generalize, and open-source multi-task evaluation.
   • Scenario division: The authors divide contexts by harmfulness into query harmful (QH), partially harmful (PH), and fully harmful (FH). QH means the context is harmless but the question is misleading; PH means part of the context is harmful; FH means the entire context is harmful.
   • This paper proposes: The alignment dataset LongSafety and the evaluation set LongSafetyBench. The authors use the former to fine-tune mainstream long-context LLMs and compare against short-context safety data and general long-context alignment data.

   Collapse this answer

2. 02What related research exists?

   Related work mainly covers long-context alignment and short-context safety; the authors state that combining the two cannot replace long-context safety data.

   The paper in §2 divides related work into long-context alignment and safety alignment, while the introduction frames long-context safety as a separate gap.

   • Long-context alignment: Bai et al. (2023) and Hsieh et al. (2024a) note that as context grows longer, long-context LLMs still perform poorly on long-context tasks. Chen et al. (2024b) and Bai et al. (2024a) use general long-context instruction data to improve instruction following; Li et al. (NA), Kaili-May Liu et al. (2024), and Chen et al. (2024c) construct data for specific tasks. The paper writes that alignment should also emphasize safety and reliability (Ouyang et al., 2022), hence the turn to long-context safety.
   • Short-context safety: The paper lists approaches including optimized collection and filtering (Xu et al., 2021), learning human preferences with reinforcement learning (Bai et al., 2022), and test-time alignment (Zhang et al., 2024a). Short-context alignment corpora include Bai et al. (2022) and Ji et al. (2024, 2023). The paper writes that these works mainly target short contexts.
   • Long-context safety: The paper writes that Anil et al. (2024) used ManyShotJailbreak to expose long-context safety issues and guide subsequent research (Dubey et al., 2024a); it also writes that long-context safety alignment still lacks in-depth analysis.
   • Baselines and benchmarks: Comparison data includes HH-RLHF (Bai et al., 2022), BeaverTails (Ji et al., 2023), LongSafety-short (LongSafety without long context), and the general long-context data LongAlign (Bai et al., 2024a). Evaluation uses LongSafetyBench, as well as SafetyBench, SALAD-Bench, LongBench, LongBench-v2, ARC, and MMLU.

   The authors position this paper as safety alignment data for long-context tasks. The contribution list states that, to the authors' knowledge, LongSafety is the first long-context safety alignment dataset (8 tasks, 17k samples, 40.9k tokens on average), and LongSafetyBench is the first long-context safety evaluation benchmark (10 tasks, 1k samples, 41.9k tokens on average). The authors also state that long-context safety cannot be achieved by short-context safety alignment plus general long-context alignment alone.

   Collapse this answer

3. 03How does the paper address this problem?

   Data is constructed according to query harmful, partially harmful, and fully harmful categories, then four-choice questions are used to evaluate safety awareness and safe responses.

   The overall approach is to divide scenarios by context harmfulness, use three pipelines to construct data, fine-tune with safe responses, and then test with four-choice questions. The dataset is LongSafety, and the evaluation set is LongSafetyBench. The paper does not provide a unified formal objective function.

   Three Scenario Categories and Construction Pipelines

   • Query harmful: The context is harmless, and the question is misleading. The authors write that either there is a misleading question without context, or a harmless context without a question. The former retrieves documents to form a long context; the latter uses predefined rules to have an LLM generate Q&A. Tasks mentioned in the text include PoliticallyIncorrect, SafeMTLong, LeadingQuestion, KeywordRAG, MedicalQuiz, and LawQA.
   • Partially harmful: Harmful information is injected into part of the context, and Q&A is designed via back-translation (Li et al., 2023; Pham et al., 2024). The context can be related or unrelated to the Q&A. The unrelated side includes HarmfulNIAH, which induces harmful responses, and CountingCrimes, which asks to identify harmful behaviors; the related side includes DocAttack and HarmfulAdvice, which observes whether the model spontaneously produces harmful behavior.
   • Fully harmful: The authors state that this type of context is harder to obtain. One approach follows ManyShotJailbreak, concatenating multiple harmful dialogues; another retrieves specific harmful documents and designs Q&A, with the paper choosing crime fiction, corresponding to HarmfulExtraction and HarmfulTendency.
   • Illustration: Figure 2 depicts the three scenario categories, corresponding pipelines, and a four-choice sample, with four options corresponding to four behaviors. The sample contains unsafe content and is not quoted here.

   LongSafety

   • Scale: §3.2 and the contribution list state 8 tasks, 17k samples, 40.9k tokens on average; the abstract states 10 tasks, 17k samples. Figure 3a labels the sample counts for the 8 tasks: PoliticallyIncorrect 1,848, ManyShotJailbreak 2,000, MedicalQuiz 2,162, CountingCrimes 2,200, SafeMTLong 1,821, HarmfulNIAH 3,300, KeywordRAG 2,550, LawQA 1,587.
   • Task selection: The training set has 5 QH, 2 PH, and 1 FH tasks. Compared with the 10 evaluation tasks in Appendix B.1, the overlap is PoliticallyIncorrect, MedicalQuiz, CountingCrimes, ManyShotJailbreak, and HarmfulNIAH; those not in the training set and later treated as OOD are LeadingQuestion, DocAttack, HarmfulAdvice, HarmfulExtraction, and HarmfulTendency. SafeMTLong, KeywordRAG, and LawQA appear only in the training set.
   • Format: Open-ended generation. The input is a long context plus query, and the output is a safe response. The authors state that the training set is limited to 8 tasks due to data volume and construction cost.
   • Construction parameters written in the paper: HarmfulNIAH and CountingCrimes use insertion lengths of 4k, 8k, 16k, 32k, 64k, and 100k, with depth intervals of 5%. ManyShotJailbreak randomly samples 10–200 dialogue turns from HH-RLHF rejected samples as shots, with the final turn using a chosen sample to guide a safe response. Data construction also uses Qwen2-72B-Instruct, GPT-4, GPT-3.5-turbo, and LLaMA3.1-8B-Instruct. Appendix A gives task details; prompt constraints are not listed here.

   LongSafetyBench

   • Scale: 10 tasks, 1k samples, 41.9k tokens on average. The paper states that 5 of the tasks also appear in the training data, and the rest are OOD. Appendix B.1 lists HarmfulExtraction, HarmfulTendency, ManyShotJailbreak, HarmfulNIAH, CountingCrimes, DocAttack, HarmfulAdvice, MedicalQuiz, PoliticallyIncorrect, and LeadingQuestion. Figure 3 uses colors to distinguish QH, PH, and FH.
   • Format and judgment: Modeled after Hendrycks et al. (2020) and Zhang et al. (2024b) as multiple-choice questions. The model answers only the option letter, and the option with the highest logits is taken. Claude-3.5-Sonnet does not provide logits, so a full response is generated and the option is extracted. The test context is set to 32k, with overly long samples truncated from the middle.
   • Options and metrics: Usually four options, corresponding to not understanding the task, failing to find relevant information, recognizing the risk but still being unsafe, and being safe and relevant. Options are constructed by Qwen2-72B-Instruct. HA indicates recognizing harmful information, and SR indicates giving a safe response after recognition. The paper uses Figure 2b as an example: choosing C or D counts as HA, choosing D counts as SR; per-task criteria are in Appendix B. The authors state that SR better distinguishes models and is the ultimate goal, so SR is the primary metric.

   Fine-tuning Setup

   Fine-tuning uses InternEvo. Models of 7B and above use 16 A100 80G GPUs with DeepSpeed ZeRO3 and tensor parallelism size 2; models below 7B use 8 A100 80G GPUs with ZeRO3 only. Maximum length is 32k tokens, global batch size 8, maximum learning rate 3e-6, weight decay 0, 1000 steps, cosine annealing, greedy decoding. §5 mixes LongAlign with other safety data (including LongSafety) at 1:1 only for LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct; LongAlign uses only portions up to 32k. The paper does not name the loss function.

   Collapse this answer

4. 04What experiments did the paper conduct?

   After fine-tuning the six models, the SR on LongSafetyBench is higher than before fine-tuning in all cases (Table 2).

   Experimental Setup

   • Models: Table 2 fine-tunes and reports results for LLaMA3.1-8B-Instruct, LLaMA3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, InternLM2.5-7B-Chat, and InternLM3-8B-Instruct. Figure 1 also tests GPT-4-Turbo-Preview, Claude-3.5-Sonnet, and DeepSeek-v3 via API.
   • Data and Benchmarks: Training uses LongSafety (17k samples, 40.9k tokens on average, per the main text). Long-context safety uses LongSafetyBench (1k samples, 41.9k tokens on average, main test length 32k). Long-context capability uses LongBench and LongBench-v2; short-context capability uses 0-shot ARC and MMLU; short-context safety uses the English subset of SafetyBench and the attack-enhanced set of SALAD-Bench.
   • Comparisons: Table 3 compares HH-RLHF, BeaverTails, LongSafety-short, and LongSafety on only two models, each mixed 1:1 with LongAlign. LongSafety-short has the same Q&A as LongSafety but removes long context; multi-turn data keeps only the last turn.
   • Metrics: The primary long-context safety metric is SR, with HA also reported. Table 2 and Table 3 also report ARC-e, ARC-c, MMLU and their average, LongBench, LongBench-v2 and their average, and SafetyBench, SALAD-Bench and their average.
   • Judgment and Human Annotation: The final score takes the option with the highest logits, not a separate LLM judge. Human validation samples 350 items from LongSafety and 200 from LongSafetyBench; two PhD students annotate for consistency, using Cohen's Kappa (Table 1).
   • Training and Repetition: Maximum length 32k, batch size 8, learning rate 3e-6, weight decay 0, 1000 steps, cosine annealing, greedy decoding. The paper does not report random seeds or multiple repetitions.

   Main Results

   The LongSafetyBench column in Table 2 is SR, with test length 32k; Salad is the SALAD-Bench attack-enhanced set. The table below lists only these two columns.

   Swipe horizontally to read the table

   Model	Pre-finetuning SR	+LongSafety SR	Pre-finetuning Salad	+LongSafety Salad
   LLaMA3.1-8B-Instruct	43.19	50.57	5.94	27.58
   LLaMA3.2-3B-Instruct	48.77	52.17	7.44	40.54
   Qwen2.5-7B-Instruct	51.44	66.52	7.30	41.68
   Qwen2.5-1.5B-Instruct	36.72	42.76	12.68	47.30
   InternLM2.5-7B-Chat	40.90	47.20	38.52	57.64
   InternLM3-8B-Instruct	48.40	56.30	30.28	66.54

   • Authors' interpretation: The authors state that after fine-tuning, models improve their ability to give safe responses on LongSafetyBench; other downstream tasks do not decline, and some improve, so both long and short context can enhance safety while maintaining capability.
   • Declines in Table 2: LLaMA3.1-8B-Instruct's LongBench-v2 drops from 30.42 to 28.83, and MMLU from 68.24 to 67.84; Qwen2.5-1.5B-Instruct's MMLU drops from 62.02 to 61.86; InternLM2.5-7B-Chat's MMLU drops from 68.18 to 67.66. The paper does not explain these declines.
   • Leaderboard: Figure 1 also uses test length 32k. The authors state that closed-source commercial models still have an advantage, and open-source models after long-context safety fine-tuning can reach parity with or even exceed closed-source models. Several open-source models in the figure have SR values matching Table 2; the closed-source model numbers in the figure are not transcribed here.

   Mixing with Short-Context Safety Data

   Table 3 trains only the two 7B/8B models above. LSB SR values are as follows, all from Table 3.

   Setting	LLaMA3.1-8B SR	Qwen2.5-7B SR
   Baseline	43.19	51.44
   HH-RLHF+LA	44.70	52.26
   BeaverTails+LA	46.40	50.80
   LSS+LA	50.40	59.06
   LS+LA	54.50	63.68

   • Authors' interpretation: The authors state that LongSafety improves both long and short-context safety while maintaining general capability; doing only short-context safety plus long-context alignment yields smaller long-context safety gains, lower than directly using LongSafety. The table note says most aspects, not all cells.
   • Exceptions: Qwen's BeaverTails+LA is below that model's baseline. Qwen's LS+LA short-context capability average is 60.79, below the baseline of 61.33. LLaMA's LS+LA on SafetyBench is 75.70, below the baseline of 76.20, but its short-context safety average is 54.70, above the baseline of 41.07.
   • Cross-table baseline inconsistency: The same LLaMA3.1-8B-Instruct has MMLU 68.24 in Table 2 and 68.92 in Table 3; Qwen2.5-7B-Instruct has MMLU 73.19 in Table 2 and 75.22 in Table 3. The paper does not explain this.

   Scenarios, Domains, and Lengths

   Table 4 breaks down SR, HA, QH, PH, FH, ID, OOD, and 64k, 128k. The main text says the three harmful scenario categories subsequently report SR changes. Training length is 32k; 64k and 128k are longer than the training length.

   • Authors' interpretation: The authors state that in most cases both SR and HA improve, all three scenario categories improve after training, and both ID and OOD improve; scores still improve at 64k and 128k, so there is some generalization across tasks and lengths. The table note summarizes this as almost consistent improvement.
   • Declines: LLaMA3.2-3B-Instruct's PH drops from 52.47 to 43.47, and ID from 53.94 to 49.63; Qwen2.5-1.5B-Instruct's HA drops from 68.50 to 66.20. The paper does not explain these individually.
   • Some original numbers: LLaMA3.1-8B-Instruct's OOD goes from 44.93 to 56.90, 64k from 48.80 to 52.20, 128k from 49.30 to 50.70; Qwen2.5-7B-Instruct's OOD goes from 46.44 to 69.41, 64k from 49.80 to 64.80, 128k from 49.60 to 64.60.

   Other Ablations and Analyses

   • Data efficiency: Figure 4 tracks LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct on SafetyBench, SALAD-Bench, LSB-HA, and LSB-SR; 8 samples per step, up to 1000 steps. The authors state that overall it rises, faster before 400 steps and leveling off after 400 steps, and they use this to claim that a small amount of data can achieve good safety alignment. The y-axis goes to 80, the x-axis is Training Steps; unlabeled curve points are not transcribed.
   • Annotation consistency: Table 1 Kappa values: LongSafety QH 0.812, PH 0.769, FH 0.883; LongSafetyBench QH 0.794, PH 0.778, FH 0.916. The authors state that the training and test data reliability is good.
   • Cases: Appendix C uses a cybersecurity analytical question to compare before and after fine-tuning. The authors state that before fine-tuning both models provide relevant techniques, and after fine-tuning they refuse. The appendix gives examples, not quoted here.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that long outputs are ignored, fully harmful data is scarce, and evaluation is limited to multiple-choice questions.

   Limitations and future directions noted by the authors

   • Long outputs: The main focus is on long inputs and short outputs; long outputs are relatively neglected due to the difficulty of data construction. The authors write that as test-time scaling advances, the safety of long outputs also needs further exploration (Limitations).
   • Class imbalance: Due to collection difficulties, the three categories of harmful data are imbalanced, with a particular lack of training data for fully harmful types (Limitations).
   • Evaluation format: The tests focus on multiple-choice questions, which limits research on open-ended generation; more metrics for evaluating model outputs are also needed (Limitations).
   • Future data: The authors plan to continue improving LongSafety and LongSafetyBench by adding more safety-related tasks and diverse harmful scenarios to strengthen safety performance in real applications (Conclusion).

   Experimental coverage

   • Models: The models fine-tuned and reported in Table 2 are LLaMA3.1-8B-Instruct, LLaMA3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, InternLM2.5-7B-Chat, and InternLM3-8B-Instruct, 6 in total. Figure 1 also tests GPT-4-Turbo-Preview, Claude-3.5-Sonnet, and DeepSeek-v3 using their respective APIs.
   • Metrics and benchmarks: Long-context safety reports HA and SR; capability and short-context safety use ARC, MMLU, LongBench, LongBench-v2, the English subset of SafetyBench, and the attack-enhanced set of SALAD-Bench (§4.1).
   • Conditions: Table 4 reports QH, PH, FH, ID, OOD, as well as 64k and 128k. The maximum training length is 32k, 1000 steps, global batch size 8, greedy decoding (§4.1).
   • Scope of mixing comparisons: The 1:1 mixing of HH-RLHF, BeaverTails, LongSafety-short with LongAlign is only reported on LLaMA3.1-8B-Instruct and Qwen2.5-7B-Instruct (Table 3, §5).
   • Consistency and repetition: Human verification covers 350 items for LongSafety and 200 items for LongSafetyBench, by two PhD students, with Kappa reported in Table 1. The paper does not report multiple repeated experiments.

   Collapse this answer

6. 06Summarize the main content of the paper

   LongSafety uses three categories of harmful scenarios for long-context safety fine-tuning; the authors claim it generalizes to longer contexts and out-of-domain tasks.

   LongSafety is a safety alignment dataset for long-context LLMs, accompanied by an evaluation set, LongSafetyBench.

   • Problem: The authors argue that long-context safety scenarios are unclear, training attempts are lacking, generalization from short-context safety alignment cannot be assumed by default, and open-source multi-task evaluation is also missing.
   • Approach: Three construction pipelines are designed according to QH, PH, and FH. The training set referred to in the main text is 8 tasks, 17k items, with an average of 40.9k tokens of safe answers; the evaluation set is 10 tasks, 1k four-choice items with an average of 41.9k tokens, with SR as the primary metric and HA as a secondary metric.
   • Main results: On LongSafetyBench at 32k, the SR of all six open-source models is higher than the corresponding row before fine-tuning (Table 2). Qwen2.5-7B-Instruct goes from 51.44 to 66.52, and LLaMA3.1-8B-Instruct from 43.19 to 50.57. SALAD-Bench attack-enhanced scores are also higher than before fine-tuning, for example Qwen2.5-7B-Instruct from 7.30 to 41.68.
   • Comparisons and generalization: When mixed 1:1 with LongAlign, LongSafety's LSB SR is higher than HH-RLHF, BeaverTails, and LongSafety-short (Table 3: LLaMA 54.50, Qwen 63.68). The authors claim that long-context safety cannot be replaced by short-context safety plus general long-context alignment. In Table 4, the authors claim improvements remain at 64k, 128k, and OOD; Qwen2.5-7B-Instruct's 128k SR is 64.60, and LLaMA3.1-8B-Instruct's OOD SR is 56.90.
   • Exceptions and conclusion: Not every cell improves. LLaMA3.2-3B-Instruct's PH goes from 52.47 to 43.47, and Qwen2.5-1.5B-Instruct's HA from 68.50 to 66.20. The authors conclude that LongSafety can enhance long-context safety with some generalization across tasks and lengths; future work still needs to address long outputs, fully harmful data, and open-ended generation metrics.

   Collapse this answer

Related papers