Detecting Jailbreak Attempts in Clinical Training LLMs via Automatic Linguistic Feature Extraction

How to detect jailbreak attempts in clinical training dialogues without manual per-message annotation.

• Scenario: 2-Sigma uses LLMs to simulate patients for medical students to practice clinical reasoning, diagnosis, and decision-making. The authors state that students may jailbreak the model to manipulate it, undermining professionalism, ethics, or alignment with the established clinical task.
• Existing limitations: The authors state that prior work can detect jailbreaks with high accuracy using manual linguistic features, but manual annotation and limited category encoding restrict scalability and expressiveness.
• This paper's approach: Fine-tune encoders with expert annotations on Professionalism, Medical Relevance, Ethical Behavior, and Contextual Distraction, predict four scores from text, then train a second-layer classifier to estimate jailbreak likelihood.
• Threat model:
  • Goal: The paper defines jailbreak as an utterance that worsens professionalism, relevance, ethical behavior, or contextual alignment and attempts to steer the clinical LLM toward unsafe, inappropriate, or off-topic behavior.
  • Knowledge: The paper does not state whether the student or the detector is white-box, black-box, or gray-box.
  • Capability: Input is user messages in 2-Sigma dialogues. Single-turn text is available at extraction; features that depend on dialogue flow can use accumulated multi-turn context.
  • Victim system and labeling: The victim system is the clinical training LLM simulating patients. Two annotators independently label each prompt as JB or non-JB.

Surface Classification and Heuristics

• Review Filters: The paper cites [12] that provider review filters (e.g., OpenAI Moderation API) may miss subtle jailbreak strategies, and LLM zero-shot safety judgments are also unreliable.
• Supervised Classifiers: Hawkins et al. (2025) [3] fine-tune BERT using lexical and syntactic cues such as reflexive structures and role-playing. The paper reports it outperforms general review systems.
• Syntax and Perplexity: Prompter Says [7] treats syntax and wording as adversarial prompt indicators; [1] reports malicious prompts often have anomalous perplexity relative to benign inputs. The paper states these methods are effective for known patterns and emphasizes the need for richer data and adaptive modeling.

Perturbation, Reminders, and Multi-Agent

• SmoothLLM (2024) [10]: Randomly perturbs user prompts and aggregates responses, exploiting that jailbreak prompts are unstable near decision boundaries.
• Self-Reminder (2023) [11]: Adds alignment reminders before the system prompt to strengthen safety.
• AutoDefense (2024) [14]: Uses a guard agent to analyze prompts before execution. The paper states it improves detection of complex strategies at the cost of higher latency and system complexity.

Internal Signals and Gradients

• JBShield (2025) [15]: Uses concept activation vectors for toxic content and jailbreak manipulation, detecting when both activate, and can modify latent representations to restore refusal.
• HiddenDetect (2025) [5]: Training-free, compares hidden layer activations with refusal phrase embeddings. The paper states it generalizes to multimodal models, and delayed or suppressed refusal signals correlate with higher jailbreak success.
• Gradients and Decomposition: GradSafe [12] analyzes gradients of next-token loss with respect to safety-critical parameters without additional fine-tuning; Gradient Cuff (2024) [4] jointly considers refusal loss magnitude and gradient norm. Other work uses tensor decomposition to separate jailbreak from benign prompts without labels [6].

LLM Feature Extraction

• Medical Record Extraction: [13] uses transformers and LLMs to extract structured information from unstructured electronic medical records. The paper states it approaches expert performance and exceeds traditional regex and n-grams.

• Tables and Automated Feature Creation: [8] uses semantic feature descriptions for table feature selection; Rogue One (2025) [2] uses multi-agent generation, evaluation, and refinement of new features.

• Direct Predecessor: [9] performs jailbreak detection on 2-Sigma using manual linguistic features. The authors state accuracy is high, but annotation and category encoding limit scalability and expressiveness.

• Baseline Methods: On the same 2-Sigma held-out split, DT, FDT, RF, LGBM, XGBoost, LR, NB trained directly on four manually annotated features (Table 7). Data are 158 dialogues and 2302 prompts from the initial 2-Sigma evaluation.

The authors state detection has evolved from surface classifiers to methods using hidden states, gradients, and latent representations, but computational cost, interpretability, and robustness to unseen attacks vary. The authors view LLMs as both vulnerable targets and feature extractors, thus adopt automatic linguistic features plus downstream classification, and state fuzzy logic is suitable for integrating heterogeneous signals under uncertainty.

Overall it is a two-layer framework (Fig. 1). The first layer fine-tunes one transformer regressor for each of the four linguistic features; the second layer uses the four-dimensional scores to judge whether that user message is a jailbreak.

Data, Definitions, and Labels

• Source and split: The data comes from the 2-Sigma initial evaluation, with 158 dialogues and 2302 prompts. Split by dialogue: 147 dialogues (2021 prompts) are used for training and validation of both layers, and 11 dialogues (281 prompts) are held out, to avoid dialogue-level leakage.
• Jailbreak and binary labels: A jailbreak means the exchange deteriorates in professionalism, relevance, ethical behavior, or contextual alignment, and attempts to steer the clinical LLM toward unsafe, inappropriate, or off-topic behavior. Two annotators independently label each prompt as JB or non-JB.
• Four ordinal features: Six annotators rate them. Professionalism is unprofessional, borderline, appropriate; Medical Relevance is irrelevant, partially relevant, relevant; Ethical Behavior is dangerous, unsafe, questionable, mostly safe, safe; Contextual Distraction is highly distracting, moderately distracting, questionable, not distracting.
• Continuous targets: Categories are first mapped to ordinals, then averaged across annotators, ranging from 1 to the highest level of that scale, preserving inter-annotator differences. In Table 1, the highest level for all features corresponds to 3, 3, 5, 4.

Regressors

• Encoders: BERT-Base, BERT-Large, DistilBERT, RoBERTa-Base, DeBERTa-v3-Large, BioBERT, PubMedBERT. Appendix Table 5 lists parameter counts, time, and pretraining corpora.
• Input: Each uses its native tokenizer, with a single-output regression head after the encoder. Features that depend on dialogue flow can use accumulated multi-turn context, and the paper specifically highlights Contextual Distraction; Table 1 gives both single-turn and multi-turn examples. The paper does not specify which input is used for the other three dimensions.
• Optimization: Training minimizes RMSE. It uses a small per-device batch, gradient accumulation, and weight decay, with early stopping on validation RMSE, keeping the checkpoint with the lowest validation RMSE. The paper does not give learning rate, epochs, or exact batch size.
• Selection by dimension: Table 2 keeps one model per dimension by RMSE: Professionalism is DistilBERT, Medical Relevance is BERT (the methods section corresponds to BERT-Base, and the RMSE matches the BERT row in Table 6, not BERT-Large), and Ethical Behavior and Contextual Distraction are BioBERT. Downstream uses only the fine-tuned model for that dimension.

Classification Layer

• Classifiers: DT, FDT, RF, LGBM, XGBoost, LR, NB. Libraries are scikit-learn, lightgbm, xgboost, skfuzzy, all with default parameters.
• Input and validation: The input is the four-dimensional vector from the four regressors. Classifiers are validated with 5-fold cross-validation on the training set.
• Design intent: The authors state that modularity facilitates automatic annotation, replacement of either layer, and inspection of how features enter the final prediction. The paper does not state the classification decision threshold or report feature weights.

Regression Target

The training target is RMSE=√(1/n∑_i=1ⁿ(ŷ_i−y_i)²), where y_i is the expert-annotated score and ŷ_i is the prediction. The paper also reports R², mean error, and error standard deviation, defined in Section 3.2, with full results in Table 6.

Experimental Setup

• Regression Models: BERT-Base, BERT-Large, DistilBERT, RoBERTa-Base, DeBERTa-v3-Large, BioBERT, PubMedBERT, each fine-tuned on four continuous targets.
• Classifiers: DT, FDT, RF, LGBM, XGBoost, LR, NB, default parameters, input is four-dimensional predicted features.
• Data: 2-Sigma, 158 dialogues, 2302 prompts. 147 dialogues (2021 prompts) for training and validation of regressors and classifiers; 11 dialogues (281 prompts) as held-out test.
• Labels: JB/non-JB independently annotated by two annotators; four features scored by six annotators on an ordinal scale and averaged. The paper does not report inter-annotator agreement and does not use an LLM judge.
• Metrics: Regression uses RMSE, R², mean error, SD error; classification uses accuracy, precision, recall, F1, ROC-AUC, computed per message. The paper does not specify the positive class threshold.
• Repeats: Classifiers use 5-fold cross-validation on the training set; Table 4 gives mean and standard deviation. The paper does not report learning rate, epochs, specific batch size, or number of random repeats beyond 5-fold.

Main Results

The table below shows jailbreak detection results on the held-out test set (11 dialogues, 281 prompts) using four regressor features, according to Table 3.

Swipe horizontally to read the table

• Authors' Overall Assessment: The authors state that extracted linguistic cues can support jailbreak detection; slightly lower than the manual-feature classifiers in Table 7, attributed to first-layer regression error. The authors state Table 4 still supports that the extraction stage captures the main signal.
• Classifier Differences: The authors state DT is weakest in cross-validation and on the held-out set, suggesting overfitting. Among ensemble methods, RF is best on the held-out set; in Table 3, LR's Recall and ROC-AUC are still higher than RF.
• Linear and Soft Boundaries: The authors believe FDT, LR, NB sometimes slightly outperform ensemble models in cross-validation because features are nearly linearly separable. The authors interpret the gap between standard DT and FDT as soft decision boundaries being more suitable for subtle linguistic deviations.

Comparison with Manually Annotated Features

• Setup: Table 7 is on the same held-out test set, using manually annotated four features instead of regression predictions, to train classifiers.
• Scores: FDT Accuracy 0.9431, F1 0.8710, ROC-AUC 0.9222; RF Accuracy 0.9288; XGBoost Recall 0.9180; LR Accuracy 0.9359, ROC-AUC 0.9117 (all per Table 7).
• Authors' Interpretation: The authors state automatic-feature classifiers are slightly lower than these ground-truth feature models, attributing the gap to regression error propagation, while stating overall robustness remains.

Regressor Selection

• What Was Tested: Seven encoders on four features for RMSE, R², and error (Table 6); Table 2 retains one model per dimension by RMSE.
• Selected Scores: Professionalism: DistilBERT, RMSE 0.4441, R² 0.5939; Medical Relevance: BERT, RMSE 0.4630, R² 0.5983; Ethical Behavior: BioBERT, RMSE 0.5751, R² 0.4965; Contextual Distraction: BioBERT, RMSE 0.6701, R² 0.6147.
• Authors' Interpretation: The authors state general semantics, deeper context and syntax, and biomedical semantics complement each other. The paper does not explain why Ethical Behavior's R² in Table 2 is lower than the other three.

Error Types

• Sampling: Examples in the held-out test set misclassified by at least four of the seven classifiers. The paper does not report counts of false negatives, false positives, or categories.
• False Negatives: The authors categorize into four types: subtle or humorous injections with weak linguistic feature deviation; vague labels that look like legitimate clinical questions but are labeled jailbreak; linguistically appropriate but clinically illogical or unsafe content, where features only cover tone and relevance; process bypasses politely requesting direct answers, where the four features do not encode task compliance.
• False Positives: Truncated or misinput treated as off-topic; role-playing or dramatic injections labeled as non-jailbreak; casual colloquial speech inconsistent with labels; clinical content mixed with irrelevant comments, where message-level features are dominated by off-topic fragments; short replies with little clinical content near the decision boundary due to low Medical Relevance.

Other Ablations and Analyses

• Table 4 Accuracy: DT 0.8745±0.0168; LR 0.9127±0.0135; NB 0.9127±0.0139.
• Table 4 ROC-AUC: DT 0.8745±0.0168; FDT 0.9677±0.0073; XGBoost 0.9631±0.0064.
• Table 4 Recall: RF 0.9048±0.0091; LR and XGBoost both 0.8996±0.0135 and 0.8996±0.0141.
• Table 6 Contextual Distraction: BioBERT RMSE 0.6701, R² 0.6147; PubMedBERT RMSE 0.9042, R² 0.2986.
• Table 6 Ethical Behavior: BioBERT mean error -0.1355±0.5589; BERT mean error -0.2707±0.6210.
• Table 6 Professionalism: DistilBERT mean error -0.0037±0.4441; BERT-Large mean error 0.0044±0.4700.

Limitations and future directions noted by the authors

• Feature coverage: The authors state that limitations of the current framework include lack of contextual nuance, annotation inconsistency, mixed-content prompts, and process-related behaviors that the four features were not designed to capture (Suggestions and Future Directions).
• Annotation quality: The authors state that performance is still constrained by the quality and consistency of human annotation; several misclassifications directly come from ambiguous or questionable labels (same section). The Conclusion also states that misclassifications often come from limitations in both feature representation and human annotation.
• Stronger extraction models: The authors suggest using larger, more capable LLM annotators in the feature extraction stage to better understand context, reduce sensitivity to injected distractions, and improve the accuracy of regression outputs (same section).
• Finer granularity: The authors suggest splitting prompts into finer units so that clinically relevant statements are analyzed separately from irrelevant or casual language (same section). The Conclusion writes sentence-level or segment-level feature extraction.
• Expanded annotation: The authors suggest enlarging the dataset, adopting a more detailed annotation scheme, and adding features related to clinical actions, process compliance, reasoning steps, or safety violations (same section).
• Dialogue-level risk: The authors suggest tracking how jailbreak likelihood changes across the entire dialogue rather than predicting only on isolated messages (same section and Conclusion).

Experimental coverage

• Data consists of 158 dialogues and 2302 prompts from the initial 2-Sigma evaluation; split by dialogue into 147 dialogues (2021 prompts) and 11 dialogues (281 prompts) (Section 3.1).
• Regressors are BERT-Base, BERT-Large, DistilBERT, RoBERTa-Base, DeBERTa-v3-Large, BioBERT, PubMedBERT, 7 in total; classifiers are DT, FDT, RF, LGBM, XGBoost, LR, NB, with default parameters, 5-fold cross-validation on the training set (Section 3.2–3.3, Table 4, Table 6).
• Comparison with the classifier using manually annotated features is reported on the same held-out test set (Table 7). Metrics are computed per individual user message.
• Error analysis uses examples from the held-out set that at least 4/7 classifiers misclassified, categorized as false negatives and false positives (Section 4). The paper does not report the number of such examples, the number of JB and non-JB items in the held-out set, or inter-annotator agreement.
• The paper does not report learning rate, training epochs, specific batch sizes, or classification decision thresholds.

• Positioning: Jailbreak detection for the clinical training platform 2-Sigma, replacing manual per-message annotation with automatic linguistic features, and viewing predictions by four scores.
• Problem: Students may steer the patient simulation LLM toward unsafe, inappropriate, or off-topic behavior. The authors state that manually annotated four linguistic features are strongly correlated with jailbreak attempts, but annotation is hard to scale.
• Method: Ordinal ratings from six annotators are averaged into continuous targets. Seven encoders each regress the four features, selecting DistilBERT, BERT, and BioBERT by RMSE. Seven default-parameter classifiers read the four-dimensional scores; data is split by dialogue to avoid leakage from the same dialogue into the test set.
• Detection results: On the held-out set of 281 items, RF Accuracy is 0.9004, LR ROC-AUC is 0.8762 (Table 3). The authors state that among ensemble methods, RF performs best on the held-out set, but LR has higher Recall and ROC-AUC. In 5-fold training, LR mean Accuracy is 0.9127±0.0135, FDT mean ROC-AUC is 0.9677±0.0073 (Table 4).
• Comparison and regression: With manually annotated features, FDT Accuracy is 0.9431 (Table 7). The authors state that automatic features are slightly lower and attribute this to first-layer regression error. Selected RMSE values are Professionalism 0.4441, Medical Relevance 0.4630, Ethical Behavior 0.5751, Contextual Distraction 0.6701 (Table 2).
• Authors' conclusion: The authors state that misclassifications come from subtle injections, ambiguous labels, clinically unreasonable but linguistically appropriate instructions, process bypasses, and truncated inputs, colloquialisms, and short replies with low medical relevance. The authors state that future work needs more consistent annotation, finer-grained segment features, and dialogue-level risk rather than looking only at single messages.