Cracks in the Embedding: Robustness and Stability of JEPA-Based Chest X-ray Representations
Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations
The authors test the robustness of four JEPA chest X-ray encoders under six perturbations: CosSim drops to as low as 0.33, adversarial fine-tuning of the predictor improves AUROC, while a VQ codebook trained only on clean images performs worse.
- JEPA-type self-supervised encoders are being used for chest X-ray analysis, but their robustness under real image variations (noise, brightness, contrast) and black-box adversarial attacks lacks measurement, and there is no mature downstream-head repair method after damage.
- Four JEPA chest X-ray encoders are frozen, and only a small residual predictor and linear classifier on top are trained; evaluation is on NIH ChestX-ray14 with two black-box attacks, three image degradations and one few-label test; then two predictors that change only the training method are compared: adversarial fine-tuning on clean + perturbed images, and a VQ codebook trained only on clean images.
- Perturbations cause damage that varies greatly across models: I-JEPA is damaged most (AUROC drops 18.46% under Boundary, CosSim drops to 0.33 under Gaussian Noise); adversarial fine-tuning improves AUROC for most models (I-JEPA averages 0.68); the codebook instead degrades across the board (I-JEPA drops 21.43% on average).
- The authors note that the codebook setup is limited (only 128 entries, trained only on clean images, encoder not updated); because the official weights of X-WIN are not public, it was pretrained by themselves on LIDC-IDRI; the evaluation uses only one dataset, NIH ChestX-ray14, and some results are based on a 1,000-image test subset, and the authors caution that small-sample differences may be due to chance.
- Threat model
- The attacker can only query the deployed model's outputs and cannot access internal gradients (black-box), using SPSA and Boundary Attack; the encoder is frozen throughout, and only the predictor and classifier are trainable.
- Evaluated models
- CheXWorldX-WINRadJEPAI-JEPA
- Benchmarks
- NIH ChestX-ray14LIDC-IDRI
- Metrics
- Macro AUROCEmbedding Cosine Similarity
Four JEPA chest X-ray encoders trained on NIH ChestX-ray14 had a minimum cosine similarity of 0.33 under black-box adversarial attacks and image corruption, and I-JEPA's AUROC dropped by at most 0.12. Adversarial fine-tuning brought embedding consistency to almost all above 0.97, and I-JEPA's AUROC improved by at most 0.10; the Vector-Quantized codebooks trained only on clean images all degraded, and I-JEPA dropped by 0.15 on average.
Paper analysis
What problem does this paper attempt to solve?
The robustness of JEPA chest X-ray encoders under real image perturbations and black-box attacks has not been measured, and there is a lack of repair methods once they are degraded.
Core question: whether JEPA-type self-supervised chest X-ray encoders remain reliable when input images are corrupted or attacked, and how to repair them after degradation, remains an open question.
- Setting and motivation: Chest X-rays vary greatly in brightness, contrast and noise across scanners, hospitals and acquisition conditions; deployed systems may also be probed by attackers who can only query outputs. The authors argue that safety-critical models should degrade gracefully rather than fail silently.
- Gaps in existing work: The authors say REMEDIS, Bundele et al. and others evaluated the data efficiency and generalization of self-supervised medical encoders, but did not measure how representations behave when the input images themselves are corrupted or attacked, nor provide recovery methods after performance degradation.
- The paper's two gaps: First, the robustness of JEPA chest X-ray encoders under real corruption and black-box attacks has never been measured; robustness is often assumed rather than tested. Second, even when AUROC drops are found, there is no mature method for training downstream heads to cope.
- The paper's approach: Freeze four existing JEPA chest X-ray encoders, train only a small predictor and classifier on top, measure embedding changes and classification AUROC under six tests, and compare two training methods that touch only this lightweight head: adversarial fine-tuning and a Vector-Quantised codebook.
- Threat model:
- Attacker goal: Make the model's predictions on perturbed images diverge from those on clean images.
- Knowledge: black-box, able to query only model outputs, with no access to internal gradients (unlike white-box attacks such as FGSM).
- Capability: SPSA estimates the attack direction through random probing; Boundary Attack shrinks the perturbation based only on the model's final decision (80 queries).
- Victim system: A frozen JEPA encoder plus its predictor/classifier, for 14-class chest X-ray disease classification.
What related research exists?
Related work falls into four groups: JEPA models, self-supervised representation learning, evaluation of self-supervised models, and black-box adversarial attacks and robustness.
JEPA models
- LeCun (2022) proposed JEPA: predicting representations of masked regions in embedding space rather than reconstructing pixels. I-JEPA (2023) implements this idea with a Vision Transformer.
- CheXWorld (2025), X-WIN (2026), and RadJEPA (2026) apply the JEPA idea to chest X-rays, learning the general structure of chest X-rays; this paper tests exactly these four encoders.
Self-supervised representation learning
- SimCLR (2020), MoCo (2020), and BYOL (2020) learn representations by comparing augmented views of the same image; Masked Autoencoders (2022) directly reconstruct masked pixels, a design JEPA deliberately avoids.
Evaluation of self-supervised models
- REMEDIS (2023) shows self-supervised pretraining can improve the robustness and data efficiency of medical imaging, but the paper notes it studies only a single representation-learning recipe and does not compare different paradigms.
- Bundele et al. (2024) systematically evaluate medical SSL's label efficiency, cross-dataset generalization, and OOD detection via MedMNIST; the paper notes they do not measure representation behavior under input corruptions or attacks, nor provide correction methods. This paper's few-shot tests (100%/10%/1% labels) follow SimCLR's standard low-shot protocol.
Black-box attacks and perturbation robustness
- White-box attacks FGSM (2014), Carlini & Wagner (2017), and PGD adversarial training (2018) assume access to model internals; this paper uses only query-level black-box attacks SPSA (2018) and Boundary Attack (2017).
- Hendrycks & Dietterich's (2019) common corruption benchmark inspired this paper's noise/brightness/contrast tests; Szegedy et al. (2013) first pointed out neural networks' vulnerability to small input changes.
- VQ-related: Mao et al. (2021), Yang et al. (2023), and He & Singhal (2024) report that VQ representations can improve robustness; this paper tests whether a discrete bottleneck alone, without training on perturbed images, is sufficient.
Positioning: the authors state this paper is the first to connect black-box attack and corruption robustness research to JEPA chest X-ray models, and repairs damage only through two training approaches for the lightweight predictor head (adversarial fine-tuning, codebook) rather than retraining the encoder.
How does the paper address this problem?
Freeze the encoder, train only a residual predictor plus a linear classifier, measure robustness with six tests, then compare two training methods: adversarial fine-tuning and a VQ codebook.
Overall approach: all encoders are frozen throughout, and only a small residual predictor and a linear classifier are trained on top; first train on clean images to obtain the "Original" baseline, then measure the damage caused by perturbations with six tests, and finally compare two methods that change only the predictor training ( adversarial fine-tuning and VQ codebook).
Problem Setting and Architecture
- Predictor: residual network zp = z + W3σ(W2σ(W1 z)), where z is the frozen encoder's embedding and σ is GELU; the residual connection means the predictor only needs to learn small adjustments to z. Weight sizes are D×512, 512×512, 512×D.
- Classifier: a single linear layer outputting probabilities for 14 chest X-ray disease labels.
- Original baseline: the predictor and classifier are trained only on clean images for 300 epochs (learning rate 10⁻², Adam, weight decay 10⁻⁴), then frozen, serving as the control for all robustness tests.
Six Robustness Tests
- Two black-box attacks: SPSA (ε = 0.02/0.05/0.10/0.20) and Boundary Attack (fixed 80 queries); the paper emphasizes using only black-box attacks because real-world attackers on deployed medical systems typically have only query access.
- Three image corruptions: Gaussian Noise (σ = 0.05–0.50), Brightness Shift (factor 0.3–2.2), Contrast Shift (factor 0.2–2.5), each at four intensity levels (Table 1).
- Few-shot tests: no image perturbation; train a fresh linear classifier (no predictor) with 100%/10%/1% training labels to measure the label efficiency of the original embeddings.
- Metrics: Macro AUROC (average over 14 classes) and Cosine Similarity between clean/perturbed embeddings (Equation 2); both are used only for evaluation and are not directly optimized.
Adversarial Fine-tuning
- Procedure: warm-start from the clean-trained weights and train for 200 epochs on clean images plus copies generated with each perturbation's strongest level (learning rate 10⁻³).
- Loss: L = 1/2BCE(y,ŷ) + 1/2BCE(y,ỹ̂) + λ(1 − CosSim(zp, z̃p)), λ=1; the first two terms require correct classification of both clean and perturbed images, and the third constrains the two embeddings to be close.
- Efficiency handling: training caches for SPSA/Boundary are generated only on a large subset of the training set (multiple forward passes are costly), while simple corruptions are generated on the full training set; evaluation still uses the full strongest settings in Table 1.
VQ Codebook Predictor
- Structure: instead of pooling encoder outputs, operate directly on patch tokens (14×14 grid for CheXWorld/X-WIN, 16×16 for RadJEPA/I-JEPA), reduce to 128 channels via 3×3, stride-2 convolution, quantize each spatial position independently to a 128-entry codebook (Equation 5), and pass gradients with a straight-through estimator.
- Initialization and update: the codebook is initialized with k-means cluster centers of 1,000 clean training images, updated during training with EMA (decay 0.99), and rarely used entries are periodically reset; the quantized features are added residually to the pre-quantization features and then passed to a single linear classifier.
- Key setting: trained only on 78,468 clean training images for 300 epochs (batch 128, learning rate 10⁻³ cosine annealed to 10⁻⁵), never exposed to perturbed images during training, with loss BCE plus a commitment term (β=0.25, Equation 7); the question to answer is whether a discrete bottleneck can provide robustness through structure alone.
Data and Models
- Data: official NIH ChestX-ray14 split, 78,468 training images; evaluation uses perturbed versions of the held-out test set (1,000-image subset for Table 2, 3,000 for Table 3).
- Four encoders: CheXWorld, RadJEPA, and I-JEPA use officially released weights; X-WIN's official weights are not public, so the authors pretrained it themselves on the LIDC-IDRI CT dataset for 100 epochs. I-JEPA is the only encoder pretrained on natural images (ImageNet).
What experiments did the paper conduct?
Under the six tests, all four encoders are damaged and the degree varies by model; adversarial fine-tuning is mostly effective, the codebook degrades across the board; under few-shot, I-JEPA's embeddings are best.
Experimental Setup
- Models tested: four frozen JEPA encoders—CheXWorld, X-WIN, RadJEPA, I-JEPA—each paired with an independently trained predictor/classifier.
- Dataset: NIH ChestX-ray14 (78,468 training images); X-WIN pretrained on LIDC-IDRI.
- Test conditions: SPSA, Boundary Attack, Gaussian Noise, Brightness, Contrast, each at its strongest level, plus few-shot tests (Table 1).
- Metrics: Macro AUROC and embedding CosSim; test set size varies by experiment—Table 2 and Figures 3/4 use a 1,000-image held-out subset, Table 3 uses 3,000.
- Baselines: an "Original" predictor trained only on clean images; the two comparison variants are an adversarially fine-tuned predictor and a 128-entry codebook predictor.
- Repeats: main results are single-seed (42); Section 5.5 repeats adversarial fine-tuning with three other seeds (123, 456, 789).
Main Results
Per Table 2 (Original predictor, strongest level, 1,000-image test subset, values in parentheses are AUROC drops relative to clean images):
Swipe horizontally to read the table
Test CheXWorld X-WIN RadJEPA I-JEPA Clean 0.58 0.63 0.63 0.65 SPSA (0.20) 0.58 (0.00%) 0.63 (0.00%) 0.63 (0.00%) 0.60 (7.69%) Boundary (80 q) 0.57 (1.72%) 0.63 (0.00%) 0.63 (0.00%) 0.53 (18.46%) Gaussian Noise (0.50) 0.57 (1.72%) 0.62 (1.59%) 0.64 (−1.59%) 0.54 (16.92%) Brightness (2.2) 0.50 (13.79%) 0.57 (9.52%) 0.53 (15.87%) 0.63 (3.08%) Contrast (2.5) 0.47 (18.97%) 0.59 (6.35%) 0.59 (6.35%) 0.62 (4.62%) Average 0.54 (6.90%) 0.61 (3.17%) 0.60 (4.76%) 0.58 (10.77%) - Authors' interpretation: damage varies by model and test; I-JEPA is damaged most; CheXWorld, X-WIN, and RadJEPA are nearly unchanged under SPSA and Gaussian Noise, while I-JEPA's Gaussian-Noise CosSim drops to 0.33 (Figure 4).
- Exception: RadJEPA's AUROC slightly increases under Gaussian Noise (0.63→0.64); the authors explicitly state this does not mean noise is beneficial, and with 1,000 test images such a difference may be chance.
- Non-monotonic phenomenon: under Brightness and Contrast, embedding similarity is often lowest at the mildest level (CheXWorld and X-WIN have CosSim of 0.39 and 0.41 at Level 1, respectively), while AUROC keeps decreasing to Level 4.
Codebook Tests
- What was tested: AUROC of the 128-entry codebook predictor (trained only on clean images) under the strongest-level perturbations (Table 3, 3,000-image test subset).
- Results: all four models degrade, with average drops of CheXWorld 10.00%, X-WIN 13.56%, RadJEPA 9.38%, I-JEPA 21.43%; I-JEPA drops 24.29% under Gaussian Noise. Note that the codebook's clean AUROC differs from Original (e.g., CheXWorld 0.70 vs 0.58), and drops are computed relative to each model's own clean baseline.
- Authors' interpretation: the discrete bottleneck alone is insufficient to resist unseen perturbations; the authors suggest the gap from prior VQ robustness results [15, 27] may stem from their more restricted setting—the codebook is trained only on clean images, sits inside a small predictor, has only 128 entries, and the encoder is not updated.
Adversarial Fine-tuning
- What was tested: AUROC of predictors trained on a mix of clean and perturbed images under the strongest level (Table 4).
- Results: average AUROC across the five tests is CheXWorld 0.62, X-WIN 0.63, RadJEPA 0.62, I-JEPA 0.68; the authors state three of the four models improve, with the most-damaged I-JEPA improving most (up to +0.10 relative to the perturbed baseline in Table 2), and RadJEPA essentially unchanged.
- Authors' interpretation: training the predictor jointly on clean and perturbed images improves both classification and embedding consistency; the abstract states embedding consistency rises above 0.97 in almost all cases.
Few-shot and Seed Stability
- Few-shot (Figure 5): I-JEPA's AUROC at 100%/10%/1% labels is 0.75/0.73/0.65, best at every budget; X-WIN is highest among medical encoders at full labels with 0.67, but drops to 0.55 at 1%, while CheXWorld and RadJEPA are both 0.60 at 1%. The authors state the general-purpose encoder transfers surprisingly well; fine-tuning or codebook training does not reliably improve low-label AUROC.
- Seed stability (Figure 6): X-WIN and RadJEPA are almost unaffected by seed (std ≤ 0.001); I-JEPA std = 0.006 (mean 0.671 vs 0.677 for seed 42); CheXWorld is most sensitive (std = 0.014, three-seed mean 0.579 below 0.616 for seed 42). The authors state this supports that the fine-tuning gains are not an artifact of a single initialization, but CheXWorld's results should be interpreted in light of its seed sensitivity.
Other Ablations and Analyses
- No other independent ablations; full curves for the four intensity levels of each perturbation are in Figure 3 (AUROC) and Figure 4 (CosSim), and the main text gives only the strongest-level summary.
What are some points that can be further explored?
The authors note that the codebook setup is limited and that X-WIN was pretrained by themselves; the evaluation is concentrated on a single dataset and black-box attacks.
Limitations and Future Directions Noted by the Authors
- Restricted codebook setting (Section 5.2): the authors state the codebook results fall short of prior VQ work, possibly because it is trained only on clean images, sits inside a small predictor rather than a large trainable backbone, has only 128 entries, and the encoder is never updated.
- Chance in small-sample tests (Section 5.1): the authors point out that RadJEPA's slight AUROC increase under Gaussian Noise does not mean noise is beneficial, and with 1,000 test images a difference of this magnitude may be due to chance.
- CheXWorld's seed sensitivity (Section 5.5): the authors state CheXWorld's fine-tuning results should be interpreted in light of its higher seed sensitivity (std = 0.014).
- X-WIN weights unavailable (Section 4): X-WIN's official pretrained weights are not public, and its original training data NLST could not be used due to compute constraints, so the authors instead pretrained on LIDC-IDRI for 100 epochs, differing in source from the other three encoders with official weights.
- The paper has no dedicated Limitations or Future Work section; the above are limitations explicitly stated in the main text.
Scope of Experiments
- The models tested are four JEPA encoders: CheXWorld, X-WIN, RadJEPA, I-JEPA, all frozen, with only the predictor/classifier trained.
- Attack evaluation includes only two black-box attacks (SPSA, Boundary Attack); the paper explicitly states it does not use white-box attacks such as FGSM, on the grounds that real-world attackers have only query access.
- Corruption tests are Gaussian Noise, Brightness, and Contrast, each at four intensity levels; the only evaluation dataset is NIH ChestX-ray14.
- Main result tables are based on 1,000-image (Table 2) or 3,000-image (Table 3) test subsets; the seed check for adversarial fine-tuning uses three additional seeds, and other results are single-seed (42).
- The paper does not report the actual number of queries per attack (except Boundary's fixed 80), SPSA's step budget, or training/evaluation time.
Summarize the main content of the paper
All four JEPA chest X-ray encoders are non-robust under perturbations and the degree varies by model; adversarially fine-tuning the predictor is mostly effective, while a pure codebook is ineffective.
An empirical study on the robustness of self-supervised medical imaging models: the authors measure the stability of four JEPA-type chest X-ray encoders under real image variations and black-box attacks, and compare two repair methods that train only downstream lightweight heads.
- Problem: JEPA encoders are used for chest X-ray analysis, but their robustness under corruptions and black-box attacks has never been measured, and there is a lack of repair methods that do not retrain the encoder after damage.
- Method: freeze the encoder, train a residual predictor plus a linear classifier; evaluate on NIH ChestX-ray14 with SPSA, Boundary Attack, Gaussian Noise, Brightness, Contrast (each at multiple intensity levels) and few-label tests, with Macro AUROC and embedding CosSim as metrics.
- Damage is widespread and varies greatly across models: at the strongest level, average AUROC drops range from 3.17% for X-WIN to 10.77% for I-JEPA (Table 2); I-JEPA drops 18.46% under Boundary, and its CosSim under Gaussian Noise is as low as 0.33.
- Adversarial fine-tuning is effective: after training the predictor on a mix of clean and perturbed images, I-JEPA's average AUROC across the five tests reaches 0.68, three of the four models improve, and the conclusion holds under three additional seeds (CheXWorld has higher seed sensitivity).
- Codebook is ineffective: a 128-entry VQ codebook trained only on clean images further degrades all models, with I-JEPA dropping 21.43% on average; the authors state the discrete bottleneck alone is insufficient to provide robustness.
- Few-shot results: I-JEPA's original embeddings are best at every label budget (0.75 at 100%), the authors state the general-purpose encoder transfers well, and perturbation-robustness training should not be assumed to help label efficiency.
- Authors' conclusion: robustness cannot be assumed from pretraining or architecture and must be measured directly; training against perturbations is more reliable than a purely structural discrete bottleneck.