SCIB: Robust Representation Learning for Spiking Neural Networks Based on the Conditional Information Bottleneck

1. 01What problem does this paper attempt to solve?

   The authors argue that standard SNNs retain label-conditioned input information, so they use a conditional information bottleneck for robust training.

   Standard-trained SNNs retain a large amount of input-related representation after label-prediction information stabilizes, making them susceptible to environmental noise, hardware noise, and adversarial perturbations at deployment.

   • Setting: The authors argue that SNNs perform event-driven computation with discrete spikes and can be deployed on Intel Loihi, IBM TrueNorth, and Tianjic. Environmental noise, device non-idealities, and substitute-gradient adversarial attacks perturb spike timing, membrane potentials, or input spike patterns.
   • Existing shortcomings: The authors group prior work by perturbation source: encoding optimization, constructed noise, frequency-domain encoding, stochastic neurons/synapses, and spectral/gradient regularization and adversarial training. The authors argue that these are often tied to a specific noise or attack, and that adversarial training additionally depends on attack samples and inner optimization.
   • Observation: On the information plane in Figure 1, for standard SNNs, after I(S;Y) is relatively stable, I(X;S) remains in a high-information region or rises slowly. Under Y↔X↔S, I(X;S)=I(S;Y)+I(X;S|Y), so directly penalizing I(X;S) may compress discriminative information along with it.
   • Proposal: The authors propose SCIB, which uses the conditional mutual information I(X;S|Y) as the compression target and a learnable class-conditional prior, as an objective-level regularizer that does not change the architecture or neuron dynamics.
   • Threat model:
     • Evaluation target: Image classification accuracy under three types of heterogeneous perturbations. Adversarial robustness is regarded by the authors as a side effect of representation regularization; training does not use adversarial samples.
     • Knowledge: Adversarial experiments use substitute gradients and BPTT to approximate non-differentiable spikes. The paper does not use the terms white-box or black-box.
     • Applicable perturbations: Environmental noise is added to pixels; hardware noise is added to synaptic updates or membrane potentials; adversarial perturbations are limited to L∞, ε∈{4/255, 8/255}, with methods FGSM, PGD, BIM.
     • Victim system: An SNN classifier trained directly with substitute gradients, with a VGG-9 LIF backbone and default time steps T=4.

   Collapse this answer

2. 02What related research exists?

   The authors distinguish SCIB from defenses against specific perturbations and from conventional IB that directly compresses I(X;S).

   SNN Robustness and Training

   • Training routes: The authors list ANN-to-SNN conversion, membrane-potential-based learning, and direct training with substitute gradients (Wu et al., 2019; Neftci et al., 2019). The authors state that substitute gradients have become the main training method for deep SNNs, but their differentiable approximation also makes the network susceptible to gradient-based adversarial attacks.
   • Methods by perturbation source: Discrete input encoding is reported to have partial intrinsic robustness (Sharmin et al., 2020); HIRE-SNN uses constructed input noise (Kundu et al., 2021); FEEL-SNN uses frequency encoding and evolvable leak factors (Xu et al., 2024). On the hardware side, there are stochastic gating, stochastic heterogeneous neurons, and dynamic synaptic noise. On the adversarial side, there are SNN-RAT's Parseval regularization and hybrid adversarial training, as well as sparse gradient methods.
   • The authors' grouped critique: The authors argue that these methods are mostly tied to a specific perturbation model, attack process, or structural modification, so there remains room for representation-level learning objectives. This is the authors' view of this group of work, not the conclusions of the cited papers themselves.

   Information Bottleneck in Deep Networks

   • IB and variational implementations: Tishby et al. (2000; 2015) describe minimal sufficient representations with I(S;Y)−β I(X;S). VIB (Alemi et al., 2016) uses variational inference to make high-dimensional networks optimizable. The authors also list nonlinear IB, supervised IB, information bottleneck distillation, and gated IB as extensions to generalization, adversarial robustness, distillation, and sequential decision-making.
   • Conditional form: Conditional Entropy Bottleneck (Fischer, 2020) formalizes supervised compression with the form I(Z;X|Y). Tang and Yu (2025) use variational IB and conditional mutual information to extract shared representations from two dependent sequences.
   • Information plane: Shwartz-Ziv and Tishby (2017), Geiger (2021), and others are used by the authors as empirical tools for observing the training trajectories of I(X;S) and I(S;Y).

   IB in the Spike Domain

   • SNIB (Yang and Chen, 2023): Applies a nonlinear information bottleneck regularizer to spike learning, constrains the dependence between input and representation, and reports improved SNN robustness. SIB and QIB in this paper's noise baselines are the squared and quartic variants of this framework.
   • SeLHIB (Yang et al., 2024): Applies a higher-order information bottleneck to self-supervised event optical flow. The authors state that these two lines of work mainly follow conventional compression forms and have not yet systematically addressed conditional redundancy and class-structure variational priors in spike representations.

   Baselines and Datasets

   • Noise comparisons: w.o. IB, IB (prior N(0,I)), NBP-SNN, SIB, QIB; datasets are Fashion-MNIST, CIFAR-10, CIFAR-100.
   • Adversarial comparisons: Vanilla SNN, FEEL-SNN; datasets are CIFAR-10, CIFAR-100.

   The authors position SCIB as an objective-level regularizer that does not change the architecture and does not depend on a specific noise model: the compression target is changed from I(X;S) to I(X;S|Y), and a learnable class-conditional prior replaces the label-independent isotropic Gaussian.

   Collapse this answer

3. 03How does the paper address this problem?

   SCIB preserves prediction information with cross-entropy and compresses I(X;S|Y) with a KL upper bound on class-conditional Gaussians.

   SCIB does not change the SNN structure or LIF dynamics; it only adds conditional compression to substitute-gradient training: cross-entropy preserves label-prediction information, and the KL term compresses the input information that remains after conditioning on the label.

   Problem Setting

   • Markov chain: During training, Y↔X↔S, with S generated from X; at inference, Y is no longer read, so I(S;Y|X)=0. The chain rule is written as I(X;S)=I(S;Y)+I(X;S|Y).
   • Minimal sufficient representation: The authors require I(S*;Y)=I(X;Y) and minimize residual input information among sufficient representations. When I(X;S*|Y)=0, I(X;S*)=I(X;Y).
   • Two limitations: Directly penalizing I(X;S) does not distinguish the prediction term from label-conditioned redundancy; standard VIB commonly uses N(0,I), pulling all class posteriors toward the same anchor. Figure 2 depicts SCIB's compression target as the residual term after conditioning.
   • Encoder: The basic unit is LIF, V_t=τ V_{t-1}+W·S_{t-1}, S_t=H(V_t−V_th). Integrating over T steps yields the stochastic encoding distribution p_θ(s|x).

   Objective and Variational Bounds

   • Information objective: Maximize 𝒥_SCIB=max I(S;Y)−β· I(X;S|Y). β controls the strength of conditional compression.
   • Compression upper bound: Since p_θ(s|x,y)=p_θ(s|x), the conditional mutual information is written as the expected KL of the posterior against the true p(s|y). p(s|y) is unavailable, so a variational prior r_φ(s|y) gives a computable upper bound; the authors state that the bound is tight when r_φ=p.
   • Prediction lower bound: A classifier q_ψ(y|s) approximates p(y|s). The dataset label entropy is constant during training, so maximizing this lower bound is equivalent to minimizing cross-entropy.
   • Surrogate loss: Minimize ℒ_SCIB=ℒ_CE+β𝔼[D_KL(p_θ(s|x)|r_ϕ(s|y))].

   Learnable Class-Conditional Prior

   • Parameterization: Each of the K classes has mean μ_k and diagonal variance σ_k², r_φ(s|y=k)=N(s; μ_k, diag(σ_k²)). After marginalizing over labels, the authors interpret this as a Gaussian mixture.
   • Posterior and KL: The encoder outputs μ_x, σ_x², sampled via reparameterization. Both sides are diagonal Gaussians, so the KL has a closed form and does not rely on high-variance Monte Carlo.
   • Role stated by the authors: Pull sample posteriors toward the class center and constrain variance; different classes have independent anchors, which together with cross-entropy maintains inter-class separability.

   Optimization and Inference

   • Procedure: Figure 3 and Algorithm 1: input is encoded over T steps, the label indexes the prior during training, cross-entropy and KL are summed, followed by BPTT. θ, ψ, φ are updated jointly.
   • Default settings: SGD, momentum 0.9, initial learning rate 0.1, cosine annealing, batch size 128. β defaults to 0.4; hardware noise experiments use 0.2. The prior is initialized with mean ~ N(0, 0.01I) and log-variance ~ N(0, 0.01I).
   • Inference: The class-conditional prior is not used; prediction goes through the standard forward SNN.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors state that SCIB has higher accuracy on most noise and the evaluated iterative attacks, and that no adversarial training was done.

   Experimental Setup

   • Implementation: PyTorch 1.12, Python 3.9, Ubuntu 20.04, NVIDIA A100. SGD momentum 0.9, initial learning rate 0.1, cosine annealing, batch size 128.
   • Network and data: backbone is a VGG-9 SNN, 7 Conv+BN+LIF plus 1 fully connected, LIF, default T=4; adversarial experiments also test T=8. Datasets are Fashion-MNIST, CIFAR-10, CIFAR-100. The paper does not state sample sizes.
   • Baselines: noise group is w.o. IB, IB (prior N(0,I)), NBP-SNN, SIB, QIB. Adversarial group is Vanilla SNN and FEEL-SNN; the latter has r=[16, 14, 12, 10] at T=4, and additionally 8, 6, 4, 2 at T=8.
   • Perturbations: external noise is salt-and-pepper, Gaussian, Rayleigh, each with two NNI levels. Hardware noise is synaptic weight noise and membrane potential noise, NNI=1 and 5, with β=0.2 for this group. Adversarial is FGSM, PGD, BIM under L∞, ε∈{4/255, 8/255}; PGD/BIM step size 0.01, 7 steps.
   • Metrics: classification accuracy. Energy is a SynOps analytical proxy, using 45 nm estimates, MAC 4.6 pJ, AC 0.9 pJ, counting only computation energy. No LLM review, no human agreement. The paper reports no number of repetitions or statistical tests.
   • SCIB: default β=0.4, training does not use adversarial examples.

   Main Results

   The main adversarial table covers only CIFAR-10, T=4. Accuracy (%), per Table 2.

   Swipe horizontally to read the table

   Method	Clean	FGSM ε=4/255	PGD ε=4/255	BIM ε=4/255	PGD ε=8/255
   Vanilla SNN	92.94	19.63	0.78	0.67	0.02
   FEEL-SNN	88.07	39.17	21.33	21.10	1.73
   SCIB	89.69	52.08	47.58	47.73	26.65

   • The authors state that on CIFAR-10, T=4, relative to Vanilla, PGD/BIM (ε=4/255) is higher by 46.80 and 47.06 percentage points, and relative to FEEL-SNN higher by 26.25 and 26.63 percentage points; at ε=8/255 relative to FEEL-SNN it is 24.92 and 24.66 percentage points (Section 5.4).
   • The authors state that these numbers are incidental gains from representation regularization, and should not be taken to mean SCIB beats dedicated adversarial training defenses. SCIB training does not use adversarial examples.
   • Exception: on CIFAR-100, T=4, ε=4/255 FGSM, FEEL-SNN is 19.51% and SCIB is 14.08% (Table 2). In every Clean cell of Table 2, Vanilla is higher than SCIB in the same cell.

   External Noise

   • Salt-and-pepper: at NNI=0.3, SCIB is the highest in its row on all three datasets, CIFAR-100 at 58.40%; the authors state it is 0.91 percentage points higher than SIB (Table 1). On CIFAR-100 at NNI=0.1, SIB 63.27%, QIB 62.92%, SCIB 62.91%.
   • Gaussian: at NNI=1.0, SCIB is the highest in its row on all three sets, Fashion-MNIST 89.86%, CIFAR-10 81.78%, CIFAR-100 51.32%. On CIFAR-10 at NNI=0.25, IB is 88.52%, higher than SCIB's 88.44%.
   • Rayleigh: the authors state that on CIFAR-10 and CIFAR-100 SCIB is highest across the four levels. On CIFAR-100 at NNI=1.0, SCIB is 46.60%, the unconstrained baseline 45.91%, IB 45.39%; the authors state IB is 0.52 percentage points lower than the unconstrained baseline. On Fashion-MNIST at NNI=1.0, IB 85.07% is higher than SCIB 84.99%. The authors interpret the comparison on CIFAR as the class-conditional prior being better suited to this asymmetric noise than a standard Gaussian independent of the label.

   Hardware Noise and Energy

   • Membrane potential noise: the main text gives, on CIFAR-10 at NNI=1, SCIB 88.18% and IB 88.20%; at NNI=5, IB goes from 88.20% to 64.08%, and SCIB is 64.45% (Section 5.3). The authors state that on CIFAR-100 it is highest at both levels, but do not give the accuracies for these two levels. Figure 4 shows training curves; the main text does not list all endpoints.
   • Synaptic noise: the authors state it is highest or near highest in most configurations. On CIFAR-100 at NNI=5, SCIB is 62.98%, which the authors state is about 0.9 percentage points above the second place; the paper does not give the second-place value (Section 5.3, Figure 5).
   • Energy: single-sample SynOps proxy on CIFAR-10 is ANN 9.02×10^8 pJ, w.o. IB 5.31×10^8 pJ, IB 6.11×10^8 pJ, SIB 4.10×10^8 pJ, SCIB 3.99×10^8 pJ (Section 5.5). The authors state about 55.8% lower than ANN and 24.9% lower than w.o. IB. In Table 3, the firing rates of Conv#6 and Conv#7 are 2.4516 and 2.6551.

   Other Ablations and Analyses

   • The information plane compares only the standard SNN and conventional IB, on Fashion-MNIST and CIFAR-10; the authors state that the standard SNN shows no compression phase, while in conventional IB, I(X;S) falls in the middle-to-late stage while I(S;Y) stays relatively high (Section 3, Figure 1). The main text gives no trajectory endpoint values.
   • Prior visualization is on Fashion-MNIST: the authors state that at Epoch 0 the means are within about ±0.5 and the ellipses overlap, after Epoch 80 they separate in PCA, and at Epoch 120 they are tighter; at Epoch 0 the off-diagonal KL is < 0.5 nats (Figure 7). The neighbor relations of top and shoe classes are the authors' reading of the figure; the text gives no projection coordinates.
   • The β sweep is only on CIFAR-10, VGG-9: Gaussian NNI is 0.25, 0.50, 0.75, 1.00, membrane potential NNI is 2, 4, 6, 8, and β is 0.2, 0.4, 0.6, 0.8, 1.0. The authors state that under Gaussian noise it is insensitive to β, while under membrane potential noise accuracy decreases monotonically as β increases (Figure 8). The main text gives no per-cell accuracies.
   • Hardware-noise curves are trained for 200 epochs, and the figure plots only the first 150 epochs (Section 5.3).
   • T=8, CIFAR-10, ε=4/255: SCIB's PGD/BIM is 44.88%/45.10%, FEEL-SNN is 17.67%/17.71%, Vanilla is 0.56%/0.52% (Table 2).
   • T=8, CIFAR-100 Clean: FEEL-SNN 61.68%, SCIB 61.53%; in the same setting at ε=8/255, PGD/BIM, SCIB is 7.55%/7.18%, Vanilla is 0.02%/0.00% (Table 2).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that it is not optimized against adversarial samples, and they plan hierarchical priors and chip validation.

   Limitations and Future Directions Noted by the Authors

   • Not optimized against adversarial samples: The authors state that, therefore, robustness may still be limited under some single-step attacks (Section 6).
   • Prior form: Currently one independent Gaussian component per class; the authors believe datasets with many classes and fine granularity may require more careful design (Section 6).
   • Hierarchy and sample efficiency: The authors plan to explore more sample-efficient and hierarchical prior structures (Section 6).
   • Combination with adversarial training: The authors plan to study complementary combinations of SCIB and adversarial training (Section 6).
   • Hardware validation: The authors plan to validate the framework on physical neuromorphic hardware (Section 6).

   Scope of Experiments

   • Classification and noise experiments use a VGG-9 SNN, LIF, default T=4, with datasets Fashion-MNIST, CIFAR-10, CIFAR-100 (Section 5.1).
   • Noise comparisons are w.o. IB, IB, NBP-SNN, SIB, QIB; adversarial comparisons are Vanilla SNN and FEEL-SNN, reported only on CIFAR-10 and CIFAR-100 at T=4 and T=8 (Table 1, Table 2).
   • Adversarial settings are L∞, FGSM/PGD/BIM, ε∈{4/255, 8/255}, with PGD/BIM step size 0.01 and 7 steps; SCIB training does not use adversarial samples (Sections 5.1, 5.4).
   • Energy figures come from the SynOps analytical model for 45 nm MAC/AC, counting only computation energy (Section 5.5). The paper does not report the number of repetitions, variance, or statistical tests.
   • The information plane is for standard SNNs and conventional IB on Fashion-MNIST and CIFAR-10; prior visualization is on Fashion-MNIST; the β sweep is on CIFAR-10 (Sections 3, 5.6).

   Collapse this answer

6. 06Summarize the main content of the paper

   SCIB regularizes an SNN with a conditional mutual information upper bound on class-conditional priors; the author claims greater robustness under most perturbations and fewer spikes.

   SCIB is a conditional information bottleneck added to the SNN training objective, used to address environmental noise, hardware noise, and gradient adversarial perturbations, without changing the LIF or network structure.

   • Problem: the authors believe standard SNNs retain a large amount of input-related information after label information stabilizes; under Y↔X↔S, directly compressing I(X;S) compresses both the prediction term and the residual term conditioned on the label. A conventional isotropic Gaussian prior also does not distinguish classes.
   • Approach: the ideal objective is to maximize I(S;Y)−β I(X;S|Y). It is implemented as cross-entropy plus a closed-form KL from the encoder posterior to a learnable class-conditional Gaussian prior. Labels index the prior only during training; inference uses the ordinary forward pass. Default β=0.4, and 0.2 for hardware-noise experiments.
   • External noise: Table 1 has 18 configurations in total. The authors state that in most cases SCIB is higher than the no-IB and IB-type baselines. Cases in the table that are not the highest in their row include: CIFAR-100 with salt-and-pepper NNI=0.1 (SCIB 62.91%, SIB 63.27%), CIFAR-10 with Gaussian NNI=0.25 (IB 88.52%, SCIB 88.44%), and Fashion-MNIST with Rayleigh NNI=1.0 (IB 85.07%, SCIB 84.99%).
   • Adversarial: CIFAR-10, T=4, ε=4/255, SCIB's PGD/BIM is 47.58%/47.73%, Vanilla is 0.78%/0.67% (Table 2). The authors state that training did not use adversarial examples, and this should not be taken as evidence of beating dedicated adversarial training. On CIFAR-100, T=4, ε=4/255 FGSM, FEEL-SNN's 19.51% is higher than SCIB's 14.08%. In every Clean cell, Vanilla is higher than SCIB.
   • Energy: single-sample SynOps proxy on CIFAR-10, SCIB is 3.99×10^8 pJ, w.o. IB is 5.31×10^8 pJ. The authors state a 24.9% reduction relative to the unconstrained SNN and about 55.8% relative to the ANN.
   • Authors' conclusion: conditional compression can, in the evaluated settings, both improve robustness and reduce spiking activity. The authors also write that it may still be limited under single-step attacks, that independent Gaussian priors for fine-grained multiple classes need redesign, and that physical chip validation is left for future work.

   Collapse this answer

Related papers