Skip to content

Breaking Fair Image Search with Generative Adversarial Perturbations

Subverting Fair Image Search with Generative Adversarial Perturbations

arXiv· arXiv:2205.02414· FAccT 2022· Avijit Ghosh, Matthew Jagielski, Christo WilsonNortheastern University; Google ResearchIndexed by this site · 23 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose attacking the reranking algorithm of fair image search with generated adversarial perturbations (GAP), and the authors state that under a strict threat model it can bias results toward light-skinned men while barely affecting relevance.

Threat modelThe attacker is a malicious image curator who cannot poison the training data, does not know the victim search engine's retrieval model or fairness algorithm, and can only add perturbed images to the database after model training; the paper also considers the stronger case where the search engine is fully compromised.

Questions
After a ranking model is calibrated to achieve fairness, can an external attacker make it behave unfairly without access to the model and training data? The authors study this question using fair image search as a case study.
Methods
The authors build an image search engine based on MMT retrieval and FMMR fair re-ranking, and train a Class-Targeted GAP against demographic inference models (Deepface, FairFace) so that images are misclassified, thereby misleading the re-ranking.
Experiments and results
The authors state the attack increases the Skew and Attention unfairness toward light-skinned men across multiple variables, with attention unfairness increasing by over 50% at pr=100% for some k, while NDCG changes by no more than 0.7%.
Limitations and future work
The authors note: only 4 groups covering two races and two genders are included, the dataset is smaller than a real search engine, the attack is tuned for FairFace and Deepface, it can only create an advantage for the already dominant group, and the effect varies across queries (§6.1).
Experimental setupFMMR、DetConstSort et al. · MS-COCO(Zhao et al. 标注子集,8,692 张单人图像) · Skew、Attention et al.
Evaluated models
FMMRDetConstSortFaster R-CNNInceptionV3ResNet18
Benchmarks
MS-COCO(Zhao et al. 标注子集,8,692 张单人图像)
Metrics
SkewAttentionNDCGAttack Effectiveness η
AI summaryThe researchers propose and demonstrate an attack that undermines fair image search: a generative adversarial perturbation (GAP) model maliciously modifies images so that a fair reranking algorithm unfairly boosts the ranking of images of people from an attacker-chosen subgroup. The attack targets a state-of-the-art fairness-aware image search engine that they built, and it still succeeds under a strict threat model in which the attacker has no access to the model or the training data. Experiments show that, compared with the fair ranking baseline, the attack can give a significant unfair advantage to the majority group and remains robust across multiple variables, with almost zero impact on search result relevance. The authors point out that deploying fair machine learning algorithms in real-world settings is dangerous when the data needed to achieve fairness may be adversarially manipulated and the models themselves are not robust.

The researchers propose and demonstrate an attack that undermines fair image search: a generative adversarial perturbation (GAP) model maliciously modifies images so that a fair reranking algorithm unfairly boosts the ranking of images of people from an attacker-chosen subgroup. The attack targets a state-of-the-art fairness-aware image search engine that they built, and it still succeeds under a strict threat model in which the attacker has no access to the model or the training data. Experiments show that, compared with the fair ranking baseline, the attack can give a significant unfair advantage to the majority group and remains robust across multiple variables, with almost zero impact on search result relevance. The authors point out that deploying fair machine learning algorithms in real-world settings is dangerous when the data needed to achieve fairness may be adversarially manipulated and the models themselves are not robust.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Once a fair ranking model is calibrated, can an external attacker make it unfair without access to the model and training data?

    Core question: when a ranking model has been calibrated to satisfy some fairness definition, can an external attacker make the ranking model behave unfairly without access to the model or training data? The authors use fair image search as a case study for this question.

    • Scenario and importance: fair ML systems are being deployed by companies in real scenarios, and such systems mostly rely on demographic data to control bias; the authors believe that if this data can be adversarially manipulated, fairness guarantees may be "weaponized".
    • Gaps in existing work: prior work on adversarially attacking fairness is mostly at the training stage (data poisoning); test-stage attacks targeting fairness are relatively unexplored in ranking scenarios (§2.2).
    • Core observation: fair reranking algorithms such as FMMR do not explicitly use demographic labels, but perform indirect demographic inference based on image embeddings, so perturbations trained against demographic inference models may transfer to the reranking algorithm.
    • The paper proposes: an attack on fair image search—using generated adversarial perturbations (GAP) to modify images so that the fair reranking algorithm unfairly boosts the ranking of an attacker-chosen subgroup (e.g., light-skinned men); and builds a complete search engine for case validation.
    • Threat model (§3.1):
      • Goal: perturb images in one's own database to break the fairness guarantees of a downstream retrieval system.
      • Knowledge: does not know the internal details of the ranking system (retrieval model, other images in the index, the fairness algorithm used).
      • Capability: cannot poison the training data; can only add images to the victim database after the retrieval model is trained; the paper also considers the stronger case where the search engine is compromised and the attacker gains the model and all data.
      • Victim system: an image search engine using the MMT retrieval model and FMMR fair reranking.
  2. What related research exists?

    Related work is divided into three groups: fair ranking, adversarial ML, and adversarial attacks against fairness; the authors state that test-time attacks on fair ranking have not yet been explored.

    Fair ranking

    • Celis et al. [28], Zehlike et al. [98]: Proposed fair ranking under a two-group setting; later Singh and Joachims [86] solved ranking fairness using constrained learning.
    • Geyik et al. [38] (LinkedIn): Proposed the post-processing re-ranking algorithm DetConstSort to achieve representational parity, deployed in LinkedIn talent search.
    • Karako and Manggala [51] (Shopify): Proposed FMMR, which performs fair re-ranking based on embeddings without explicitly using demographic labels; the main target of the attack in this paper.
    • Ghosh et al. [39]: The paper notes that they show demographic attribute inference errors can break fairness guarantees in IR; this paper follows their DetConstSort evaluation pipeline.

    Adversarial machine learning

    • Adversarial examples and defenses: Szegedy et al. [89], Carlini and Wagner [24], and others show that models are vulnerable to malicious inputs; various defenses have been broken [8, 91], and adversarial training is computationally expensive and reduces standard accuracy [35, 54, 84].
    • Attacks on IR systems: Liu et al. [61], Zhou et al. [101] attack graph-to-graph search; Li et al. [57] attack image retrieval with universal adversarial perturbations; Raval and Verma [77] attack text retrieval; He et al. [46] attack personalized ranking. The paper notes that none of these works consider text-to-image models or group fairness objectives.

    Adversarial attacks against fairness

    • Training-time attacks: Jagielski et al. [48], Mehrabi et al. [64], Chang et al. [29], Solans et al. [87] harm the accuracy of specific subgroups or exacerbate unfairness by poisoning training data.
    • Test-time attacks: Nanda et al. [71] show that adversarial attacks cause greater harm to some subgroups in classification tasks; the paper notes that their harm concerns only inequality among adversarially corrupted samples, whereas this paper shows that test-time attacks can harm the fairness of benign data in ranking scenarios.

    Positioning

    The authors state that this paper examines the test-time robustness of Shopify's and LinkedIn's fair ranking algorithms, a setting not previously studied.

  3. How does the paper address this problem?

    Builds a fair image search engine with MMT+FMMR, and trains CGAP against demographic inference models so that images are misclassified, misleading fair re-ranking.

    Overall approach: first build a realistic fair image search engine as the victim system, then train a Class-Targeted Generative Adversarial Perturbation (CGAP) model that generates human-imperceptible perturbations against demographic inference models, causing the fair re-ranking algorithm to misjudge the demographic attributes of images.

    Victim system: image search engine (§3.2)

    • Retrieval model: MultiModal Transformer (MMT) [37], which first uses embeddings for fast retrieval (cosine distance to take a candidate set), then re-scores candidates with a joint model; MMT itself contains no fairness objective and is not the attack target.
    • Fair re-ranking: FMMR [51], which changes MMR's "similarity" heuristic into a demographic difference, making the next selected item as different as possible in embedding from already selected items; it does not require explicit demographic labels but can be viewed as performing indirect demographic inference. The trade-off parameter λ is fixed at 0.14 (following the FMMR original source).
    • Embedding models: FMMR's similarity is computed using embeddings generated by three pretrained models: Faster R-CNN, InceptionV3, and ResNet18.

    Attack construction (§3.3)

    • Attack target models: Two demographic inference models, Deepface [90] (race and gender, public wrapper fine-tuned on about 22,000 samples) and FairFace [52] (trained on 108,000 images); race labels are mapped to skin tone per the paper (White, East Asian, Middle Eastern→Light; Black, South Asian, Hispanic→Dark).
    • Why GAP: Per-image optimization is infeasible at database scale; GAP training is a one-time cost, after which only forward passes are needed to perturb large numbers of images; the authors consider that UAP requires all images to be the same size, which is unrealistic for real image collections.
    • CGAP definition: Given a loss ℓ, target model f_targ, source class y_s, and target class y_t, CGAP minimizes ℓsCGAP(D) = E[ℓ(fCGAP(x), yt; ftarg) | y = ys], i.e., misclassifying source-class samples into the target class while preserving classification performance on non-source-class samples. Extensions include multiple target classes (the extreme being assigning all samples to one class) and untargeted attacks.
    • Key design rationale: Merely making the inference model err does not necessarily affect fairness—if the two groups swap labels, the fair ranking result is unchanged; therefore the attack must be directionally targeted at subgroups.

    Experimental setup (§4.2)

    • Data: The subset of MS-COCO 2014 annotated by Zhao et al. [100], taking 8,692 single-person images, containing 5,216 Light Men, 2,536 Light Women, 714 Dark Men, and 226 Dark Women.
    • Queries: "Tennis Player", "Person eating Pizza", "Person at Table"; baseline retrieval upper limit 200 images; the three queries return 131, 75, and 124 images respectively.
    • Training: After removing the 330 images in the oracle list, 8,362 remain, split 8:2 into training/test; CGAP is trained for 10 epochs with an L∞ bound of 10.
    • Variables: Attack probability pr ∈ {0.2, 0.5, 0.7, 1.0} (the proportion of relevant images the attacker can perturb), top k ∈ {10, 15, ..., 50}, and four training objectives (Any→Light Men, Light Men→Any, Dark Men→Light Men, Light Men→Dark Men).

    Evaluation methods (§4.3)

    • Skew: The ratio of a subgroup's share in the top k to its share in the overall population; the ideal value is 1.
    • Attention: Models attention decay with a geometric distribution (p=0.36, derived from research showing top results receive 36.4% of clicks), computing the average attention for each subgroup.
    • NDCG: Measures ranking utility, ranging from 0–1.
    • Attack Effectiveness η: η(m, g) = the percentage change of group g on metric m − the smallest percentage change among other subgroups; in this paper g is Light Men, and all η and NDCG changes are measured relative to the oracle (fair re-ranking) list.
  4. What experiments did the paper conduct?

    The authors state the attack increases the advantage of light-skinned men across variables such as pr, k, embedding model, training objective, and query, with NDCG changing by no more than 0.7%.

    Experimental Setup

    • Victim system: MMT retrieval + FMMR reranking (embeddings use Faster R-CNN, InceptionV3, ResNet18); the appendix additionally tests DetConstSort.
    • Dataset: MS-COCO 2014 annotated subset, 8,692 single-person images (§4.1).
    • Attack training: CGAP trained against Deepface or FairFace, 10 epochs, L∞=10; four training objectives.
    • Metrics: Skew, Attention, NDCG, η (relative to the oracle fair list, g=Light Men).
    • Variables: pr ∈ {0.2, 0.5, 0.7, 1.0}, k ∈ {10,...,50}; when focusing on one variable, results are averaged over the other variables and the three queries.
    • Baselines: baseline list (no fair reranking) and oracle list (FMMR reranking without perturbation, 3 queries × 3 embeddings, 9 in total).

    Main Results

    Main results are presented as figures (Figure 3–5, 7); key numbers given in the paper:

    ConditionResultSource
    pr=1.0, some kattention unfairness increases by over 50%§5.1
    pr=0.2attention unfairness increases by about 30%§5.1
    any k, prNDCG changes by no more than 0.7%§5.1
    pr=0.5, "Person eating Pizza"only 35 images need to be compromised§5.1
    • Authors' interpretation: the larger pr, the more effective the attack; as k increases, Skew unfairness decreases (longer lists make the composition fairer) but attention unfairness rises slightly; the authors state the attack leaves the top of the FMMR reranked list still unfair.
    • Authors' interpretation: the attack is stealthy—it has almost no effect on search result relevance.

    Effect of Training Objectives (§5.2, Figure 5)

    • What was tested: comparison of the effects of four CGAP training objectives (Dark Men→Light Men, Light Men→Dark Men, Any→Light Men, Light Men→Any).
    • Results: the four attacks harm fairness similarly on Skew and Attention, and all remain stealthy on NDCG; the authors state the observation that Dark Men→Light Men and its reverse attack have similar effects is surprising.
    • Authors' interpretation: explained with the example in Figure 6—misclassifying minority group members into the majority group shrinks the minority group's overall proportion, so FMMR only needs to place fewer minority group members at the top; both directions ultimately give Light Men an unfair advantage.
    • Authors' interpretation: based on Figure 5, the authors believe their attack cannot create an advantage for minority groups.

    Embedding Models and Attack Training Algorithm (§5.3, §5.1 continued)

    • Embedding models (Figure 4): attack effectiveness is stable when the FMMR embedding model is changed; ResNet embeddings are slightly more robust, F-RCNN slightly more vulnerable; the authors note that ResNet's robustness appears when its architecture is most similar to FairFace.
    • Attack training algorithm (Figure 8): when CGAP is trained against Deepface and FairFace respectively, attack effectiveness is similar; the authors state attack effectiveness is largely independent of the inference model choice, and both remain stealthy.

    Effect of Queries and DetConstSort (§5.4, Appendix A.1)

    • Queries (Figure 7): the attack succeeds under all three queries but with different effects; after the attack "tennis" is the most unfair, which the authors attribute to the largest utility score difference between the Light and Dark groups in its baseline; "pizza" is the most robust on attention, which the authors attribute to minority group utility scores being higher than majority group in its baseline (Figure 2).
    • DetConstSort (Figure 9): at pr=0 the η of Skew and Attention is already high (unfair), and does not change as pr increases; the authors state the correct interpretation is not that DetConstSort resists the attack, but that it is already unfair because it uses inaccurate inferred labels, and the attack cannot make it worse; evaluating the attack presupposes an accurate demographic inference model.

    Other Ablations and Analyses

    • Baseline distribution (Figure 2): in the top 40 of the three queries, Light Men make up the vast majority and have high utility scores; in the top 40 of "tennis player" there are no Dark Women.
  5. What are some points that can be further explored?

    The authors list limitations such as group categories, data scale, model generalization, and only being able to benefit dominant groups, and call for developing more robust fair ML.

    Limitations and Future Directions Noted by the Authors

    • Limited group categories (§6.1): the analysis is limited to 4 discrete groups from two races × two genders; the authors state it is unclear how the attack performs with more than 4 discrete protected groups, continuous attributes, multiple or partial group memberships, or when the overall distribution differs greatly from their dataset.
    • Data scale (§6.1): although the dataset is sufficient to demonstrate the attack, it is smaller than the databases retrieved by real image search engines.
    • Generalizability (§6.1): the proof of concept is tuned for FairFace and Deepface; the authors state it is unclear whether CGAP can generalize to other models or real deployed systems.
    • Attack direction (§6.1): the attack can only create an unfair advantage for already dominant groups; the authors also note this does not mitigate the real harm it may cause to marginalized groups.
    • Query variation (§6.1): attack effectiveness varies with the query; the authors state attackers can partially mitigate this by concentrating resources on high-value queries, but since they did not attack a deployed search engine, how much additional effort is actually required is unclear.
    • Future directions (§6): the authors believe future work is needed to develop more robust fair ML interventions, listing value-sensitive design, adversarial perturbation hardened models, audit checklists and tools; and emphasize that high-quality demographic data is fundamental—allowing attackers to influence demographic metadata is the underlying flaw enabling the attack's success.

    Scope of Experimental Coverage

    • The victim system is built by the authors: MMT retrieval + FMMR (3 embedding models), with DetConstSort additionally tested in the appendix; no deployed search engine was attacked (§6.1).
    • The data is the MS-COCO annotated subset of 8,692 single-person images, 3 queries (§4.1, §4.2).
    • The attack is trained against 2 demographic inference models (Deepface, FairFace), with 4 training objectives, 4 levels of pr, and 9 levels of k (§4.2).
    • The paper does not report the number of repetitions or confidence intervals for each variable combination; when focusing on a single variable, results are averages across the other variables and queries (§5).
    • The paper does not report evaluations against adaptive attacks or specific defenses; mitigations are discussed only as directions in §6.
  6. Summarize the main content of the paper

    The authors attack fair image search with GAP and claim that under a strict threat model they can bias results toward light-skinned men with almost no loss in relevance.

    An attack paper on the intersection of fairness and robustness: after a ranking model is calibrated to fairness, can an external attacker make it unfair without access to the model and training data?

    • Method: The authors build an image search engine with MMT retrieval + FMMR fair re-ranking, and train CGAP against demographic inference models (Deepface, FairFace) to perturb images into being misclassified, thereby misleading FMMR's indirect demographic inference; under the threat model, the attacker cannot poison training data, does not know the model used by the victim, and can only add images to the database.
    • Main results (as claimed by the authors): At pr=100%, attention unfairness increases by more than 50% for some k; at pr=20%, it increases by about 30%; for any k and pr, NDCG changes by no more than 0.7%, i.e., the attack is stealthy.
    • Robustness (as claimed by the authors): The attack effect remains stable across FMMR embedding models (Faster R-CNN, InceptionV3, ResNet18), the inference models used to train the attack, four training objectives, and three queries, differing only in degree.
    • Exception: The attack can only create an advantage for the already dominant group (Light Men); it is ineffective against DetConstSort—the authors state that because its inferred labels are inaccurate it is already unfair, and the attack cannot make it worse.
    • Authors' conclusion: In the absence of defenses, fairness interventions may be weaponized by malicious parties; until fair ML methods are robust to adversarial attacks, policymakers may be unable to safely mandate the use of fair ML algorithms. The authors plan to release code and data upon researcher request.

Related papers

Show 3 more papers
Read originalFAccT 2022