GeoCloak: An agent framework for semantic-preserving geolocation privacy via peripheral cue manipulation

LVLMs can infer location from a single photo without metadata, and users lack image-level control that preserves semantics.

• Occurrence scenario: the authors state that sharing photos often contains unintentionally leaked clues such as street signs and buildings; a passive attacker with black-box LVLM access can locate within minutes, and retrieval, tool-based agents, and geographic-reward post-training further strengthen this capability.
• Existing shortcomings: the authors state that output auditing and prompt refusal depend on the service provider, and attackers can switch to unprotected models; fixed perturbations such as M-Attack, AnyAttack, GeoShield, and ReasonBreak rely on post-hoc utility evaluation and leave perceptible artifacts (Section 5.4).
• Core observation: the authors state there is a visual-geolocation asymmetry: decisive clues are often visually peripheral. Figure 1 marks a Khmer script street sign as 1.0, with Culture 0.7, Culture 0.9, and Spatial 0.4.
• This paper proposes: proposes GeoCloak, which identifies clues per image, performs semantic edits on peripheral clues, and configures adversarial perturbations under utility feedback. The authors state that semantic editing is a new protection mechanism directly targeting these clues.
• Threat model:
  • Goal: infer the city, state, or country-level location of images users share on public platforms.
  • Knowledge: only the protected image, without the original; the summary calls it black-box LVLM access, and inference uses SOTA LVLMs and multiple methods.
  • Victims: user images intended for sharing on public platforms such as social media.
  • Defender: submits the image and true geographic location before sharing as a stopping condition. When going through external APIs, the trust assumption extends to the service handling the image and ground truth; open-source configurations can also be deployed on one's own infrastructure.

Section 2 divides related work into geolocation inference and protection methods.

Geolocation Inference

• Prompts, retrieval, and tools: zero-shot queries to structured chain-of-thought (Liu et al., 2024; Mendes et al., 2024; Li et al., 2025; Luo et al., 2025; Song et al., 2025); retrieval augmentation (Jia et al., 2024; Zhou et al., 2024); agents with web search, zoom, and street view (Li et al., 2026; Zhang et al., 2025).
• Post-training: supervised fine-tuning or reinforcement learning with geographic rewards (Li et al., 2024; Jia et al., 2025; Li et al., 2025; Xu et al., 2025). The authors state that agents and post-training have converged, including end-to-end training of geolocation agents (Jia et al., 2026; GeoVista by Wang et al., 2025).
• Retrieval and classification: GeoCLIP (Cepeda et al., 2023), IM2GPS (Vo et al., 2017), PIGEON (Haas et al., 2024), PlaNet (Weyand et al., 2016). The authors state that although these methods require millions of labeled images, they are still empirically inferior to LVLMs.

Protection Methods

• Traditional and model-level: the authors state that metadata stripping and blurring have limited effect on LVLMs that infer location directly from visual content. Output auditing (Mendes et al., 2024) and prompt refusal (Luo et al., 2025) restrict location disclosure during inference, but require deployment on the service side, giving image owners no direct control.
• Image-level perturbation: targeted M-Attack (Li et al., 2025) and AnyAttack (Zhang et al., 2025) steer predictions toward specified wrong locations; untargeted GeoShield (Liu et al., 2026) and ReasonBreak (Zhang et al., 2025) do not specify a target. The authors state that they place utility in post-hoc evaluation, resulting in perceptible artifacts (Figure 3, Table 6, Appendix G).

Baselines and Datasets

• Baseline methods: the authors reproduce the state-of-the-art image-level defenses M-Attack (targeted) and GeoShield (untargeted) using released code and all original hyperparameters.
• Benchmarks: Im2GPS3k, GeoBench, Dataset-GSS, Dataset-PCW, used for city, state, and country-level evaluation.

The authors distinguish this paper from fixed perturbation pipelines: they select semantic edits and adversarial perturbations per image, and place utility constraints inside the iterative protection, rather than uniformly applying the same perturbation pipeline.

GeoCloak uses an Orchestrator to coordinate ClueAgent and EditAgent, iteratively modifying the image until it no longer matches the ground truth at the target granularity and passes utility checks, or reaches the maximum steps.

Stopping conditions

• Two objectives: The privacy objective is to prevent correct location inference; the utility objective is to preserve semantics and visual quality. The ground-truth location is the stopping criterion for the loop.
• Utility check: If the main object set changes, utility is 0; otherwise it multiplies by the CLIP image cosine. That is, U(I^(t))=1[obj(I^(t))=obj(I⁽⁰⁾)]·cos(ϕ_I(I^(t)),ϕ_I(I⁽⁰⁾)). It means editing the main object causes immediate failure; otherwise it measures residual visual deviation. Threshold τ=0.85. The paper does not name the specific LVLM used for main object detection.
• Protection check: GeoMatch judges whether predictions match ground truth at city, state, and country levels. The main experiment sets the target granularity gtgt to country; the authors say country-level mismatch is the strongest protection level.
• Termination and fallback: It stops as soon as "no longer matches and utility is not below τ"; maximum rounds N=20, temperature 0.5. If it runs all N, it reports the strongest level achieved by country, state, city. If the image is unchanged throughout, a fallback perturbation is applied where CLIP cosine remains not below τ. If utility fails, it rolls back and changes the plan.

ClueAgent

• Clues: Analyzes the current image; types are landmark, text&signage, culture, spatial (following Li et al.). Each contains a description, pixel box, and ranking score from 0 to 1. The description is the main input to EditAgent.
• Internal attacker: Uses all clues to predict a place name. Editing stops once it no longer equals ground truth. The authors say this does not indicate all clues are removed; protection beyond ClueAgent is evaluated by the external attacker in Section 5.1.
• GT-guided: If it cannot match ground truth on the original image, the entire run instead provides ground truth to clue analysis until no ground-truth-related clues are recognized. The mode is fixed after t=0 and does not switch mid-run.

Semantic editing

• Five strategies: ReplaceText, ReplaceObject, EraseObject, AddDecoy, StyleTransfer. Prioritizes clues with high scores and small areas based on the ratio of score to region area. Erasing does not distinguish text from objects; replacement does. Appendix B gives full prompts, Appendix C gives examples.
• Constraints: Modifications must match scene lighting, perspective, and style; replacements use geographically nearby regions; do not cover main subjects detected by the LVLM; switch strategies after the same strategy fails. Figure 7 shows a multi-round example: after utility failure, roll back and switch strategy.

Adversarial perturbation

• Configuration: Used for large-scale or structural clues that are hard for semantic editing. Reference is region_focus (clue box plus entire image) or full_image; loss is feature_only (λ=0, pushing away CLIP features of the reference crop) or dual_loss (λ=1, also pulling toward the non-geographic description written by EditAgent).
• Update and feedback: Sign stochastic gradient ascent, step size ε/10, projected to an ℓ∞ ball of radius ε, for K steps. On first use, initial reference, loss, K, ε, and text weight μ are set based on whether there is a box and whether the scene has clear visual content. If utility is below τ, roll back and tighten; if utility is sufficient but privacy is insufficient, strengthen. The main text does not give a single fixed ε. Appendix B gives the full prompts.

Experimental setup

• Attack-side models: Gemini-2.5-Pro, Gemini-3.1-Pro-Preview (Gemini-3.1-Pro in tables), GPT-5.4, o3, Qwen-3.5-Plus (Qwen3.5-Plus in headers), and GeoVista's own trained model. Also evaluate GeoCLIP and commercial black-box tool GeoSeer on Im2GPS3k; fine-tuning setup uses Qwen3.5-9B. Claude-Sonnet-4.6 is not included in official results due to size limits; in an initial Dataset-PCW run, 40.0% of samples exceeded 5MB.
• Defense configuration: Main results use Gemini-2.5-Pro for clue, planning, and code, and Gemini-2.5-Flash-Image for editing. There is also o3+Seedream-4.5, and Qwen3.6-35B-A3B+Qwen-Image-Edit-2511. Each agent has up to 20 rounds, temperature 0.5.
• Data: GeoBench 512 images, Dataset-PCW 267 images are all available samples; Im2GPS3k and Dataset-GSS each randomly sample 500. Total 1,779 images, 108 countries, 576 cities (Table 10). The authors say the full design covers over 75,000 runs across 11 attack configurations and 4 conditions.
• Scope: Sections 3 and 5.2 exclude visually dominant globally recognizable landmarks. The pipeline uses U2-Net, Florence-2, and Gemini-2.5-Pro; at least one tier-A and judged intentional marks Exempt. Table 1 exemption rates: GeoBench 37.5%, Im2GPS3k 53.4%, Dataset-GSS 21.0%, Dataset-PCW 55.3%. Table 3 caption does not mention Exempt; its Gemini-2.5-Pro zero-shot aggregate 48.3% matches Table 2's Non-Exempt.
• Baselines and metrics: M-Attack and GeoShield reproduced with original hyperparameters: CLIP ensemble ViT-B/16, ViT-B/32, CLIP-Laion, ε=8/255, step 1/255, 100-step PGD; GeoShield's non-geographic descriptions generated by GPT-5-mini. Main metric is city-level accuracy, plus state, country, and @1km, @25km, @100km. Success is defined as location matching, not LLM scoring. The paper does not report the number of repetitions for the entire evaluation.
• Inference methods: Zero-shot, two-stage GeoMiner, tool-augmented GeoVista, plus five LVLMs forming 11 configurations. GeoMiner and GeoVista use official default settings. Zero-shot prompts are in Section D.2, not repeated here.

Main results

The table below is the city-level accuracy aggregated across four datasets from Table 3. Zero-shot rows and the GeoVista row take the Overall segment; the last row is the paper's aggregate over 11 configurations. Lower means attackers find it harder to locate.

Swipe horizontally to read the table

Model	Initial	M-Attack	GeoShield	GeoCloak
Gemini-3.1-Pro (zero-shot)	51.3%	37.3%	33.2%	13.8%
Gemini-2.5-Pro (zero-shot)	48.3%	32.3%	26.0%	9.5%
o3 (zero-shot)	46.9%	25.2%	17.7%	13.3%
GPT-5.4 (zero-shot)	39.5%	21.5%	17.1%	12.2%
Qwen3.5-Plus (zero-shot)	31.1%	15.3%	11.8%	6.5%
GeoVista (same-named method)	35.2%	21.4%	15.6%	11.1%
Paper Overall (11 configs)	41.2%	24.5%	19.7%	9.9%

• Authors' interpretation: The authors say Gemini-3.1-Pro is the strongest, and zero-shot is already strong, not relying on specialized inference pipelines. They say GeoVista is weaker because it is based on Qwen2.5-VL-7B-Instruct with reinforcement learning, while the other evaluated models come from stronger variants from their providers.
• Aggregate and counterexamples: The authors say GeoCloak reduces the aggregate accuracy to about half of the best baseline in the table, and claim superiority over both baselines on all settings. Table 3 has counterexamples: on Dataset-GSS with zero-shot GPT-5.4, GeoShield is 9.1%, GeoCloak 12.9%; on Dataset-PCW with GeoMiner×Qwen3.5-Plus, GeoCloak is 6.1%, higher than GeoShield's 4.4% and M-Attack's 5.3%.
• Same backbone and refusals: In the table, the Gemini-2.5-Pro row has the attack model same as the ClueAgent backbone; the Gemini-3.1-Pro row does not. With o3 with GeoMiner, the no-defense safe refusal rate is 28.1%, and under M-Attack, GeoShield, GeoCloak it is 15.2%, 21.6%, 18.1% (Appendix I). The authors say refusals are counted as failures in all defenses, so they do not favor any one method.

Utility and human perception

• What was measured: 100 source images uniformly from four datasets, each with three defenses, totaling 300 tasks. Two annotators, unaware of defense identity, rate naturalness, semantic preservation, and overall quality on a five-point Likert scale. Quadratic weighted Cohen's κ is 0.709.
• Results: Table 5 overall quality is GeoCloak 4.43, M-Attack 2.63, GeoShield 2.53; naturalness 4.57, 2.31, 2.23; semantic preservation 4.45, 3.77, 3.68. Table 6 aggregates CLIP similarity as 0.9197, 0.7719, 0.4209; BERT Score 0.9158, 0.9124, 0.9047; BLEU 0.1136, 0.0968, 0.0766; ROUGE-L 0.4054, 0.3839, 0.3480.
• Authors' interpretation: The authors say GeoShield and M-Attack show perceptible artifacts such as cropping, floating text, and separated heads, while GeoCloak completes protection with local edits. The text metric descriptions are generated by an LVLM, but the paper does not state which one.

Cross-mechanism transfer and adaptive attacks

• Non-LVLM: On Im2GPS3k, GeoCLIP city-level accuracy drops from 12.1% to 5.0%, and commercial tool GeoSeer from 30.7% to 5.7% (Table 4). The authors say this supports transfer to different localization mechanisms.
• Informed prompts and compression: Table 8 is for five LVLMs zero-shot. Naive attacker Initial 43.4%, GeoCloak 11.1%, then JPEG 11.5%; conflict-aware prompt Initial 41.6%, GeoCloak 15.9%. The authors say that prompt has extra attack capability, but on unprotected images goes from 43.4% to 41.6%, and argue that treating clues indiscriminately as suspicious loses real evidence.
• Fine-tuning and purification: On an 80/20 split of protected images, GeoCLIP goes from 17.0% unprotected to 7.4% protected, then 9.6% after fine-tuning; Qwen3.5-9B from 24.5%, 6.4%, then 11.3% after LoRA. The paper does not state which dataset that split comes from. The authors say fine-tuning only partially recovers. DiffPure's guided diffusion, DDPM, and Stable Diffusion variants blur details and perturbations remain visible; the paper does not report post-DiffPure city-level accuracy.

Other ablations and analysis

• Table 7 on Im2GPS3k, average city-level attack success across 11 configs: Initial 24.1%, removing semantic editing 15.4%, removing perturbation 5.1%, o3+Seedream-4.5 6.3%, open-source config 7.4%, full system 4.0%.
• Figure 5: Erase Object accounts for 24.9% of calls, Replace Text 21.4%, adversarial perturbation 20.4%; utility preservation rates 85.6%, 81.8%, 13.5%. Both skills used 58.4%, semantic editing only 39.9%.
• Figure 5b successful rounds: city-level Erase Object 37.3%, dropping to 16.9% at country level; country-level Replace Text 36.5%. The authors say city-level often relies on removing a single clue, while country-level relies more on recontextualization.
• Of non-exempt images, 12.1% enter GT-guided; on Im2GPS3k, the attacker still locates 2.9% of those, and 11.8% with a one-round debate scheme. Median cost is $0.32/image, 8.3 minutes; the authors say Batch API is about $0.16. Median rounds are 2.0 for city and state, 5.0 for country.
• Table 12 groups by number of successful attackers without defense. High identifiability (8–11/11) mean goes from 9.84 to GeoCloak 2.75, GeoShield 5.69, M-Attack 6.91; low identifiability (1–4/11) is 2.18, 0.35, 0.39, 0.57. The authors say the advantage is less pronounced on low-identifiability images.
• Table 13: Gemini-3.1-Pro zero-shot, four-dataset aggregated country-level accuracy: Initial 86.3%, GeoCloak 24.9%, GeoShield 42.1%, M-Attack 45.0%. Landmark manual verification precision is 86.7%, inter-annotator agreement 93.3%; the paper does not report the sample size for that verification.

Limitations and future directions noted by the authors

• Backbone capability: The authors say effectiveness depends on the underlying LVLM's visual reasoning; stronger models can identify more complete clues. The open-source Qwen configuration's attack success rate is 7.4%, higher than the main Gemini configuration's 4.0% (Table 7). Improving protection with smaller, resource-constrained open-source backbones is written as future work (Section 6).
• Post-hoc audit: Running a GT-guided audit after the standard mode stops may check residual clues; the authors say its additional protective benefit and computational cost remain to be evaluated (Section 6).
• Prompts: Without changing agent roles, rewriting prompts has negligible impact on protection, the authors say. They say frameworks like DSPy are not directly applicable to multi-round dynamic prompts, and systematic prompt optimization is left as future work (Section 6).
• User intent: The authors say existing datasets have no ground-truth intent, so evaluation approximates with visual landmark analysis, and automatic utility checks estimate acceptable modifications. An interactive version lets users specify unmodifiable objects and accept, roll back, or stop incrementally. The authors say future work will incorporate real user feedback (Section 6).
• Architecture dependence: The authors say clues can only be discovered at inference time, so a fixed pipeline struggles to preserve both privacy and utility; this is an argument for the agent structure, not a separately listed open problem (Section 6).

Experimental coverage

• Sampled benchmarks are Im2GPS3k, GeoBench, Dataset-GSS, Dataset-PCW totaling 1,779 images, 108 countries, 576 cities (Section 5.1, Table 10); exemption rates in Table 1, localization difficulty for exempt and non-exempt in Table 2.
• LVLM attack configurations are zero-shot and GeoMiner for five models, plus GeoVista, totaling 11 (Table 3); Im2GPS3k also has GeoCLIP and GeoSeer (Table 4).
• Image-level baselines are M-Attack and GeoShield. Adaptive settings include conflict-aware prompts, JPEG, qualitative observations of three DiffPure variants, and fine-tuning of GeoCLIP and Qwen3.5-9B (Section 5.8, Table 8).
• Main configuration is Gemini-2.5-Pro and Gemini-2.5-Flash-Image; Table 7 additionally reports o3+Seedream-4.5 and Qwen3.6-35B-A3B+Qwen-Image-Edit-2511. Human utility evaluation is 100 source images with two annotators (Section 5.4).
• The paper does not report repetitions for the whole evaluation, post-DiffPure city-level accuracy, the number of samples for landmark manual verification, or the name of the LVLM used for text utility descriptions.

GeoCloak is a semantics-preserving privacy protection for LVLM image geolocation.

• Problem: black-box LVLMs can infer city to country from a single photo without metadata. The authors believe model-level refusal can be bypassed by switching models, and fixed adversarial perturbations leave perceptible artifacts. The authors consider images needing protection as those whose location can be inferred from inconspicuous clues, where the user did not intend to disclose the location.
• Method: ClueAgent lists peripheral but high-scoring clues and predicts location; EditAgent chooses among five local edits and CLIP adversarial perturbation; Orchestrator requires the main object to remain unchanged, CLIP cosine no lower than 0.85, target granularity country, up to 20 rounds. When localization fails on the original image, it switches to using ground truth to guide finding residual clues.
• Main results: in Table 3, aggregated city-level accuracy across 11 attack configurations drops from 41.2% to 9.9%, with M-Attack and GeoShield at 24.5% and 19.7%. For zero-shot Gemini-3.1-Pro, which is not a protection backbone, it drops from 51.3% to 13.8%. In some dataset × model cells, GeoShield or M-Attack has lower city-level accuracy than GeoCloak.
• Utility and transfer: human overall quality on 100 images is 4.43/5 (Table 5), and aggregated CLIP similarity across four datasets is 0.9197 (Table 6). On Im2GPS3k, GeoSeer drops from 30.7% to 5.7%. After conflict-aware prompting, the aggregated accuracy of five LVLMs zero-shot is 15.9% (Table 8).
• Authors' conclusions: the authors believe this visual-location asymmetry allows targeted editing to weaken localization with smaller visual changes, and state that they have released an interactive version that users can accept or roll back step by step. The authors also state that open-source backbones remain weaker than the main configuration, with smaller open-source models, real user feedback, and post-hoc ground-truth auditing left for future work.