Polish social media misogynistic speech detection dataset: transferability of hate speech detection from generic to specific hate types
A Dataset for Misogyny Detection in Polish Social Media: Transferability of Hateful Speech Detection from General to Specific Types of Hate
The authors build a dataset of 867 Polish misogynistic comments and find that transfer from general harmful language detection to misogyny detection is inconsistent, with safety-specific models transferring better.
- It is unclear whether general hate speech detection models transfer to more specific misogyny detection, and Polish lacks dedicated misogyny detection resources, especially for implicit misogyny.
- 867 Facebook comments surrounding a controversy about a Polish female public figure were collected and annotated into three classes: misogynistic/gender-stereotyping/neutral; GPT-5.5 and Grok-4.3 annotated independently, with expert adjudication of disagreements, and then encoder models fine-tuned on BAN-PL, safety-specialized models and Polish generative LLMs were used for cross-domain and three-class evaluation.
- When BERT/HerBERT are transferred from BAN-PL to the misogyny data, F1 drops by 0.307 and 0.323, respectively; the safety-specific HerBERT-Guard achieves zero-shot transfer F1 of 0.686 and three-class F1 of 0.742, outperforming the other tested models; the authors state that general detection capability does not transfer uniformly to misogyny detection.
- The authors note that the data come from a single platform and a single controversy event, that the stereotype class accounts for only 5.1%, and that category boundaries involve subjectivity; the experiments cover only Polish and 6 classification models, and the expert validation subset is 100 items.
- Evaluated models
- GPT-5.5Grok-4.3Polish BERT (bert-base-polish-cased-v1)HerBERT (herbert-base-cased)Bielik-Guard (Bielik-Guard-0.5B-v1.1)HerBERT-Guard (HerBERT-PL-Guard)Bielik-7B-v0.1-awqPLLuM 8x7B-2025
- Benchmarks
- BAN-PL本文提出的波兰语厌女数据集(Misogyny corpus)
- Metrics
- Accuracymacro-F1Cohen's κpercentage agreement
The researchers constructed a new dataset of Polish social media comments, annotating comments for misogynistic speech, a type of gender-targeted hate speech. Experiments show that strong performance on generic harmful language detection does not transfer uniformly to misogyny identification, and safety-specialized models show markedly stronger generalization ability. The paper first evaluates the reliability of LLM-assisted annotation, then compares three types of models: models trained on generic harmful language data, safety-specialized models, and models fine-tuned directly for misogyny detection. The study contributes a new Polish-language gender-directed harmful language resource and shows that the scope and granularity of prior safety training are important factors in the transfer of harmful language detection from generic to specific hate forms.
Paper analysis
What problem does this paper attempt to solve?
Can general harmful language detection capability transfer to specific misogyny detection? The authors build a dedicated dataset for Polish and evaluate transferability.
Core question: can models that perform well on general harmful language detection transfer their capability to the more specific task of misogyny detection? The authors study this general-to-specific transfer problem in the Polish-language setting.
- Setting and motivation: Polish is under-resourced in hate speech detection research; existing datasets mostly cover broad categories such as cyberbullying and offensive language, while resources targeting gender hostility, especially implicit misogyny (hostility expressed through stereotypes and insinuation), are scarce; the authors argue that implicit misogyny poses a challenge to contemporary NLP systems.
- Shortcomings of existing methods (as stated in the paper): existing Polish benchmarks treat harmful language as a broad phenomenon, making it difficult to examine hostility targeting specific groups; models trained on general datasets may fail to recognize content that does not fit typical hate speech patterns.
- Core observation and hypothesis: it is unclear whether general and specific forms of harmful language share transferable representations; the authors hypothesize that safety-oriented specialization may provide more transferable representations than general training.
- What the paper proposes: a new dataset of Polish misogynistic social media comments (867 items, three annotation classes: misogynistic/gender-stereotyping/neutral), annotated independently by two frontier LLMs with expert adjudication of disagreements; on this basis, the transfer performance of general models, safety-specialized models and directly fine-tuned models is evaluated.
- Research questions: RQ1 agreement of frontier LLM annotation of Polish misogynistic content; RQ2 degree of transfer from general and safety-oriented classification settings to misogyny detection; RQ3 effectiveness of different model families in detecting Polish misogynistic language.
What related research exists?
Related work falls into three groups: Polish harmful language detection resources, misogyny detection research, and Polish safety classifiers; this paper fills the gap in Polish misogyny detection.
Polish harmful language detection resources
- PolEval 2019 shared task (Ogrodniczuk and Kobyliński, 2019) — an early benchmark for Polish cyberbullying and hate speech detection, covering binary harmful content detection and distinguishing bullying from hate speech.
- BAN-PL (Kolos et al., 2024) — content removed by professional moderators from the Wykop.pl platform, one of the largest public Polish harmful language resources; this paper uses it as the training set for cross-domain transfer.
- The paper notes that these Polish resources are limited overall and mostly treat harmful language as a broad phenomenon, with insufficient coverage of hostility toward specific social groups.
Misogyny detection
- Shared tasks and corpora: IberEval 2018, SemEval-2022 Task 5 (Fersini et al., 2018, 2022), and others provide misogyny-annotated corpora and finer-grained category systems (stereotyping, objectification, domination, threats of violence, etc.).
- Multilingual and cross-lingual transfer: Frenda et al. (2018) and Pamungkas et al. (2020) extend misogyny detection to multilingual settings; for Arabic there are ArMI (Almanea and Poesio, 2022) and LeT-Mi (Mulki and Ghanem, 2021).
- The paper notes that Polish is largely absent from this line of research, and existing Polish datasets were not designed for misogynistic discourse.
Polish safety classification and multilingual safety evaluation
- PL-Guard / HerBERT-Guard (Krasnodebska et al., 2025) — a Polish safety classifier based on HerBERT; the paper relays its finding that compact safety classifiers can outperform larger generative models (including under adversarial evaluation).
- Bielik-Guard (Wrobel et al., 2026) — a compact Polish safety classifier trained on multiple harmful content categories.
- Multilingual safety evaluation: Krasnodebska et al. (2026) note that safety benchmarks are English-centric; Dadas et al. (2025) argue that evaluation needs to target language- and culture-specific capabilities.
Baselines and benchmarks
The baselines for the cross-domain evaluation are BERT and HerBERT fine-tuned on BAN-PL; the benchmarks are BAN-PL (training/in-domain test) and the misogyny dataset proposed in this paper.
The authors position this paper as extending the above argument for multilingual safety evaluation from language and cultural capability to detection of specific subtypes of harmful language, and as filling the gap in Polish misogyny detection resources.
How does the paper address this problem?
Build a dataset of 867 Polish misogynistic comments (LLM annotation + expert adjudication), then run two sets of evaluations: cross-domain transfer and three-class classification.
Overall approach: first build a Polish misogyny detection dataset, then use it as a benchmark to compare three types of systems—models fine-tuned on general harmful language data, safety-specialized models, and models fine-tuned directly on the misogyny data—thereby quantifying general-to-specific transfer.
Corpus construction and annotation
- Data source: comments on Facebook surrounding a media controversy about the Polish female public figure Anna Lewandowska; the authors say they chose this domain because such discussions often trigger evaluations of women's social roles and are rich in implicit hostility that keyword methods struggle to capture.
- Annotation scheme: three mutually exclusive labels—Misogyny (hostility, contempt or prejudice toward women), Stereotyping (attributing generalized traits to women as a group, without overt hostility), Neutral.
- Annotation process: GPT-5.5 and Grok-4.3 annotated independently using the same annotation guidelines; samples with conflicting labels were adjudicated by experts to determine the final label; separately, experts independently annotated a randomly sampled subset of 100 items for quality evaluation.
- Agreement measures: percentage agreement and Cohen's κ, under both three-class and binary (misogyny + stereotyping merged as harmful) settings.
Classification experiment setup
- Two groups of systems: (1) supervised Transformer encoders—base models Polish BERT, HerBERT, and safety-pretrained models Bielik-Guard, HerBERT-Guard; (2) instruction-tuned Polish generative LLMs—Bielik and PLLuM, zero-shot, with the task definition given in the prompt (the full prompt is given in Appendix A).
- Cross-domain transfer protocol: after training/freezing on BAN-PL, models are applied directly to the misogyny corpus without any fine-tuning, measuring zero-shot cross-domain generalization; cross-domain evaluation uses 20% of the misogyny data as the test set.
- In-domain protocol: BAN-PL is split 800/200/200 into train/validation/test; the misogyny corpus is split 60%/20%/20% (693/174/174, Table 1), with the validation set used only for model selection and hyperparameters.
- Metrics: Accuracy and macro-F1, with error analysis of confusion between misogyny and stereotyping.
Training details
- Hyperparameters for the three-class task: encoder model learning rate candidates 1×10⁻⁵, 3×10⁻⁵, 5×10⁻⁵, training for 10–20 epochs, with configurations selected by validation set performance.
- Training background of the safety models (as recounted in the paper): HerBERT-Guard was developed with Polish safety data plus PolyGuard and WildGuard resources, with unsafe categories including hate; Bielik-Guard was trained on the Polish Sójka resource, including hate/aggression dimensions covering discrimination based on characteristics such as gender; neither Guard model was fine-tuned on this paper's misogyny dataset.
What experiments did the paper conduct?
Three groups of experiments: LLM annotation agreement, binary cross-domain transfer, and three-class misogyny detection; safety-specific models perform best in both transfer and three-class performance.
Experimental Setup
- Models evaluated: encoders Polish BERT (bert-base-polish-cased-v1), HerBERT (herbert-base-cased); safety classifiers HerBERT-Guard, Bielik-Guard; generative PLLuM 8x7B-2025, Bielik-7B-v0.1-awq (three-class task); for annotation, GPT-5.5, Grok-4.3.
- Datasets: BAN-PL (training/in-domain test, 800/200/200); this paper's misogyny dataset of 867 entries (Neutral 446 entries 51.4%, Misogyny 377 entries 43.5%, Stereotyping 44 entries 5.1%), split 693/174/174 (Table 1).
- Metrics: Accuracy, macro-F1 (classification); percentage agreement, Cohen's κ (annotation agreement).
- Annotation quality assessment: experts independently annotated a random subset of 100 entries, compared with each LLM's annotations.
- Number of repetitions: not reported in the paper.
Main Results
Binary cross-domain transfer results (according to Table 2):
Swipe horizontally to read the table
Model Training data Test data Accuracy F1 BERT BAN-PL BAN-PL 0.875 0.875 BERT BAN-PL Misogyny 0.580 0.568 BERT Misogyny Misogyny 0.753 0.753 HerBERT BAN-PL BAN-PL 0.830 0.830 HerBERT BAN-PL Misogyny 0.523 0.507 HerBERT Misogyny Misogyny 0.764 0.760 HerBERT-Guard PolyGuard/WildGuard BAN-PL 0.750 0.749 HerBERT-Guard PolyGuard/WildGuard Misogyny 0.701 0.686 Bielik-Guard Sójka BAN-PL 0.875 0.875 Bielik-Guard Sójka Misogyny 0.655 0.632 - The authors state that the F1 transfer gap for BERT from BAN-PL to the misogyny data is 0.307, and for HerBERT 0.323, and that general harmful-language detection ability does not transfer uniformly to misogyny detection.
- The authors state that training directly on the misogyny data remains the most effective strategy, and that the poor cross-domain results cannot be attributed to the difficulty of the target dataset itself.
- The authors state that HerBERT-Guard's zero-shot transfer performance shows that broader safety-oriented training can improve transfer to a specific form of harmful discourse.
LLM Annotation Agreement (RQ1)
- What was measured: agreement of GPT-5.5 and Grok-4.3 annotations with the final corpus labels, and with the 100-entry expert-annotated subset.
- Results: compared with the expert subset, GPT's accuracy is 0.79, Cohen's κ = 0.60, and Grok's is 0.73 and κ = 0.53; for both models agreement improves after merging misogyny and stereotyping into a binary classification.
- Authors' interpretation: agreement is moderate to substantial, which the authors consider confirms the reliability of the dataset; part of the annotation disagreement occurs at the boundary between different gender-oriented forms of harmful discourse, rather than in distinguishing harmful from neutral.
Three-Class Misogyny Detection (RQ3)
- What was measured: comparison of encoders, safety classifiers and generative LLMs on the three-class misogyny/stereotyping/neutral task (according to Table 3).
- Results: HerBERT-Guard is highest (Accuracy 0.782, F1 0.742, stereotyping-class F1 0.667); HerBERT 0.753/0.619, BERT 0.747/0.573; PLLuM 8x7B-2025 is 0.517/0.429, and Bielik-7B-v0.1-awq is lowest (0.431/0.364, stereotyping-class F1 only 0.097).
- Authors' interpretation: the base encoders perform reasonably on the two majority classes but decline on the stereotyping class (BERT's F1 for that class is 0.200); the authors consider the low macro-F1 mainly reflects weak performance on the minority class (5.1%); both Polish generative LLMs are below both the supervised encoders and the frontier LLMs, which the authors state further indicates the effectiveness of task-specific and safety-specific training in this setting.
Other Ablations and Analyses
- Binary vs three-class annotation agreement: GPT's agreement with the final labels differs very little between the two settings, and the authors state it largely maintains its classification performance when distinguishing misogyny from stereotyping.
- Cross-task reference: the authors note that the GPT-5.5 and Grok-4.3 annotation experiments use a different evaluation subset from the Table 3 classification results and are not directly comparable, but on accuracy alone the supervised encoders are comparable to these two larger frontier LLMs.
- Negative results: Bielik-7B-v0.1-awq is overall weakest on the three-class task; PLLuM's F1 on neutral comments (0.327) is far below that on the misogyny class (0.627), and the paper does not explain why.
What are some points that can be further explored?
The authors list limitations such as a single data source and class imbalance, and propose future directions such as a cross-target transfer matrix.
Limitations and future directions noted by the authors
- Single data source (Limitations): the dataset comes from a single social media context surrounding a controversy about one female public figure, and its linguistic patterns may partly reflect that topic and platform; it should not be assumed to represent Polish misogynistic discourse as a whole.
- Class imbalance (Limitations): the gender-stereotyping class accounts for only 5.1%, limiting the reliability of conclusions about this class.
- Class boundaries are partly subjective (Limitations): despite expert adjudication and independent expert validation, the distinction between misogynistic, stereotyping and neutral still depends partly on interpretation.
- Future direction (Discussion and Future Work): examine general-to-specific transfer for other hate targets, build a cross-category transfer matrix, and distinguish category transfer from cross-lingual and cross-domain transfer.
- Future direction (Discussion and Future Work): the authors suggest moving toward an evaluation paradigm that assesses moderation systems not only by aggregate capability but also by the degree to which that capability transfers across forms of hate expression.
Scope of the experiments
- 6 classification models were tested: BERT, HerBERT, HerBERT-Guard, Bielik-Guard (binary classification), and PLLuM 8x7B-2025, Bielik-7B-v0.1-awq (three-class classification only).
- Language coverage is Polish only; data comes from 867 comments on a single controversy on Facebook.
- The expert subset for annotation quality evaluation is 100 items; the annotation models are GPT-5.5 and Grok-4.3.
- Cross-domain transfer is evaluated only in one direction, from BAN-PL (and the Guard models' original safety training data) to the misogyny data.
- The paper does not report the number of repetitions or confidence intervals for the classification experiments, nor full classification metrics for GPT-5.5 and Grok-4.3 on the three-class task.
Summarize the main content of the paper
A new Polish misogyny detection dataset plus a transferability evaluation: general detection transfers inconsistently, and safety-specific models generalize better.
An evaluation/benchmark paper studying whether general harmful language detection capability transfers to specific misogyny detection, in the resource-scarce setting of Polish.
- Problem: Existing Polish hate speech resources mostly cover broad toxicity and lack dedicated data targeting misogyny, especially implicit misogyny; it is unclear whether performance on general benchmarks indicates model performance on specific forms of harm.
- Method: 867 Facebook comments were collected around controversies involving Polish female public figures and annotated into three classes: misogyny (43.5%), gender stereotypes (5.1%), and neutral (51.4%); GPT-5.5 and Grok-4.3 annotated independently, disagreements were adjudicated by experts, and this was validated with a 100-item expert subset (κ = 0.60 for GPT, κ = 0.53 for Grok).
- Transfer results: When BERT and HerBERT were transferred directly from BAN-PL to the misogyny data, F1 dropped from 0.875 to 0.568 and from 0.830 to 0.507, respectively (Table 2); the authors state that general detection capability does not transfer uniformly to misogyny detection.
- Safety-specific models: HerBERT-Guard zero-shot transfer achieved F1 of 0.686, a smaller drop than the base models trained on BAN-PL; on the three-class task, HerBERT-Guard exceeded all tested models with Accuracy 0.782 and F1 0.742 (Table 3).
- Other findings: HerBERT fine-tuned directly on the misogyny data reached 0.760 F1; two Polish generative LLMs (PLLuM, Bielik-7B) were below supervised encoders on the three-class task.
- Authors' conclusion: The scope and granularity of safety-oriented training are important factors affecting transfer from general to specific harm detection; the authors argue that the relationship between general and specific harm detection should be treated as an independent research question, and plan to extend this kind of transfer analysis to other hate targets.