Industry & Economics
What counts as harm? Feminist experiments with AI-driven visual content moderation to detect misogynistic violence in mainstream pornography.
Open Access: Yes.
Abstract
This paper offers a feminist critique of algorithmic systems of harm-detection by evaluating the visual recognition capabilities of Machine Learning as a Service (MLaaS) tools and their effectiveness in detecting misogynistic violence within online pornography. It asks what counts as harm in commercial artificial intelligence (AI) moderation systems and what happens when these are used to identify misogynistic violence in pornography. While previous research has examined gender bias in AI-based moderation, the detection of male violence against women (MVAW) in visual pornography remains unexplored. This study addresses this gap through an analysis of 100 pornographic thumbnails, combined with visual transitivity analysis. It shows that MLaaS tools miss acts such as strangulation but flag guns as violent, demonstrating that they are ineffective in classifying harm as defined by feminist scholars. From a feminist standpoint, pornography is structured by misogyny. Thus, it functions as a site to test technology’s understanding of the structural nature of MVAW, showing that this failure is not a technical oversight but evidence of a broader issue within digital technologies. A combination of training datasets shaped by sexist assumptions, male-centred moderation standards, and biased developers creates a system that sees violence and pornography as mutually exclusive, failing to recognise harm. Without a feminist intervention, the technology defaults to norms that normalise and eroticise pornography as legitimate entertainment. This study contributes to debates on AI bias, technology-facilitated image-based MVAW, and pornography regulation. It provides insights for researchers, policymakers, and platforms seeking an informed basis for adopting a feminist approach to harm-detection.
Relevance
“Addressing the first question, the violence recognised by AI moderation tools does not correspond to feminist understandings of harm. Although officially designed to detect all forms of violence, these systems are not equipped to recognise misogynistic violence in pornography. For example, they can tag weapons or blood, but not strangulation, gagging, or restraint as violence.” In this way, we must “start treating the misclassification of violent pornography [by AI moderation tools] not as a glitch but as a symptom of the systemic failure of technology to register routine and structural MVAW [male violence against women] in pornographic content.”
“This leads to the second question which addresses what happens when the ideological frameworks that define violence – and train models to detect it – are shaped by a few powerful corporations influenced by misogynistic ideologies, while feminist understandings of harm remain marginal. The answer is that misogyny continues to spread and persist online. As shown in this study, when commercial AI moderation systems are confronted with pornographic material, MVAW becomes invisible to classifiers and, therefore, also to users. While this reasoning may appear paradoxical, it becomes plausible once we recognise that violence can remain visible in circulation yet invisible as harm. By being allowed to circulate freely through a system that cannot see it, violence becomes increasingly normalised…and even overtly aggressive acts become unworthy of scrutiny. In other words, the silent but systematic exclusion of MVAW from detection and moderation means that misogynistic violence remains invisible in plain sight. In this sense, the systematic exclusion of misogynistic violence from AI-based moderation constitutes a form of technology-facilitated image-based MVAW.”
“The inability of current tools to detect domestic and sexual violence, particularly, though not exclusively, in pornography, exposes the depths of misogyny in digital technologies and the limits of relying solely on technical solutions, without feminist insight, to address social harms.”
“Male-centred definitions of harm and moderation standards, combined with the structural misogyny embedded in the development of the technology itself – from tech culture to training datasets – mean that misogyny is integrated into these systems at every level, leaving moderation tools ill-equipped to detect or classify misogynistic violence or to engage with feminist definitions of harm. In other words, in AI-based moderation tools, MVAW is excluded by design, and – as a practice shaped by misogynistic violence – pornography makes this failure particularly visible.”
“Moreover, flawed content moderation based on narrow definitions of harm can be exploited by Big Tech companies to justify inaction and evade social responsibility, while increasing profits by reducing labour costs.”
Citation
Tranchese, A. (2026). What counts as harm? Feminist experiments with AI-driven visual content moderation to detect misogynistic violence in mainstream pornography. Big Data & Society, 13(3). https://doi.org/10.1177/20539517261447833
