On 17 November, Meta Platforms announced changes to its recommendation systems, specifically for India, to keep sexually suggestive content out of users’ recommendations. It has improved its detection system using AI models to ensure content is checked against its recommendation guidelines before it is pushed to users.
Meta also said that users can now control what they see through the content preferences option, where they can choose not to watch sensitive content or political content.
Despite the announcement, worries persist that tackling non-consensual deepfakes through regulation has become nearly impossible because identifying the individuals who shared the media is difficult, and controlling the generation of such content is harder as numerous tools exist and have become easier to access.
A recent report published by the Centre for Communication Governance (CCG) at National Law University (NLU) Delhi raises questions about how to control the creation of non-consensual deepfakes and the gaps that still exist in the digital space, making it difficult to regulate the spread of such content.
How Non-Consensual Synthetic Imagery and Deepfakes Have Become Easy To Create
According to the report titled “But that's not me!": Sexualised Deepfakes in Law, Tech and Society, a combination of factors make it easy for NCSI (Non-Consensual Synthetic Imagery) to be created and disseminated.
One of the contributors is the low to no-cost tools, such as GrokAI and Nudify, that make it easy to create and push content with minimal repercussions.
Furthermore, training data for numerous AI models includes intimate imagery, making it easy to create non-consensual content using existing images and videos shared by users on social media.
Recommendation algorithms promote content to new audiences and are only focused on its potential for engagement. However, this translates to NCSI being pushed to audiences before it gets detected, a key example being the Grok AI case where X pushed the content at scale. This is how NCSI get recommended to people whether they actively search for it or not.
They also point out that the internet doesn’t differentiate between NCSI content and other content, which helps recommendation algorithms push the content. This also means payment processors do not have any restrictions on the transfer of such content. App stores also play a role, by hosting such apps.
The existence of a supply chain that provides monetary incentives for creating such content also promotes it further and keeps the system alive.
Content moderation, which is how platforms evaluate and take action against questionable content, is based on platform policies and regulations. However, these are unhelpful in numerous situations since these policies are subjective, making them difficult to implement. They are also not comprehensive and don’t have separate policies for NCSI apart from the general ones for sexually exploitative content.
Grievance redressal on these platforms fails to be user-friendly, with the process of filing a complaint being difficult to comprehend for users. Additionally, the actions following a complaint are also not adequate at times. Furthermore, these systems fail to understand the sheer amount of NCSI content that a user can be faced with, making it difficult to report every single user sharing the content.
The Intersectionality of It in the Indian Context
In the Indian context, gendered violence through NCSI doesn’t exist independently. The intersections of caste, class, religion and sexuality change how women are impacted by such content. The report by CCG highlights that in Indian society, deepfakes function as a weapon that threatens women's social status, dignity, honour and even livelihood.
For example, for a Muslim woman, the intersection of gender and religion makes non-consensual synthetic media a tool for further abuse and harassment.
The Quint conducted a deep-dive into how female Muslim journalists faced harassment and gendered disinformation due to their gender and religious identity, which highlighted that this took various forms, “including doctored images and videos that misrepresent the journalists' views or actions.”
Dalit women are also subjected to harassment, which is not limited to sexist comments but includes casteist elements as well. As stated in the report on how NCSI impacts minority, “for Dalit and Adivasi women, it can reproduce historical patterns of caste-based sexual violence, humiliation and narratives of ‘pollution’ that reaffirm caste hierarchy.”
In a country that is hostile towards the queer community, individuals already exist in a digital space that is characterised by hate and harassment. With AI being used as a tool for online abuse, the digital space may amplify the situation.
Lack of Technology-Based Support
As mentioned earlier, the current system in place for grievance redressal and how content is managed doesn’t provide proper support to victims of non-consensual synthetic content.
Content management policies, being subjective and not comprehensive, are implemented inconsistently. According to the report, the takedown process is a flawed system for preventing the spread of non-consensual synthetic content.
The report states, “Even when takedown systems are in place, they do not catch 100% of the material the companies prohibit. And in that gap, real harms persist.”
Another flaw with content takedown is that once content is shared on a platform, it can be downloaded, reshared or reuploaded on another platform with less stringent rules.
The Quint had conducted a deep dive on X’s Grok’s alarming trend of generating images of real women in minimal clothing when prompted with ‘put her in a bikini’. According to the article, a report published by the Centre for Countering Digital Hate (CCDH) stated that, based on a sample they estimated, Grok produced nearly 30 lakh sexualised images between 29 December 2025 and 9 January 2026.
The article also notes that while the platform’s policy prohibits users from sharing non-consensual intimate media, the policy doesn’t lay down rules for AI-generated non-consensual intimate imagery, highlighting a gap in the policy.
Grievance redressal, in addition to being complicated, poses a different challenge in the Indian context. The different level of digital literacy among different demographic groups makes grievance redressal even more difficult. If an English-speaking, digitally native user can struggle with the interface, then it would make it even more difficult for users with different language proficiency and digital literacy.
Lack of Proper Enforcement of Regulations
According to a study published in the International Journal of Legal Science and Innovation, Deepfake Regulations in India: Challenges and Way Forward, despite the amendments to the IT Rules, significant gaps exist.
One of them is that personality rights protection is celebrity-centred. Historically, the protection of personality rights was much more important for celebrities’ identities, considering there was market value associated with it. However, the growth of AI has shown that this is not a threat limited to celebrities, and a common person can be affected by the same.
Another challenge, according to the report, is the lack of an independent cause of action for appropriating someone’s identity, or likeness through AI-generated media. The wrongdoing will have to be handled through existing remedies such as privacy violations, copyright or trademark, or defamation.
The report also points out the rigorous process that a victim of unauthorised creation or sharing of AI content has to undergo, including filing a complaint with the hosting platform, filing a complaint with the police and one with a civil court, and initiating criminal proceedings. The process becomes painstaking for a victim and can delay any possible outcome from the procedure.
The way forward would require creating a safe digital space through regulatory systems offering comprehensive regulatory systems and safeguards that will protect victims against the threat of online harassment through generative AI.
