As the world's leading streaming platform Twitch continues to grow, so do the problems it encounters with all sorts of malfeasance: 2022 was perhaps the most dramatic year in Twitch history (opens in new tab), and 2023 will be another kind of serious It opened with a scandal: popular streamer Atrioc live-streamed a visit to a so-called deepfake website that posted blatantly fake images of female streamers, including Maya Higa and Pokimane

.

Atrioc later apologized (open in new tab) and stepped back from streaming for a while, but the damage was done, and the pain caused to the streamers on the site became public and obvious; Twitch now has a "Deep Fake Porn" campaign, both on and off the platform, which is "a new way of saying that the Deep Fake Porn" incident and has outlined a plan (opens in new tab) to address the term itself, beginning with addressing.

The term "deepfake porn" is widespread, but following the advice of experts such as Danielle Keats Citron (law professor and vice president of the Cyber Civil Rights Initiative) and the UK Revenge Porn Helpline, Twitch has decided to call this material "synthetic non-consensual exploitation images" or "synthetic NCEI". Here's why:

"Calling synthetic NCEI content "pornography" is incorrect. Pornography should be consensual (although it is not allowed on Twitch) and should feature people who are aware that they are participating in an act that others will see. That is not what is happening here, which is why it is important to clearly reinforce that this content was created without consent.

Incidentally, this is somewhat similar to what is currently happening with the term "revenge pornography," which is considered problematic by experts in the abuse field: not least because "revenge" implies that the victim did something to deserve it. In this case, the term currently favored in the field is "image-based abuse," which is more clinical and precise about the nature of the offense and has less victim-blaming connotations.

Finally, on the issue of terminology, Twitch notes that while "deepfake" is something of a modern buzzword, it is not necessarily an accurate description of the methods used to generate such content.

This is all important because language is a policy component. Synthetic NCEIs are prohibited under Twitch's current policy and will of course continue to be prohibited, but in light of the events of January, the policy language on two sites will be changed:

1. The policy on adult sexual violence and exploitation will be updated to state that synthetic NCEIs are intentionally advertised, created, or shared, to make it more clear that a first offense may result in indefinite suspension.

2. Update the Adult Nudity Policy to include synthetic NCEIs. Even if the NCEI is only displayed for a short time or, for example, to express outrage or disapproval of the content, it will be removed and subject to enforcement.

Twitch is currently hosting a creator camp on March 14 to gather further community and expert input on how to address these issues in the future. The camp will be led by Zara Ward, an NCEI expert and manager of the Revenge Porn Helpline, and herself a streamer and member of Twitch's Safety Team Twitch will also be the first point of contact for those affected by synthetic NCEIs a list of resources (opens in new tab) that they should contact first.

These are clearly new technologies and can be used for more than just synthetic NCEIs; Twitch is stepping into uncharted territory in terms of proactively addressing NCEIs and acknowledges that there may be more nuanced incidents besides the obvious horrific ones acknowledges: "Not all synthetically created content is sexual, and not all is nonconsensual. The creation, promotion, or viewing of such content is not welcome on Twitch."

The new policy will be implemented this month. Following the Creator Camp, Twitch promises further updates on how its approach will evolve. This will be an interesting area for more borderline AI-dependent and difficult-to-analyze content on Twitch, rather than the more obviously heinous stuff that was explicitly covered here. This year, an AI-led "Seinfeld Next Door" show debuted, lasted a few days, and then was taken off the air due to jokes (opens in new tab) making fun of homophobia and transphobia Welcome to the future of AI: It's a mess.