To curb online bullying, Instagram has to spot it

These efforts are still unproven, and, like any Facebook-related promises, they’re best taken with a heaping handful of salt. But Instagram seems to be more aggressive about this than competing platforms like Twitter and Snapchat.

If you want to stamp out bullying, you first have to know what forms it takes. So late last year, Instagram began assembling focus groups of teenagers and parents and gathering feedback about what types of unwanted behavior they encountered on the platform.

Some were the predictable types of threats and insults — like rating users’ attractiveness on a 1-to-10 scale, a practice that Instagram already prohibits — while others were more unexpected.

Some teenagers reported feeling bullied when their exes showed off new boyfriends or girlfriends in a menacing way — for example, by tagging the jilted ex in the photo to trigger a notification and rub in the fact that they had moved on to someone new.

Instagram came up with a name for this category of bullying — “betrayals” — and started training an algorithm to detect it.

“One of the things we learned early on is that how we were defining bullying in our Community Guidelines doesn’t necessarily capture all the ways people feel like they’re being bullied,” said Karina Newton, Instagram’s global head of public policy.

Not all of these behaviors necessarily violate Instagram’s rules. The company has not yet decided where to draw every line; for now, it is just trying to understand bullying’s many flavors and teach machines to flag them for human reviewers, who then decide whether or not they violate the platform’s rules.

Facebook and Instagram already use AI to detect various types of off-limits content, including nudity, child exploitation and terrorism-related material. But classifying bullying is a bigger challenge, because doing so often depends on the context of a social interaction.

Take one of the examples used by the executives during Tuesday’s briefing: a photo of two teenage girls that was posted to Instagram with the caption “love you hoe.”

Normally, Instagram’s systems would pick up on the derogatory word “hoe” and flag the post to a human reviewer. But in context, it’s clear that the user meant it as a term of endearment, so the correct action would be to leave the post up.

Or consider a hypothetical photo of a teenage couple at the beach, posted to Instagram with the caption “Wish you were here, Amanda!”

Normally, that post would be bland and inoffensive. But you can imagine contexts in which it would constitute bullying:

Are the people in the photo mocking Amanda for being the only senior not invited to Beach Week? If so, it could constitute “intentional inducement of FOMO.”

Is Amanda the ex-girlfriend of the boy in the photo, being taunted by the new girlfriend? If so, it could be classified as a form of betrayal.

Is there a whale in the background that is tagged as Amanda, as a cruel joke about her weight? If so, it could be classified as an insult.

It’s odd to realize that what Instagram is describing — a planetary-scale AI surveillance system for detecting and classifying various forms of teenage drama — is both technically possible and, sadly, maybe necessary. It should make us all question whether a single company should have so much power over our social relationships, or whether any platform of Instagram’s size can be effectively governed at all.

But if you have to have an Instagram-size platform, there are arguments in favor of using AI to seek out bad behavior, rather than wait for users to report it. One reason, Instagram’s executives said, is that teenagers often don’t report bullying when it happens to them. Some fear social repercussions or retaliation from their bullies, while others fear that their parents will take away their phones.

Eventually, the company hopes its AI will be good enough to identify and remove all types of bullying on its own, without the need for human review. But, executives cautioned, that day may be distant, especially outside the English-speaking world, where it has fewer moderators and less local-language data available to train algorithms.

“Our algorithms aren’t yet as good as people when it comes to understanding context,” Fairman said.

Instagram’s critics probably won’t be satisfied that, after making billions of dollars in profits and contributing to what researchers say is an epidemic of teenage depression and anxiety, the company is now trying to dismantle the culture of social media bullying it helped to create.

“Where were they five years ago? It’s about time, honestly,” said Jim Steyer, the chief executive of Common Sense Media, a nonprofit watchdog group that advocates better protections on children’s technology. “This has been a huge issue for years, and most of these companies buried their heads in the sand until they were under pressure to do something about it.”

It’s true that Instagram’s anti-bullying effort may be useful for generating good publicity, and that the company seems to be making up some of the details as it goes along. It’s also true that Instagram has a multitude of serious problems on its hands — including anti-vaccine misinformation and rampant hate speech and extremism — and that building AI to detect bullying is probably a more convenient challenge than rethinking the ad-driven business model and platform design issues that encourage anti-social behavior in the first place.

But better too little, too late than nothing, ever. Instagram’s bully-detecting AI is a good idea, and a step toward giving young people an easier time navigating the vicissitudes of 21st-century adolescence. For their sake, let’s hope it works.