To a parent, few things are scarier than something wrong with your child’s health. When Mark (no last name given for privacy reasons) noticed his son had an inflamed and what appeared to be painful sore on his penis, he monitored it and contacted a nurse. The nurse asked for photos, which Mark’s wife sent along to their doctor. Thanks to those, the doctor made a diagnosis, prescribed antibiotics, and the issue was resolved.
Unfortunately, for Mark, that wasn’t the end of his family’s problems. It was the beginning of an issue that would permanently disrupt his digital life, make him the subject of a police investigation, and even put him at risk of losing custody of his child.
Google had flagged him for distributing harmful content involving minors, usually called “child sexual abuse material,” or just “CSAM.” With that, he lost access to everything online, and, even after a police investigation proved his innocence, Google refused to help.
Google flagged and punished an innocent man, and now refuses to fix their mistake. Mark’s far from alone. It has to do with the unique methods Google uses to monitor CSAM, which is potentially more prone to false positives and longstanding issues than more traditional CSAM-scanning methods.
In This Article:
False Flagging
Mark had only been trying to help out his son. He even followed the guidelines of a doctor. He noticed a problem, made the efforts to track it, got a doctor to diagnose the issue, and helped his child. However, this involved using his Android phone. While both he and his wife had the photos on their phones, her iPhone did not flag her as a criminal, and rightfully so. Mark’s only “crime” was trying to get healthcare for his son, but Google wedged itself between doctors and patients, uprooting Mark’s life.
“I knew that these companies were watching and that privacy is not what we would hope it to be, but I haven’t done anything wrong.”
– Mark
Google had already been looking into Mark, apparently. An older video on his device featured his wife and baby in bed. The wife was unclothed. Mark says he doesn’t remember the specifics, but admitted to not sleeping with pajamas, saying, “If only we slept with pajamas on, this all could have been avoided.” He forgot the cardinal rule of modern life: always wear clothes around your house, Google could be watching.
“I applaud Google for what they’re doing. We do have a horrible problem. Unfortunately, it got tied up with parents trying to do right by their kids.”
– Dr. Suzanne Haney, chair of American Academy of Pediatrics’ Council on Child Abuse and Neglect
Mark wasn’t alone. Cassio (just his first name again), had a toddler with a similar issue. At his doctor’s request, he sent photos of his child’s genitals to the doctor’s office. Like Mark, this was enough to get him in trouble. It came at a terrible time. Cassio was trying to buy a house, and the broker became suspicious when he had to use a different email. Cassio, like Mark, had his entire digital life shut down when Google thought he was a criminal.
Both Mark and Cassio were investigated. Neither were found to be in possession of CSAM or guilty of any wrongdoing. Despite this, Google refused to reinstate their accounts, and hasn’t explained why. When asked for comments, Google only gave publications a generic answer.
“Child sexual abuse material is abhorrent and we’re committed to preventing the spread of it on our platforms.”
– Google’s Response
Jon Callas, of the Electronic Frontier Foundation (EFF) pointed out that, “There could be tens, hundreds, thousands more of these.” Google claims to have flagged 600,000 instances of child abuse and 270,000 users in 2021 alone. How many were falsely accused? According to a Facebook study on CSAM reports, 75% were “non-malicious”. How many had their lives uprooted because they were trying to have a private conversation with their child’s doctor?
What Happens When Someone’s Flagged?
Both Mark and Cassio found themselves locked out of all of their Google services. This was especially damaging for Mark, who was a bit of a Google superuser. Mark had email, contacts, photos, videos, and even his cellular service through Google. Because Mark was on Google Fi for cellular service, when he lost access to his Google account, he lost his entire phone. He couldn’t even easily contact Google support anymore, as Google locked him out of phone calls, text messages, and emails. Even his other accounts, from other companies, were locked. When you use two-factor authentication tied to Google Authenticator or text message tied to Google Fi, you lose everything when Google locks you out.
The San Francisco Police Department (SFPD) investigated Mark and cleared him of any wrongdoing. However, that investigation took 10 months. That’s 10 months in the dark. Once Mark was cleared, he thought he would be able to get his account back. He appealed to Google, now with proof of his innocence in hand, and they ignored his request. It didn’t matter that he was innocent, Google didn’t care. The same happened to Cassio when the Houston Police Department finished their investigation. Victims of Google’s false claims get nothing from the company, not even their own data, photos, or emails back.
“…systems like this could report on vulnerable minorities, including LGBT parents in locations where police and community members are not friendly to them.”
– Joe Mullin, writing for the EFF
Losing your accounts seems like nothing though, compared to how bad this could have been. Kate Klonick, a law professor at St. John’s University, says Mark and Cassio could have lost custody of of their children. We know policing isn’t always fair in this country. Racial minorities may not get the same trust as white suspects, and LGBTQ people find themselves fighting both bigotry and Google’s accusations. There’s a chance families that look different than Mark and Cassio’s could have been hurt far worse.
A false positive is one thing. But refusing to correct your mistake later puts people in a terrible position.
How Did It Go Wrong?
The standard way of scanning for and flagging CSAM is by hash comparisons. Put simply, actual images of abuse are never compared. Instead, an algorithm takes an image and creates a hash, a value made up of seemingly random symbols. This hash is mostly unique. Because they want to catch people who flip the image, crop it, remove the color, or distort it in other ways, the hash is not a strict one-to-one comparison, like hashed passwords or keys. This means there are occasional false flags. When this happens, a person compares blurred versions of both photos, to see if they appear to be the same image, and passes it on to the authorities.
Google doesn’t do this, or they don’t do just this. Instead, they introduced AI, and that introduced its own problems.
AI Trained on What, Exactly?
Since 2018, Google has used AI to scan for exploitative images of children in people’s images and videos. They released this as the “Content Safety API.” Google’s not alone in using it, Facebook also uses the service, and possibly Meta’s services, Instagram, WhatsApp, and Oculus as well. Though Facebook has a questionable record of protecting children.
There are multiple problems with this. First, AI isn’t precise, and could have even more false positives than a hashing method. Seeing as Google doesn’t follow up on cases, that can be devastating to innocent victims of Google’s mishandling of their data. Secondly, how exactly are they making this AI? To make an AI able to recognize particular images, you use a process known as machine learning. This involves processing large datasets in which a computer can eventually recognize patterns. There are multiple ways to do this, and I am oversimplifying it greatly, but basically put, they couldn’t make an incredibly accurate CSAM-detecting AI without abusive material of minors. Without it, it would just find potentially nude photos of children, and, as a former baby with a mother, I can tell you there exists many “bath time” photos of me. Is my mom a perv? No. It’s weird, sure, but parents become desensitized to it because they don’t sexualize their children and change their diapers often.
A poorly trained AI increases the odds of photos of parents and their children being flagged as abusive. Unfortunately for the people in the photos Google flags, the problem doesn’t end with their own issues from that false flagging, like a police investigation, possible loss of their children, and losing access to potentially all of their digital accounts. Google also uploads their private content to CyberTipline. Here, those images are stored, hashed, and used to find future violations of child abuse. Do they only do this for photos that police verify are CSAM, or for everything they flag? Does Google ever upload false positives? An intimate family moment used only to embarrass teenagers when they bring their significant other over could now be stored on a server somewhere.
Google says they have a human in the process who looks at the images and makes an assessment. But they default to assuming the worst because an investigation is safer than allowing abuse to continue. Although, creating an AI that hunts for photos of naked children to show to a person is not the defense Google thinks it is.
The Family Photo Album Should be Private
Google says they only scan items when a user takes an “affirmative action,” but to Google, that involves not turning off one of their services. It’s so hard to opt out of all of Google’s tracking, especially on an Android device. It seems every third time I open the photos app on my Android phone, it prompts me to upload my images to their cloud. I use my phone as a work device, I cannot and do not want my photos or work-related images ending up on Google’s servers. It’s hard to avoid them.
Then there’s the fact that families often involve more than one person. People share these images with each other. In fact, in both Mark and Cassio’s cases, their wives also had the images. At least in Mark’s case, we know she wasn’t flagged because she had an iPhone. Apple only uses AI to scan for nudity on children’s phones, and only in messages, to prevent them getting unsolicited nudes.
“This is precisely the nightmare that we are all concerned about. They’re going to scan my family album, and then I’m going to get into trouble.”
– Jon Callas, a technologist at the Electronic Frontier Foundation (EFF)
Google seemingly would rather default to false positives, cut people off of their service, and create headaches for those investigating actual abuse cases than let that abuse stay on their servers. But Google could be worried that admitting false positives happen could open them up to concepts like false negatives and other investigations into their storage practices. It’s probably easier for them to pretend it works perfectly and ignore any criticism.
Police have the contents of Mark’s Google account on a flash drive, private and intimate photos and all. They found no wrongdoing and are currently working to get him that data. At least his family photos from the time of his child’s first year of life aren’t digitally shredded. Google tried to cut him off from those memories, but the SFPD will be able to return them.
A Necessary Service, But With Room to Improve
It’s hard to take a stand against this kind of scanning, even when the results of false positives are devastating. In 2021, CyberTipline reported over 4,260 potential new victims. That potentially includes false positives, like Mark and Cassio’s, but it likely includes many children saved from abuse as well. Thousands of children saved in just one year. The system certainly can protect children, but it doesn’t need to put people in harm’s way.
Google and Facebook could go the route of Apple, using human investigators to flag images of CSAM and only comparing hashed images. This protects privacy. However, Apple’s method was flawed as well, and made room for tyrannical government abuse with images that aren’t CSAM to flag potential political dissonance. There was also an issue with false positives. Apple went back to the drawing board over their system last year. Google hasn’t had such reservations.
Google shouldn’t use AI for this process. At the very lease, they shouldn’t do it in-house. It should be a completely separate third party that has interest in protecting children and families, not a brand. Still, if they’re going to refuse to split off their process for the sake of protecting more families, they could at least improve that process. They should wait until authorities have investigated potential criminals before adding photos to CyberTipline. They also should not lock users out of their accounts when suspected of distributing or storing CSAM. Instead, they should just limit their accounts so they cannot upload new photos or videos anywhere across Google’s services. Finally, they need to admit when they’ve done wrong. If a false positive hurt someone, Google needs to make it right. When an investigation reveals someone’s innocence, Google needs to unlock their account so they can recover what they’ve lost and possibly repair their reputation. Google’s responsibility is only to flag potential problems, not dole out punishment. That’s what our justice system is for. Google can choose who’s on their platforms, but doing so for this reason after someone is proven innocent feels akin to digital slander. Many of these people don’t even want to talk about this issue because even a false accusation could ruin their reputation.
The way Google is doing this now gets in the way of parents trying to talk to their doctors about their children’s healthcare. With an increasingly remote presence, and doctors realizing they can help more of their patients with technology, parents may want to use those technologies to get help for their children immediately. With Google’s current system, parents have to worry if the corporation will have their kids taken away for trying to help them. No one should be allowed to get between people and their doctors.
Nail the Distributors
While it would be nice to be able to scan every piece of media that could be CSAM safely, only Apple’s method came close to respecting privacy, and still had false negatives and abuse potential that made it a risk. However, if companies only applied their methods to repeatedly shared images, the risk is lower. While this could help stop distributors of CSAM, it still potentially gets in the way of people talking to their doctors, though less than current processes.
There’s no perfect solution yet, but scanning people’s personal photos and communications with their doctors is definitely not the right path. The EFF has had some solutions that respect privacy and still improve our scanning capabilities.
If Google had complete confidence in their technology, they could tell the police there is an emergency and send them to kick down a person’s door the moment they detect something. They don’t do this, possibly because they don’t trust their system enough to have that certainty. So why don’t they fix the problems they make? If Google can’t trust their system, why should we?
Sources:
- Grace Dean, Business Insider
- Kashmir Hill, NY Times
- Joe Mullin, EFF
- Andrew Orr, AppleInsider
- Emma Roth and Richard Lawler, The Verge