Apple, however, has made the argument that it has set up multiple fail-safes to stop this situation from ever really happening.
For one thing, the CSAM hash database encoded into future iPhone operating systems is encrypted, Apple says. This means that there is very little chance of an attacker discovering and replicating signatures that resemble the images contained within it unless they themselves are in possession of actual child porn, which is a federal crime.
Apple also argues that its system is specifically set up to identify collections of child pornography—as it is only triggered when 30 different hashes have been identified. This fact makes the event of a random false-positive trigger highly unlikely, the company has argued.
Finally, if other mechanisms somehow fail, a human reviewer is tasked with looking over any flagged cases of CSAM before the case is sent on to NCMEC (who would then tip-off police). In such a situation, a false-positive could be weeded out manually before law enforcement ever ostensibly gets involved.
In short, Apple and its defenders argue that a scenario in which a user is accidentally flagged or “framed” for having CSAM is somewhat hard to imagine.
Jonathan Mayer, an assistant professor of computer science and public affairs at Princeton University, told Gizmodo that the fears surrounding a false-positive may be somewhat overblown, though there are much broader concerns about Apple’s new system that are legitimate. Mayer would know, as he helped design the system that Apple’s CSAM-detection tech is actually based on.
Mayer was part of a team that recently conducted research into how algorithmic scanning could be deployed to search for harmful content on devices while maintaining end-to-end encryption. According to Mayer, this system had obvious shortcomings. Most alarmingly, researchers noted that it could be easily co-opted by a government or other powerful entity, which might repurpose its surveillance tech to look for other kinds of content. “Our system could easily be repurposed for surveillance and censorship,” writes Mayer and his research partner, Anunay Kulshrestha, in an op-ed in the Washington Post. “The design wasn’t restricted to a specific category of content; a service could simply swap in any content-matching data base, and the person using that service would be none the wiser.”
The researchers were “so disturbed” by their findings that they subsequently declared the system dangerous, and warned that it shouldn’t be adopted by a company or organization until more research could be done to curtail the potential dangers it presented. However, not long afterward, Apple announced its plans to roll out a nearly identical system to over 1.5 billion devices, in an effort to scan iCloud for CSAM. The op-ed ultimately notes that Apple is “gambling with security, privacy and free speech worldwide” by implementing a similar system in such a hasty, slapdash way.
Matthew Green, a well-known cybersecurity professional, has similar concerns. In a call with Gizmodo, Green said that not only is there an opportunity for this tool to be exploited by a bad actor, but that Apple’s decision to launch such an invasive technology so swiftly and unthinkingly is a major liability for consumers. The fact that Apple says it has built safety nets around this feature is not comforting at all, he added.
“You can always build safety nets underneath a broken system,” said Green, noting that it doesn’t ultimately fix the problem. “I have a lot of issues with this [new system]. I don’t think it’s something that we should be jumping into—this idea that local files on your device will be scanned.” Green further affirmed the idea that Apple had rushed this experimental system into production, comparing it to an untested airplane whose engines are held together via duct tape. “It’s like Apple has decided we’re all going to go on this airplane and we’re going to fly. Don’t worry [they say], the airplane has parachutes,” he said.
A lot of other people share Green and Mayer’s concerns. This week, some 90 different policy groups signed a petition, urging Apple to abandon its plan for the new features. “Once this capability is built into Apple products, the company and its competitors will face enormous pressure — and potentially legal requirements — from governments around the world to scan photos not just for CSAM, but also for other images a government finds objectionable,” the letter notes. “We urge Apple to abandon those changes and to reaffirm the company’s commitment to protecting its users with end-to-end encryption.”