Yup, you got it, the content itself will remain only in the device, the hashing is done in-browser, and the only part of the original content that makes it into the system is the hashes. Once a platform that is part of the program downloads those hashes and is able to match content, you need to apply some amount of verification. It’s on the participating companies themselves to review the content that matches the hash, to see if it actually violates their policies on NCII.

It’s somewhere in the FAQ at https://stopncii.org/faq/, but It’s PDQ for photos (https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5 for videos. PDQ is resistant to some modifications (it focuses on the ones that come from regular usage, such as changing the format from gif to jpg, or a filter changing colors or brightness), but it’s not as resistant to modifications as you could get by training dedicated classifiers or other approaches that you might do with the original media or by storing more context, which StopNCII chose not to do.

Yup, PDQ (https://github.com/facebook/ThreatExchange/tree/main/pdq) and MD5- https://stopncii.org/faq/. MD5 is a cryptographic hash, which means that even a single bit changed gives you an entirely different hash. This usually precludes any attacks to try and deliberately generate collisions, but also means it's harder to match “benign” changes - for example, many platforms re-encode videos as part of the upload process (reducing resolution, changing formats, etc). However, many platforms skip re-encoding, so MD5 can have better results than you might think at first glance. The more bits you take from the original content, the harder you can make it to bypass, but the more you might have to worry about capturing enough that you might compromise the privacy of the original submitter. For the program, StopNCII picked a set of tradeoffs between those two tensions, and are keeping a close eye on how effective it is, and will iterate if need be.

It’s a real challenge - you can try and do more pre-processing on a submitter’s device to try and avoid mistakes or malicious use of the system. The big limitations are:

1. How much processing time can you put on what might be a mobile phone - it’s using MD5 for Videos, but StopNCII did also try other things that were just too slow for phones.

2. StopNCII chose to do verification on platforms after a match rather than do it up front to prioritize the privacy of the user. This decision came after a lot of feedback from victims and experts who represent them. There are all sorts of techniques you can use to process server side (thus saving the person’s phone’s CPU), but it deliberately doesn’t retain enough context to make them viable.

Another commenter (“ahahahahah”) described it as the equivalent of a “report image” button, which I thought was apt - it preemptively reports on all the platforms that are participating with StopNCII. It comes down to each platform: Meta has specific rules against NCII (https://transparency.fb.com/policies/community-standards/sex...) - we prioritize potential NCII content as high severity and requiring immediate review to verify. However, just like hitting the report button on each of those platforms, they might have different rules.

I like the comparison of the program to hitting “report image” on a platform. Advantages of StopNCII is that you can report proactively, and to report it to all the participating platforms at the same time.

Misuse of the StopNCII platform is something that we (Meta/Facebook), UK Revenge Porn Helpline and the full StopNCII team discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-positives, because now you can use your platform-specific ML, or in-house photo detection algorithms.

> "Misuse of the StopNCII platform is something that we (Meta/Facebook) discussed a lot. Once you have confirmed that a piece of content violates your platform’s policy, it’s easier to find other instances and filter out false-po"

Okay, and what was the conclusion then? Can I use the service to take down arbitrary content online that has nothing to do with the subject at hand? Can I just incriminate other devices, users and accounts that store innocent data and get it deleted without anybody's consent?

Hey everyone! I’m a software engineer at Meta, and I helped work on the program that the newsroom post is about. Sorry I’m late, I only just found the thread! I might be able to answer some questions about the technology - many of the building blocks are open source, like the PDQ photo hashing algorithm (https://github.com/facebook/ThreatExchange/tree/main/pdq) mentioned in FAQ (https://stopncii.org/faq/). I might not be able to answer every question, but I’ll do my best!