At what point do the mathematical limits of data mining kick in here? How useful is all this information?
I'm not an expert in this area of mathematics, so I could be wrong, but my impression is that as the haystack becomes larger the problem of false positives becomes more and more severe.
As a data miner, what you want is the maximum number of "hits" (of whatever you're trying to hit) with the minimum number of misses and the minimum number of false positives. My impression is that this becomes progressively harder-- the golden region between too many false positives and too many false negatives becomes smaller and smaller and harder to hit.
Eventually you either miss important hits, namely the next terrorist attack, or you get swamped with false positives that you have to manually investigate and rule out.
I'd love someone who does know more here to chip in, but my personal suspicion is that this actually has a pretty huge pork angle to it. How much money are the contractors getting for building this stuff?
so let's say there's a law that says "any American company doing business with a company that does business with a known terrorist organization will have a bad day"
you don't need to use some kind of fancy data mining algorithm for this to work (generating false positives), you just need a ho-hum graph traversal algorithm and unbelievable amounts of graph data to generate "candidates for investigation".
US Company A -> intermediate 1 -> known terrorist group B
US Company A -> intermediate 2 -> known terrorist group B
US Company A -> intermediate 3 -> known terrorist group B
Each set of links is just one lead to investigate, but having a giant graph to work off of would make generating those leads simply. You might find out that intermediate 1 is a local falafel delivery place that "US Company A" uses for lunch catering. Can probably strike that one off the list. intermediate 2 is a utility (no choice but to use the local water monopoly), but intermediate 3 is a material supplier that employs several low level delivery guys from known terrorist group B, and the founder of the company is a cousin of the founder of known terrorist group B.
So I'd wager it's not as simple as just running an algorithm and automatically sending out Skynet drones to blow things up. There's some kind of more subtle assessment being made, with the systems just providing help to the analysts.
The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours."
I'm not an expert in this area of mathematics, so I could be wrong, but my impression is that as the haystack becomes larger the problem of false positives becomes more and more severe.
As a data miner, what you want is the maximum number of "hits" (of whatever you're trying to hit) with the minimum number of misses and the minimum number of false positives. My impression is that this becomes progressively harder-- the golden region between too many false positives and too many false negatives becomes smaller and smaller and harder to hit.
Eventually you either miss important hits, namely the next terrorist attack, or you get swamped with false positives that you have to manually investigate and rule out.
I'd love someone who does know more here to chip in, but my personal suspicion is that this actually has a pretty huge pork angle to it. How much money are the contractors getting for building this stuff?