This much resorces used on such activities? All for what? Boggles my mind. If its for defense intelligence, isnt it simply better and less costly to have spies and snitches ....?
The problem with America's intelligence agencies (and many other intelligence agencies) is compound. First and foremost - there are too many damned agencies. They need to comprise the various intelligence agencies and 'security' agencies into one. FBI, CIA, NSA, TSA, etc need to become one agency. While the CIA was formed and supposed to be distinct in that it only monitored foreign activity - that is not a realistic distinction in the digital world with air travel.
Along with that - much of that needs to be dedicated to what is called "Human Intelligence Networks."
Basically - spies and informants. The awesome thing about spies and informants is that, when properly placed, human intelligence networks are comprehensive in their coverage of human activity.
An informant within a large technology firm, such as Google, could relay information about suspicious accounts or account behavior to a network handler who would then index the lead or act on it with a warrant. Which is how the system is supposed to work.
In a perfect world - we could do without any kind of central agency (and I suspect we -could- get along just fine without any central security agency - private firms acting in protection of their own assets and those of their customers would file reports with the police regarding possible criminal plotting - but that is my near-anarchist streak speaking) - but in a realistic one, this is how you keep it functional and limited without sacrificing reliability.
Even with all of this data - some of which has been used to stop crimes - much of it was based on the information given to them by human intelligence networks. Thousands of terabytes of stuff to search through; thousands of hours of video with billions of faces to scan with software... how the hell can you know what is relevant?
In theory - it is possible to cache the information and then have it analyzed by algorithms within a supercomputer that could automatically compile a profile of individual identities from around the world - but the technical reality of such a project makes it unlikely such an endeavor is practical.
For a project like that to work - you are looking at roughly a year of design, a year of testing, and a year for construction and installation of supercomputing networks.
In that amount of time, the amount of internet traffic you would need to collect in a given day has nearly quadrupled. The amount of data your computer must be able to access at any given moment for proper analysis expands geometrically over time. Adding more computers to the network also means adding more storage capacity in a manner that can keep the processing units fed with data. So if they started on some grand project even as recently as 2010 - it is horribly insufficient for their goals.
Which means they have to fall back on using human intelligence to reconcile against the mass of shit they have collected.
So you may as well go back to relying purely upon human intelligence networks that can permeate anything human beings do.
I remember not too long ago when I got my first phone when my parents adopted a family plan on Cingular - even before roll over minutes. The "internet" on phones was a clunky abomination, at the time. Most of the 'games' were silly little things contrived by the marketer that were often there to make you wish the damned thing came pre-loaded with "snake" - which was a better time killer.
I pretty much used it to keep in contact with my parents when I was taking courses at the college and started a part time job and needed to organize car rides.
Five years later, I was cursing our cell tower for being too slow to properly browse the internet in pre-3G America, and was texting up a storm with a girl I met (which later turned into a mess - but, hey).
Five years later, still, I am cursing the lack of 4G towers to properly stream videos on my phone - which is now more powerful than the computer I had ten years ago with a touch screen display that makes the hand-held game console a thing of the past. I have physics simulators on the thing, CAD programs, like three different free navigation programs that make the Garmin my dad bought several years ago an obsolete thing.
The way we use technology and the rate at which we consume information has changed so radically that it makes any attempt at information gathering without specific targeting/profiling naive and foolish.
A few years before I got my first cell phone? I was on a computer, talking on internet forums over dial-up internet. If I got 10Kb/s on downloading anything - I was sure that some part of the modem or wiring was going to melt down or start spitting sparks. There was no "MySpace." The closest thing to "YouTube" was some variety of file sharing service. Most internet shopping was a glorified form of mail-order that companies were still trying to figure out. Amazon was getting there - but they were still up and coming.
The world I was raised in has almost become obsolete. I don't go to a store to buy a computer. People don't exchange phone numbers like they used to - they send face-book friend requests. People hardly write e-mails, anymore, outside of business interactions. Letters sent by mail are reserved for the most formal of affairs and invites.
While part of it is because my family was literally destroyed - I haven't seen a Christmas card in at least four years. Even things like wedding invitations are starting to go more informal and digital.
Our world is transforming faster than intelligence agencies can devise ways of searching through it without permission or oversight.
Yet if they were utilizing human intelligence networks - all of this would already be taken into account. The informants would migrate to Facebook and would keep their ear to the ground on their twitter feeds. They would make note of provocative individuals in forums and take note of when someone's youtube channel starts advocating genocide, or something.
A solid human intelligence network is where it is at.
[video=youtube;1c3YIbEt2wg]http://www.youtube.com/watch?v=1c3YIbEt2wg[/video]
