CIA / NSA Top Secrets Leaked by Traitor - One Example

[Reality Check]

Top Secret - 32 Page - Training Document - that discloses Foreign Intelligence Sources and Methods

http://www.theguardian.com/world/intera ... esentation

Those familiar with technology, such as John and perhaps Aedens, will be able to read between the lines and deduce just how the "Soft Intelligence" referenced here is collected and stored.

The global map of server locations is very telling.

[John]

You have to apply a common sense test to articles like this.

* How much storage would be required to store every e-mail message
from every user in the world, including many spam messages?

* How much processing power would be required to analyze each
message for whatever keyword searches they're doing?

* Same with online chats and all the other data?

* How much disk storage would be required to store all that?

* Assuming that the mean time between failure of disks is a
couple of years, how would the issue of backing up all this
data be handled?

* How many computers would be required to do all this
processing?

* How much network bandwidth would all this suck up?

* How big a staff would be required to keep this facility up?

* How many hardware and software vendors would have to know
what was going on?

* How many journalists are sticking their noses into these
stories?

* What's the probability that all this is going on, and there
hasn't yet been a credible leak?

* How long would it take for the NSA's IT people to become
completely overwhelmed by this?

And so on and so forth.

Officials are saying that what's going on is selective data
collection. That's plausible. Collecting everything
is not plausible.

Meanwhile, the IRS is setting up a monster database of all financial
and medical information about every citizen in the country, and we
already know that there are almost no controls about how this
information is used to attack political enemies, and possibly personal
enemies.

The IRS is the problem. The NSA problem is tiny by comparison.

[Reality Check]

John,

You need to think of this from the stand point of how you would address some of the issues you raised, if you were trying to design a system to do what they are trying to do.

And then re-read the linked training document with special attention to the difference between "hard data" and "soft data".

You might also want to note the process of deriving hard data from soft data, and then using that derived hard data to link targeted soft data to previously collected hard data, so hard data can be used to index/target only the very miniscule amounts of soft data you need to focus on to achieve your analysis goal.

It becomes very clear that without the hard data ( the meta data, the index data ), the soft data has only a fraction of it's worth.

As to storage capacity and the length of time you can store various types of data. Hard data ( both collected Meta Data and derived Meta Data ) is nothing more than indexes into the "Soft Data".


Indexes, both collected and derived, take up little space and can be stored indefinitely if your budget for hard disk replacements for your massive RAID systems are in the 10s of Billions of dollars a year.

As to storage of ALL the soft data filtered, they are talking - in the training document - about doing analysis on soft data in real time on ALL the soft data stored for a period of hours or stored for a finite number of days.

But that is for ALL the soft data. The same hard data ( indexes ) used for analysis can also be used for selecting ( targeting ) smaller amounts of targeted soft data to be extracted from the stream of soft data and those small fractional amounts of the total soft data can be stored for a few days, months, years or even decades, as needed.

Finally, different types of soft data have much different storage needs. All the way from text message content, instant text message content, and the text portion of e-mail content, which could be stored virtually forever in a huge data storage facility, through voice communications content which requires larger storage requirements ( and/or shorter periods of storage ), up through still pictures, the old Broadcast TV standards video, and Blue Ray quality High Definition Video which is probably why the gargantuan new NSA data storage facility is being built in Utah.

I am sure you can read between the lines in this document, and so can some of the people in foreign countries whose hard data and soft data is being filtered and collected.

[Reality Check]

Meanwhile, the IRS is setting up a monster database of all financial
and medical information about every citizen in the country, and we
already know that there are almost no controls about how this
information is used to attack political enemies, and possibly personal
enemies.

The IRS is the problem. The NSA problem is tiny by comparison.

Both the new IRS database which includes limited data from a number of different government and credit bureau sources is just one of the new federal government databases related to personal medical data that has been collected created by the Obama administration.

The other database which is much more invasive, and vastly more comprehensive, is the one created by the Stimulus package to record all medical records. When you sit down with your doctor these days they are either in real time, or very near real time, entering everything you tell them into a database which is not covered by medical confidentiality laws. The government can not ask the doctor what your told them, but the federal privacy laws have an exception which gives the government access to the new medical database under the business records exception to unreasonable search and seizure protections of the U.S. Constitution.

The stimulus packaged provided monetary incentives to doctors, hospitals and insurance companies to implement these voluntary systems ( and all so penalties for not voluntarily implementing them ).

The Obamacare law made it mandatory for all doctors to participate in the database by 2014. Which is one reason why some doctors are choosing to close their private practices and retire ( or take a job in a larger medical practice ) rather than invest the money in buying such a software system and also avoiding having to train themselves and their staff in it's use.

The point you are missing here is these are not separate, competing databases with the NSA database, these are just additional business databases that feed into the NSA databases under the business records exception. From a foreign intelligence gathering standpoint I do not see much need for anything other than hard data ( meta data ) from them. But from an internal security services stand point the ATF could use soft data, in the form of mental health diagnosis related information to more strictly enforce the long existing instant check for gun purchasers.