NSA Data Collection - The Common Sense Deniers

[Reality Check]

.
.
.

Many people, who should know better, are in a state of denial regarding what the Foreign Intelligence data collection capabilities of the NSA really are.

Example:

You have to apply a common sense test to articles like this.

* How much storage would be required to store every e-mail message
from every user in the world, including many spam messages?

* How much processing power would be required to analyze each
message for whatever keyword searches they're doing?

* Same with online chats and all the other data?

* How much disk storage would be required to store all that?

* Assuming that the mean time between failure of disks is a
couple of years, how would the issue of backing up all this
data be handled?

* How many computers would be required to do all this
processing?

* How much network bandwidth would all this suck up?

* How big a staff would be required to keep this facility up?

* How many hardware and software vendors would have to know
what was going on?

...

The above are the wrong questions if you want to arrive at the truth.

Some Examples of the questions that should be ask, and then ground checked for truth:

* How much network bandwidth would all this ( this being Filtering all the data moving on the Internet ) suck up?

Would it be more Band Width than currently exists on the Internet ?

Is the World Wide Internet Architecture topology a distributed network of equal size data pipes, or is it a hierarchical structure with distinct massive backbone pipes ?

Are there a number of collocated backbone routers ( at a relatively small number of collocation communications sites ) that interconnect the different proprietary Backbone pipes ?

Are there massive numbers of dark ( unused ) fibers running between many of the collocated backbone inter tie locations ?

Are there also large numbers of dark ( unused ) fibers running in parallel to many of the proprietary backbones ?

Given the answers to the above ( and your desire to avoid duplicating the entire internet ) where would you locate the data storage locations where the "soft data" filtered would be stored for a few hours or a few days ?

Does this ground truth against the sever locations in the top secret data leaked by the CIA / NSA traitor ?

* How much processing power would be required to analyze each message for whatever keyword searches they're doing?

How do you design databases to avoid searching through every character in every record stored in a database to find the text string your are looking for ?

What is the schema you would use to structure the template you would use to filter the data being filtered to analyze it ?

Where would you get that schema ( packet structures ) necessary to create a template for the data stream you are capturing and storing ( for whatever short period of time ) ?

How would you index the data stream you are filtering so that you could use meta data obtained from a different source to look at just the records in your data base related to the packets in the stream you have both stored and are interested in ( so you do not need to do keyword text searchers of every record in the database).

Where would you get that meta-data ?

How would you index the the indexes so you could tie data coming from a workstation computer, telephone switch or communications server ( such as an SMTP server, or a Instant Messaging server, or a Face Book Server, etc. ) to one or several records of the steam ( soft data ) you temporarily stored in your database ?

Where would you obtain the Meta Data ( Hard Data ) you needed to create the Index of the Index ?

How would you index the index of the index to tie a specific telephone number ( or a specific e-mail address, or a specific face book user, etc.) to the index of a specific telephone switch, or specific work station, or specific server ?

Where would you obtain the Meta Data ( Hard Data ) you needed to create the index of the Index of the index?

How would you generate other third level indexes on the fly by associating patterns in the relations ship of stream records associated with the same first and second level indexes during the same time period they are associated with related third level indexes you are interested in ?

Does this ground truth against the pattern of leaks, confirmations and non-denial denials of the leaks, as they regard the collection of Meta Data ( index information ) from various technology companies ( the leaks, confirmations and non-denial denials, having occurred over the last few weeks and months ) ?

* How much storage would be required to store every e-mail message ( or just the text portion of the e-mail message ) from every user in the world, including many spam messages?

* Same with online ( text ) chats ( and text instant messaging, and skype text messaging, and face book text messaging ) and all the other ( text ) data?

* How much disk storage would be required to store all that?

What is the total amount of data stored on hard disks in online servers, online computer workstations, telephone switches, and other online devices at anyone time ?

What percentage of that is text data? Music ? Digital Voice Data ( like telephone calls ) ? Low Quality Video ( like the old National Broadcast TV standard ) ? DVD quality video ? High Definition Blue Ray Quality video ?


What percentage of that total data stored on "online disks" travels across the internet in an hour ? in a day ? in a week ? in a month ?


What is the percentage of Internet content is text data ?

What form of data storage get's cheaper and cheaper per Giga-byte as the amount of data stored scales larger and larger ?

What form of data storage becomes virtually immune from hard disk failures as a form of data loss as the amount of data stored scales larger and larger ?

What is the acronym for Redundant Arrays of Inexpensive Disks ?

Who created RAID Systems and when were they created ?

Who funded, and continues to fund, the development of ever more capable, ever larger, ever more efficient, and ever more robust ( immune from performance degradation due to the failures of individual disks ) Raid designs ?

Given what a tiny percentage of all the internet content data in the world is text data, sent as messages during any given period of time, does the original question have any meaning related to the NSA ?

Did the below linked presentation even claim that ALL Internet text content data ( text soft data ) was stored for years, let alone decades - or forever ?

http://www.theguardian.com/world/intera ... esentation

I believe the above is more than enough examples.

[John]

I'm not sure what you think I'm in a state of denial about. I'm just
saying that the IRS database project is far more dangerous than the
NSA database project, and there's absolutely no question that's true.

The NSA project that you describe is not technically sustainable.
Even if you imagine that the NSA has achieved the technology to
collect every e-mail message and every online chat in a database,
along with all sorts of documents in the cloud, the maintenance issues
are overwhelming. Every time there's a new e-mail service, or any
kind of software service, or a change in APIs somewhere, the NSA IT
people would have to react, and I can assure you that's not possible.
Every time the data analysis needs of the NSA change (e.g., new
keywords, new political situations), the NSA IT people would have to
react, and they'd quickly be months behind schedule in reacting to
support tickets. In the end, the NSA could not capture much more
data, on a continuing basis, than people's phone bills.

A lot of the questions that you're asking are close to fantasy, at
least today. Would the required bandwidth be larger than the entire
internet? Well, no, but it would be so enormous that it would be
widely noticed and reported on. And then various entities (countries,
companies, terrorists) would make minor changes to their own
infrastructures that would require enormous changes to the NSA
collection technology. It's simply not sustainable.

By contrast, he technology involved with the IRS is quite achievable,
because the data is much more static. You might send 20 e-mail
messages a day, but you only see the doctor a couple of times a year,
so data collection on an ongoing basis is quite sustainable.

What's unsustainable in the case of the IRS is privacy restrictions.
They might announce with grand fanfare that they've implemented
security restrictions on the Navigators, but those security
restrictions would start falling apart within a few days of actual
usage. There will quickly be published loopholes that permit any
navigator or contractor to see everyone's data.

When the Snowden story broke, people wondered how a kid could have
access to so much intelligence data. I knew the answer to that
question the nanosecond that I read the story. There's no way to
protect data from the IT people. It's like telling your brain surgeon
that he's allowed to operate, but he can't have access to your brain.
It's a contradiction.

A few years ago I worked for a company that provided a software
service, and the company maintained a database of personal information
on several thousand people across the country. The customers who
purchased the service had restrictions on the data they could access.
But I, and any other employee of the company I worked for, could have
done just what Snowden and Manning did -- download all the data and
sell it to the highest bidder.

So even if you believe that the NSA has achieved Orson Welles's 1984
model today, it would start to crumble tomorrow, and would fall apart
within a year or two. The big danger continues to be the IRS.

[at99sy]

With respect to the storage capacity question, I don't see that as a problem. IBM and others have made huge advances in capacity and size
http://www.digitaltrends.com/computing/ ... ge-device/

and data storage sites have been developed around the world.
Back in the 70's we lived in Huntsville,AL and my dad worked for NASA, Computer Sciences and the Redstone Arsenal at various times as a computer programmer etc... emphasis on the etc.. One day he pointed out this nondescript building that we had driven past for years but was not even marked in any way and had few windows. He told me that it was a data storage building and that it had 1 million sq.ft of space and that 9 levels went underground. Each floor was full of the old reel to reel magnetic tape devices and it was where they collected "data."

Today a million square feet of space would hold a heck of a lot of bytes.

John is spot on with the threats that the IRS is capable of. Not unlike the NKVD and Gestapo. The smart despot doesn't show his cards until he has all the Aces.
What really disturbs me is the rapid militarization of the civilian police forces around the country.
http://www.digitaltrends.com/computing/ ... ge-device/
http://www.salon.com/2013/07/10/militar ... %E2%80%9D/
http://www.huffingtonpost.com/tag/police-militarization

With the continuous assaults on gun ownership, anti-christian attitudes, embracing every fringe minority group in existence while rejecting founding values, demonizing patriots, and those who ask questions and demand explanations from the dear leaders, coupled with rapidly increasing costs of living and the 1% and 47% bleeding the country dry, we're heading for a shock.

I see the working people bringing the system down due to either becoming so apathetic that they "go Galt" and stop consuming and or working to their potential, or they begin a social movement via mass protest and demand "changes." This is why the gov't is stocking up on special items and equipping the police, while trying to disarm the public. "It's for the children!"

cheers
sy

[John]

> One day he pointed out this nondescript building that we had
> driven past for years but was not even marked in any way and had
> few windows. He told me that it was a data storage building and
> that it had 1 million sq.ft of space and that 9 levels went
> underground. Each floor was full of the old reel to reel magnetic
> tape devices and it was where they collected "data."

Exactly. The contrast is with online data. If the NSA wanted to
collect all that data and burn it onto a zillion blu-ray disks, and
then store all the disks in your father's underground vault, then
that's doable, but it would take hours or days or weeks or months to
retrieve any particular item of data.

To keep all that data online, available for immediate access, is
literally impossible and unsustainable.

[at99sy]

Exactly. The contrast is with online data.
To keep all that data online, available for immediate access, is
literally impossible and unsustainable.

I see no reason to keep it all online. Would it not be more efficient to download it into massive storage sites
that could be accessed when needed or wanted by any "official" from where ever they happened to be working or watching porn?

[Reality Check]

I'm not sure what you think I'm in a state of denial about.

You have answered your own question, in spades.

First denial:

the IRS database project is far ( larger and therefore more dangerous ) than the
NSA database project, and there's absolutely no question that's true ( sort of sounds like a Global Warming true believer talking about settled science).

Second denial:

The NSA project that ( is described by the leaked top secret documents ) is not technically sustainable.

Third denial:

Even if ... the NSA has achieved the technology to
collect every e-mail message and every online chat in a database,
along with all sorts of documents in the cloud, the maintenance issues
are ( impossible to overcome with the level of technical expertise at NSA and the NSA budget ).

Fourth denial:

Every time there's a new e-mail service, or any
kind of software service, or a change in APIs somewhere, the NSA IT
people would have to react, and I can assure you that's ( impossible to overcome with the level of technical expertise at NSA and NSA budget ).

Fifth denial:

Every time the data analysis needs of the NSA change (e.g., new
keywords, new political situations), the NSA IT people would have to
react, and they'd quickly be months behind schedule in reacting to
support tickets.

Sixth denial:

In the end, the NSA could not capture much more
data, on a continuing basis, than people's phone bills ( sort of sounds like someone defending the flat earth theory after an astronaut circled the globe - high definition on demand video storage farms sort of shoots that one down).

And on, and on.

The problem here is you are thinking in terms of a denier trying to find reasons, even those reasons based on false premiss, to prove something cannot be done.

Top Secret documents are now all over the international media ( and U.S. Media ABC and CNN ) claiming these things not only can be done, but have been done ( and also claiming that most of these things have been being done for decades as part of foreign intelligence gathering operations ).

If you are even a fairly good systems analyst, you should be able to approach it from the stand point of how would I find a way to do it.

But so far you are taking this approach:

Regarding one very small part, of only one of your denials, related to filtering and temporarily storing all the Internet data filtering through one of several geographic points on Earth. You postulate

1. There are only two, mutually exclusive ways to capture and perform analyses on ( and also selective extractions for long term storage ) on this data.

2. One, mutually exclusive way, is to search all the data as one giant blob.

3. The other, mutually exclusive way, is to dismantle every single e-mail, from every single switched network packet, and every single mime component of that e-mail, and store it in a static record format for each and every single mime type, so that it can be searched in the old fashioned way.

And you then conclude it is impossible to achieve your objectives if postulations are correct.

However, a systems analyst would instead look for ways to achieve a goal that would achieve what is already being reported as fact in leaked top secret documents.

In fact all of the above postulates are false. By using technology that has been in existence for decades ( stateful inspection of switched data packets ) it is possible to store packets as indexed records ( indexed by telephone number, email address, user name, social security number, street address, legal name, alias etc. etc. ) with the indexes being created on the fly from matching meta data already in databases to ip addresses and other packet identifiers and then use the indexes to retrieve, and search and locate using your blob method, only those switched data packets you are interested in, so that they can be extracted for longer term storage and/or real time analysis.

Again, anything you come up with as an analyst could be "denied" by using false postulate, as you have been previously doing in this thread.

But I am challenging you to actually read the top secret documents and use your knowledge and systems analyst skill to reverse engineer how to do it.

Please keep in mind the NSA has existed for many decades as a foreign intelligence gathering operations and none of the sophisticated capabilities they truly have ( and they do have some capabilities developed over decades - even if you believe the Top Secret documents are total fakes ) were NOT created by Generation X programers in the last decade, nor were they developed by the incompetent programmers at the IRS.

The NSA is a military organization - not a civilian civil service organization.

[John]

Well, RC, you'd better watch out:

> The FBI develops some hacking tools internally and purchases
> others from the private sector. With such technology, the bureau
> can remotely activate the microphones in phones running Google
> Inc.'s Android software to record conversations, one former
> U.S. official said. It can do the same to microphones in laptops
> without the user knowing, the person said. Google declined to
> comment.

> http://online.wsj.com/article_email/SB1 ... DEyWj.html

So it's much, much worse than even you are saying. The FBI is now
recording EVERY CONVERSATION, everywhere in the world, and storing it
in the online database, presumably in MP3 format. So anything you say
to anybody, no matter where, and no matter how intimate, is now on
file with the FBI, and they can bring it up and listen to it at any
time. Also, they have voice recognition software, so they're keeping
track of everything you say, and if you ever say anything they don't
like, they're gonna come and get you, RC.