How will we analyze the new 15 petabytes of data created every day?

[Eric Garland]

Hey intelligence people: watch this video, and tell me, what will 15 PETABYTES of new data every day do to the profession of understanding changing markets.

I'm reminded of Arik Johnson's observation on Intelligence 2.0 - competition will be asymmetry of analysis, not asymmetry of information.

[Eric Garland]

I'm thinking that lots of business success will be a contest between the analytical haves and the have nots. If you have the capacity or the internal talent to derive insights from a rapidly increasing set of data, then you will win over those who are content to use the odd focus group or market research.

There's another tension here as well - the housing and financial debacle in the United States showed us the limits of reliance on mathematical formulae to make critical strategic decisions. That said, this kind of data will clearly require new algorithms and insightful analysts (note the distinction) to make sense of the world. So maybe the problem isn't math, but a particularly myopic and narrow view of using statistics to reach pre-formulated conclusions.

Another aspect here - much of that flood of data is collected directly from US as individuals, often without our consent. Listen to that comment: "Facebook is data." Sure, but half the time it's data gleaned from private emails. (One reason I have ceased using the service.) The product IBM is selling back to is our very own behavior. That can also be incredibly useful, but we ought to be quite aware of what this means.

In any event, we are in the right profession.

[Arik Johnson]

I think that's a major reason why we've been referring to the broader umbrella meme as the Intelligence Collaborative and not the Business Intelligence Collaborative or Competitive Intelligence Collaborative or Market Intelligence Collaborative... because all of that differentiated marketing nonsense is just so much artificial nomenclature designed to obscure (not expose!) the core problems of uncertainty and confidence in the decisions our institutions are making every day and their propensity toward over-assumption and consequent mistakes.

I'm kind of over all that now... I don't need to micro-define my work that way any more. I'm intellectually promiscuous enough to embrace all of these ideas.

One of the guys in the video - Jeff Jonas - and I have been trying to get together for beers for a few months now after having been introduced electronically, so I sent him the link to this discussion to see if he'd like to join in. Here's hoping he will! ;-)

Finally, check out this story about the so-called Facebook Whisperer from Fast Company - when the asymmetries of interpreting data (I personally prefer "interpretation" to "analysis" as a similarly broad memetic umbrella) outweigh access to the data itself (anyone could backward engineer Facebook's database... I know plenty by name who could query scrape a couple hundred million records like that) the insights that might be derived are so... apocalyptic (e.g., revealing) that it scares the suits... even when the broader social good might be a higher order goal.

[Eric Garland]

Funny you mention analysis versus interpretation - I quite like this thin slice.

Analysis makes something sound quite official - "Boss, we ran it through spectral analysis - it seems that this sandwich is loaded with uranium, as you can see on this graph. In analysis, there are numbers, pie charts, Excel, and big (usually unexamined) assumptions.

Interpretation, on the other hand, is pretty direct - "Here's what I think this means. That sounds less authoritative, but it's probably more useful. Less impressive, but more honest.

[Arik Johnson]

Right! It sounds... pseudo-intellectual - plus, it's imprecise for what we actually do; there are lots of complimentary methods in my toolkit that analysis technically doesn't include - synthesis (converse of analysis), antithesis, et cetera.