By Brian Lucas
This one is for Mari a truly wonderful person who I am privileged to have met.
“The only sure bulwark of continuing liberty is a government strong enough to protect the interests of the people, and a people strong enough and well enough informed to maintain its sovereign control over the government.” – Franklin D. Roosevelt
Being agile requires information. Today both business and government are collecting massive amounts of data on many of our activities. Predicting human behavior by tracking data, sometimes vast amounts of data has been the subject of much science fiction, but today the science fiction is a reality. This is such a hot topic there are already a plethora of terms and sub categories each with their own set of multi-billion dollar industries attached. I am going to spare you the heady exploration of quantum computers and indeterminate state analysis. Instead let’s look at technology that is already in wide use.
Take Predictive Analytics as an example. Predictive analytics comprises the following disciplines of artificial intelligence, data mining, machine learning, modeling and statistics to analyze patterns in historical facts and current activity in order to make predictions about future events. Predictive analytics is heavily used in actuarial science, financial services, healthcare, insurance, marketing, pharmaceuticals, the retail industry, telecommunications, and travel amongst others.
Deep data mining has risen to prominence lately. It has been touted as the future by many from Dean Abbott, President of Abbott Analytics, Inc. to Eric Siegel, PhD, founder of Predictive Analytics World. Deep data mining takes predictive analytics one step further into the realm of influence. It refers to the strategy of data collection involving the sophisticated analysis of patterns across a broad spectrum of data domains. It is used to not only predict, but to sway behavior. This technology is being used by businesses and governments alike to predict the psychological profiles of individuals. Look at the way Pandora analyzes an internet user’s personality by tracking their favorite music and movies. If this sounds like Big Brother know that it has definitely raised concerns from privacy advocates.
Big data is the term for a collection of contrived data sets whose resulting size and complexity often exceed the capacity of traditional data processing applications. The trend to gather greater meaning from ever larger data sets is an irresistible and irreversible one. The amount of inference derivable from the analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allows incredibly complex correlations to be made. These perform various functions from the innocuous spotting of business trends to prevent diseases and combating crime.
Two of the technological revolutions that made this possible were the explosion in hardware data storage and software data manipulation. As of 2012, it was feasible to process in a reasonable amount of time exabytes of data. An exabyte is one quintillion bytes or 1,000,000,000,000,000,000B or 1018 bytes. The world’s technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s; every day 2.5 exabytes of data are created.
People often confuse Business Intelligence and technologies like Big Data. Business Intelligence uses intrinsically descriptive statistics on data that has high information density in order to measure things and detect trends. Big data on the other hand uses inductive statistics and nonlinear system identification concepts to infer regressions, nonlinear relationships, and causal effects from massive data sets of relatively low information density in order to reveal dependencies, relationships, and predict outcomes.
The bottom line is that everyone in business and government is looking for patterns. Looking for patterns is nothing new; it is as old as reasoning itself. It is at the heart of most of our technological advances throughout history. The difference is that now when have massively complex, large and sophisticated machines to aid us in this effort. This is indeed a two edged sword.
Since most of our activities are done online, there is so much data collected about our activities that personal privacy becomes doubtful. There are both wild and credible stories of scientists and mathematicians creating predictive algorithms of which the intelligence community immediately takes possession. How will we ever know when some entity crosses the line? In fact there probably is no single defining line. It is more of a grey band stretching from moral into immoral use of personal data. This represents a challenge for everyone, but we in the Information Technology industry bear a greater responsibility.
Banning the practice of collecting data and making predictions from it is not the answer. There is far too much good that can come from this activity. And after all we need this data in order to be truly agile and make intelligent decisions. What can we do? The answer is to get involved and learn more. The more you learn the more light you can shed on this vital subject with others. The more informed your votes and your actions will be. The totalitarianism of a Big Brother state can only arrive when the populace is ignorant and uncaring. So get involved and get informed and remember till next time – keep agile!
 See Siegel, Eric (2013). Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die
 See http://www.deep-data-mining.com/2013/05/the-10-most-influential-people-in-data-analytics.html
 See “IBM What is big data? — Bringing big data to the enterprise”
Thanks for clearing this up Brian. I have found the terms confusing. Do you have any data you can share on the use of these technologies to guide reorganization efforts? Particularly the more fluid agile structures you wrote about previously.
Again, thank you for an informative and thought-provoking post. These issues are important as we see what’s happening with the digital footprints we leave behind; and how they affect our future.
Hi Brian: Thanks for putting this in perspective! There seems to be so much overlapping technology here that it gets confusing thanks for making it clear.
Brian, I am ashamed to admit I have been lurking here since I commented on your agile organization post. I am glad you posted this since we have been barraged by vendors with conflicting claims that I believe are overselling and over promising. Are you up for an email on this subject with a few questions?.
Nice post Brian! I struggle with understanding all the new technology we keep getting hit with and trying to understand its impact. Any tips you care to share on how you do it?
Wish I was an informed as you are! Keep Posting Brian!
Knowledge is the key to any success today! Peter Drucker wrote about knowledge workers, but it has gone beyond that. Today all you have to do is look at a preschooler navigating a tablet and see them using skills that we in the previous generation learned on-the-job. Knowledge if technology and computers is an absolute must to working in any industry now. I realize this is a bit off topic. I do agree with this post and think it could have gone even farther.
I am so on board with your thinking here Brian! Would love to have the opportunity of working with you some day!
Brian, how do you keep so informed? I read your blog and others all the time. I am overwhelmed… How do you keep up with all the latest without suffering from information overload? I am not in the computer industry as you are. I love how you think though which is why I follow your blog. You seem to be able to apply intelligent thinking to any subject. Do you have any tips for me? Would you mind if I emailed you? I’d like some career advice on an opportunity I have.