By Brian Lucas

This one is for Mari a truly wonderful person who I am privileged to have met.

“The only sure bulwark of continuing liberty is a government strong enough to protect the interests of the people, and a people strong enough and well enough informed to maintain its sovereign control over the government.” – Franklin D. Roosevelt

Being agile requires information.  Today both business and government are collecting massive amounts of data on many of our activities.  Predicting human behavior by tracking data, sometimes vast amounts of data has been the subject of much science fiction, but today the science fiction is a reality.  This is such a hot topic there are already a plethora of terms and sub categories each with their own set of multi-billion dollar industries attached.  I am going to spare you the heady exploration of quantum computers and indeterminate state analysis.  Instead let’s look at technology that is already in wide use.

Take Predictive Analytics as an example.  Predictive analytics comprises the following disciplines of artificial intelligence, data mining, machine learning, modeling and statistics to analyze patterns in historical facts and current activity in order to make predictions about future events[1].  Predictive analytics is heavily used in actuarial science, financial services, healthcare, insurance, marketing, pharmaceuticals, the retail industry, telecommunications, and travel amongst others.

Deep data mining has risen to prominence lately[2].  It has been touted as the future by many from Dean Abbott, President of Abbott Analytics, Inc. to Eric Siegel, PhD, founder of Predictive Analytics World.  Deep data mining takes predictive analytics one step further into the realm of influence.  It refers to the strategy of data collection involving the sophisticated analysis of patterns across a broad spectrum of data domains.  It is used to not only predict, but to sway behavior.  This technology is being used by businesses and governments alike to predict the psychological profiles of individuals.  Look at the way Pandora analyzes an internet user’s personality by tracking their favorite music and movies.  If this sounds like Big Brother know that it has definitely raised concerns from privacy advocates.

Big data is the term for a collection of contrived data sets whose resulting size and complexity often exceed the capacity of traditional data processing applications.  The trend to gather greater meaning from ever larger data sets is an irresistible and irreversible one.  The amount of inference derivable from the analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allows incredibly complex correlations to be made.  These perform various functions from the innocuous spotting of business trends to prevent diseases and combating crime.

Two of the technological revolutions that made this possible were the explosion in hardware data storage and software data manipulation.  As of 2012, it was feasible to process in a reasonable amount of time exabytes of data.  An exabyte is one quintillion bytes or 1,000,000,000,000,000,000B or 10¹⁸ bytes.  The world’s technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s; every day 2.5 exabytes of data are created[3].

People often confuse Business Intelligence and technologies like Big Data.  Business Intelligence uses intrinsically descriptive statistics on data that has high information density in order to measure things and detect trends.  Big data on the other hand uses inductive statistics and nonlinear system identification concepts to infer regressions, nonlinear relationships, and causal effects from massive data sets of relatively low information density in order to reveal dependencies, relationships, and predict outcomes.

The bottom line is that everyone in business and government is looking for patterns.  Looking for patterns is nothing new; it is as old as reasoning itself.  It is at the heart of most of our technological advances throughout history.  The difference is that now when have massively complex, large and sophisticated machines to aid us in this effort.  This is indeed a two edged sword.

Since most of our activities are done online, there is so much data collected about our activities that personal privacy becomes doubtful.  There are both wild and credible stories of scientists and mathematicians creating predictive algorithms of which the intelligence community immediately takes possession[4].  How will we ever know when some entity crosses the line?  In fact there probably is no single defining line.  It is more of a grey band stretching from moral into immoral use of personal data.  This represents a challenge for everyone, but we in the Information Technology industry bear a greater responsibility.

Banning the practice of collecting data and making predictions from it is not the answer.  There is far too much good that can come from this activity.  And after all we need this data in order to be truly agile and make intelligent decisions.  What can we do?  The answer is to get involved and learn more.  The more you learn the more light you can shed on this vital subject with others.  The more informed your votes and your actions will be.  The totalitarianism of a Big Brother state can only arrive when the populace is ignorant and uncaring.  So get involved and get informed and remember till next time – keep agile!