This question about who coined the term Big Data has intrigued me for quite sometime. Especially when it doesn’t make much sense in the context of a whole new economy it seems to spawn around it. Data has always been there in one form or the other and of course, the spark of the digital revolution in the last 5 years has changed the nature of data being recognized as not only text but also audio, video and graphics as well. The evolution of the nature of data has also made progress with respect to how it is stored, retrieved and renewed; moving from structured (hierarchical and relational) to unstructured methods and we know Google has made a business out of just searching the data that is now created across the world in terabyte loads by the second; responsibly and irresponsibly. My question on Google that has evolved from a Boolean search to a context based search yielded several relevant results of which I was able to pick only one well-researched article on the subject for those who might be interested. (http://bits.blogs.nytimes.com/2013/02/01/the-origins-of-big-data-an-etymological-detective-story/?_r=0) However, my curiosity about the relevance of the term Big Data and the Big Hoopla around it remains an enigma.
It is very evident that the Nature, Form and Scope of Data have dramatically evolved to this point of arrival in the digital revolution age. But what I don’t understand is how it suddenly became Big? Is it a reflection of Volume or the Complexity associated with the management of data? Big data is the term defined in Wikipedia for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to “spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions. But my question is, isn’t technology required to reduce the complexity as opposed to increase it? There may be some agencies like the defense and space and government that may have to deal with this complexity as a part of its professional hazard. But is there a need for businesses even in a global economy with the B2B, B2C, C2C and P2P concurrent business models in operation to the conventional brick and mortar model?
Please refer to the above illustration. Even when the sub-processes of an enterprise is integrated with its Customer Universe, it is possible to process data no matter how complex and voluminous, in smaller manageable chunks locally and aggregating it to a global level. In fact, a federated structure such as the locally autonomous global business model is what would probably succeed in the future. Why then the frantic frenzy to spend money in a direction that is of no immediate consequence in terms of strategic outcomes? Inversely, is there an explicit understanding that concurrently businesses will be scaled to global models and operating with multiple strategic and tactical models with respect to Product, Process and Services offered by the business enterprise taking into consideration the Demographic and Psychographic insights of Customers and Consumers in Real- Time?
There is an underlying assumption here that the mechanism of Big Data would be driven by algorithmic fuzzy logic that would scale vertical industry based data on top of the logic and rules that already support enterprise business processes to drive out transaction data. The algorithmic fuzzy logic would replace human experience with artificial intelligence in being able to raise questions off a transacted data and provide insights that would contribute to a quick re-alignment of Product, Process and / or Services to serve existing and New Customers in New Markets and New Segments to generate New Revenue. I don’t think non-proprietary data is in scope here as Google and Apple are learning to their consternation.
Therefore when data is drawn and inferred to the context and relevance of a business enterprise in the local markets and to the specific cultural alignment and aspirations of the customers in real-time, this whole exercise of Big Data seems to one in futility. For examples:
- An R&D Manager on realization that the packaging he created for a product is not accepted by a certain demography of customers and therefore can reverse to other designs and order production while other functions such as Trade and Marketing can shift the design to a market that can accept the design and its related costs of production in real-time.
- A Service Provider is able to engage customers in real-time to not only provide services virtually because most of the service is data and information based anyway; but is actually blending in as an extended arm of the business and forms an integrated support process such as shared service.
- Customers in any part of the world can co-create car designs with a car manufacturer and supply chain teams will find sources for material locally and make the cars available for release in the same year across the globe; again in real-time.
As a society and responsible business enterprise, aren’t we putting the cart before the horse? Business Processes derive data but so far enterprises have been force-fitting data onto processes in the explicit belief that there is integrity of data at an enterprise level. There are very few corporations in the world who have successfully undertaken an exercise at Master Data Management (MDM) and are therefore capable of having a unified view of their Customers across all Products, Processes and Services that their enterprise have to offer. It would probably make sense for this small minority to embark on the next threshold of data maturity but what about the vast majority?
We have seen consistently in the last two decades that Business Technology Solutions have really not matured with associate concepts of Services Oriented Architecture (SOA) and Cloud Computing not able to cross the Proof Of Concept stage. More critically, legacy applications are still managed and maintained in corporations, not because they lend themselves to the Transactional Processing environment but rather they provide the data required for the day-to-day operations. It is a very unfortunate world we live where we create and propagate expensive technologies for the problems we have willingly conspired to create and try to solve them; while the core of Transformation and Innovation remains unaddressed due to lack of resources; be it money, talent or time. Isn’t it time to investigate the motivations and the competency of the analysts and others who allow such propaganda to seep into our system and distract us from the real purpose with their hype and hyperbole?
Here is my summary on how this phenomenon called “Big Data” needs to be addressed with respect to qualifying for implementation in an enterprise:
Enterprises that have implemented Master Data Management (MDM) that is consistent with Globally Unified Enterprise Business Process.
Enterprises would check themselves in for the above if they have implemented globally unified systems with Services Oriented Architecture (SOA) and Cloud Computing as methods of a seamless global infrastructure.
These enterprises have created a Learning Organization where people correlate Cognitive Intelligence with Business Intelligence to drive mature market decisions in real-time.