Changing Lifestyle in Big Data World

Are you going from a data famine to a data glut? A sustainable data lifestyle is needed in the ‘Big Data’ world.

Unless one has been away from media for years, maybe on a secret mission to Mars, it’s hard to avoid the din of exhortation around big data. By now, most businesses have understood the power of data in disrupting and transforming their businesses. Even the common man has gained an understanding of how data influences politics (elections in India, the Republicans catching up in the recent mid-term cycle with the data expertise of Obama campaign, etc) or how data shapes Hollywood content (Netflix and Amazon now fund movie plots and actors based on big data). The question is no longer “should you?” but “how do you?” execute on big data strategy for an organization.

Developing a Healthy Data Lifestyle

If we take for granted that data is a core, strategic asset of an organization and not just plumbing for the IT department, what comes next? As Boards, CEOs, and CFOs loosen the purse strings for investment in collecting, analyzing and providing data, there is a real risk of going from a data famine to a data glut. Starvation to gluttony is rarely a good change and it’s important to develop a healthy data lifestyle.

Going from data to big data definitely involves scaling up collection, management, presentation, and consumption of data. The growth to big data is often viewed in terms of the 3Vs—Volume, Velocity, and Variety of data. There is the additional and the most important V—Value to business. One can manage the volume, velocity, and variety explosion with the right business context by focusing on the value to business and using a business benefits tracking model to simplify the complexity and manage costs of data initiatives.

Big Data Strategy

Big data strategy is not just about collecting large amounts of data but also about the right data. If handled poorly, a simple change to data collection can break the largest budgets. For example, a large global financial institution decided to go beyond transactional data and collect data on human and algorithmic pre-events before the transactions. For example, a simple derivative purchase or a forex trade may have hundreds of thousands of pre-events. The underlying systems were ill-equipped to handle and exploded the volume and velocity of data, adding tens of millions of dollars to the data budget. A simple relook at the right types of data to collect and a holistic review of the data lifecycle with a focus on business value dramatically reduced these unnecessary costs.

An important driver for big data initiatives is to use new insights to increase customer intimacy, the intelligence and flexibility of the supply chain or the pace of innovation and introduction of products and services. This most often requires collecting a wide variety of data in addition to large volumes. Unlike variety in a normal diet that can make the food healthy, blind collection of a variety of data in a data lifecycle can lead to big problems. For example, a single customer upsell event can lead to hundreds of thousands of pre-events a lot of which are unstructured data sources such as social interactions, media impressions, evaluations, product reviews, etc. It’s important to ensure that collection of such a variety of data be intelligently handled with a business value filter. This filtering needs to be balanced with the need to support tomorrow’s business models and therefore keeping some of the data that may not be needed today. One can also add to the variety of data incrementally in phases rather than ingest all new data sources in a big bang.

The Collection of Data

Another important area to watch is the collection of data from such sources as Industrial Internet of Things (IIOT). IIOTs are amazing enablers of digital transformation. For example, continuous collection of data such as temperature from sensors, operating characteristics of an engine or video images checking for equipment cracks can be extremely helpful in predictive maintenance and safety. While there are all these benefits, a single jet engine on a single long haul flight can generate petabytes of data, more than all the data in all the books in print in human history. Its imperative therefore for these new sources of real time, continuous data to be smartly pre-processed and compressed or transformed with business context. Just because we can collect data does not mean we should—the data lifecycle needs to be constantly evolved based on changing business needs and not just technical capabilities.

There are tremendous advances in technical capabilities by many product vendors and the open source community to power business needs for big data. Only relentless focus on business innovation and organization objectives can bring sanity to this explosion of choices and allow you to manage the transition from a data famine to a data feast.

The power of big data is not just in its availability but also ultimately in the use and consumption of data for behaving differently and producing very different outcomes. Data is consumed in many ways to eventually drive actions or changes.

True transformation of customer experience or business outcomes comes from the consumption of relevant data at the right time by the right people, process or application.

A healthy data lifestyle therefore is as much about consumption as it is about collecting and processing data. Significant advances in data visualization have transformed the way data can be consumed by human beings. A big data strategy therefore needs an integral data visualization initiative. Such diverse organizations as sports media to dentists and health professionals have shown the power of rich data visualization to gain new insights and make behavioral changes.

On a side note, data science that taps statistics and data visualization that goes beyond math and technical skills to tap into creative and visual skills open new vistas for different talent pools. There is a severe shortage of skills in these areas and hugely successful careers lie ahead for those that can help organizations develop a healthy data lifestyle with these skills that go beyond just technology.

Conclusion

Finally, a sustaining data lifestyle that powers organization success and innovation in the big data world requires a transformed organization culture and behavior. Decisions need to be evidence based powered by data. Data governance needs to be modernized to balance collaboration and risk management with deep involvement of business, operations and technology teams.

In a recent meeting, a CIO reminded me of management guru Pater Drucker’s words of wisdom, paraphrased often as ‘organization culture eats strategy for breakfast’. The best data strategy without an organization culture that values and consumes data appropriately will lead nowhere. A lot of big data initiatives fail not because there is a famine or glut of data but because the organization culture does not adapt to a data driven culture, proactively making decisions and learning from experimentation. It’s therefore important to embark on the big data journey by focusing on the people, core values and culture as much as on the technology and processes.

Big data is here now and it’s an opportunity and a threat we all face today. Developing a healthy data lifestyle can power your success through data.