Start thinking 'small' data

How often do we see Big Data deliver its promise? According to McKinsey’s quarterly report titled, ‘Getting Big Impact from Big Data,’ very few companies have achieved what we would call ‘big impact through big data,’ or impact at scale. Sometimes the expectations are too big or at times, the information so huge that it can’t be tamed and managed as per expectations. We have heard of cases where organizations take up large big data projects spending huge amounts of resources, but the projects fail to take off or the results take a long time to show up.

Yes, big data means a lot of data. Most of the times valuable chunks of data, that can provide deep business insights. But all that data can be daunting at times. Organizations might spend millions and still not get the desired results, or some may not have the millions to spend in the first place. The more important point here is—‘big data’ is not what you need all the time, when the answer to many business problems could be found in some small, simple ways. The answer could be ‘Small Data.’ Interestingly, while big data still looks like an elusive dream, organizations are waking up to the benefits of small data.

The reason is quite simple. The whole objective behind the data revolution is to get actionable insights that help in quicker and better decision making. Analysis of massive data sets is often a time-consuming process and requires huge amounts of specialized resources. This is especially true in case of medium- and small-sized enterprises that can’t get into costly big data plays. That’s where theimportance of small data comes into picture.

To define small data, it is nothing but some specific sets of data that are decentralized and easily accessible to end users. Most importantly, it is ‘actionable’ data. Despite all the hype around big data, many a time it is just a few data attributes that are required to resolve issues or gather customer insights. Organizations are hence realizing the importance of thinking small.

“The small data revolution is gradually picking up steam. It can be helpful for organizations to understand larger trends by understanding its constituents. For many practical problems faced by companies today, small data could be enough,” says Sunil Jose, Managing Director Teradata India. Some examples of small data could be data from Excel spreadsheets, catalogues from vendors or internal lists.

The significance of small data comes from the ability to decentralize data. It allows you to break down large information to smaller pieces that can be spread across people and organizations. “This can help people and organizations collaborate, build, and integrate components, making it a mass-participation activity,” adds Jose.

WHY BIG DATA IS NOT FOR ALL

A global study sponsored by Teradata Corporation and conducted by The Economist Intelligence Unit in 2014, reveals that CEOs often have an inflated view of their big data initiatives. In reality, the picture could be different. The success of big data initiatives requires flow of data to be smooth and timely to users. While the top management might think that this is the case, the end users often have a different opinion.

The report states: “While 47% of CEOs believe that all employees have access to the data they need, only 27% of all respondents agree that they do. Similarly, 43% of CEOs think relevant data are captured and made available in real time, compared to 29% of all respondents. CEOs are also more likely to think that employees extract relevant insights from data—38% of them hold this belief, as compared to 24% of all respondents and only 19% of senior vice presidents, vice presidents, and directors.”

It all boils down to the point that big data has no value unless it offers meaningful insights to end users who can leverage the data and draw insights from it. Lack of coordination between teams and non-availability of information to workforce often fails the whole big data objective.

“The three Vs—Velocity, Variety, and Volume make big data a difficult proposition for organizations that lack scalability in terms of right resources, budget, and processes to optimize the information flow,” say, Dr Pradeep K Mukherji, President & Managing Partner, Asia Pacific and EMEA and Alok Ranjan, Marketing Specialist, Avasant.

THE POWER OF SMALL

On the other hand, what makes small data tick is that it is decentralized and includes separate pieces of information that is generated almost every day, and thus is easily available. “Small data is an answer for organizations looking to resolve complex business issues by partitioning problems across people, process, and technology and helps move away from centralized silos created by big data,” add Mukherji and Ranjan. The advantage for small data is that it does not require complex, high capacity engine for processing. Finding data scientists and specialists for large data projects could be a challenge for many organizations. Small data fills this gap by enabling non-technical workforce to derive insights relating to processes and customers, which can help drive decision making across different levels. “Small data allows organizations to drive personalization by activating accessible data in small packets, thereby initiating small data revolution,” point out Mukherji and Ranjan. Instead of mining through large sets of data, the small data concept relies on using the tiny bits of information in a more powerful way. This scenario will become more relevant in the IoT (Internet-of-Things) era,as experts emphasize that many IoT use cases only require small data points.

Having said that, it is not really the size of data that is going to make a difference in the long run, but identifying what can be done with it. Data will be all around us, all the time. Understanding the impact areas and using what works best is the key. For instance, small data can be used to provide information on how a particular machine part is performing at a given time. Now to find answers to ‘why’ it is performing in that way, we might need big data. So, going ahead we will see the ‘big and small’ complimenting each other.