"The small data revolution is gradually picking up steam"

While we are still surrounded by the Big data hypes and realities, a Small data revolution is slowly gaining steam. In an interaction with Sunil Jose, Managing Director, Teradata India, we try to understand what’s powering the trend and what is it that enterprises need to watch out for. Excerpts.

How is Small data gradually gaining more relevance in a business scenario where big data has always been so fascinating?
The growth in Internet of Things (IoT) will far exceed that of other connected devices. By 2020, the emergence of mass market smartphones and tablets, combined with the mature PC market, will result in an installed base of about 7.3 billion units, which compares with the expected human population of 7.7 billion in that year (based on information from the United Nations Population Division). In contrast, IoT will have expanded at a much faster rate, resulting in an installed base of about 26 billion units at that time. Installed base is important because it drives the value of service revenue, aggregate communications bandwidth and data center activity. Due to the low cost of adding IoT capability to consumer products, we expect that ghost devices with unused connectivity will be common.

In addition, enterprises will make extensive use of IoT technology, and there will be a wide range of products sold into various markets. The most popular IoT sensors initially installed are those on items that provide remote monitoring (eg. wind turbines, ATMs/ kiosks, heavy equipment, drilling equipment, etc.) or are mobile (autos, aircraft & engines, people, packages, rail, medical equipment, etc.). This too will evolve quickly.

Now, even though there is a lot of talk going around Big Data, Small Data is gradually gaining relevance. While Big Data is centralized, small data is decentralized that makes it more accessible, informative and actionable. Organizations will start to leverage both Small and Big data in tandem to
derive value across functions.

The small data revolution is gradually picking up steam, it can be helpful for organizations to understand larger trends by understanding its constituents. For many practical problems faced by companies today, Small Data could be enough. It could be helpful in dividing problems across people and organizations by creating smaller components out of massive centralized information. This can help people and organizations to collaborate, build and integrate components, making it a mass-participation activity. In simpler terms, what would matter here is the amount of data that an average employee could handle on their own workstation or laptop. Some examples of small data could be data from excel spreadsheets, catalogues from vendors or internal lists.

Does it require a different strategy to handle Small data as compared to Big Data? How is it different?

Big Data if done correctly can help enterprises improve productivity, enhance customer experience and provide them with a competitive advantage. Small data on the other hand could in fact be difficult to manage as it comprises smaller components across systems. Companies need to pursue a unified data strategy and governance approach to help accomplish valuable insights. Organizations can address these problems by integrating small data into this overall data management process. They would also have to process this data in a way that leads to business intelligence. This will happen only if these smaller datasets are integrated as a part of the larger data strategies. This I believe could help them in achieving a competitive advantage.

For instance, let me pick up the eg of Automotive sector. As I’m sure you can imagine, the bill of material for assembly of a vehicle is critical to many aspects of business operations. The average vehicle contains 25,000 or more components. The relationship between base parts, sub-assemblies, subsystems and the final vehicle is essential to understand. From a data perspective these relationships are hierarchical and can be complex to represent accurately and conveniently. Add to this the complexity that parts are very often reusable across multiple vehicle models and option combinations and that “look across” is also important to understand.

This means all the small data needs to be analyzed as a whole to see correlations that cause quality problems.

Are organizations aware of the small data advantage and are having the right strategies in place to take advantage of it?

The concept of small data is increasingly resonating within the industry and the value of thinking small is gaining importance. The strategy is to make data highly consumable and to bridge the worlds of big and small data. A straightforward approach that one can adopt would be by addressing this question- how do we productise Data? Small Data is good way to productise data. Organizations are increasingly looking at vendors who can provide both small data and big data under one umbrella. This enables them to use a combination of different technologies which allow Data Scientists to look at small size data to bring value and then they can run algorithms on the larger data sets to derive patterns and take action.

How developed is the Vendor community in this space? Does it also require different strategies/capabilities on the part of the vendors to help clients take advantage of Small Data?

Currently, enterprises' primary business case is leveraging the Internet of Things to optimize the performance of their large assets. Although there are many other ways in which the Internet of Things can help enterprises (for example, remote operation, extending services or usage-based billing), this optimization leads to reduced operating costs (for example, fewer energy costs), improved availability (for example, reduced downtime from failures) and increased yield (for example, more output from the same operating costs).

Vendors in this space are focused on getting access to the source of this small data and making it available for analysis. Going forward the focus is going to increase on exploiting this data for efficiency and competitive advantage and this requires vendors to build differentiated capabilities that will allow customers to do this with ease.