The Big Data Gamble

The IDC Digital Universe study a while back stated that the "Big data phenomenon seems to have appeared from almost nowhere, but in reality, some aspects of this emerging trend are not new". That's what the CIOs are saying. It's not new!

Dataquest spoke to some of the leading CIOs in the country and they looked rather perplexed at the undue hype about big data. Some verticals like telecom and retail have been using big data for the last many years. Much of the market confusion stems from the fact that almost all vendors had taken the big data plunge, hence making it one of the hotly debated technologies in enterprise computing.

Vendor Optimism

The bullishness with which the vendors talk about big data stems from the current levels of data growth and enterprise IT's readiness to manage, manipulate and arrive at patterns using internal and external data. Vendors say conventional and traditional BI tools are no good in this age of data divergence. IDC says that the amount of information created and replicated will surpass 1.9 zettabytes (1.8 tn gigabytes) in 2011, growing by almost 9 times in just 5 years.

 

Further, analysts say that the other aspects of big data are indeed new. The variety of the data sources is growing at a rapid rate, particularly as we move into the semi-structured and unstructured realms (eg, social media interactions, rich media files and geospatial information). The other emerging factor that organizations need to contend with is the increased velocity at which data is being generated (eg, real-time sensor data feeds).

"These new aspects of big data are creating unprecedented levels of complexity for IT executives, particularly as they realize that these massive data sets cannot be processed, managed and analyzed using traditional databases and architectures," says Philip Carter, associate vice president, IDC Asia/Pacific.

"What is becoming clearer is that the real value from big data will be derived from the high-end analytics, predominantly using data mining, statistics, optimization, and forecasting type of capabilities to proactively turn this data into intelligence to drive business benefits and better decision making capabilities," he adds.

The vendors make for a convincing argument on why big data makes sense for enterprises and say that the hype is well deserved. Srikanth Karnakota, director, server and cloud business, Microsoft India, asserts firmly, "Big data has certainly created a huge challenge for the enterprises but at the same time, opened new vistas of opportunities for companies, which by analyzing and mining the historical information are able to identify new trends. By analyzing the big data, we are able to do predictive analysis of what is going to happen in the future apart from improving company's efficiency."

But the road to big data is indeed a challenging one. While big data is all about taming the unstructured data and in its ambit includes whole lot of stuff. One of the leading CIOs expressed his concern over meshing his BI with social networks and the challenges on that front as it needs special tools to crawl data from social media. So clearly big data also dictates changes at every level - IT policies, compliance, GRC frameworks, and indeed significant investments.

Vendors agree that the challenges are many but the solutions lies in a judicious blend of strategy and technologies. Says Karnakota, "I see enterprise facing plethora of challenges and the significant ones are: One, data explosion - driven by declining hardware cost and almost 90% of the organization's data created and stored being digital. Two, greater variety of data - customers need to analyze both relational and non-relational data. Over 80% of data captured is unstructured and three, increased velocity of data requiring organizations to respond quickly to rapidly changing data. The need is to explore data interactively leading their information workers with insightful decision making."

This in a way is the crux of the whole big data debate and makes for the need for a proactive analytical backbone that can churn, slice and dice big data. Quips John Haddad, director of Enterprise Data Integration, Informatica, "Big data is being driven by the nexus of three secular technology megatrends - cloud computing, social computing and mobile computing. Cloud computing changed the economics of computing. Social computing enabled the next generation of data-centric applications and, finally, mobile computing, by empowering users with context-aware and location-based services, represents the next surge in device sensor interaction data."

The New Normal

Experts say that the fragile global economy and increasingly fierce competition are putting enormous pressure on the value chains of many organizations across the world. Relatively dynamic industries like banking and insurance, healthcare, and retail, and even the more conservative ones, like manufacturing, are striving hard to change their status quo and looking beyond the four walls of the enterprise and must explore concepts like big data.

 

 

 

Reflecting on this, Jaskiran Bhatia, country manager, information management software group, IBM India/South Asia says, "The trend of CIOs transforming their IT infrastructure into a nimble, automated environment that can create a futuristic enterprise had in fact started a few years back. It was driven by the realization that many organizations would find it difficult to support the enterprise of the future with their existing IT infrastructures, which typically include silos of data, applications and hardware that can slow change. However, the new transformation agenda for CIOs transcends the realms of IT." Agrees Arun Ramachandran, country manager, data computing division, EMC India and Saarc, "First of all, CIOs will have to realize the significance of big data and how the evolving economic environment facilitates IT to become a business transformation function. In the right hands and handled strategically, the massive amounts of information companies collect today can become a valuable new asset. Players seeking additional organic revenue streams should consider tapping their data trove to power a new information services growth engine."

Clearly there needs to be lots of awareness on big data and vendors must come out of the clichd definitions and make numerous use case scenarios that can help CIOs arrive at informed decisions. Putting the entire debate in the backdrop, Dr Mukund Deshpande, associate vice president and head, BI and analytics competency, Persistent Systems makes a very valid point. He says, "If an insurance company wants to improve its customer service and claims settlements process based on customer feedback, the sheer amount of communication that needs to be analyzed to understand the customer sentiment is huge. Here a traditional data infrastructure would not be sufficient to carry out the analysis of textual data. With big data technologies, text as well as unstructured data can be analyzed with commodity hardware infrastructure - which simply means there is no need to invest in a supercomputer."

 

Industry experts say that CIOs are in various stages of exploration in terms of big data. Says Ramanujam S Komanduri, director, BCS, HP India, "In our conversations with the CIOs, they are seriously looking at Big Data and in the ongoing year we will see interest levels on Big Data going up. I think this is one area that will command greater attention due to the broad range of tangible benefits."

Outlook

The vendors remain bullish about big data and each has their solutions portfolio. In the ongoing year one will hear more often about big data and every vendor will try to prove their supremacy in big data analytics. While India specific, 2012 will be a year of awareness and vendors will try many ways to influence the CIOs on the potential impact Big Data will have on their IT capabilities.

According to Gartner, data will grow by 800% in 5 years, with 80% of it will be unstructured. Part of that is the trend called "the collective," which includes data from groups and communities and social networks outside the business. Mining the collective has become a very common thing to do, and it's a great way to understand what the marketplace looks like. The need for data storage means money, and big data storage means big money. Even during the slowdown the flow and growth of data remained unabated. This is where cloud computing eased the burden of a restricted enterprise IT budgets.

Cloud has set the stage to allow big data to be leveraged. As per a report conducted by EMC and Zinnov Management Consulting, cloud computing market to grow to $4.5 bn by 2015. In one sense, the move to cloud (whether private, public or hybrid) sets the stage for big data. Reflecting on this, EMC's Arun Ramachandran sums it up, "We are today at a stage where such technologies are a reality. Really, the time now is about the cloud' meeting big data'. Cloud computing makes big data possible by providing an elastic pool of resources to handle the massive scale of big data. Through cloud computing, IT resources are more efficient and IT teams are more productive - freeing up resources to invest in big data."