Taming Big Data

In the recent times of technology, big data is one of the most talked subjects. The amount of digital data being created globally is doubling every two years, and the majority of it is generated by consumers, in the form of movie downloads, VOIP calls, emails, cell-phone location readings, and so on, according to analyst firm IDC.

Big data has become all pervasive, as the organizations have come to realize the value of insights that can be gained from analysis of a single large set of related data, allowing correlations to be found that can help spot business trends. It is therefore the analytics aspect of big data that is beneficial to the growth of businesses. In the UK for example, supermarket chain Tesco is analyzing its huge data sets of customer behavior information, collected in its Clubcard loyalty scheme to determine promotions and price adjustments.

IDC estimates that the big data solutions market in India will grow annually at 37.8%, from $58.4 mn in 2011 to $153.1 mn in India by 2014.

About a third of Indian organizations are witnessing a 60% YoY growth in big data. However, analysts expect that while interest in big data is strong in India and the Asia-Pacific region, vendor hype and a lack of understanding could drive ‘irrational exuberance and unreasonable expectations'.

The IDC study has highlights that Asia Pacific have markets with unique traits, such as population ‘mega centers,' distributed manufacturing hubs, and fluid regulations on data sharing, which are creating significant new opportunities with big data.

However in Asia, many of the big data initiatives are under way in the enterprises center on building big data repositories. While as an important first step, this won't allow organizations to realize the true benefits of the concept. Purchasing all the servers requires processing and mining the data-even though that processing may only require a few hours in a week or month-it is costly and will lead to resources not being fully utilized at other times.

Getting More Out of Big Data

To make economic and strategic sense, CIOs in India must combine the power of the three IT megatrends-big data, virtualization, and the cloud. Virtualization and the cloud are enablers of putting big data to use-making it possible to create large and highly automated pools of computing that can scale up and down with the volume of data to be processed. Combining these will create a flexible, scalable, and intelligent foundation for big data applications.

Thus, enterprises can economically ‘lease' a virtually limitless capacity of storage from cloud providers in the form of infrastructure-as-a-service, paying only for what they use. So the problem of storage and processing capacity is solved.

However moving large data sets to and from cloud data centers will reveal the weak link in the ‘big data ecosystem'-a network that isn't built to handle such large quantities of data. As with all networking capacity problems, this issue cannot be simply addressed by installing or leasing larger data pipes, because of the bewildering array of new services and the inexorable rise in data.

Processing big data effectively is served using a virtual data center architecture, where the physical walls of individual data centers are effectively broken down to connect multiple data centers as one logical entity. Putting it the other way, a ‘data center without walls' is created, that uses a high performing ‘cloud backbone' network to seamlessly connect a larger pool of resources shared between both enterprise and provider data centers.

By virtualizing and pooling all data centers and network assets, organizations can allow flexible placement and migration of workloads according to changing needs. This is the only IT set-up that can economically meet the infrastructure challenge posed by big data and allow this phenomenal trend to deliver on its promise.

PROCESSING BIG DATA

According to the IDC study, more than a third of Indian enterprises see a prevalence of unstructured data, which they say often delays decision making. Indian enterprises need a cost-effective mean to process big data to meet business objectives.

To make big data work for your organization, such that computing power can easily be engaged to process data sets, you need a high-performance connection from your data center to the cloud provider's data center(s). But beyond a static connectivity service, which can be expensive to have enabled all the time, a ‘performance-on-demand' connectivity service can allow you to turn up the bandwidth when you need it and turn it back down when it is not required.

So make sure you have an IT infrastructure that cannot only leverage virtual resources but, is also built on performance-on-demand based connectivity so you pay only for what you use.