Big Data is defined as large amounts of data that requires us to re-think how we store, manage and retrieve data. How big is big? One way to think of it is that it’s so big that the tools we use today cannot cope with it. This is the reason why core thinking around how we ingest data and transform it into insight and this is the differentiating factor that a Big Data solution brings to table. Big Data is about doing things differently and enabling breakthroughs by getting to information faster
Today, the issue we are facing with big data is that there is so much hype around it in the IT industry that it is spreading to general business discussions and infiltrating almost all media, which further adds to the hype. Yet, despite (or because of) this hype, we need to get our heads around it. So, let’s break it down into manageable chunks to help us understand it.
Data is getting bigger
Irrespective of the industry or sector an organization is in, one thing is certain for 2013 – every enterprise will have to pay attention to managing this data growth. Whether it is labelled as big data or not, we cannot deny that there is a huge amount of machine data being generated today due to mobile devices, tracking systems, RFID, sensor networks and social networks to name a few. Only recently, Facebook claimed it had over one billion active users, all creating their own big data spheres of content and making the social network ever-complicated to sift through.
The same is happening in organizations and businesses all over the world, as we have seen with eBay and NASA. Information on sales, customers, partners, products, employees, contracts and more is doubling and tripling, making it all the more harder to find the right and relevant information quickly at the right time. While regulations and compliance mean it is vital companies retain all information for use if and when they are called upon, there is a secondary benefit to being able to quickly access data and map it against a different data set for comparisons – namely competitive advantage. By analysing data to spot opportunities, anomalies and make discoveries about your business, the business can be run more effectively and efficiently.
Big Data – Complimenting Existing Information Architectures
Even today organizations do not have a formulated strategy to manage Big Data. In fact Big Data systems are most commonly seen as complimentary to an organization’s existing data infrastructure rather than being a replacement.
The significant historic investment in centralized and localized data warehouses along with concerns on data security and protection are some of the reasons why there is no investment in a separate strategy for Big Data. Additionally businesses face the complexity of using existing software tools to deploy and manage Big Data systems like Hadoop; the current generation of Big Data management tools is less evolved than SQL data access tools. This gap is further enhanced with shortage of IT skills with current Big Data management systems like Hadoop, NoSQL, and Google BigQuery
The Role of Analytics in Big Data
Big Data solutions in isolation provide little value to an organization unless that data can be acted upon to support the decision-making process. While much has been written and talked about the underlying technologies that offer storage of and access to extreme volumes and variations of data using distributed computing capabilities, it is only in the analysis of that data that real value is extracted. While this is true for data of any size or type, it is particularly relevant in the Big Data arena. As Wayne Eckerson from TechTarget points out in his research paper entitled “Big Data Analytics: Profiling the Use of Analytical Platforms in User Organizations”, “A valuable characteristic of ‘big’ data is that it contains more patters and interesting anomalies than ‘small’ data. Thus, organizations can gain greater value by mining larger data volumes than small ones.”
Who actually works with big data?
It might seem quite simple to access the data and compare and contrast it with other data sets, however, until fairly recently, this was only feasible if you had an IT department or a particularly scientific mind that can process mathematical equations. By relying on the IT department to find the relevant information, format it and process it with the other required data set, the information was often already out of date before it found its way into the business user’s hands.
With a wider access to data for users, IT is freed up and big data sets can get the big business brains behind them to actually make use of the insights garnered for competitive advantage. It is now possible, for instance, to analyse retail sales trends with massive transactional data from the data warehouse to reveal customer buying patterns. This in turn can inform sales promotions to make them more effective and to foster better customer loyalty as they are targeted in a more tailored manner.
Technology for big data management
We are now getting up to speed with big data – technology exists to deliver analysis from varied data sources. Market has a choice today with self-service new generation data discovery tools BI like Business Discovery Platform. A business user can now make direct queries into big data combined with data loaded in memory, meaning no information has to be downloaded.
Big data will always exist; although Gartner believes the hype will subside in a couple of years as more technologies and solutions appear to help organizations better manage it. The key is to remember that big data is but a sum of its parts. Broken down into byte-sized bits it can deliver all kind of insight and business discoveries. The important factors are the volume, velocity and variety of the big data – as well as the technology and business brains behind the analysis.
The ‘last mile’ with Big Data
One of the big challenges in telecom is the “last mile” – bringing the telephone, cable, or Internet service to its end point in the home. It is expensive for the service provider to fan out the network from the trunk or backbone – to roll out trucks, dig trenches, and install lines. As a result, in some cases they pass high installation costs down to the end customer – or neglect to go the last mile at all. There is a “last mile” problem in Big Data, too.
Today, most vendors working on the problems of Big Data are focused on processing the data – they are focused on the backbone, to use the telecom analogy. The last mile: this is where Business Discovery Platform fits into the picture. The new generation data discovery tools is a great complement to the capabilities of vendors focused on processing Big Data and truly provides that high value, highly relevant component of Big Data, namely providing analytics and meaning to their data, for everyone.