Big Data Analysis— the value it brings, the debate it opens

DQI Bureau
Updated On
New Update

Over the year's organization have used data for varied business logics, say to generate a credit report, helping businesses to make better decisions for granting or denying a loan or providing services. Traditional data management processes are struggling to cope with the emergence of such mammoth amount of information from various sources, their variety and interdependency. The speed with which this is created has changed the landscape of data management. Organizations are struggling with the challenge of managing the combination of structured and unstructured data and identifying what information is relevant and critical to decision making.


An interesting thing to note here is that anyone who has worked with the data in an effort to make better decisions knows that most of the data that we collect is useless; it is a noise which disturbs us. We are in a world where there are multiple dimensions of information it may be 10, may be 15; each dimension has its own value and sometimes these dimensions yells at the analyst "this is important", "this is a trend", there is a pattern", "you cannot ignore this" for his consideration. Today, technology innovation such as business intelligence, predictive analysis and pattern generation allows us to look at the data differently. In the midst of all this, analyzing these pieces of information together by connecting the dots and building a story out of it - to make decisions, predict the behavior and make the right move & strategies, makes Big Data Analysis interesting.

Businesses are fascinated by the strength of these technologies as it helps them to make the right marketing decisions, understand consumer behavior, develop right strategies and forecast the future. The data helps businesses to exactly provide the right product, at the right place, with the right quantity and at the right time. And consumers are captivated by this push mechanism as they now don't struggle even at the beach where they can get SMS on their mobile phones on information about the best quality swim suits. Data Analyst are only able to do this by collecting mammoth amount of information, trending them together, building analytics and building correlation logics helping businesses to take informed decisions. Yes privacy may be a concern but from a business view point, they collected the data because individuals provided the same by filling up a form or accepting a user agreement which provides consent or even making the data publically available through the website they browse, searches they make, application they use, data they provide.

While commercial interest has been the main objective for the businesses, safety and security of citizens has been the driver for Law Enforcement Agencies (LEA). As criminals are becoming increasingly technology savvy and sophisticated, LEA needs to be more proactive and alert. They need to analyze and make fast & accurate detection from all intelligence that they gather be it people lifestyle - what they do, where they do, how they do, or there connections - friends, family, business partners, or any other information source. Time and information are critical factors for law enforcement investigations. They need to connect distinct pieces of information and create a pattern which may be hiding behind the data-the challenge for them is chasing for it and putting it all together.

LEAs are understanding this value and are looking out for Big Data solution that not only collects, and analyzes large data sets from multiple sources, which might be hidden and have non-obvious relationship but visualizes the data to uncover the complex patterns of connections between people, places, and events. These connections are a key component of both proactive policing as well as digital forensics investigations. For example, many state police have launched a program on open source intelligence realizing the value of social media. Using innovative, intelligent data modeling, keyword searches, semantic and sentiment analysis, law enforcement agencies are able to analyze databases with billions of records in seconds, they divulge in patterns that reveal the insights and act as a foundation for predictive analysis critical for accelerating the investigation by generation of actionable intelligence and find criminals faster.

The usage of such solutions often opens up the debate for privacy and security which sometimes become too emotional for people to absorb. LEAs argue that the data mostly is collected from the public domain and they are just linking the same for better policing and safety of the citizens who are all part of the ecosystem. Although difficult to digest, the analogy can be related to a familiar business scenario, where the information provided by the employees to the businesses, add to that the log information collected by the application they access, the network they use, the communication they make is aggregated to generate intelligence for overcoming vulnerabilities and threats to the ecosystem. Most CISOs responsible for protecting the information assets from rogue elements or vulnerabilities in the system make use of such intelligent solution, analyzing large sets of data.

The crux of the issue is that this is an information age and data analysis cannot be ignored, be it for commercial purpose or for law enforcement reasons. Although both have privacy and security implication, we need to forfeit certain rights based on our needs and demands to make utmost utilization of the technology.