Business Data Lake – Key to Successful Digital Transformation

Today, a majority of companies have significant frustrations with their current delivery methods within IT as they do not realize the value they expect. Only those who break out of traditional information silos such as the enterprise data warehouse (EDW) manage to see significant new business value from IT.

One provocative view is that it is not surprising that the EDW has failed to achieve data democratization. Its goal was to be the single place for all the end-to-end enterprise information, empowering individuals across the organization, the reality, however, is that EDW-based approaches are biased towards committee-based decision-making. In an EDW, there is a single schema defining the data; experience shows that this definition usually ends up serving the needs of a narrow subset of the organization, while the rest generally have to make do with views of the data that represent a significant compromise of various conflicting or misaligned requirements. As a result of which spreadsheets remain the single most successful analytics tool, because they work for the individual not by committee.

As a result it is necessary for organizations to take innovative approaches, which can free up data from silos by enabling both the sharing and the continuous improvement of insights across the business.  We have always taken a business rather than technically focused approach to information, and this has  resulted in Business Data Lake (BDL), which was co-innovated with Pivotal and has been adopted by EMC Federation (including Hortonworks, SAS and Cloudera) and Informatica.

The BDL represents the new approach to the creation of analytical insights for the business, from the acceleration of traditional enterprise reporting through to new analytics driven by data science. It works with high volumes of structured and unstructured data, storing them at low cost and making insights rapidly available throughout the enterprise. It can coexist with earlier investments, accelerating the evolution of the information landscape.

By introducing the BDL, and moving towards a more iterative and agile approach to analytics and insight delivery, CIOs can evolve their information landscape to reposition the IT function as the business’s “insight enabler”. As a result, decision-makers at all levels of the business will be able to seize the opportunities of big and fast data more rapidly and so gain competitive advantage.

The Business Data Lake approach has five basic principles are:

Store Everything (Ingest any kind of data at scale, structured or unstructured data sets – from any internal, partner, IoT or open data)

Store for both near real-time and long-term analysis at the lowest possible cost

Distill on demand (Analyze in bath or real-time, Complement with data science tooling, Concentrate on the local view)

Encourage local requirements (Surface insight into operational processes, provide the view the business users want, for each of the business users)

Govern where it matters (Focus on Master, Meta and Reference Data, Govern only where required by compliance or where collaboration is required)

The BDL subsumes the existing EDW, rather than requiring it to be replaced. If an EDW is satisfying its current data capture and base reporting requirements, the BDL complements and extends to provide to more powerful predictive analytics engine and to enable new information sources to be adopted more simply and cost effectively. It also draws information from existing systems and legacy data solutions and provides a platform on which new insight driven services can be developed. These insights can be integrated back into operations or turned into entire new data monetization business lines. It Having an evolutionary approach towards the new data landscape provides a more assured and reliable path to the next generation of insights.

Compared to older generation approaches, the Business Data Lake enables organizations to progress much faster towards their goals. It democratizes insight by giving the business the capabilities it needs to seize opportunities rapidly, providing a common analytical engine for the business. It is also designed to overcome the limitations of traditional business intelligence (BI) systems and provide an evolutionary approach that helps organizations augment and transition to a data-centric business.

The BDL has the following fundamental advantages. It:

* Enables a focus on local requirements, whether from the board or the warehouse manager

* Provides access to everything by storing it in a single connected substrate

* Provides key insights from data where and when it’s needed

* Enables the appropriate level of governance to accelerate business collaboration

The key success factor for digital transformation is to have a clear strategy, driven from the centre, to ensure organizations work towards the returns they expect. The BDL approach gives the CIO and IT function the ability to align and accelerate the digital strategy for the business and enables it into operations by providing a robust and flexible platform for the creation of business insights.