Let’s take a look at DMS problems from the perspective of the DW data concerns: Data Accuracy, Ease of Accessibility, Security etc.
The typical problems of many organizations stem from not staying true to the primary objectives of the data storage.
- Many copies of the same data, with sometimes conflicting attribute values. It’s not uncommon to see 17 data warehouses where Transaction Processing Systems struggle to send out different shapes of the same data, and this data then gets forwarded to all kinds of data lakes and data swamps.
Storage may be cheap these days, but it’s not where the true costs are paid – with so many versions of truth, you can argue that decision making is almost always made based on dirty data, and this can easily cost millions of dollars of lost opportunities, or worse – revenue losses or regulatory penalties.
Redundancy is easiest problem to overlook, that’s why it’s so common.
- Poor data quality. Many organizations don’t have processes in place to validate their data or – this would be even a better solution – design schema that prevents poor data from entering the warehouse.
This violates the Accuracy concern of data.
- Poor interface. This violates the Easy Accessibility concern. Imagine walking into a Home Renovation warehouse and seeing no product tags, confusing shelve arrangements, rotten products and staff that struggles to explain where to find items.
Too often organizations forgo data dictionary and good schema design because of tactical decisions and needs to meet artificial deadlines. This forces users of data quickly lose interest and revert to old ways of relying on their own copies of data, which leads to proliferation of point-to-point feeds and tons of technical debt. This spreads like cancer in the organization, sucking out vital energy of organization.
- Lack of reality. Often people who run a data warehouse distance themselves from business so much that they cannot answer real-life business questions.
The data needs to reflect complexity of the real life, to enable business, as opposed to continuing to exist for its own sake. That’s why it’s so important to have business buy-in and continuous feedback from the very beginning of the initiative, and during the life of the data warehouse.
- Delays. Data can quickly lose its value, and if data is stale or too old, it serves no business purpose.
The days of batch-processing last month’s files are almost gone. The most successful organizations strive to build real-time pipelines of data that enable making decisions on the fly.