There are many scenarios that showcase the pains organizations are going through with their data platfoms. Of course, everyone is different, but we wanted to draw your attention to just three use cases that may cover big chunk of the business landscape.
Can we use our Netezza instance to store a little bit of transactional data, please?
Case one: bad initial decision to save money and run a single database for both daily transactions and batch reporting needs.
This is an initially cheap arrangement that pretty much inevitably costs a lot in the end. Usually this arrangement is justified by the fact that the database is utilized very lightly, if at all, after the business hours, so why not add an after-hours load to it and save money by avoiding a purchase of a separate database instance.
As one grows their business, the amount of data accumulats, their database becomes slow, queries take much longer and they may not be able to finish all of the end-of-day reports before the new business day starts.
This can pose a significant challenge for the daily business operations since the database all of the sudden is used for both transaction processing and batch reporting at the same time.
This will almost inevitably mean business losses, and as the priority of dealing with the database load rises to the top, the costs of the solutions will significantly increase because of urgency of the problem and unpreparedness of the system for a sudden upgrade or migration.
Will Hadoop save us?
Case two: Reporting is run off a separate database instance, which is populated in periodic batches.
But capacity planning either was not done or the amount of data outgrew the physical limitations of vertical growth. Naturally, in a lot of cases, a solution is proposed to either load data into Hadoop or into an MPP database, or maybe a NoSQL database.
The choice needs to be carefully considered an driven by the business needs. It’s very easy to start growing your Hadoop farm and end up with 5,000 commodity servers that are now sitting under-utilized and cost arm and a leg to maintain, patch and operate.
How do you know if you are heading for a disaster?
Case three: not paying attention to the database metrics.
The answer to this is you need to know where you are, if you want to know where you are heading to. So collect your metrics on a regular basis, pay attention to the spikes patterns, utilization and grow rates – are they irregular, linear or exponential? Then plan your future accordingly. Calculate costs of owning or leasing a cluster when you need it.
Architecting a solution in such cases as above is prone to errors since a customer is often unaware of the actual business needs. Often there’s a broken relationship between technology teams and the business, to the point that the business hires their own technology team.