Is Hadoop Killing the EDW? Fair question since in it’s 11th year Hadoop is known as the innovative kid on the block for analyzing large data sets. If the Hadoop ecosystem can analyze large data sets will it kill the EDW?
The Enterprise Data Warehouse has ruled the data center for the past couple of decades. One of the biggest question big data question I get is what’s up with the EDW. Most database developers and architects want to know what is the future of the EDW.
In this video I will give my views on if Hadoop is killing the EDW!
Transcript
(forgive any errors text was transcribed by a machine)
Hi I’m Thomas Henson with thomashenson.com and today is another episode of big data big questions today’s question this made it a little bit controversial but it is big data killing the enterprise data warehouse let’s find out so is the death of universe data warehouse coming all because of big data the simple answer is in the short-term the medium-term no but it really is hampering the growth of those enterprise traditional data warehouses right and part of the reason is the deluge of all this unstructured data so 80% of all the data in the data center and in the world is all unstructured data and if you think about enterprise data warehouses they’re very structured and they’re very structured because they need to be fast right so they support our applications and I support our dashboards but when it comes to you know trying to analyze that data and trying to get that unstructured data into a structured version it really starts to blow up your storage requirements on your enterprise data warehouse and so part of the reason that the enterprise data warehouse growth is slow is because of 70% of that data that’s in those enterprise data warehouses is all really cold data so really you know only thirty percent of the data in your enterprise data warehouse is what’s used and normally that’s your newest date so that cold data is sitting there on some of your premium fast storage you know taking up that space that has your licensing fees for your enterprise data warehouse and also the premium storage and a premium hardware that is sitting on then couple that with the fact that we talked about 80% of all new data that’s coming in and data is created in the world is all in structured data right so they could clean up data from Facebook any kind of social media platforms but video and you know log files and some of your semi structured data that’s coming you know often for your Fitbit or any kind of those IOT or any of the new emerging technologies and so all this data if you’re trying to pack it into your presentation warehouse just going to explode that license fee and then also that hardware and then you don’t even know if this data has any value to it soon and that’s where big data in Hadoop and spark and that whole ecosystem comes because we can store that data that unstructured on local storage and be able to analyze that data before we need to you know put it into the dashboard or some kind of application that’s supporting it so in the long term I think that the enterprise data warehouse will start to sunset and we’re starting to see that right now but for the immediate term still you’re seeing a lot of people doing enterprise data warehouse uploads so they’re taking some of that 70% of that cold data the transfer in Hadoop environment to save on calls to the sable net licensee but also to marry that with this new data this new instruction data that’s coming in from whether it be from sensors social media or anywhere in the world and they’re marrying that data to see if they can pull any insights from it then once they have insights depending on the workload sometimes they’re pushing it back up to the enterprise data warehouse and sometimes they’re using some of the newer projects to actually use their new environment and they’re you know big data architecture to support those production you know type enterprise data warehouse applications so so that’s all we have for today if you have any questions make sure you submit up to big data big questions you can do that in the comments below or you can do it on my website thomashenson.com thanks and I’ll see you again next time.