BlogThe promise of big data is enormous. The idea that you could automatically discover unforeseen value in the sea of data you collect about your business sounds great. But in practice, big data projects often underdeliver, and the hype around the topic makes it tough to find real information about the efficacy of big data projects. Of course, big data vendors have a strong motive to make buyers feel like big data is simple to implement, and buyers aren't exactly scrambling to talk about it when that turns out not to be true. It's embarrassing to spend a boatload of money on a platform only to find you can't actually use it that well.
We teamed up with TechValidate to survey 113 IT Directors and other tech professionals to find out how they were using big data and whether the value was living up to the hype. Questions included:
- How long has it taken you to go from a Big Data idea to a solution that delivers value in your organization?
- Where are you using data analytics effectively within your business to improve decision making?
- What are your main challenges in regards to analyzing your business and IT operations data?
- And more.
IT directors and professionals describe their biggest hurdles in implementing big data initiatives.
The survey responses were mixed, but hopeful. While many respondents reported getting some value out of their big data initiatives, very few reported having "mastered" big data, or even being able to get their implementations fully operational. Respondents also got quite specific about why they were struggling to succeed with big data. That's great, because when you know what's wrong, you can work to fix it.
We'll go over a few of the findings below, and you can download the full report of the survey results here.
"We have too few specialists to analyze and interpret data"
41% of respondents said that having too few specialists to analyze and interpret data was the main challenge they faced. This is likely a factor that is under-accounted-for as teams plan out big data projects. Qualified analysts aren't common or cheap, and unless you have a big data platform that does most of the work for you, you're likely to need dedicated personnel to get the most value out of a big data initiative.
The Solution: When you're researching big data platforms, be sure you understand how reliant you'll need to be on data scientists or specialists to actually get value. Get references from other companies that use the same products, and find out how many dedicated staff they have for analyzing the data. Needing dedicated staff isn't a dealbreaker, but you should incorporate the added cost into your decision-making process.
"We can't combine and correlate our data sets"
40% of respondents said that they were unable to extract the data they needed and combine and correlate it with other data sets to find value. This is a classic problem in every field that deals in data. Your data lives in silos, in different formats (often proprietary), controlled by different teams. Correlating different datasets is the linchpin of successful big data initiatives, and if you can't do that, you'll always be behind.
The Solution: Resist the urge to buy solutions that lock you into a proprietary data format or restrict the way you use your data. Vendor lock-in is a huge barrier to succeeding with big data. So when you're vetting tools to use for any purpose in your data environment, choose ones that make it easier, not harder, to get your data into other systems so you can analyze and visualize it however you choose.
"We have the data, but we can't use it"
34% of respondents said that they weren't able to operationalize the data they were getting from their big data initiative. No matter how good your data is, if you can't act on it, it isn't valuable.
The Solution: There's no single way to address this issue, but this is a great motivator to help your teams work more closely together to build processes that make it easier to see data, understand how to act on it, and take action in a timely manner.
It is also crucial to start with good questions. Don't go in thinking, "If we combine all this data, we're sure to find great insights." Start smaller, with questions you already have. Look for those answers in the data, then expand. The business should drive how you use the data, not the other way around. If you're thinking, "We need to get some more value out of this data to justify all the money we spent on this big data platform," then you need to step back and reassess your priorities.
Get the full survey results report to learn how your peers in IT are using big data, what their biggest struggles are, their most feared consequences when making decisions with too little data, and where in the business they're getting the most value from big data initiatives.
