Solving Big Data Problems with Hadoop
This is the continuation of the transcript of the DM Radio show "Avoiding Bottlenecks and Hurdles in Data Delivery." InetSoft's Principal Technologist, Byron Igoe, joined industry analysts and other data management software vendors for a discussion about current issues and solutions for information management.
Whether it’s changing because of competitiveness or regulatory issues, or that ability to respond quickly without having to build up new types of materialized views is critical for them to be able to respond in a cost effective way. That real time response is absolutely one of the driving features.
I think I agree with Philip, that the big data crisis that was in the past is being addressed. Technologies are there. They can solve big data problems with Hadoop and other types of things, but it is that blending of those different streams of those different data sources that are really the challenge, that these new technologies like data virtualization and data mashup address in a cost effective manner.
Eric Kavanagh: Yeah, let's see Philip any comment from you on where we are here?
Philip: Yeah, well we have mentioned data federation few times here. And also I would like to tie federation back what I almost was talking about, what Byron Igoe of InetSoft was talking about. He was talking about self-service BI and the need to sort of deliver the product of BI sooner, and I do see data federation used with this kind of person bottleneck.
Self-service for the Mildly Technical
We do see companies just with the pace of business nowadays the business people just can't sit around for 90 days waiting for the BI team to create a new report. For them self-service reporting is important. But we are also seeing a kind of practice called agile BI, and agile BI has some overlap here, and so in both agile BI and self-service BI, I think federation is used. Let me explain how.
With self-service reporting the users are typically mildly technical or not technical at all, so you got to give them some database layer and a sort of federated data layer if it really works well in self-service BI. In fact we did publish a TDWI best practice report on self-service BI a few months ago, and people can go to our Web site and get that for free and read it.
It’s one of our big 40 pagers, and I commissioned Claudia Imhoff and Colin White to research and write that. So obviously they have a lot of insights so they talk about the use of federation coupled with self-service BI. Also with agile BI typically it’s not self-service. You do have technical people involved. It is just they get to create reports faster.
They have got to create the data sheets behind them faster. And we are finding that agile BI works very well with federation and data mashup because you think one of the agile method things is you want to get a prototype of a report in front of the business user. He is going to consume it for the most part as early on as possible, and you want to sort of focus on the presentation, but not necessarily have all the data together.
Data Mashup and Agile BI
So data mashup is really absolutely wonderful for providing that sort of data in a hurry for these early prototypes and then as you do the iterations, that agility requires you can come back and polish work from the data layer behind it. So I am saying data mashup is very useful with these practices agile BI and self-service BI to help us speed up that person bottleneck.
Another key aspect of federated data access is its ability to break down silos within organizations. Traditionally, data has been stored in disparate systems, making it difficult for business users to gain a holistic view of operations. Federation enables seamless integration of these sources, allowing users to query and analyze data across multiple platforms without the need for complex ETL processes. This not only accelerates decision-making but also reduces the dependency on IT teams for data preparation and integration tasks.
Moreover, as organizations increasingly adopt cloud-based solutions alongside on-premises systems, the importance of federated data access continues to grow. Modern data virtualization platforms can connect to both cloud and legacy databases, providing a unified interface for analytics and reporting. This flexibility ensures that businesses can leverage their existing investments while embracing new technologies, ultimately supporting scalability and adaptability in a rapidly changing environment.
Finally, security and governance remain critical considerations when implementing federated data solutions. By centralizing access controls and monitoring data usage across sources, organizations can maintain compliance with regulatory requirements and protect sensitive information. Advanced federation tools often include robust auditing and role-based access features, empowering organizations to strike a balance between agility and control as they pursue self-service and agile BI initiatives.