The Challenges of Big Data for BI
Below is the continuation of the transcript of a Webinar hosted by InetSoft on the topic of Traditional BI vs Agile BI. The presenter is Mark Flaherty, CMO at InetSoft.
Mark Flaherty (MF): In terms of social networking people are trying to understand who are the main influencers in social networks in order to market to them. Then they want to know how effective those marketing campaigns are and whether or not those are the right influencers to target those other people in the social network, in other words to cause the ripple behavior across the social network and get more sales.
In terms of sensor data, well I would say most definitely supply chain and distribution channel optimization is the are of focus. Its also relevant from a green computing perspective, and also a cost saving perspective. People are analyzing where money is being spent in terms of energy usage, and where you can save it, and where you can reduce your overall costs.
The bottom line is we are dealing with large volumes of data, and in addition to that, we have got low latency requirements to get that data in order to make quick decisions or present offers in real-time, etc. Large data volumes tend to present problems for traditional kinds of analysis.
Another area is sentiment analytics where people look at external data sources to see what customers are saying about their products and services to see if there is something they need to understand or how many complaints there are, any faults there that regularly occur in a product that maybe get voiced on social sites. Another area of interest for unstructured content is competitive analysis. There is a whole range of applications emerging focused on that kind of data.
Big data presents problems for BI systems that perhaps are not seen with the smaller volumes of data within the enterprise. One is the ability to capture low latency data. For example, a few of the applications I have already mentioned, online advertising, location specific advertising on mobile devices, these are applications that need to look at large volumes of data, but need a very, very low latency. That can be a challenge with the data management system.
Big data also requires data integration tools to connect to the large data sources. For example connectivity to Web logs is probably mainstream these days. But connectivity to online gaming data sources is probably not, and therefore you have got to look to see if data integration vendors can support those requirements.
Data transformation and integration on large data volumes is a challenge. It's all and well good to have a database that can scale up for large data volumes, but what if you can’t get the in there because of a problem that’s caused by data integration, you’ll be prevented from analyzing it.
Your analysis tools can’t afford to do full refresh. Instead they need to be able to detect the changes. Perhaps you need power loading capability in order to be able to get that data in rapidly. And then the question is also where do you put the data in order to be able to analyze it? Should it go into a relational database? Should it go somewhere else, and what happens if you want to maintain history on these kinds of data volumes? Do we have to analyze everything online or can we do some of these large data volume analyses in batch? There is another term that comes up with big data and business intelligence which is Hadoop for BI.
What is meant by Hadoop for BI?
Hadoop is a batch programming framework that can be used in order to scale analytical processing across thousands of machines. And effectively, what you are doing therefore is a kind of batch processing on a large volume of data. But it's also possible to create analytical functions using MapReduce which can process large volumes of data.
Some people think a lot of this technology is useful for absolutely everything, I mean if you really want low latency data, and that’s critical to your business, then this kind of technology is not really geared up for it. If you are looking for a small subset of data out of a very large data set, then again I would look to see, is there a place to store your data. It might be better to reduce the data down to manageable data blocks rather than relying only on access via a relational database.
Data Quality and Consistency Challenges
Another challenge organizations face is the difficulty of validating data quality at scale. As data volumes grow, so does the likelihood of inconsistencies, missing fields, duplicate records, and conflicting values across systems. Traditional data cleansing methods often cannot keep pace with the velocity of incoming information. To maintain trust in BI outputs, companies must invest in automated validation rules, anomaly detection, and governance workflows that continuously monitor data integrity rather than relying on periodic manual checks.
The rapid expansion of data sources also introduces complexity in maintaining consistent definitions across the enterprise. Different departments may use the same term—such as “active customer” or “qualified lead”—but apply different criteria. When these definitions collide inside a BI environment, dashboards become misleading and decision‑makers lose confidence in the insights. Establishing a centralized business glossary and enforcing semantic consistency through metadata management tools helps ensure that analytics results are comparable and reliable across teams.
Performance and Scalability Concerns
Scalability is another major concern as organizations attempt to process increasingly large datasets. BI systems that performed well with gigabytes of data may struggle or fail entirely when confronted with terabytes or petabytes. Query performance slows, dashboards take longer to load, and analysts spend more time waiting than exploring. To overcome these bottlenecks, companies often need to adopt distributed processing frameworks, columnar storage engines, or cloud‑native architectures that can elastically scale with demand.
Security and compliance risks also intensify as data ecosystems expand. Sensitive information may be replicated across multiple platforms, exported into spreadsheets, or accessed by users who do not fully understand regulatory requirements. Without strong access controls, encryption policies, and audit trails, organizations expose themselves to breaches and compliance violations. Embedding security into every layer of the BI pipeline—from ingestion to visualization—helps reduce these risks while still enabling broad analytical access.
Finally, the human element remains one of the most persistent challenges in big data environments. Even with advanced BI tools, many employees lack the training or confidence to interpret complex datasets effectively. This skills gap can lead to misinterpretation, overreliance on default visualizations, or avoidance of data tools altogether. Investing in data literacy programs, hands‑on workshops, and role‑specific training ensures that users not only have access to big data but can also translate it into meaningful business decisions.