Stale Data – Is the tale getting stale?


Manager: That shipment is still here? We had agreed to expedite!

Team member: It went out at 5:00 AM today. The warehouse confirmed over the phone. It won’t be there in your report but should show up when you download tomorrow.

Sounds familiar? It will be to anyone who monitors business on an hourly or even daily basis.  Over the years, enterprise users have come to accept limitations of business systems and have lived with outdated data.  When needed, adjustments based on educated guesswork and offline info bring in verisimilitude.  The exercise is riddled with inaccuracies and inefficiencies but everyone in the industry works with these handicaps, and hence there is no erosion of relative competitiveness.

The status quo is about to change. Proven solutions for most of the past limitations are available today.  More importantly, many business users are aware of these solutions, and have started demanding real-time data from IT. The rancor gets especially intense on occasions when incomplete or outdated information leads to major disruptions and unhappy customers.

Improvements in both data acquisition and analysis have led to the reduction in information latency. The progress on the analysis front has been primarily due to the advent of in-memory technologies and innovative database designs for continuous data streaming, transformation and analysis.  Past posts on this forum discussed these topics.  So, today’s focus will be on advancements in data acquisition.

Enterprise Transactions

Mostly teams access information in transaction systems through spreadsheets or data-warehouse where overnight batch reports push the data. Of course, the data will be many hours old when the team uses it. However, this sacrifice in timeliness prevents slowing down the system of record due to frequent data access or extraction.

But today, tools that continuously push out changes from transaction systems without materially impacting its performance are available. Any Operational Intelligence (OI) platform worth its salt will include such near real-time connectors for common enterprise systems in its offering. As for systems developed in-house, OI platforms also come with connectors to most databases. Efficient scripts continuously monitor tables for changes and push out the incremental data in near real-time to the analysis platform.

Partner Data

Often partner data acquisition is the hardest part for an organization trying to reduce information latency. Typically, vendors, contract manufacturers, distributors, retailers and other partners provide data daily/weekly through EDI, spreadsheets or a partner portal.  Negotiating for real-time data is hard since it involves IT effort on the partner’s side and of course, the issue of trust is always there. The good news is that change is on its way.

Thanks to initiatives from powerful retailers like Walmart, many companies have experienced the benefits of collaborative practices like ECR, VMI and CPFR. Providing partners with information like production plans, forecasts, detailed sales/ inventory levels etc. in near real-time is imperative for the success of such programs. Companies are also realizing that the IT effort on this front were over-estimated. In many cases, the techniques used for data exchange within the Enterprise can be tweaked for cross-enterprise transfers too.

Social Media

For many consumer-facing teams, real-time monitoring of social media isn’t avant-garde. It helps companies engage customers when they are focused on the product/ service… or in some cases like this when they are pointing out major gaffes in a marketing initiative.

Industrial grade real-time social media data acquisition can be more complicated. Nevertheless, the effort is worth it – more revenue by factoring in customer impulse into activities like price setting or inventory rebalancing. Sophisticated crawlers and scrapers for such purposes are gaining traction and should be commonplace soon. 


Of late, a number of platforms to collect store and analyze live data-streams from IoT sensors have emerged. When these sensor data streams are integrated with transactional data, real-time operational intelligence to drive and improve business execution takes hold. Some techniques are discussed in this post. In short – for most part the technology has matured to the level where wide scale use is possible.

Overall, it is not a stretch to say that an “only real-time” world is not far away. Many would argue that most applications do not require real-time information.  But, it doesn’t hurt them either.  Today stale data is a reality in the enterprise because of real and perceived IT limitations.  At current rate, this tale could be stale by the turn of the decade.