Time Series Data + AI + Cloud

Nikunj Mehta
Feb 1, 2024

Key takeaways:

Integration between data management and AI is simply not scalable unless you solve both storage and processing problems equally well
IIoT Clouds such as from AWS, Azure and Cognite require varying degrees of IT efforts to design and build individual applications as well as the right platform
Real-time AI Cloud effortlessly exploits diverse real-time data forms like IIoT, telemetry, and sensor data, driving impactful outcomes in Smart Manufacturing and beyond

Water, water everywhere, not a drop to drink. We are really good at producing real-time data but we are really poor at putting it to work. This is a serious problem because, according to IDC, 50% of all data being produced in the world will originate in IoT and be from machines, sensors, and industrial tools. There is no doubt that we need to go beyond conventional means of analysis for Smart Manufacturing.

Real-time data takes various forms – operational data, IIoT data, telemetry data, SCADA data, PLC data, machine data, and sensor data – and share the common characteristics of time series data. Traditional BI approaches such as charts and dashboards are ineffective at leading to actionable insights from this complex data type mainly because of their complexity and volume.

This article, therefore, talks about an architecture and technical approach to solving this conundrum for manufacturers, industrial facilities, producers, mobility providers, and heavy asset operators. All the focus in this article is on putting their real-time data to work.

Why AI for real-time data?

Before we dig into this further, just a little context about us. I (and Falkonry CTO, Dan Kearns) have have been trying to crack the time series data code for 14 years through two startups. For the first decade, we focused on patterns since we had seen people explaining their problems through visible time series patterns. We realized about 4 years ago that the number of patterns people knew of in their domain was vanishingly small compared with the number of problems they wanted to solve. The patterns they didn’t know existed – the unknown unknowns and surprise patterns – were the ones that always tripped them up. No matter how many alarms they developed to get ahead of the problems, there were many many more gremlins to eliminate.

Big Data approaches for Industrial IoT applied Machine Learning (ML) to real-time data, which worked well for use cases with labeled data and consistent behavior. These were highly dependent on data science teams, required intensive coordination between IT and OT teams, and minimal changes to operations. However, economic and technical constraints have made ML impractical for all but a few IIoT needs. Without AI, it has become impossible to find the known and unknown patterns in such colossal quantities of real-time data.

This is where a recent statement by Ali Ghodsi, CEO of the leading data company Databricks, is prescient “AI will be native to wherever you have data.” One needs to put intelligence as close as possible to time series data. What intelligence is that?

So Falkonry developed an AI for automated time series anomaly detection to locate sequences, values, distributions, and trends that were out of place in real-time data. This AI gets to work as soon as 10,000 values accumulate (it can do it with fewer if we have very little or slow data). There is no manual set up or labeling and anomalies can be effortlessly found wherever real-time data is available. This AI is native to time series data and it is designed to reside near it. You can see an example of this AI in the video below.

To get the most out of time series data, we have integrated the time series data ingest and anomaly detection paths in the Falkonry Time Series AI Cloud.

Why AI Cloud?

The kind of integration between data management and AI is simply not scalable unless you keep them close together and solve both storage and processing problems equally well. An AI Cloud doesn’t behave like a database but instead optimizes for scalable AI.

Real-time AI Cloud is a data and analytics technology designed specifically for time series data, originating from PLC, SCADA, telemetry, sensors, etc. Many other structured data systems can also supply time series data such as quality results, energy prices, and attendance records. Yet other time series data may be of a categorical nature such as strings and IDs. Most time series data is not identified for each record other than by time stamps. Therefore most time series data doesn’t fit neatly into a database. (This is the reason time series databases have done so poorly outside crypto and e-commerce.) On the other hand, there is a lot of analytical value to time series data but such analytics are hard to perform from outside the data platform.

Processing extremely large volumes of time series data very efficiently is much more important than ensuring every single data point is reliably stored. While structured data is written once and read many times, with real-time data, it is written once and then recalled perhaps 1.001 times. This is because one doesn’t ordinarily query time series data, but rather seeks to interpret it immediately and automatically. Typical database performance criteria have very little if any meaning to the field of real-time data and its analogs in telecom, energy, manufacturing, data centers, and defense. Because it is dealing with real-time data, latency of processing is also important and the Real-time AI Cloud.

So we decided to create an elastic and always-on Cloud to provide the following capabilities in a business model that is based solely on number of data points processed:

Receive and parse time series
Calculate derived time series data using mathematical expressions
Store time series data and their time aggregates
Track data quality and flow characteristics
Organize time series by domain properties
Visualize time series and their statistical summaries
Retrieve time series based on time ranges
Score time series data in real-time against models
Highlight and explain anomalies in time series
Compare time series for properties
Look for correlations among time series and anomalies in such correlations
Locate and explain patterns in time series
Estimate future value of time series and time to a future value
Locate and explain time series that match predicates such as thresholds and persistence
Trigger business logic-based event handlers to initiate requests to other systems

Over time, a Time Series Cloud would grow even more native capabilities to optimally exploit computer architecture options and the nature of time series data. Externally developed models may also be applied in real-time against time series data. The video below shows just some of the key characteristics of the underlying services for time series storage, organization, and visualization.

IIoT Cloud vs. Real-time AI Cloud

Just like Snowflake designed a Cloud data warehouse using AWS and Azure platform services to have the right economics, usability, and versatility, Falkonry has built an AI Cloud for Real-Time data to enable end users to effortlessly put their real-time data to work. This AI Cloud is built on a dozen distinct patents to achieve the ideal mix of efficiency and efficacy.

Summary

30% of the data generated will be consumed in real-time in industrial, automotive and navigation operations. It is time to graduate from home grown platforms and assembled kits for real-time data management and processing to a native AI Cloud for real-time data.

With an AI Cloud to boost Smart Manufacturing, Smart Fab, Smart Oilfield, Smart Factory and Smart Navy, we can shift our attention from technology to business. We can focus on human factors that limit AI impact and design business models that better match motivation to incentives.

With this, I now declare the Real-time AI Cloud Olympics open! We hope you will join us for the exhilarating performances to follow.