Preventing Blackouts: Real-Time Data Processing for Millisecond-Level Fault Handling

At Ververica, we specialize in helping companies ingest, transform, analyze, and act on large volumes of data in real time. Customers like ING, Microsoft, Booking.com, and Alibaba rely on our Unified Streaming Data Platform to power critical applications, including fraud detection and cybersecurity.

Reflecting on Ververica’s experience helping companies understand threats, act quickly and prevent losses combined with my personal experience in energy engineering, this blog explores how millisecond-level fault detection and real-time data processing can help build a more resilient grid that integrates even higher levels of zero-carbon renewable energy while, quite literally, keeping the lights on.

Power Grid Monitoring: Black Magic Or Delicate Balance?

The power systems we use today were mostly designed in an era where almost all energy was produced from thermal sources, which can generally be throttled at will. If the grid needs more power, you can increase the output of a gas turbine or a coal boiler by simply increasing the amount of fuel that flows into the system. In principle, it’s similar to stepping on the gas pedal of a gas-powered car that is run by an internal combustion engine.

Electric grids are inherently delicate: at every moment, the total energy produced and input into the grid must precisely match the energy that is consumed by all the users drawing power from it. One additional person turning on a kettle means that somewhere, somehow, the production of one of the power plants connected to the grid at that point must also increase by approximately 1 kilowatt.

One can’t help but wonder:

• How can my grid operator know that I really need some tea now?
• How can they notice and respond to my personal choices?

The answer is by carefully monitoring grid frequency: when more power is drawn from the grid than fed into it, the grid frequency dips - and this is a clear-cut signal to increase power generation. A delicate balance between power supply and demand maintains the grid frequency at (or very close to) its design frequency: 50Hz in most countries (mainly Europe, Africa, Asia) and 60Hz in others (mostly in the Americas). This is absolutely crucial because electrical and electronic equipment will malfunction if it is fed electric power at a different frequency than what it is designed to handle.

Figure One: Power Grid Monitoring and Grid Reliability

For example, in 2018, a frequency slip in the Central European grid made all grid-connected clocks lag in some parts of Europe, since they rely on the grid frequency to count the passing of time. This meant that for the duration of the event, critical systems like trains, radars, and others were unreliable or rendered completely unusable. Because the power electronics that control large electric motors and actuators can be damaged by higher or lower frequencies than what they are designed for, they will shut down preventively in those conditions to avoid further damage. Electronics like computers and microcontrollers are also very sensitive to frequency deviations, and the impact of these types of protective measures means loss of service and interruptions.

Renewables Enter The Game

The scientific consensus is firm about climate change and how human actions, in particular greenhouse gas emissions, are accelerating it. Simply put, it is humanity’s duty to stop and reverse this, if we are to survive as a species. Renewable energy is probably the brightest development in this area, allowing us to grow our energy usage by over 400% since the 1950s, while at the same time halving the average emissions from 900 gCO2e/kWh to below 450 gCO2e/kWh. However, renewable energy introduces a whole new set of challenges to the energy grid. By its very nature, renewable energy is not ‘dispatchable’, i.e., it cannot be turned on and off at will (with the notable exception of hydropower). This means that the rest of the grid must constantly adjust to its availability, one second at a time.

Figure Two: Energy Usage and Emissions Comparison, 1950-2025.

Forecast models have gotten very good at predicting the long-term output of a renewable resource (for example, public models exist that forecast how much solar energy your roof will produce if fitted with PV panels, month by month), but it is notoriously difficult to forecast output in the very short term. In addition, any weather variation can have an immediate impact; cloud cover, for example, can reduce the output of a solar plant from 100% to 37% in 1 minute.

To make up for this unpredictability, power grids have multiple reserve mechanisms that step in and fill in the gaps left by renewable power sources like solar and wind that decrease output momentarily, like a solitary cloud passing overhead, or more permanently at sunset, or if a power line fails. Many thermal plants do not run at full throttle - or in some cases, don’t run at all - just to be ready to ramp up and pick up the load in case that is needed. These supply the reserve margin of the power system, and most countries generally provide sufficient incentives for firms to profitably build and operate the plants that provide this reserve capacity. However, when either primary production or reserves are insufficient, the end result is a blackout.

The Iberian Peninsula Blackout Of April 2025

On Monday, April 28, 2025, at 12:33 CEST, the Iberian grid, which covers Spain and Portugal, experienced a sudden fifteen gigawatt drop—about 60% of Spain’s load—in just five seconds, triggering a peninsula-wide blackout lasting ten hours. This is nothing short of catastrophic: European nations are accustomed to continuous, on-demand electricity access, and it is a foundational commodity the community relies on. From traffic lights to data centers, electric vehicles to televisions, everything that is considered a part of basic existence stopped working suddenly with no explanation or indication of when it would be available again.

Figure Three: Iberian Peninsula Blackout of April 2025 Affected Region

Currently, the proximate cause of the blackout is still being investigated, and a report will be published by the authorities in due time. Thus far, the Spanish Transmission System Operator (REE) has acknowledged the following: the sudden drop of fifteen gigawatts of generation capacity in the southeast of Spain caused the frequency to drop below acceptable limits (press release, in Spanish). This, in turn, forced other generating units connected to the grid via inverters to also disconnect from the grid to protect themselves, starting an adverse domino effect that made Spain and Portugal go dark within a few seconds.

Why Millisecond Detection Matters for Power Grid Monitoring

The protective relays and circuit breakers that are fitted to European grids operate in the 10–100 millisecond (ms) range. This is mind-blowingly fast: as my friend and Ververica’s Field CTO Ben Gamble likes to point out, a human blink lasts about 250 milliseconds.

To put this speed in perspective, Figure four demonstrates another common use case of real-time data processing: a financial transaction with a payment card. In the time it takes to blink an eye, to keep business and customer satisfaction high, an entire process quickly occurs:

• The initial request is received and processed.
• Fraud detection ensures the legitimacy of the transaction,
• And the appropriate response is received back at the request source.

Figure Four: Real-Time Data Processing at the Speed of a Blink of an Eye

Likewise, in less than half a blink, a well-designed electrical protection and smart grid monitoring system can detect a fault and trigger the appropriate response. This instant fault detection is possible because the protective relays and circuit breaker devices work on the principle of fault isolation. When something is “off” in a node, like the current, voltage, or frequency registers, dangerously high or low, the node disconnects and is sacrificed to preserve the integrity of the entire system and to prevent the fault from cascading through adjacent nodes.

However, the Supervisory Control and Data Acquisition (SCADA) systems that govern generators and some other electrical equipment have polling cycles of 1–2 seconds, which makes them comparatively incredibly slow to detect and register a fault. Additionally, once a fault is registered, delays continue to add up. A trained operator must be alerted, assess the situation, and issue a response. From the moment the fault occurs to the point where a command is executed by the controller, this round-trip process can take over 5 seconds, which is far too slow to prevent cascading blackouts.

And the threat to energy grids is constant, with a myriad of factors that can trigger a fault. From the mundane, like a software glitch, to the catastrophic, like a tree falling on top of a transmission line, or the truly bizarre, like a bullet striking a power line. What is common to all faults is that they must be detected, understood, and mitigated very quickly to prevent further damage.

Figure Five: Constant Threat to Energy Grids = Fault Detection and Predictive Maintenance Must Be Constant

Streaming Fundamentals for Smart Grid Monitoring and Data-Driven Fault Handling

Real-time grid monitoring hinges on streaming (processing data as it arrives) versus batch (processing large, scheduled chunks). Streaming pipelines target sub-100 millisecond latencies, while batch jobs typically operate on seconds or minutes of delay (though they can run much longer).

With Ververica’s Unified Streaming Data Platform (powered by the VERA engine, and built by the original creators of Apache Flink®), you can build a responsive, real-time streaming architecture that supports modern smart grid monitoring and prevents grid instability, ingesting Phasor Measurement Units (PMUs) as a foundational data source.

Phasor Measurement Units (commonly referred to as PMUs) are specialized devices that provide high-precision, time-synchronized measurements of electrical waves on a power grid. Unlike traditional measurements that sample steady-state values every second or more, PMUs capture the instantaneous magnitude and phase angle of voltage and current waveforms at rates of 30–60 samples per second. This granularity and synchronicity make them indispensable for real-time monitoring, control, and protection of modern grids. They are orders of magnitude faster than SCADA systems, which typically poll every 1–2 seconds.

Acting as the foundational data source in a real-time streaming architecture, PMUs offer:

• High-resolution time series data: PMU phasors can flow into Kafka topics at sub-100 ms intervals.
• Event timestamps: Ververica’s Unified Streaming Data Platform aligns events by their timestamps, which ensures out-of-order packets still fit correctly into temporal analyses.
• Data richness: The amount and resolution of magnitude and phase data provided by PMUs can be very useful to detect voltage sags, angle jumps, or frequency dips in under 50 ms, triggering alerts or automated isolation commands before cascading failures can propagate.

By providing high-fidelity, time-synchronized measurements, PMUs are the linchpin for any system aiming to detect, alert, and handle grid faults in real time, and protect against rapid cascades like those resulting in the April 28, 2025, Iberian Peninsula blackout. Ververica’s stateful architecture retains context (e.g., last N snapshots), and its exactly-once guarantee ensures consistency even under failures, making it well-suited for sub-50 ms decision loops at grid scale.

Fault Detection in Milliseconds

To catch grid anomalies in under 50 ms, and prevent cascading faults and blackouts, streaming pipelines must ingest, correlate, and act on phasor data faster than a human blink. A performant and reliable streaming data pipeline that achieves this might look like the following:

• High-Velocity Ingestion
• PMUs → Kafka via Kafka Connect: sub-millisecond writes into partitioned topics keyed by substation.
• Exactly-once semantics: Kafka’s acknowledgments paired with Ververica’s checkpoint guarantee no data loss or duplication, even under flash crowds (e.g., during a widespread fault).
• Pattern Detection with Complex Event Processing (CEP)
• Define temporal patterns (e.g., voltage sag + angle jump) with CEP rules.
• Windowing & Event-Time Semantics: Watermarks align out-of-order PMU streams so that late data can still trigger alarms within a bounded lateness (e.g., 20 ms).
• Adaptive ML Models
• Online anomaly detectors (e.g., streaming K-means) can run alongside CEP, spotting novel fault signatures that rules might miss.
• Model updates in-flight: We can use Ververica’s dynamic CEP to define new patterns and update thresholds and rulesets on the fly with no downtime required. For example, immediate adaptation to daily forecasted production and consumption, seasonal variations, or renewable intermittency.

Instant Fault Detection and Automated Handling

Detecting a fault is only half the battle, as automated systems must then isolate the affected nodes, reconfigure the network, and restore power in a tightly controlled sequence to prevent a small disturbance from cascading into a regional blackout like the one that affected Spain and Portugal.

In an environment like the one we have described in this blog, where incident detection and alerts can happen in milliseconds by detecting fault patterns in the PMU readings, the process for fault handling could resemble the following:

1. Fault isolation

   When a high-severity fault is detected, the first automated response is to isolate the affected segment by tripping the nearest circuit breaker. Modern substations use the IEC 61850 GOOSE (Generic Object Oriented Substation Events) protocol for peer-to-peer protection messaging, and relays publish a “trip” command over Ethernet so that subscribing breakers act within a few milliseconds. Because GOOSE messages are multicast and pre-configured in the substation network, there’s no need for centralized polling, and as a result, latencies as low as 5-10 ms are routinely achieved (source).

2. Grid topology reconfiguration

   After isolation, the next step involves reconfiguring the topology of the grid to route power away from the affected node to maintain service continuity. Typically, an Advanced Distribution Management System (ADMS) holds the real-time network topology and load forecasts. Upon receiving a breaker-open event, the ADMS computes alternate feeder paths within 100–200 ms and pushes new switch-closing commands back through the same control network. This achieves the goal of maintaining power flow from producers to consumers while the faulted node remains de-energized, minimizing possible damages to equipment and disruptions to grid users.

3. Black start (if necessary)

   If the outage is severe enough to trigger a blackout, a planned and coordinated black-start sequence must be carried out when grid integrity has been verified. This involves remotely activating specific power plants that are capable of starting with no external power, and providing the power needed for larger grid-forming units to start up themselves. Automated restoration plans, stored in the Energy Management System (EMS), list these black-start units (including hydro plants, diesel gensets, or battery banks). These units are critical to the disaster recovery capacity of a power system and constitute the last line of defense. The EMS issues start commands to black-start sources in a pre-defined order, and as each generator comes online, its voltage and frequency are stabilized to match the grid design conditions. Progressively, new transmission lines are energized, paving the way for more generation units and consumers to come back online.

Building a Power Grid Monitoring Architecture

Figure Six: Building a Power Grid Monitoring Architecture and Proposed Real-Time Data Processing Architecture with Ververica

Summary

As the Iberian Peninsula Blackout of April 28, 2025, demonstrates, split-second faults can cascade across thousands of kilometers of grid if they aren’t caught and contained in real time. Power grids with high penetration of renewables are more unpredictable and require faster actions and responses than legacy systems are designed to handle. For that reason, adopting streaming-first architectures and a wide range of actions to stabilize and secure the grid in real time is a critical step to guaranteeing energy supply in decarbonizing energy systems.

In this blog, we propose a solution that joins high-fidelity PMU telemetry with Ververica’s Unified Streaming Data Platform, giving grid operators the millisecond-level visibility and control they need to detect anomalies, isolate faults, and restore services automatically before a local disturbance becomes a regional crisis. Whether you’re piloting edge-deployed Flink clusters in remote substations or fine-tuning CEP patterns and online ML models for adaptive protection, the path to blackout-proof power systems runs through real-time data pipelines.

More Resources

• Learn more about Complex Event Processing
• Explore more real-time use cases
• Learn more about the Iberian Peninsula Blackout of 2025, including lessons learned and early analysis.
• Ready to build a resilient energy grid? Contact Ververica.

About The Author

Jaime López is a Marketing leader with over a decade of success transforming Marketing at top B2B and SaaS companies, like the industry giant Wärtsilä and the unicorn scaleup Aiven. Jaime holds an M.Sc. (Tech) in Energy Engineering from the Technical University of Madrid and a MicroMasters in Statistics and Data Science from the Massachusetts Institute of Technology.

During his career, he has worked on multiple projects that contributed to the design, expansion, and hardening of electrical grids around the world, from policy design to commercial implementation of generation solutions, with a particular emphasis on the integration of growing shares of renewable energy. Being part of projects in Myanmar, the UAE, Saudi Arabia, Spain, California, El Salvador, Texas, the southwestern USA, and Mexico has afforded him a unique opportunity to understand in depth the challenges that stem from decarbonizing large energy systems.