Skip to content

Ververica Cloud, a fully-managed cloud service for stream processing!

Learn more

How Kcell and GetInData built a stream analytics platform using Apache Flink® as a core component


by

Telecommunications companies, such as Kcell, hold vast amounts of data at their disposal. Whether arriving from billing, data usage, roaming, or location events, being able to access and react instantly to these streams of data is paramount for modern enterprises.GetInData worked with Kcell on developing an innovative real-time platform for complex event processing that enables the telecommunications provider to build different applications on top of it.

Example use cases of this new real-time platform:

Balance top-upBalance top up caseUser experience is one of the most important service elements of Kcell’s offering. Imagine a situation where a subscriber tops-up his or her balance, very often in a short period of time. This might be due to intensive internet consumption even if the user is not subscribed to any special tariff, as one example. The platform can recognize this type of subscriber behavior and send a text message with a tailored offer. With real-time analytics, Kcell can do this instantly, as the subscriber is still going through the balance top-up process that leads to higher chances for successful message response and increased subscriber satisfaction.

Fraud detection while roaming

Fraud case in roamingDetecting fraudulent actions and responding to them in real time can prevent significant losses for a telecommunications service provider and provide a better user experience. Let’s imagine a situation where one of Kcell’s subscribers registers to a roaming service while his or her balance is zero, a scenario that is impossible in standard use cases. Using a real-time streaming platform allows Kcell to act on those scenarios by, for example, instantly sending a notification to the anti-fraud unit for follow up.

 

The solution

GetInData and Kcell worked collaboratively on deploying a new solution using complex event processing with Apache Flink as its core event processing engine. Apache Flink was the best fit due to the following features of the framework:

Other key elements of the solution included Apache Kafka® as the central event hub that all data comes through and Apache Nifi® that acts as both the data consumer and the data production engine, allowing for a flexible adjustment of incoming and outgoing data pipelines through a simple GUI. Additional components of the solution include monitoring and logging for performance measurement of the stream analytics platform and debugging of any potential issues. Nifi solution diagram

Solution implementation and technical requirements

Before starting the deployment, extensive planning and prioritization took place to ensure the system would be successful and provide clear ROI to Kcell.

A few requirements that were taken into account in the approach were the following:

  • Large-scale

Kcell has more than 10 million subscribers and an average of 160,000 events per second (300,000 events per second at peak times). This leads to the development of a platform that can handle 23TB of data per month with this number set to increase as more and more applications and use cases come in the future.

  • Reliable and 24/7 stream processing

Due to the nature of the planned use cases and because the data events flow continuously through the streaming platform, it is crucial that the architecture is both performant and fault-tolerant. This means a framework that is able to not only run 24/7 but can also restart and rewind the internal state to the same value after a potential failure so that all computations are 100% accurate. Another crucial element in the Kcell case was the variety of data the company holds.Ingesting data from different sources, and in different formats, such as billing, data usage, roaming or location events makes it necessary for the streaming platform to be reliable and flexible, allowing an appropriate response to different use cases and types of applications.

  • Accuracy, flexibility, and usability

Another factor taken into account is that the system should be working on event time. This is key for the planned applications so that all customer communications are timely and relevant. In addition, the system should be flexible, allowing for separate layers of incoming and outgoing data pipelines with elasticity in mind.

Finally, the system needs to be user-friendly and easy-to-configure for Kcell’s engineers that have various levels of programming proficiency.

  • Constant end-to-end testing and monitoring

A final but necessary step was to develop a framework that would allow for constant end-to-end testing to ensure the integrity of all components involved and that the business requirements are met.Solution impementationOn top of that, the team built a monitoring system including metrics and logging. Metrics are collected into InfluxDB, a dedicated database for time series data. Grafana is used as a metrics visualization and alerting tool. In a similar manner, logs are collected from all components using the widely known Elasticsearch + Logstash + Kibana stack for debugging purposes.

New call-to-action

New call-to-action

About GetInData

GetInData logo

Founded in 2014 by former Spotify data engineers, GetInData is a Warsaw-based company that helps other organizations implement their Big Data projects by providing outsourcing, consulting and training services. GetInData consists of a team of passionate veterans in Big Data and Fast Data Analytics. The company has a multi-year experience in implementing Big Data projects with various customers ranging from fast-growing startups to global corporations from the telecommunications, banking, retail, media, and healthcare sectors.

GetInData’s people strive for a continuous improvement and challenge of the status quo in data analytics technologies to help customers achieve true ROI from their data processing [1].

About Kcell

Kcell logo Kcell is part of the largest Scandinavian telecommunications holding - TeliaCompany (www.teliacomany.com). The company was founded in 1998 and is now a market leader in Kazakhstan, providing cellular services under the brand name of Kcell, which mainly focuses on the B2B segment and active, which focuses on the B2C segment. As of December 31st, 2016 Kcell’s subscriber base amounted to almost 10 million people[2].

This article was initially posted on GetInData’s blog. Special credits go to the project team at Kcell and GetInData for allowing us to use the story on the data Artisans blog and to Alexey Brodovshuk, Senior Software Engineer and Technical Lead at Kcell.

Sources:[1] http://getindata.com[2] http://investors.kcell.kz/en/article/show/189/false?navipageId=2102

Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.

Flink Forward 2024 conference only ticket

Topics:
Dawid Wysakowicz
Article by:

Dawid Wysakowicz

Comments

Our Latest Blogs

The Streamhouse Evolution featured image
by Giannis Polyzos 03 October 2024

The Streamhouse Evolution

A Glimpse Into the House of Streams In addition to being the heart of Ververica’s Streaming Data Platform, Apache Flink®️ has been the de facto gold standard for stream processing. Throughout the...
Read More
VERA Blog Series Part 3: Full Stream Ahead! featured image
by Karin Landers with Ben Gamble 13 August 2024

VERA Blog Series Part 3: Full Stream Ahead!

VERA: The Cloud Native Engine Revolutionizing Apache Flink® Blog Series Welcome to the final installment of the three-part blog series that introduces Ververica Runtime Assembly (VERA), the...
Read More
VERA Blog Series Part 2: Under the Hood: VERA's 3 Core Pillars featured image
by Karin Landers with Ben Gamble 08 August 2024

VERA Blog Series Part 2: Under the Hood: VERA's 3 Core Pillars

VERA: The Cloud Native Engine Revolutionizing Apache Flink® Blog Series Welcome to part two of our three-part blog and video series that introduces Ververica Runtime Assembly (VERA), the...
Read More