This publish was cowritten with Steven Aerts and Reza Radmehr from Airties.
Airties is a wi-fi networking firm that gives AI-driven options for enhancing residence connectivity. Based in 2004, Airties makes a speciality of growing software program and {hardware} for wi-fi residence networking, together with Wi-Fi mesh techniques, extenders, and routers. The flagship software program as a service (SaaS) product, Airties House, is an AI-driven platform designed to automate buyer expertise administration for residence connectivity, providing proactive buyer care, community optimization, and real-time insights. Through the use of AWS managed companies, Airties can give attention to their core mission: bettering residence Wi-Fi experiences by means of automated optimization and proactive problem decision. This contains minimizing community downtime, enabling sooner diagnostic capabilities for troubleshooting, and enhancing total Wi-Fi high quality. The answer has demonstrated vital impression in lowering each the frequency of assist desk calls and common name period, resulting in improved buyer satisfaction and decreased operational prices for Airties whereas delivering enhanced service high quality to their clients and the end-users.
In 2023, Airties initiated a strategic migration from Apache Kafka working on Amazon Elastic Compute Cloud (Amazon EC2) to Amazon Kinesis Knowledge Streams. Previous to this migration, Airties operated a number of fixed-size Kafka clusters, every deployed in a single Availability Zone to reduce cross-AZ site visitors prices. Though this structure served its goal, it required fixed monitoring and handbook scaling to deal with various information hundreds. The transition to Kinesis Knowledge Streams marked a big step of their cloud optimization journey, enabling true serverless operations with automated scaling capabilities. This migration resulted in substantial infrastructure value discount whereas bettering system reliability, eliminating the necessity for handbook cluster administration and capability planning.
This publish explores the methods the Airties group employed throughout this transformation, the challenges they overcame, and the way they achieved a extra environment friendly, scalable, and maintenance-free streaming infrastructure.
Kafka use circumstances for Airties workloads
Airties constantly ingests information from tens of thousands and thousands of entry factors (comparable to modems and routers) utilizing AWS IoT Core. Earlier than the transition, these messages have been queued and saved inside a number of siloed Kafka clusters, with every cluster deployed in a separate Availability Zone to reduce cross-AZ site visitors prices. This fragmented structure created a number of operational challenges. The segmented information storage required complicated extract, remodel, and cargo (ETL) processes to consolidate data throughout clusters, rising the time to derive significant insights. The information collected serves a number of crucial functions—from real-time monitoring and reactive troubleshooting to predictive upkeep and historic evaluation. Nevertheless, the siloed nature of the information storage made it significantly difficult to carry out cross-cluster analytics and delayed the power to establish network-wide patterns and tendencies.
The information processing structure at Airties served two distinct use circumstances. The primary was a conventional streaming sample with a batch reader processing information in bulk for analytical functions. The second use case used Kafka as a queryable information retailer—a sample that, although unconventional, has turn out to be more and more frequent in large-scale information architectures.
For this second use case, Airties wanted to offer rapid entry to historic gadget information when troubleshooting buyer points or analyzing particular community occasions. This was applied by sustaining a mapping of information factors to their Kafka offsets in a database. When buyer assist or analytics groups wanted to retrieve particular historic information, they might shortly find and fetch the precise data from high-retention Kafka matters utilizing these saved offsets. This method eradicated the necessity for a separate database system whereas sustaining quick entry to historic information.
To deal with the huge scale of operations, this resolution was horizontally scaled throughout dozens of Kafka clusters, with every cluster liable for managing roughly 25 TB of data.
The next diagram illustrates the earlier Kafka-based structure.
Challenges with the Kafka-based structure
At Airties, managing and scaling Kafka clusters has introduced a number of challenges, hindering the group from specializing in delivering enterprise worth successfully:
- Operational overhead: Sustaining and monitoring Kafka clusters requires vital handbook effort and operational overhead at Airties. Duties comparable to managing cluster upgrades, dealing with {hardware} failures and rotation, and conducting load testing continually demand engineering consideration. These operational duties take away from the group’s means to focus on core enterprise capabilities and value-adding actions throughout the firm.
- Scaling complexities : The method of scaling Kafka clusters entails a number of handbook steps that create operational burden for the cloud group. These embrace configuring new brokers, rebalancing partitions throughout nodes, and offering correct information distribution—all whereas sustaining system stability. As information quantity and throughput necessities fluctuate, scaling usually entails including or eradicating complete Kafka clusters, which is a fancy and time-consuming course of for the Airties group.
- Proper-sizing cluster capability: The static nature of Kafka clusters created a “one-size-fits-none” state of affairs for Airties. For giant-scale deployments with excessive information volumes and throughput necessities, including new clusters required vital handbook work, together with capability planning, dealer configuration, and partition rebalancing, making it inefficient for dealing with dynamic scaling wants. Conversely, for smaller deployments, the usual cluster measurement was outsized, resulting in useful resource waste and pointless prices.
How the brand new structure addresses these challenges
The Airties group wanted to discover a scalable, high-performance, and cost-effective resolution for real-time information processing that might permit seamless scaling with rising information volumes. Knowledge sturdiness was a crucial requirement, as a result of dropping gadget telemetry information would create everlasting gaps in buyer analytics and historic troubleshooting capabilities. Though momentary delays in information entry could possibly be tolerated, the lack of any gadget information level was unacceptable for sustaining service high quality and buyer assist effectiveness.
To deal with these challenges, Airties applied two totally different approaches for various eventualities.
The first use case was real-time information streaming with Kinesis Knowledge Streams. Airties changed Kafka with Kinesis Knowledge Streams to deal with the continual ingestion and processing of telemetry information from tens of thousands and thousands of endpoints. This shift supplied vital benefits:
- Auto-scaling capabilities : Kinesis Knowledge Streams will be scaled by means of easy API calls, assuaging the necessity for complicated configurations and handbook interventions.
- Stream isolation : Every stream operates independently, that means scaling operations on one stream don’t have any impression on others. This alleviated the dangers related to cluster-wide modifications of their earlier Kafka setup.
- Dynamic shard administration : Not like Kafka, the place altering the variety of partitions requires creating a brand new matter, Kinesis Knowledge Streams permits including or eradicating shards dynamically with out dropping message ordering inside a partition.
- Utility Auto Scaling: Airties applied AWS Utility Auto Scaling with Kinesis Knowledge Streams, permitting the system to routinely regulate the variety of shards primarily based on precise utilization patterns and throughput necessities.
These options empowered Airties to effectively handle assets, optimizing prices in periods of decrease exercise whereas seamlessly scaling as much as deal with peak hundreds.
For offering on-demand entry to historic gadget information, Airties applied a decoupled structure that separates streaming, storage, and information entry considerations. This method changed the earlier resolution the place historic information was saved immediately in Kafka matters. The brand new structure consists of a number of key elements working collectively:
- Knowledge assortment and processing : The structure begins with a shopper utility that processes information from Kinesis Knowledge Streams. This utility implements analyzing the information, as making it obtainable for detailed historic evaluation. The results of the information evaluation is written to Amazon Knowledge Firehose, which buffers the information, writing it often to Amazon Easy Storage Service (Amazon S3), the place it could later be picked up by Amazon EMR. This path is optimized for environment friendly storage and bulk studying from Amazon S3 by Amazon EMR. For uncooked information storage, a number of uncooked information samples are batched collectively in bulk recordsdata, that are saved in a separate Amazon S3 path. This path is optimized for storage effectivity and fetching uncooked information utilizing Amazon S3 vary queries.
- Indexing and metadata administration: To allow quick information retrieval, the structure implements a classy indexing system. For every file within the uploaded bulk recordsdata, two essential items of knowledge are recorded in an Amazon DynamoDB desk: the Amazon S3 location (bucket and key) the place the majority file was written, and the sequence variety of the corresponding information file within the Kinesis Knowledge Streams queue. This indexing technique offers low-latency entry to particular information factors, environment friendly querying capabilities for each real-time and historic information, automated scaling to deal with rising information volumes, and excessive availability for metadata lookups.
- Advert-hoc information retrieval: When particular historic information must be accessed, the system follows an environment friendly retrieval course of. First, the applying queries the DynamoDB desk utilizing the related identifiers. The question returns the precise Amazon S3 location and offset the place the required information is saved. The appliance then fetches the particular information immediately from Amazon S3 utilizing vary queries. This method permits fast entry to historic information factors, minimal information switch prices by retrieving solely wanted data, environment friendly troubleshooting and evaluation workflows, and decreased latency for buyer assist operations.
This decoupled structure makes use of the strengths of every AWS service: Amazon Kinesis Knowledge Streams offers scalable and dependable real-time information streaming, whereas Amazon S3 delivers sturdy and cost-effective object storage for uncooked information, and Amazon DynamoDB permits quick and versatile storage of metadata and indexing. By separating streaming from storage and using every service for its particular strengths, Airties created a more cost effective and scalable resolution for ad-hoc information entry wants, aligning every element with its optimum AWS service. The brand new structure not solely improved information entry efficiency but additionally considerably decreased operational complexity. As a substitute of managing Kafka matters for historic information storage, Airties now advantages from totally managed AWS companies that routinely deal with scaling, sturdiness, and availability. This method has confirmed significantly beneficial for buyer assist eventualities, the place fast entry to historic gadget information is essential for resolving points effectively.
Resolution overview
Airties’s new structure entails a number of crucial elements, together with environment friendly information ingestion, indexing with AWS Lambda capabilities, optimized information aggregation and processing, and complete monitoring and administration practices utilizing Amazon CloudWatch. The next diagram illustrates this structure.
The brand new structure consists of the next key phases:
- Knowledge assortment and storage: The information journey begins with Kinesis Knowledge Streams, which ingests real-time information from thousands and thousands of entry factors. This streaming information is then processed by a shopper utility that batches the information into bulk recordsdata (also called briefcase recordsdata) for environment friendly storage in Amazon S3. This method of streaming, batching, after which storing minimizes write operations and reduces total prices, whereas offering information sturdiness by means of built-in replication in Amazon S3. When the information is in Amazon S3, it’s available for each rapid processing and long-term evaluation. The processing pipeline continues with aggregators that learn information from Amazon S3, course of it, and retailer aggregated outcomes again in Amazon S3. By integrating AWS Glue for ETL operations and Amazon Athena for SQL-based querying, Airties can course of giant volumes of information effectively and generate insights shortly and cost-effectively.
- Knowledge aggregation and bulk file creation: The aggregators play a vital function within the preliminary information processing. They combination the incoming information primarily based on predefined standards and create bulk recordsdata. This aggregation course of reduces the amount of information that must be processed in subsequent steps, optimizing the general information processing workflow. The aggregators then write these bulk recordsdata on to Amazon S3.
- Indexing: Upon profitable add of a bulk file to Amazon S3 by the aggregators, the aggregator will write an index entry for the majority file an Amazon DynamoDB desk. This indexing mechanism permits for environment friendly retrieval of information primarily based on gadget IDs and timestamps, facilitating fast entry to related information utilizing S3 vary queries on the majority recordsdata.
- Additional processing and evaluation: The majority recordsdata saved in Amazon S3 at the moment are in a format optimized for querying and evaluation. These recordsdata will be additional processed utilizing AWS Glue and analyzed utilizing Athena, permitting for complicated queries and in-depth information exploration with out the necessity for extra information transformation steps.
- Monitoring and administration: To keep up the reliability and efficiency of the Kafka-less structure, complete monitoring and administration practices have been applied. CloudWatch offers real-time monitoring of system efficiency and useful resource utilization, permitting for proactive administration of potential points. Moreover, automated alerts and notifications ensure anomalies are promptly addressed.
Outcomes and advantages
The transition to this new structure yielded vital advantages for Airties:
- Scalability and efficiency: The brand new structure empowers Airties to scale seamlessly with rising information volumes. The power to independently scale reader and author operations has decreased efficiency impacts throughout high-demand durations. This can be a vital enchancment over the earlier Kafka-based system, the place scaling usually required complicated reconfigurations and will have an effect on the whole cluster. With Kinesis Knowledge Streams, Airties can now deal with peak hundreds effortlessly whereas optimizing useful resource utilization throughout quieter durations.
- Reliability and fault tolerance: Through the use of AWS managed companies, Airties has considerably decreased system latency and improved total uptime. The automated information replication and restoration processes of Kinesis Knowledge Streams present enhanced information sturdiness, a crucial requirement for Airties’s operations. The improved excessive availability signifies that Airties can now supply extra dependable companies to their clients, minimizing disruptions and enhancing the general high quality of their residence connectivity options.
- Operational effectivity: The brand new structure has dramatically decreased the necessity for handbook intervention in capability administration. This shift has freed up beneficial engineering assets, permitting the group to give attention to delivering enterprise worth moderately than managing infrastructure. The simplified operational mannequin has elevated the group’s productiveness, empowering them to innovate sooner and reply extra shortly to buyer wants. The discount in operational overhead has additionally led to sooner deployment cycles and extra frequent function releases, enhancing Airties’s competitiveness available in the market.
- Environmental impression and sustainability: The transition to a serverless structure demonstrated vital environmental advantages, attaining a exceptional 40% discount in vitality consumption. This substantial lower in vitality utilization was achieved by eliminating the necessity for continually working EC2 situations and utilizing extra environment friendly, managed AWS companies. This enchancment in vitality effectivity aligns with Airties’s dedication to environmental sustainability and establishes them as an environmentally accountable chief within the tech business.
- Value optimization: The monetary advantages of transitioning to a Kafka-less structure are clearly demonstrated by means of complete AWS Value Explorer information. As proven within the following diagram, the whole value breakdown throughout all related companies from January to July contains EC2 situations, DynamoDB, different Amazon EC2 prices, Kinesis Knowledge Streams, Amazon S3, and Amazon Knowledge Firehose. Essentially the most notable change was a 33% discount in whole month-to-month infrastructure prices (in comparison with January baseline), primarily achieved by means of vital lower in Amazon EC2 associated prices because the migration progressed, elimination of devoted Kafka infrastructure, and environment friendly use of the AWS pay-as-you-go mannequin. Though new prices have been launched for managed companies (DynamoDB, Kinesis Knowledge Streams, Amazon Knowledge Firehose, Amazon S3), the general month-to-month AWS prices maintained a transparent downward pattern. With these value financial savings, Airties can supply extra aggressive pricing to their clients. The diagram beneath exhibits month-to-month value breakdown through the transition.
Conclusion
The transition to this new structure with Kinesis Knowledge Streams has marked a big milestone in Airties’s journey in the direction of operational excellence and sustainability. These initiatives haven’t solely enhanced system efficiency and scalability, however have additionally resulted in substantial value financial savings (33%) and vitality effectivity (40%). Through the use of superior applied sciences and modern options on AWS, the Airties group continues to set the benchmark for environment friendly, dependable, and sustainable operations, whereas paving the way in which for a sustainable future. In an effort to discover how one can modernize your streaming structure with AWS, see the Kinesis Knowledge Streams documentation and watch this re:invent session on serverless information streaming with Kinesis Knowledge Streams and AWS Lambda.
Concerning the Authors
Steven Aerts is a principal software program engineer at Airties, the place his group is liable for ingesting, processing, and analyzing the information of tens of thousands and thousands of houses to enhance their Wi-Fi expertise. He was a speaker at conferences like Devoxx and AWS Summit Dubai, and is an open supply contributor.
Reza Radmehr is a Sr. Chief of Cloud Infrastructure and Operations at Airties, the place he leads AWS infrastructure design, DevOps and SRE automation, and FinOps practices. He focuses on constructing scalable, cost-efficient, and dependable techniques, driving operational excellence by means of sensible, data-driven cloud methods. He’s enthusiastic about mixing monetary perception with technical innovation to enhance efficiency and effectivity at scale.
Ramazan Ginkaya is a Sr. Technical Account Supervisor at AWS with over 17 years of expertise in IT, telecommunications, and cloud computing. He’s a passionate problem-solver, offering technical steering to AWS clients to assist them obtain operational excellence and maximize the worth of cloud computing.
