AWS Lambda Triggers: Kinesis vs SQS
Some considerations and findings
Overview
As Serverless Computing has become an increasingly popular cloud computing execution model, many software engineers have been exploring its potential and investigating the use cases it is best suited for. Here at Lifion, by ADP, we use AWS Lambda for our serverless computing needs and we have a few situations where we need our Lambda functions to consume data from various sources. We wanted to compare and evaluate possible options that AWS provides for this especially from a scaling and cost perspective, the results of which we’ve detailed and shared here. For our specific purposes we benchmarked and observed the different characteristics between running Lambda functions that consume data from AWS Kinesis Data Streams (KDS) versus AWS Simple Queue Service (SQS).
Why did we consider KDS and SQS?
One of the key differences between the two options we identified was that KDS allows multiple consumers to listen to and process the same data stream in parallel whereas SQS delivers a message to only one consumer but is guaranteed to do so at least once.
We recognized that AWS Kinesis and SQS are targeted at slightly different use cases, yet KDS has some characteristics we were interested in exploring. SQS provides message queuing and tends to be good for decoupling services especially when there is a lot of heavy-duty, batch-oriented processing required. In addition to applications for analytics use cases, KDS can also be used to process streams of data. Consequently, we were interested to compare and contrast each option for our use case.
Our Experiment
Why Did We Run The Benchmark?
Our microservices platform powers a broad and dynamic suite of Human Capital Management applications for our users. To achieve this, in our current architecture a web page is assembled dynamically from metadata before being presented to the client. We call this assembly process “compilation” because multiple different document objects are compiled by our proprietary algorithms. The compilation process is a computationally heavy operation, and to give end users the most responsive experience possible we take great efforts to minimize the potential latency when processing requests coming from them. In an effort to achieve this, we devised an approach which we refer to as “pre-compilation”, that eagerly compiles all of the possible document objects for a client ahead of time, allowing the web page to then be served with minimal backend processing time. Thus to evaluate the feasibility of the pre-compilation approach, we prototyped and benchmarked this process to learn if we could perform the eager compilation without significant changes to our existing architecture, to better understand the potential benefits, and to measure the expected reduction in rendering latency.
How Did We Benchmark?
For the benchmark, we created equivalent AWS Lambda functions with either KDS or SQS as a Lambda function trigger. When a message was delivered containing references to a document object to be compiled, the Lambda function would retrieve the corresponding documents from AWS Aurora (Postgres), feed the documents into the compiler, and then store the compiled result in AWS Elasticache. We measured the processing duration and resource usage from AWS CloudWatch metrics and logs.
Our Findings
KDS and SQS are designed for different use cases and the differences become more clear when it comes to using them with Lambda functions. In our benchmark setup there were some small differences between KDS and SQS such as maximum batch size per execution, but the biggest difference we observed was the concurrent execution limit configuration. Even though the Lambda function’s concurrent execution limit was configured as per AWS Lambda documentation, the function containers scaled differently based on the attached service. With SQS, the Lambda function scaled up dynamically while KDS allowed the function to scale up to the number of AWS Kinesis shards in the data stream.
Because of this difference we had to add more Kinesis shards to the data stream to get the benchmark to finish our workload in an acceptable duration, i.e. 5 minutes. Even though we were able to optimize the running time by updating the Lambda function with more memory and adding more shards to the stream, the configuration was not optimal for our use case due to the increase in cost.
On the other hand, it was easy to achieve the desired duration with AWS SQS because the Lambda function scaled up (and down) dynamically. Moreover, it was more cost-efficient than the EC2 instances we were using at the time, m4.large and r4.xlarge, with our benchmark assuming the workload would run 10 times per day. For our use case, AWS Lambda with AWS SQS best met our requirements.
Is SQS always better than KDS with AWS Lambda?
Absolutely not! The Lambda function performed better with SQS for our specific benchmark and because it was a better fit for our use case but your mileage may vary. If the benchmark had involved multiple different Lambda functions that need to process the same message at the same time, AWS KDS might have performed better since it allows multiple consumers to simultaneously consume from the same stream.
SQS allows a AWS Lambda function to scale dynamically based on the volume of data to be processed, but due to its ‘at least once’ message guarantees it may deliver the same message more than once and therefore might not be appropriate if this property is undesirable. In contrast, AWS KDS allows multiple AWS Lambda functions to react simultaneously to the same message however the number of Lambda function instances is limited to scale up only to the number of Kinesis shards specified for the stream.
Conclusion
The results from our benchmark highlight some of the different scaling and cost characteristics and considerations between AWS KDS and AWS SQS when processing streams of data using AWS Lambda. For our use case, using SQS with Lambda was more appropriate so that instances could scale up to the volume of data being processed, while AWS Lambda with KDS may be more appropriate for fanout and/or broadcast style use cases. In order to design the right serverless computing architecture, understanding the nuances of each possible solution and its intended use cases are key to making the right decision.
Credited Team Members
- Jude Murphy (Solution Architect) (https://www.linkedin.com/in/jude-murphy-98bba662/)
- Bruno Macedo Pasini (Lead Software Engineer)(https://www.linkedin.com/in/bmpasini)
- Ali Zein Yousuf (https://www.linkedin.com/in/azy5030)
- Jun Sung Park (https://www.linkedin.com/in/jun-sung-park-28194557)
- Alex Vũ Nguyễn (https://www.linkedin.com/in/vnguyenio)
