.avif)
Serverless architectures are rising again in popularity, with developers increasingly leveraging cloud-based serverless tooling that supports compute, database, workflow automation, and container orchestration functionalities to name a few. AWS’ event-driven Lambda service is one such example, enabling developers to create Lambda functions to run code without provisioning or managing any of the underlying infrastructure themselves.
The Challenges with Serverless: Lambda and CloudWatch
While simple to use, Lambda functions are essentially black boxes, as they handle all the underlying compute that enables the code to run. To help provide users with visibility, AWS Lambda automatically monitors Lambda functions and generates logs associated with each individual action taken. However, Lambda logs are sent to Amazon CloudWatch by default for analysis, which is an expensive tool — especially at enterprise scale.
With Edge Delta’s Lambda Extension, Edge Delta users can easily circumvent CloudWatch by shipping Lambda logs directly into an Edge Delta Telemetry Pipeline, from where they can be easily routed to any downstream destination of their choice.
Additionally, with Edge Delta’s Lambda Pipeline Pack, teams can leverage a pre-built collection of processing steps to intelligently optimize and enrich their Lambda logs in stream with ease, keeping insight quality high while significantly reducing log volume and ultimately downstream cost.
In this blog post, we’ll walk through how the Lambda Pack is architected, what benefits it provides to your Lambda logs, and how to add it into Edge Delta’s pipelines.
Edge Delta Lambda Pack — Internals, Benefits, and Integration
Before diving into the pack’s internals, let’s cover Edge Delta’s Lambda architecture at a high level:
Our Lambda Extension sits directly within your Lambda environment and forwards Lambda logs to an Edge Delta Cloud Fleet. From there, the logs are processed and forwarded to the destination(s) of your choice, including legacy observability platforms, archival storage, and Edge Delta’s Backend.
Our Lambda Pack slots seamlessly into your Edge Delta Pipelines, and automatically structures your log data to reduce log volume and enable powerful insights downstream. Here’s how it works:
Parse JSON Attributes
The pack begins by leveraging a Parse JSON Node to isolate the most important pieces of the JSON-formatted log data:
- name: body_attributes
type: parse_json_attributes
process_field: item.body
Structure JSON Fields
Once parsed, the logs are then structured via an Extract JSON Node in the {attributes} section of the log item:
- name: extract_message
type: extract_json_field
field_path: message
keep_log_if_failed: true
Once processing is complete, the incoming Lambda log has been optimized, structured, and prepared for further downstream processing and analysis. Here’s an example of how the Lambda pack enriches the log body of a Lambda log:
Log body at ingestion:
"body": "{\"cloud\":{\"resource_id\":\"arn:aws:lambda:us-east-2:00000000000:function:example-log\",\"account_id\":\"00000000000\",\"region\":\"us-east-2\"},\"faas\":{\"name\":\"example-log\",\"version\":\"$LATEST\",\"request_id\":\"e8ba78a6-1028-40ba-ad62-21a74249cd59\"},\"timestamp\":\"2024-01-00T00:00:00.000Z\",\"log_type\":\"function\",\"message\":\"example-log\"}"
Log body post-processing:
{
"cloud": {
"account_id": "00000000000",
"region": "us-east-2",
"resource_id": "arn:aws:lambda:us-east-2:00000000000:function:example-log"
},
"ed.env.id": "d5d3b7dc-1e86-4cf7-bbc1-b505bc307d97",
"faas": {
"name": "example-log",
"request_id": "2e3b45fa-6f43-4981-a472-68dfa3010301",
"version": "$LATEST"
},
"log_type": "function",
"message": "example-log",
"timestamp": "2024-01-01T00:00:00.000Z"
}
After processing, the fields within the log body are structured and stored as a functional JSON object, allowing users to capture log groups based on their associated structured fields for visualization and analysis.
Lambda Pack in Action
To add the Lambda Pack to a running Edge Delta pipeline, follow the steps outlined below:
- Navigate to the Knowledge Center and locate the Lambda Pack
- Select {Add Pack} to save it to your pack repository
- Navigate to your Edge Delta Telemetry Pipeline and enter {Edit Mode}
- Click {Add Processor} and select the Lambda Pack
Once your Lambda logs are flowing through the pack, you can run additional, in-pipeline analysis on them — including metrics conversion and pattern extraction — to further optimize your log data before sending it to an observability backend:
Getting Started
Think Edge Delta’s Lambda Pack can help augment your observability strategy? Book a demo with our experts to learn more. Want to get a feel for Edge Delta’s UI? Check out our free to use playground.
