.avif)
Amazon CloudWatch is a monitoring and observability service from AWS that provides insights into the performance of your AWS resources and applications. It allows you to collect, track, and analyze metrics, logs, and events from a wide range of AWS services, custom applications, and on-premises systems.
CloudWatch integrates closely with many AWS services, which helps ensure visibility into the performance and security of applications and infrastructure. However, while useful, its log analysis capabilities can become impractical at scale. Many AWS tools ship all generated log data to CloudWatch by default, and a few tools are even required to do so. For large organizations, this can mean sending data into CloudWatch at the terabyte or even petabyte per day scale, which makes storing logs in the medium-to-long term extremely expensive.
What is Edge Delta’s CloudWatch Forwarder?
Instead of keeping everything in CloudWatch long-term and racking up a large bill, teams might consider shortening CloudWatch log retention where possible and forward logs to other downstream destinations for more effective and cost-efficient retention and analysis.
Edge Delta is one such downstream destination, and our new CloudWatch Forwarder allows you to do exactly that. It is an easy to install Lambda application currently published on the AWS Serverless Applications Repository, and it enables you to send your CloudWatch logs from any log group into an Edge Delta Telemetry Pipeline. From there, you can route the logs downstream into analysis platforms like Datadog and Splunk, cost-effective archival destinations like S3, or even Edge Delta’s observability platform for AI-powered intelligent log analysis.
Our forwarder is architected to handle data at the PB per day scale, and provides all the functionality needed to pull logs from the desired log groups — as well as pulling all associated tags — for easier downstream correlation and analysis.
In this post, we’ll walk through an example to demonstrate a configuration of Edge Delta’s CloudWatch Forwarder, and show how it enables you to cut costs by implementing a data tiering strategy.
Steps to Enable Edge Delta’s CloudWatch Forwarder
Create Cloud Fleet
An Edge Delta Cloud Fleet is a completely cloud-hosted Edge Delta Fleet that requires no maintenance or infrastructure to host. It provides all the same functionality as an Edge Fleet, with a slight difference in available inputs. Specifically, Cloud Fleets support data ingestion solely through HTTP/HTTPS/GRPC inputs.
To create a Cloud Fleet, you must:
- Navigate to https://app.edgedelta.com
- Log in using the appropriate credentials
- Click Pipelines
- Click New Fleet
- Select Cloud Fleet
- Click Continue
- Specify a name to identify the Fleet
- Click Deploy Cloud Fleet
After the Fleet is created:
- Click View Dashboard
- Take note of the HTTPS Endpoint
Configure Pipeline
The Edge Delta Telemetry Pipeline will send logs into the Edge Delta Log Destination, but they will not be unrolled nor extracted. For this pipeline in particular, we’ve added our CloudWatch Pack, which is a pre-built set of processing steps which extracts and normalizes CloudWatch logs into individual logs. To add the Pack:
- Log into the Edge Delta Console
- Click Pipelines
- Click Knowledge
- Click Packs
- Find AWS CloudWatch Pack
- Click Add to Pipeline
- Choose the appropriate pipeline
- Click Save Changes
- Delete the link from http_input to ed_archive
- Connect http_input to AWS CloudWatch Pack
- Connect AWS CloudWatch Pack to ed_archive
Deploy Edge Delta CloudWatch Forwarder
The application is readily available in the AWS Lambda Application store. To get started:
- Open the AWS Serverless Application Repository and click Available Applications
- Select Show apps that create custom IAM roles or resource policies
- Search for EdgeDelta
- Select EdgeDelta-Forwarder (either ARM64 or AMD64)
- Confirm the function template such as the application name
- Enter the HTTPS endpoint for the Cloud Fleet that you copied earlier
- Click I acknowledge that this app creates custom IAM roles and resource policies
- Click Deploy
Create Test AWS Lambda
This is a simple test Lambda application to demonstrate the functionality of the solution. Any AWS resource that writes to CloudWatch can be used to demonstrate and validate the functionality.
To create a test Lambda:
- Log into the AWS console
- Navigate to the AWS Lambda functions console
- Click Create function
- Select Author from scratch
- Populate Function name
- Select Runtime Python 3.xx
- Select Architecture x86_64
- Click Create function
Next, update the code:
- Select Code tab
- Replace with the example code below
Example Log Generator
Here’s an example Python script, which generates synthetic CloudWatch logs.
import time
import string
import random
import json
def lambda_handler(event, context):
letters = string.ascii_letters
x = "".join(random.sample(letters,10))
print ('cloudwatch-logs-'+x)
n = 10
for i in range(0, n):
ticks = str(time.time_ns())
print ('cloudwatch-logs-'+x+'-'+ticks)
return {
'statusCode': 200,
'body': json.dumps('cloudwatch-logs-'+x+'-'+ticks+'-response')
}
Verify logging is enabled:
- Select the Configuration tab
- Select the Monitoring and operations tools option
- Ensure Logs and metrics is Enabled
Generate Logs
AWS CloudWatch log groups are created the first time logs are generated. To generate logs, the AWS Lambda must be run. AWS Lambdas only run when triggered, but they do have a manual trigger option. To trigger the Lambda:
- Navigate to the Example Log Generator Lambda
- Click the Test tab
- Executing function: succeeded
Subscribe to Logs
Once the Lambda has run at least one time, the log group will be available. The CloudWatch Forwarder sends the logs based on a subscription to the log group. To subscribe to the log group:
- Navigate to the Edge Delta Forwarder Lambda function
- From the Function overview, Click Add trigger
- Select the source as CloudWatch Logs
- Select Example Log Generator Lambda (/aws/lambda/cloudwatch-logs)
- Enter a filter name (ie cloudwatch-logs-filter)
- Click Add
View Logs
The pipeline will then send logs to the Edge Delta Log Destination. This will allow viewing and searching of the logs. To search through the logs:
- Log into the Edge Delta Console
- Click Logs
- Select the appropriate Fleet, which will match the name of the Cloud Fleet
And just like that, your CloudWatch logs are now flowing into Edge Delta!
Once the data is moving through Edge Delta’s pipeline, you can use our Visual Pipeline Builder interface to process and send the incoming data to whichever destination you choose. For instance, we can extract patterns out of the CloudWatch data to leverage Edge Delta’s Anomaly Detection capabilities, and route a full copy of your data into S3 for compliance:
Wrap Up
AWS’s offerings are effective solutions for handling the wide variety of technological requirements and issues that your organization faces.
However, their log data flow infrastructure is suboptimal, as their in-house analysis tool, AWS CloudWatch, is expensive for storing TBs and potentially PBs of data. Using the Edge Delta CloudWatch Forwarder allows you to take back control of your telemetry data and route it out of CloudWatch to any destination you choose. Not only can you send data to more cost-optimized locations to reduce your observability costs, you can also route it to platforms like Edge Delta to perform data normalization and standardization, which greatly improves insight quality and strengthens your monitoring practices.
Want to learn more? Book a demo to see how Edge Delta’s CloudWatch Forwarder can help you reduce your bill and strengthen your observability efforts, or hop into our playground to try out our next-generation Telemetry Pipelines yourself.
