How to Forward CloudWatch Logs with Edge Delta’s Lightweight Lambda Extension

Step 1: Create an Edge Delta Cloud Fleet

An Edge Delta Cloud Fleet is a fully cloud-hosted Edge Delta Fleet that requires no user maintenance or infrastructure configuration. It exposes HTTP/HTTPS/gRPC endpoints, which you can use to forward logs to it. To create a Cloud Fleet:

1. Log into Edge Delta
2. Click Pipelines, followed by New Fleet.
3. Select Cloud Fleet and then click Continue.
4. Give your fleet a name, and then click Deploy Cloud Fleet.

Once the Fleet is created, you can click View Dashboard and see its HTTPS endpoint, which you’ll need to share with your Lambda Extension. 

Step 2: Configure Your Pipeline

Your new Edge Delta Pipeline is listening for logs on HTTPS (which is how we’re going to be transporting the Lambda logs) and gRPC, and is currently configured to route them directly into Edge Delta’s backend. But before you begin ingesting logs, you should first configure your pipeline to structure and enrich them. Edge Delta’s AWS Lambda Pack automatically unrolls and optimizes Lambda log data, and it can be easily added to any running pipeline by following these steps:

1. Log into the Edge Delta Console.
2. Click Pipelines, then click into Knowledge, and select the Packs tab.
3. Locate the AWS Lambda Pack and click Add to pipeline.
4. Choose the newly created pipeline and click Save Changes.
5. Delete the link from http_input to ed_archive and connect http_input to the AWS Lambda Pack.
6. Route the AWS Lambda Pack output to ed_archive.

Step 3: Create a Test AWS Lambda

Now that your pipeline is configured, follow these steps to create a simple test Lambda function that will generate logs:

1. Log into the AWS Console and navigate to the AWS Lambda functions view.
2. Click Create function, select Author from scratch, and populate Function name.
3. Select the Python 3.xx Runtime and the x86_64 Architecture. 
4. Click Create function.

For this example, we’ll run a simple Python script. To add it, select the Code tab and paste the example code below:

import time
import string
import random
import json
def lambda_handler(event, context):
   letters = string.ascii_letters
   x = "".join(random.sample(letters,10))
   print ('lambda-logs-'+x)
   n = 10
   for i in range(0, n):
       ticks = str(time.time_ns())
       print ('lambda-logs-'+x+'-'+ticks)
   return {
       'statusCode': 200,
       'body': json.dumps('lambda-logs-'+x+'-'+ticks+'-response')
   }

Note: this is a simple test Lambda function used for demonstration purposes. Any AWS resource that writes to CloudWatch can be used to demonstrate and validate the functionality. 

Step 4: Add the Edge Delta Lambda Extension

Once the test function is created, you’ll need to grab the Edge Delta Lambda Extension from the AWS serverless applications list. Once the extension is added to an AWS account, it can be reused in any Lambda function within it. To add the Edge Delta Lambda Extension:

1. Log into the AWS Console and navigate to the AWS Lambda Applications page. 
2. Click Applications, then click Create application, and choose Serverless application.
3. Under Public applications enter Edge Delta, and click “EdgeDelta-Lambda-Extension-Layer-AMD64” (Update the name if needed under Application settings). 

4. Take note of the ARN under the Physical ID column, and navigate to the AWS Lambda Functions page. 
5. Click Functions, then select the example function.
6. Click Layers, then click Add a layer, and then click Specify an ARN.
7. Paste in ARN from the previous step, then click Verify. 
8. Finally, click Add to officially add the extension. 

Step 5: Add Environment Variables to Edge Delta Lambda Extension

To add the appropriate environment variables, follow these steps: 

1. Navigate back to the AWS Lambda Functions page and click Functions. 
2. Select the example function, click Configuration, and click Environment Variables.
3. Create the required variables — add a PUSHER_MODE and ED_ENDPOINT variable, and assign to them “http” and <Your Edge Delta Cloud Fleet HTTPS Endpoint> respectively. If you need to add optional variables, further information can be found in our docs. 

Step 6: Generate Logs

AWS CloudWatch Log Groups are created the first time logs are generated, but in order to generate logs, the function must execute. AWS Lambdas only run when triggered, but they do have a manual trigger option. To manually trigger the Lambda, navigate to the example function we created and click on the Test tab. Then, wait until you see the following message: “Executing function: succeeded.” 

Step 7: View Lambda Logs in Edge Delta

Once the test function is running and the Edge Delta Lambda Extension is deployed, logs should be flowing into Edge Delta’s backend. 

Once the Lambda logs are flowing, you can use Edge Delta’s Live Capture feature to visualize the data as it’s moving through the pipeline in real time:

You can confirm the logs are entering Edge Delta’s backend by clicking on the Logs tab on the left-hand side of the screen in your Edge Delta account. If you see logs, that means the setup process was successful:

Once there, you can filter through logs by attribute, or create custom dashboards and monitors to help visualize and maintain the health of your Lambda environment.

Get started with the Edge Delta Lambda Extension

Amazon CloudWatch is a straightforward solution for monitoring your Lambda functions, but logging to CloudWatch is very expensive — particularly at scale. Our lightweight Lambda Extension lets you save on downstream ingestion costs by sending Lambda logs into an Edge Delta Telemetry Pipeline and optimizing your data in-flight. From there, you can send your logs to more cost-effective monitoring destinations like Edge Delta. 

If you want to learn more about how our end-to-end Telemetry Pipelines can help you manage your CloudWatch costs, check out our free playground or book a demo with one of our experts.