Introducing Edge Delta Config Packs: Monitor AWS ALB with Minimal Configuration

Good news: Edge Delta has introduced Config Packs – a new feature that allows customers to quickly gain observability into new infrastructure and cloud services with minimal configuration and a high level of reusability. Before diving into how it works, let’s provide some background on the feature.

Config Packs allow customers to onboard popular infrastructure and cloud services in a matter of clicks via close-to-ready configurations. Once you indicate the service you would like to observe, Config Packs automatically populate the processors to monitor all KPIs typically associated with the given log format. Then, you’re off to the races analyzing a system or service.

Getting started with Config Packs

Customers that already use Edge Delta can leverage Config Packs in any deployed agents – it’s as simple as adding a single import statement to the desired agent configuration.

The first service we’re supporting with Config Packs is AWS Application Load Balancer (ALB). We’ll have many more services released in the coming weeks and months. In this post, I will demonstrate how this feature works with AWS ALB.

If you’re using AWS ALB, you likely want to monitor common KPIs such as status code distributions, response times, and top URIs returning 4xx and 5xx, among others. Now, by using the AWS ALB Config Pack, Edge Delta will automatically populate the underlying configuration to extract and monitor these core statistics as raw data flows through the pipeline, routing the lightweight insights to the streaming destination of your choice.

To establish observability, you will take the following steps.

1. Set up configurations within AWS for analyzing ALB access logs

You can stream AWS ALB access logs to an Amazon S3 bucket defined within AWS Load Balancing. By doing so, the access log files will be published in the defined S3 bucket every five minutes.

However, for this use case, we want to be notified of each new log file. To set up these notifications, we can use Amazon Simple Queue Service (SQS). This document explains how you can configure the Amazon S3 bucket to send notifications to the Amazon SQS queue.

2. Execute Config Pack workflow and import AWS ALB Config Pack

As I mentioned earlier, this new feature pre-packages all the processors, including clustering, for common observability use cases. Edge Delta provides these processors within an automatic workflow to onboard the data sources. You can simply turn it on and start observing all patterns in the access logs to easily understand trends.

As you can see below you are going to define only two fields: InputLabel and IntegrationName.

- InputLabel to indicate which input you are going to analyze.

- IntegrationName to denote where you want to stream the analysis outputs (metrics, alerts, edac, clusters, etc.).

version: v2
processors:
cluster:
name: clustering
...
regexes:
- name: "2xx"
...
- name: "4xx"
...
- name: "response_time"
...
top_ks:
- name: "uri_2xx"
...
workflows:
analytics_workflow:
input_labels:
- '{{Param "InputLabel"}}'
processors:
- 2xx
...
- uri_4xx
...
- clustering
- response_time
...
destinations:
- '{{Param "IntegrationName"}}'

Edge Delta will use an Amazon S3 via Amazon SQS input to read new access logs. To configure this input within your agent config please refer to our documentation.

From there, you are going to import the AWS ALB Config pack to start analyzing the loglines. To do so, you can take the following steps:

1. Vist app.edgedelta.com.
2. Create a new configuration file for agent deployment within “Agent Settings.”
3. Paste the YAML tag below into the configuration file. When you do so, make sure to update fields, such as access_key_id, access_secret, and endpoint with the appropriate inputs.

version: v2
agent_settings:
tag: aws-alb-test

imports:
- name: 'aws-lb'
pack_id: 'edgedelta/aws-lb:v0'
params:
InputLabel: 'aws-lb-access-log'
IntegrationName: 'sumo-us-2'
inputs:
s3_sqs:
- labels: 'aws-lb-access-log'
sqs_url: 'https://sqs.us-west-2.amazonaws.com/XXXXXXXX/SQS_NAME'
access_key_id: 'ACCESS_KEY_ID'
access_secret: 'ACCESS_SECRET_KEY'
region: 'us-west-2'
log_type: alb
outputs:
streams:
- name: 'sumo-us-2'
type: 'sumologic'
endpoint: 'SUMOLOGIC_ENDPOINT'
features: 'metric, edac, topk, cluster'

While you are adding the configpack into your agent config you can see the resolved version of the agent configuration by clicking the "Show Resolved Version" checkbox.

3. Start analyzing AWS ALB loglines at the source

After those simple steps, we’re already ready to see the observation results of the AWS ALB access logs. Let's go to the "Metrics" page under the "Observability" tab.

Here you can see the AWS ALB 2xx response code status stats:

You can also review the AWS ALB response times in milliseconds stats:

And of course, you can see the anomalies:

Edge Delta Config Packs will help you more easily onboard your services to Edge Delta. There will be a huge amount of Config Packs soon. In the meantime, we’d love to hear your feedback on our AWS ALB release.