.avif)
Amazon CloudWatch is a tool used for monitoring the health and performance of your AWS applications and resources, and it integrates closely with many AWS services by default. However, storing logs in CloudWatch for the medium-to-long term is extremely expensive, especially for large organizations generating data at the terabyte or petabyte per day scale.
To help solve this problem for our customers, we built the Edge Delta CloudWatch Forwarder, available as a Lambda application on the AWS Serverless Applications Repository. With our forwarder, you can ship CloudWatch logs directly into an Edge Delta Telemetry Pipeline, and then route them wherever you choose.
To further strengthen your downstream log analysis capabilities, we’ve released the Edge Delta CloudWatch Pipeline Pack, a specialized collection of pre-built processing steps designed specifically for normalizing and enriching CloudWatch logs. Our packs are built to easily slot into your Edge Delta Telemetry Pipelines — all you need to do is navigate to the packs library, add the CloudWatch pack to a pipeline, route into it the forwarded logs, and let it begin processing.
Edge Delta’s Telemetry Pipelines are an intelligent, end-to-end pipeline solution that enables full control and flexibility over all log, metric, trace, and event data at any scale, at far lower costs.
HED: How Does the CloudWatch Pack Work?
Our CloudWatch Pack consists of a few different processing steps, each of which play a vital role in enabling teams to aggregate, analyze, and correlate their CloudWatch log data within the observability platform of their choosing.
Here’s a quick breakdown of the pack’s internals:
Log Field Extraction and Optimization
The CloudWatch Pack begins by converting logs into a structured, more useable format by:
- Using a Parse JSON Node to parse the JSON attributes from the log item’s message body, and store them as individual fields in the
attributessection of the log item - Utilizing an Extract JSON Node to capture nested JSON attributes within the log item
- Parsing the identified nested JSON values with an additional Parse JSON Node to store the parsed attributes as individual fields in the
attributessection of the log item
This conversion to structured log data that retains only the most essential information greatly simplifies the log search and analysis process.
# first JSON Parse
- name: body_attributes
type: parse_json_attributes
process_field: item.body
# Extract nested JSON fields
- name: extract_logEvents
type: extract_json_field
field_path: logEvents.[*]
keep_log_if_failed: true
# second JSON Parse on nested JSON fields
- name: body_attributes
type: parse_json_attributes
process_field: item.body
Further Optimize Log Items
The pack concludes by leveraging a Log Transform node to delete the extraneous logEvents attributes in each log item, optimize data volume, and send only what’s necessary further downstream.
- name: delete_logEvents
type: log_transform
transformations:
- field_path: attributes.logEvents
operation: delete
Amazon CloudWatch Pack in Practice
To begin using the CloudWatch Pack, you first need to forward your CloudWatch logs into an Edge Delta Telemetry Pipeline (check out our blog post “Detect Anomalies and Cut Costs with Edge Delta’s CloudWatch Forwarder” for a detailed description on how to do so). Once the logs are in, you can further process and route them in a variety of ways. For example, you might want to send all logs and derived metrics and patterns into Edge Delta for lightning fast query times and anomaly detection.
Alternatively, you might want to keep metrics and patterns in Edge Delta, but instead feed all processed CloudWatch logs into Splunk, while sending a full copy of all raw data into S3 for compliance:
Getting Started
Ready to see Edge Delta’s CloudWatch Pack in action? Visit our pipeline sandbox to try it out for free. Already a customer? Check out our packs list and add the CloudWatch Pack to any running pipeline.
