Introducing the CloudTrail Pack for Edge Delta

Since CloudTrail is standard in every account within AWS — which has more than 1 million active users in 190 countries — it’s a widely-used and incredibly valuable service. 

That’s why we’re excited to announce our new CloudTrail pipeline pack — a collection of pre-built configurations and best practices specifically designed to help you get more from your data. 

Previously, if you were looking to add CloudTrail events to EdgeDelta, it would require some manual, fine-tuned customization. Now, Edge Delta’s CloudTrail pipeline pack takes care of all the heavy lifting. 

In this post, we’ll run through our pre-designed pipeline configuration for AWS CloudTrail logs, and demonstrate how they’ll help your team.

CloudTrail events + Edge Delta Telemetry Pipelines 

As a quick refresher, CloudTrail is a key logging tool for tracking and auditing your AWS environment. Any action taken by a user, role, or AWS service is recorded as an event by CloudTrail.

The type of comprehensive logging that CloudTrail provides delivers transparency for things like operational and risk auditing, governance, and compliance of AWS accounts. 

CloudTrail offers three ways to record events:

• Event history — Provides a viewable, searchable, downloadable, and consistent record of the past 90 days of management events within an AWS region.
  
  
• CloudTrail Lake — Offers a managed data lake for capturing, storing, accessing, and analyzing user and API activity on AWS for audit and security purposes.
  
  
• Trails — Captures a record of AWS activities, and allows you to deliver and store these events in an S3 bucket, or send them to CloudWatch Logs and EventBridge. You can also input these events into your security monitoring solutions, such as a SIEM.

If you’re unfamiliar with Edge Delta’s Telemetry Pipelines, they are the only pipeline solution on the market that can capture and pre-process your logs, metrics, traces, and events data at the edge, and allow you to then easily forward any of it to any downstream destination. The result is higher efficiency for your CloudTrail data, which can have a variety of benefits, from lower costs to faster incident remediation. 

Our pipeline pack ensures seamless ingestion and optimized processing of CloudTrail data. Once added to your pack library and set up as a pipeline, it will efficiently manage the ingestion process by connecting structured JSON log data from CloudTrail to your Edge Delta pipeline, providing insights into user activities and API calls across your AWS account. 

Plus, you’ll be able to monitor, detect anomalies, and generate metrics from CloudTrail logs, which will enable proactive security and operational insights.

CloudTrail pack capabilities

The CloudTrail pack consists of a series of nodes, built around areas like data protection, privacy, compliance, querying, and prioritization. Here’s a look at the capabilities of each node:

Mask sensitive data

Our CloudTrail pack contains three nodes for masking that assist with protecting privacy and preventing unauthorized access. With these nodes, you can:

• Redact sensitive information by capturing and masking all references to the accessKeyID field, which is a unique identifier used to make programmatic requests to AWS services, via a regular expression. By doing this, you ensure compliance with data protection regulations while preventing unauthorized access.
  
  
• Mask the assumed role ID — which is used in AWS IAM to temporarily assume roles and provide limited access to resources — to maintain privacy and security of role usage.
  
  
• Mask session tokens — part of temporary security credentials provided by AWS — by locating and replacing text that matches the regex pattern for sessionToken. This helps avoid unauthorized access and leakage of security credentials.
  

Parse data structure

These nodes will help you convert your data from one format to another and reduce the size of your data sets. With these nodes, you can:

• Unroll structured JSON data from the Records field in each CloudTrail log and create a new log for each record found within the Records array. Each new log features a Record field at the top level, which contains the data of the original Records entry.
  
• Access and query individual fields easier by parsing the JSON attributes from the specified field path in CloudTrail logs and converting them into separate attributes. Any node failures are routed to the Fails output path of the Compound Node.
  
• Extract the value of the eventName field from the Record and create a new field to retain this value. If extraction fails for any reason, the log is still kept. Any node failures are routed to the Fails output path of the Compound Node.

Route and aggregate data

This node will help separate the signals from the noise by letting you filter out non-critical events and focus more on significant actions by shipping CloudTrail logs based on specified conditions.

In this process, logs are evaluated using a regex pattern to determine if they match the eventName for specific AWS operations. If the condition is met, the log is then routed to the not_important_event path, and processing stops for that log. The not-important path sends logs to the get describe list output path of the Compound node. Logs that are not filtered out (i.e. important logs) are routed to the Processed output path.

What the CloudTrail pack looks like in practice

When the pack is connected, your CloudTrail data can be routed to any destination within Edge Delta or outside of it, depending on your needs. For instance, you can connect your CloudTrail pipeline to an Edge Delta Log to Pattern node to generate patterns, which will help you quickly identify issues, explore behavioral trends across your environment, and receive alerts with automated analysis for faster troubleshooting. Alternatively, you could connect your pipeline to a simple storage option like S3, and ship your CloudTrail data there for archival or tiering purposes. 

Additionally, the Edge Delta Log Output node can be used to capture your CloudTrail logs in the Edge Delta backend, and the Debug Output node can be leveraged to check the content of any logs that failed to be processed by the pack.

Get started with the CloudTrail pack now

• Visit our Docs page for the full rundown on installation and information 
• If you’re not yet a customer, give Edge Delta a try! Sign up for a free trial or hop into our interactive playground.

‍