How to Cut CloudTrail Costs with Edge Delta

AWS offers more than 200 cloud-based services to help you solve complex problems related to storage, analytics, networking, security, and more. However, as you increasingly rely on these services for system reliability and innovation at scale, your AWS infrastructures grow more complex, which can quickly lead to diminished visibility and reduced efficiency. 

AWS CloudTrail provides detailed AWS account visibility through its operational and risk auditing, compliance, and governance capabilities. CloudTrail tracks every action taken by each user, role, and AWS service within your environment, from any location, to help ensure all components within your AWS infrastructure remain secure and healthy. For example, CloudTrail offers detailed information on:

• S3 bucket creation and deletion history
• Read and write history for S3 objects
• Network activity events, such as actions made using VPC endpoints 
• Insights events, such as suspicious API call and error rates

Analyzing this data allows you to identify and troubleshoot errors as they occur, which strengthens your overall security posture and keeps your infrastructure running smoothly.

The Challenge: Compromising on Quality Analysis vs. Cost with CloudTrail, CloudTrail Lake, and S3 

While it offers significant benefits, CloudTrail is incredibly expensive, particularly at scale. More specifically, costs skyrocket when CloudTrail data is sent into the expensive CloudTrail Lake platform for analysis, as opposed to being shipped into cost-efficient object storage. Although teams can save money by shipping CloudTrail data directly into S3, by doing so they lose the ability to query and manipulate their logs for much needed monitoring and analysis. This leaves teams in a difficult position with CloudTrail, as they must either compromise on functionality or on cost. 

The Solution: Using S3 for collection, Edge Delta for Analysis

Edge Delta’s next-generation Telemetry Pipelines are a cost-effective pipeline solution that  provides end-to-end control over telemetry data at scale. 

With Edge Delta, you can ingest data from a variety of sources — including S3 — process it, and route it to any downstream destination. Instead of shipping everything into CloudTrail Lake, you can centralize and consolidate your logs into an S3 bucket to be collected by an Edge Delta pipeline, then optimized, and finally routed to any backend (including Edge Delta’s, which delivers far lower ingestion and retention costs compared to legacy platforms) for monitoring and analysis. 

Configuration Steps

Let's walk through how to configure an S3 events trail to ship your CloudTrail logs into S3 and add a Simple Queue Service (SQS) to automatically notify your pipeline when a new file is ready to be ingested. 

Create AWS Access Key

The first thing we need to do to configure the data flows is to create an AWS Access Key. Here’s how to create one:

• Navigate to AWS Security Credentials 
• Click Create access key
• Securely save both the Access key and Secret access key

Create S3 Bucket

Next, we need to create an S3 bucket to send our CloudTrail logs into:

• Navigate to the AWS Bucket Management page
• Click Create Bucket 
• Select Bucket Type: General purpose
• Enter Bucket Name: example-bucketa
• Select ACLs disabled
• Select Block all public access
• Bucket Versioning: Disable
• Encryption type: Server-side encryption with Amazon S3 managed keys (SSE-S3)
• Bucket Key: Disable
• Click: Create Bucket

Create SQS

We now need to configure a new SQS standard queue to automatically notify your Edge Delta pipeline that a new batch of data is ready to be ingested:

• Navigate to AWS SQS Management
• Click Create Queue
• Select Type: Standard
• Enter Name: example-sqs 
• Server-side encryption: Disabled
• Encryption key type (if you want Server-side encryption): Amazon SQS key (SSE-SQS)
• Under Access policy
• Select Advanced
• Update policy replacing region, account id, bucket name, and sqs name

• Disable Redrive allow policy
• Disable Dead-letter queue

Add SQS to S3

Once both SQS and S3 are configured, we need to connect them:

• Navigate to AWS Bucket Management
• Select the newly bucket created
• Click Properties
• Under Event notifications
• Click: Create event notification
• Give the event a name, like: example-notification
• Check the box: All object create events
• Scroll down to the bottom, and select SQS Queue as the Destination
• Select Choose from your SQS Queues and pick example-sqs

Create CloudTrail

Our last step in the AWS environment is to create a new CloudTrail, and sync it up with our S3 bucket:

• Navigate to AWS CloudTrail Management
• Click Create a trail
• Enter Trail name: sqs
• Storage location: Use existing S3 bucket
• Trail log bucket name: example-bucketa
• Log file SSE-KMS encryption: Unchecked/Disabled
• Click: Next
• Event type
  
  • Check Management events
  • Check Data events
• API activity
  
  • Check Read
  • Check Write
• Data events
  
  • Resource type
    
    • S3
    • Log all events
• Click Next
• Click Create trail

Create Edge Delta Cloud Fleet

Once our AWS infrastructure is properly configured, we now need to get our Edge Delta Telemetry Pipeline up and running. Since our data source is located within AWS, we’re going to create a cloud fleet:

• Navigate to the Edge Delta Pipelines Dashboard 
• If you already have an agent fleet deployed, Click New Fleet
• Select Cloud Fleet
• Click Continue
• Enter a fleet name, for example: sqs
• Click Deploy cloud fleet
• Once the heartbeat is detected, click View Dashboard

Update Pipeline

Once the cloud fleet is created, we need to configure our sources, processing steps, and destinations, including our AWS CloudTrail Pipeline Pack:

• Navigate to the Edge Delta Pipelines Dashboard 
• Click Knowledge
• Click Packs
• Select the AWS CloudTrail Pack
• Click Add to pipeline
• Select sqs
• Click Add and then Save Changes
• Toggle Edit Mode on
• Click Add Source
• Click Pull
• Click S3 Source
• Populate form with the appropriate:
  
  • SQS URL
  • AWS Region
  • AWS Access Key ID
  • AWS Secret Key
  • Compression (gzip)
• Click Save Changes
• Connect S3 Source to AWS CloudTrail Pack
• Connect each AWS CloudTrail Pack output to the Edge Delta Destination
• Click Review Changes
• Click Save Changes

Inspect Logs

Once everything is configured, you’ll start to see logs flowing into Edge Delta:

• Navigate to Edge Delta Console
• Click Logs
• Select any log
• Inspect log

We now have logs populating the Edge Delta backend! You can easily search through and inspect individual logs with ease, and with a few clicks you can take advantage of our pattern analysis and anomaly detection capabilities.

Conclusion

AWS CloudTrail is a valuable tool for collecting key information about actions taken within your AWS environment. However, with that value also comes steep costs. 

Instead of relying on the expensive CloudTrail Lake platform for log analysis, teams can consolidate their logs into S3 and then ship them into Edge Delta’s Telemetry Pipelines for full control over processing and routing at lower costs. Once data is in the pipeline, it can be routed to any destination, including Edge Delta, where it can be used to create metrics, patterns, dashboards, monitors, and much more. 

Curious how it all works? Try our pipelines for free at play.edgedelta.com, or book a demo with one of our experts to learn how our pipelines can help reduce your CloudTrail bill.