Can a pipeline handle petabytes of data?
Each platform is tested under identical conditions across four pipeline types (pass-through, filter, mask, and lookup) using synthetic nginx-style logs, on a single EC2 instance with 80, 100, and 120 worker load tiers. The OpenTelemetry Collector runs three types only: its contrib distribution ships no CSV lookup processor, so lookup is reported as N/A. Tested versions: Edge Delta v2.20.0-rc.5, Cribl 4.18.2, OpenTelemetry Collector 0.155.0, Fluentd 1.16.11.
Pass-through throughput
| Vendor | Pass-Through | Filter | Mask | Lookup |
|---|---|---|---|---|
| Edge Delta | 71,464 | 57,510 | 50,196 | 56,482 |
| OpenTelemetry Collector | 17,726 | 13,921 | 6,502 | N/A |
| Fluentd | 17,085 | 14,902 | 4,316 | 7,244 |
| Cribl | 1,990 | 2,105 | 1,928 | 2,061 |
The OpenTelemetry Collector has no lookup case (no shipped CSV lookup processor), shown as N/A.
Resource efficiency
| Vendor | Avg CPU | Avg peak memory | Logs/sec per CPU% |
|---|---|---|---|
| Edge Delta | 417.3% | 273 MB | 141.2 |
| OpenTelemetry Collector | 351.4% | 296 MB | 36.2 |
| Fluentd | 102% | 206.2 MB | 106.7 |
| Cribl | 102.8% | 674.1 MB | 19.7 |
How the benchmark runs
Every vendor runs the same four pipeline types under identical load, so the only variable is the agent. Throughput, CPU, and memory are sampled per run and the best worker tier is selected per vendor.
- →Single c8i.2xlarge EC2 instance (Ubuntu 24.04), one vendor at a time.
- →Load tiers of 80, 100, and 120 concurrent workers, 60 seconds per tier.
- →Synthetic nginx-style logs delivered over HTTP.
- →All vendors reached 0 errors and 0 backpressure across every scenario.
| Pipeline type | What it does |
|---|---|
| Pass-Through | Minimal processing. Baseline throughput. |
| Filter | Exclude events where the color attribute equals Green. |
| Mask | PII masking of IP, email, and credit-card fields via regex. |
| Lookup | CSV lookup to enrich events, for example ip to region. |
Key findings
- →Edge Delta dominates throughput across all scenarios, processing 58,913 logs/sec on average, 4.6x more than the next-best vendor.
- →Edge Delta leads in efficiency at 141.2 logs/sec per CPU%, extracting the most throughput from each unit of compute.
- →Cribl is throughput-limited at roughly 2,000 logs/sec across all scenarios while consuming the most memory at 674 MB.
- →The OpenTelemetry Collector degrades 63% under regex masking, dropping from 17,726 to 6,502 logs/sec while consuming 547% CPU.
Put a real AI Teammate on call
Edge Delta's AI Teammates triage, investigate, and find root cause in your stack.