Performance matters for operations teams. Getting an alert a few minutes late doesn’t just impact mean time to detect (MTTD). It can also translate to frustrated customers and lost revenue. That’s why modern DevOps teams have to be laser focused on detecting customer-impacting issues quickly.
Fulfilling this need is becoming increasingly difficult as modern applications create more and more data. If your analytics tools struggle to keep up, you’ll see slower response times.
At Edge Delta, we’re obsessed with DevOps performance. So, we performed a side-by-side comparison of the top log management collectors on the market. To do so, used the open-source Test Harness tool, which was developed by Vector (formerly Timber Technologies, since acquired by Datadog).
The results show that data throughput varies widely across the different agents, as well across different functions of the agents.
Testing Methodology
In this section, we’ll detail how we went about comparing the different log collection agents, starting with the testing elements.
Testing Elements
Subject: This node indicates a server that the agent is working on.
Producer: The producer is responsible for generating log messages for the subject to process.
Consumer: The consumer collects the messages from subject output messages.
Test Cases
TCP to Blackhole Case: In this case, the test flow starts with messages from producers and terminates with the agent when they are ingested to itself.
TCP to TCP Case: In this case, the agent ingests messages from TCP producers and pushes the processed messages to a TCP consumer.
TCP to HTTP Case: In this case, the agent ingests messages from TCP producers and pushes the processed messages to a HTTP consumer.
File to TCP Case: In this case, the agent ingests messages from a file and pushes the processed messages to a TCP consumer.
Regex Parsing Case: In this case, the agent ingests messages from TCP and applies an apache common regex processor then pushes the processed messages to a TCP consumer.
We built out substantial resources on Amazon EC2 to perform the tests, and used Terraform and Ansible to automate the test deployments. While the agent is running on the subject server the OS resource stats are being collected by dsta, and these results are used to calculate I/O Throughput values. You can find more detailed information on the Vector Test Harness tool page.
The Vector team already conducted and published benchmark tests for the performance of different tools, including:
Vector
Filebeat
FluentBit
FluentD
Logstash
Splunk UF
Splunk HF
We supplemented the results obtained by Vector and extended the results table by adding Edge Delta agent results.
Results
The outstanding Edge Delta results aren’t by accident. The Edge Delta team earned these results by focusing on architecture, robust code design, and rigorous performance testing. The Edge Delta agent is built on the GoLang development language. GoLang has a great impact on performance because it produces native code to execute. Furthermore, our team cares about implementing code with the minimum complexity and using cache facilities for the right cases. Simple, robust architectures always lead to better results. Additionally, our asynchronous implementation with pipes drastically improves performance.
Product and engineering teams can often focus on features and widgets rather than core architecture. The Edge Delta team has always focused on end-to-end performance and rigorous testing because we believe this has a real impact on our customers and their businesses.
