Building a Reliable Benchmarking System for Go Microservices

Code Implementation Considerations 

During implementation, we made several decisions to ensure our microbenchmarking framework performed as expected.

For instance, it was important to make sure each benchmark function only assessed elements of the hot path in order to isolate the code under test. Take the example benchmark code snippet below:

func BenchmarkProcessor(b *testing.B) {
p := NewProcessor()
testData := []byte("test data for processing")
for i := 0; i < b.N; i++ {
/* expensive operations */
tempFile, _ := ioutil.TempFile("", "benchmark")
tempFile.Write(testData)
tempFile.Close()
fileData, _ := os.ReadFile(tempFile.Name()) 

// b.ResetTimer()

/* actual function to benchmark */
p.Process(fileData)
}
}

The setup work done before the commented line of b.ResetTimer() is not part of the Process function, which is the method we’re actually trying to assess. By uncommenting the ResetTimer call, the Go benchmark can ignore the initial file read and write operations and focus solely on the p.Process(fileData) function’s performance. 

Additionally, it was important to leverage Go’s native iteration testing management for accuracy and efficiency. Take the example code snippet below. Go’s benchmarking will automatically continue to scale up b.N until a statistically significant result is reached. This means there’s no need for us to specify an upper limit ourselves (as is done with the “throughPutTest” variable in the incorrect approach):

// incorrect way to benchmark 
func BenchmarkProcessor(b *testing.B) {
p := NewProcessor()
for i := 0; i < throughPutTest; i++ {
p.Process(fileData)
}
}
// correct way to benchmark 
func BenchmarkProcessor(b *testing.B) {
p := NewProcessor()
for i := 0; i < b.N; i++ {
p.Process(fileData)
}
}

Tuning Go Flags for Optimal Benchmarking Performance

Go’s benchmarking framework offers powerful tools for controlling test execution, but default settings often prioritize speed over consistency. We carefully tuned our benchmark flags to maximize result reliability. 

Key Flags We Use

• -benchmem: Measures memory allocations alongside execution time, helping us detect memory usage regressions
• -count=10: Runs each benchmark 10 times to ensure statistical significance
• -run=^$: Skips regular tests during benchmark runs to reduce environmental noise
• GOMAXPROCS=10: Sets a consistent number of available processors regardless of host configuration

Additionally, we leveraged Go’s internal benchstat tool for statistical analysis of benchmark results, and implemented a custom comparison threshold of 25% or higher. When the threshold is breached, the PR immediately updates its status to “blocking in review”. 

The Importance of Agent Scenarios

While the above benchmarks help with assessing individual performance, many of our agent’s operations span many different components. By testing common customer scenarios and incorporating asynchronous or hard-to-isolate components — such as goroutines, channels, worker groups, and locks — into our benchmarks, we gain valuable insight into how each component will perform holistically once merged. 

Our Approach

After each PR is merged and individual component benchmarks are verified, a CircleCI workflow triggers several pre-built scenarios using an agent executable, periodically recording CPU, memory, and allocation measurements. This approach has helped us catch several issues that didn’t have massive repercussions in isolation, but in full execution caused a significant increase in garbage collection and memory allocation. 

Maintaining test scenarios is incredibly hard, as it requires a deep understanding of evolving customer usage patterns. Still, it’s a vital part of benchmarking. By tailoring our test cases to real-world scenarios, we can remain confident in our agent’s ability to perform in even the most limiting environments. 

Why We Chose Self-Hosted

While cloud-based CI solutions offer convenience, they introduce several challenges for performance benchmarking:

1. Hardware variability — Cloud providers often use heterogeneous hardware based on cost and availability, meaning your workload might run on different CPU generations or configurations between runs.
2. Noisy neighbors — Shared infrastructure means other workloads can impact your performance measurements.
3. Resource constraints — Many cloud CI environments limit CPU and memory resources, making performance measurements less representative of production performance.

These are big problems that lead to inconsistent benchmark results between runs. The same code can produce significantly different performance metrics on different executions, making it impossible to reliably detect regressions.

We decided to self-host our CI runners to overcome these challenges. By implementing self-hosted CircleCI runners on dedicated hardware, we were able to take full control over system configurations and tuning parameters — isolating testing environments and establishing consistent CPU, memory, and storage configurations. With containers built specifically for testing and that leverage owned infrastructure as the underlying hardware, we were also able to control for thread counts, fully enabling consistent comparisons across Go benchstat runs.