Fama Achieves 100% Datadog Visibility While Automating Anomaly Detection

Founded in 2016, Fama is a leading background screening company. Its product helps organizations make better hiring decisions by scraping a candidate’s public social media presence and using machine learning to identify problematic behaviors, akin to a traditional background check. Fama supports organizations around the world, including a majority of the largest executive search firms.

The company is always looking for new, cutting-edge technologies to better support its customers. One of Fama’s core initiatives has been to streamline its infrastructure and operations by adopting a greenfield serverless architecture on AWS. As CTO, Brendten Eickstaedt puts it, “We’re a small team focused on building software, not maintaining systems or data centers. We want to leverage as much of the AWS expertise as possible by going serverless.” The company’s upcoming releases will run solely on this serverless architecture.

In a similar vein, Fama had recently modernized its observability stack, hoping to free its engineers from maintenance overhead by moving away from a self-hosted log management platform. In its place, the company adopted Datadog.

Datadog adoption leads to sticker shock

Prior to the move, Eickstaedt and his team worked with Datadog to build projections of their monthly costs. Almost immediately after deploying Datadog, it was clear these projections were off. Fama ended up spending between 2-3x each month as much as the team had forecasted.

There were two factors at the root of this issue. First, Fama is a growing company and doubled in screening volume from 2020 to 2021, by nature creating more logs. Second, and more problematically, was the size of their logs. Datadog charges customers based on how many log events they index. When Fama built their projections, most of their logs were represented by a single event. In practice, this wasn't the case. "Things that should have been one log event were actually three or four events. The cost just ballooned," Eickstaedt explains.

Initially, Fama worked with Datadog to strategize ways to lower costs. However, the approaches they came up with fell into one of two buckets:

• They didn't move the needle enough, meaning Fama would still be over budget.
• They attacked the problem by filtering out logs, but did so at such an extreme that they created significant blindspots.

Evaluating solutions to reduce index costs

It became clear that Fama needed another provider to help them reduce the volume of data indexed into Datadog. This provider would need to not only alleviate Datadog costs, but also facilitate the company's move to a modern serverless architecture. Eickstaedt started to fulfill this need by evaluating other centralized observability platforms. In doing so, he kept running into challenges.

From a cost perspective, many of the products he evaluated did not make sense. At one point, Eickstaedt thought a product with a compute-based pricing model might help him lower indexing costs. However, this product ended up inflating costs in other ways. "On the surface, it sounded great. But, when we did the cost analysis, those solutions ended up being just as expensive, if not more expensive, than Datadog," Eickstaedt explained.

Additionally, the team found that most solutions required Fama to either replace Datadog or adopt a platform that overlapped in functionality alongside it. Eickstaedt was unwilling to undo the months of work it took to get Datadog in place: "The prospect of ripping out Datadog and putting yet another logging and observability platform in its place was just not tenable for me."

Preventing Datadog overages with edge observability

That's when Fama turned to Edge Delta. Throughout its evaluation, it was clear that Edge Delta was unlike the other observability platforms Fama considered. In particular, Eickstaedt felt Edge Delta complemented and integrated with their existing Datadog implementation rather than overlap with it: "You can get data to Datadog the same way it's always been going, just in a more metered fashion." Edge Delta also provided out-of-the-box integrations to support Fama's serverless adoption, namely with Amazon CloudWatch and AWS Lambda.

Moreover, Edge Delta provided a solution to Fama's core cost challenge, without forcing them to sacrifice visibility through sampling or filtering logs. In fact, Edge Delta has helped Fama gain 100% visibility into their raw logs.

As soon as logs are generated from their AWS environments, Edge Delta analyzes the raw datasets and optimizes them into insights, statistics, and aggregates. It's these outputs that are streamed to Datadog instead of complete datasets. Meanwhile, all raw data is routed to low-cost storage and if an issue or anomaly occurs, Edge Delta automatically triggers a full-fidelity log capture in Datadog. "Edge Delta really gave us confidence that we could index less data in Datadog, but still have that full observability that we're looking for," remarks Eickstaedt.

Outside of the cost savings, Eickstaedt was impressed by Edge Delta's automated alerting and anomaly detection: "Almost immediately after turning it on, we started seeing anomalies." Before, the engineering team had to know what patterns to look for in order to detect and alert against them with Datadog. Now, that's all handled without any manual intervention, simply by ingesting the logs through Edge Delta before the outputs hit Datadog. "Edge Delta provided us a view into issues we didn't know were going on. It detected anomalies on its own without us having to specify what it should be looking for," says Eickstaedt.

Once Fama decided to move forward with Edge Delta, they received a white-glove onboarding and were up and running within days. "The Edge Delta support team was fantastic. The initial onboarding was super simple, and the continuing relationship has been open and responsive," says Eickstaedt.

The results and next steps for Fama

Since deploying Edge Delta, Fama has optimized the data being indexed into Datadog by over 80%. As a result, they've been able to cut their monthly bill by 80% – they are now well within their initial projections.

"We have a positive ROI on this – it's saving us real dollars right away," remarked Eickstaedt.

Having Edge Delta in place will be beneficial to Fama as it continues to grow. Today, Fama runs 10,000 screenings a month and is on pace to surpass 20,000 by the end of 2022. Further, the company is hoping to double the size of its engineering team. With this growth, log volumes will continue to scale. Edge Delta has given Eickstaedt and his team confidence they'll be able to stay ahead of this growth while maintaining complete visibility into their systems.