.avif)
Introduction
Telemetry Pipelines are modern data management tools which facilitate the collection, processing, and routing of data from multiple sources to several different downstream destinations. When properly functioning, they provide teams with greater control over their telemetry data and can significantly reduce both observability and security costs.
Naturally, telemetry pipelines grow in size and complexity as organizations scale. Therefore, teams must follow a structured pipeline construction approach to navigate the interconnected web of nodes and data flows properly, to maximize pipeline efficiency and ultimately enhance data insights.
Pipeline packs are a powerful tool which help teams manage this complexity. In this blog post, we’ll define what pipeline packs are, and demonstrate how they are useful in the pipeline management process. Additionally, we’ll cover a short example on how to deploy a pack specifically to an Edge Delta Telemetry Pipeline.
What are Packs, and why are they useful?
At the most fundamental level, packs are a special type of node that leverage a combination of two or more processor nodes to perform some high-level function. They serve as an abstraction on top of specific nodes; in other words, users working with packs aren’t forced to worry about how they work, and can instead focus on using them to build pipelines with the knowledge that they function as intended.
Informally, packs can be thought of as “mini-pipelines” that route data through multiple different individual nodes to achieve a specific goal, and can easily slot into your main pipeline wherever necessary.
So, what value do packs add?
Packs are an incredibly useful tool during the pipeline building process, as they greatly simplify the pipeline’s overall structure. By aggregating multiple individual processor nodes into a single pack, pipelines become far less visually cluttered, and also much more logically organized. Instead of users puzzling over what certain collections of nodes are doing to their telemetry, they can instead throw them into a pack and add a descriptive label to understand exactly what their collective function is.
Real world example
Let’s explore the following pack example in the context of Edge Delta’s visual pipeline builder, and demonstrate how easily it can be integrated into your telemetry pipelines.
One of the most popular telemetry processing use-cases is JSON parsing. Oftentimes, logs store information in their body field as a JSON string. However, in order to perform analysis on these values, the string needs to be converted into a properly formatted JSON object.
With a few processor nodes, you can easily parse a JSON string, convert it into a JSON object, and store that object in an entirely new field within the log (refer to our docs for a deeper dive https://docs.edgedelta.com/parse-json-node/).
Edge Delta has its very own pack for this case, called the JSON pack. Here’s what it looks like:
Here’s what the JSON pack might look like in a telemetry pipeline:
By using this pack, we provide a clear and concise view of what that particular segment of the pipeline does. It abstracts away the individual processor nodes (“json-route”, “parse-json”, “extract-message”, and “parse-attributes”) and clearly demonstrates their collective functionality. It also greatly reduces visual clutter, as users aren’t required to see the pack’s internals and instead only see the single node which represents it.
Get started
Visit our pipeline sandbox to get hands-on with our pre-built Edge Delta Packs, add them to your Telemetry Pipelines, and much more!
Want to make packs yourself? Start a free trial to create and publish your own unique packs. And be sure to keep an eye on packs list to see our growing collection of pre-built packs, curated for the most popular telemetry use cases.
