How Fleets Work with Telemetry Pipelines in Edge Delta

Not all telemetry pipelines are created equal. Edge Delta’s Telemetry Pipelines preprocess data at the source, which is a step forward from the traditional, centralized models of pipeline architecture. By starting the data processing stage closer to your sources within your environment, transformation and routing tasks significantly reduce strain on your resources and accelerate analysis. In turn, you also are able to reduce costs, control data, and improve your mean time to resolution (MTTR). 

One of the key components that make Edge Delta’s Telemetry Pipelines possible are Fleets.  

Within Edge Delta, Fleets are a grouping of agents deployed in a computing infrastructure, such as a Kubernetes cluster or a production environment. And one of the best parts about Fleets is how they free you from the toil of having to manually configure and manage individual agents or machines.

In this post, we’ll walk through the elements that make up a Fleet, the two kinds of Fleets you’ll find in Edge Delta, and how they work to give you more flexibility and control over your telemetry data. 

Types of Fleets within Edge Delta

Inside the Edge Delta platform there are two different types of Fleets — Edge Fleets and Cloud Fleets — that logically correspond with the environments in which they’re hosted. In this section, we’ll take a look at each one. 

Edge Fleets

Edge Fleets are fleets deployed directly within your computing infrastructures/environments, such as Kubernetes, Linux, Windows, macOS, Amazon ECS, and Docker.

The deployment approach for Edge Fleets positions them close to your data sources, enabling real-time analysis and data optimization at the edge, before anything is routed to final destinations. 
‍
There are three main components of an Edge Fleet:

• Processing agent - Executes pipelines and data pre-processing — which includes extracting insights, generating alerts, creating summarized datasets, and performing other tasks.
  
  
• Compactor agent - Compresses and encodes data into efficient formats. This helps reduce the total size of the data through compression algorithms, maintaining throughput efficiency for pipelines. It’s worth noting that compactors also assist in minimizing network load and storage requirements, and in enhancing overall system scalability and responsiveness.
  
  
• Rollup agent - Collects metric data by refining the frequency and cardinality of data points, thereby minimizing storage requirements and quickening retrieval of data. Rollups enhance data analysis performance by providing targeted summaries that streamline reporting, condensing data into actionable insights while preserving key trends and patterns. 

Each Edge Fleet is created when setting up a pipeline — either through our Visual Pipelines interface or a YAML file — and assigned a unique Pipeline ID, which it uses to communicate with Edge Delta for dashboard updates and configuration retrievals.

‍

Cloud Fleets

‍

As the name suggests, Cloud Fleets are deployed within a cloud infrastructure entirely owned and operated by Edge Delta. Cloud Fleets may best apply to situations like:

• Data sources that involve serverless workloads like AWS Lambda functions or events from Amazon Kinesis.
• Lightweight edge environments that produce smaller amounts of data, such as connected Internet of Things (IoT) devices.
• A scenario where you simply don’t want to manage the resources for hosting an added workload. 
• When security or regulatory compliance restrictions prevent the deployment of an agent directly within your environment.

On a functional level, Cloud Fleets also allow for push or pull data input integrations. For example, a push model might involve a server or data source automatically sending you telemetry data. More specifically, the Cloud Fleet can be configured to expose an HTTPS endpoint which external services can push data to. This scenario is best for those situations when real-time status updates are needed based on constantly changing data.  

Alternatively, in the pull method, you can manually request data from a server by specifying an input within the pipeline configuration, which might make sense in those scenarios where you need more hands-on control over the frequency of retrieval or when updates are less frequent. 

In either case, when setting up a Cloud Fleet, you should choose a configuration that best aligns with the location of your data source, such as being in the same region or hosted by the same provider.

Deploying Edge and Cloud Fleets

If you’re used to manually configuring homegrown pipelines systems, Edge Delta saves you a lot of time and resources. 

Using the dashboard interface within Edge Delta, you can create, configure, and delete both Edge and Cloud Fleets with ease. Here’s a bit more on that, based on which type of fleet you’re working with. 

Deploying a New Edge Fleet

During the Edge Delta Fleet installation, there are two main pieces involved:

1. The Pipeline: The Edge Delta platform provides a default pipeline that can be used without additional configuration. An Edge Fleet, for example, configured with a default pipeline will begin sending environment data to Edge Delta, allowing you to manage and test the pipeline from directly within our interface. Any changes made to the pipeline within the platform — such as adding data sources and destinations — are automatically applied.
2. The Pipeline ID: Each Edge Fleet is linked to a specific Pipeline ID. This identifier is used during installation to specify the appropriate pipeline and authenticate the connection to your Edge Delta backend.

As you might expect, the fleet deployment process varies depending on the specific environment you’re working within, and it’s helpful to consult our documentation for specific instructions for a more in-depth breakdown on paths, configurations, and options. 

For instance, an Edge Fleet being deployed to a  Kubernetes environment can be installed using Helm. In this scenario, the processing agent is installed as a DaemonSet, deploying a pod to each node within the cluster. From there, it collects logs from each pod and node, as well as cluster-level metrics, analyzes them, and streams them to the configured destinations.

As a general example though, here is what the process within Edge Delta looks like for deploying an Edge Fleet in Docker:

1. Click Pipelines.
2. Click New Fleet.
3. Select Edge Fleet.
4. Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
5. Click Continue.
6. Select the appropriate template and click Continue.
7. Specify a name to identify the Fleet.
8. Click Generate Config.
9. Execute the installation commands, they include the unique ID for the Fleet.
10. Expand the namespaces and select the input sources you want to monitor.
11. Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
12. Click Continue.
13. Click View Dashboard.

With Edge Delta, multiple Fleets can all share the same pipeline. Check out our docs for full guidance. 

Deploying a Cloud Fleet:

Since Edge Delta hosts Cloud Fleets, this process can be more streamlined.

To get started, from within the Pipeline Dashboard:

1. Click New Fleet.
2. Select Cloud Fleet
3. Optionally, expand Advanced fleet configuration and choose a pipeline configuration to duplicate for the cloud fleet. If you don’t select one, a default configuration will be used.
4. Click Continue.
5. Specify a name to identify the Fleet.
6. Optionally, expand Advanced Settings and select Compute Units based on your estimated traffic volume. This is the maximum bandwidth our agent can handle before notifying you about an error. The number of compute units used per hour counts towards your plan usage.
7. Optionally, specify an agent version. The interface lists the current stable version (the latest version number) and most recent candidate version (containing rc). Choose the current stable version. If this configuration doesn’t work you can contact Edge Delta support to experiment with the candidate.
8. Optionally, for Cloud Fleets, select protocols for endpoints on the Cloud Fleet that your sources can push data to. The default is an HTTPS endpoint.
9. Click Deploy Cloud Fleet.

‍

Inside Edge Delta, within the Pipelines section, you’ll see the fleets you have deployed displayed underneath the Optimization Summary, and you can filter by Edge or Cloud Fleet. 

Managing Fleets

As mentioned earlier, within the Edge Delta platform, Visual Pipelines is a tool you can use to view and modify your Pipeline configurations. Note that the Visual Pipelines tool requires Fleets with an agent version of 0.1.71 or higher. As is the case, some configuration parameters are advanced or in beta, and require them to be configured in YAML using a text editor rather than Visual Pipelines. See here for more information.

To view or make changes to a pipeline with Visual Pipelines while in Edge Delta:

• Click Pipelines.
• Select the Fleet with the Pipeline you want to view.
• Click View/Edit Pipeline.

Afterwards, the configuration settings will be automatically applied within a few minutes.

Note, too, that Edge Delta’s global agent is one of the most performant on the planet, and there are a number of optional settings you can use to ensure it’s delivering for your organization’s specific needs.

Dive Deeper into Edge Delta Fleets

Ready to test out Edge Fleets and Cloud Fleets? Visit our pipeline sandbox to try them out for free.

‍