Syslog is the de facto protocol for transmitting infrastructure-level log data to centralized log management platforms. It enables teams to collect and aggregate logs from routers, firewalls, servers, operating systems, and other infrastructure components, which can then be used for incident detection and remediation workflows.
However, the Syslog protocol presents several challenges in complex environments. For instance, Syslog’s primary message format RFC5424 isn’t strictly defined, which leads to inconsistencies in structure and content across Syslog messages from different services. Additionally, traditional Syslog management solutions like syslog-ng struggle to collect and process data at modern scale.
With Edge Delta’s end-to-end Telemetry Pipelines, teams can collect, normalize, enrich, and route Syslog messages from any source to any destination at petabyte scale with ease. Our pre-built Syslog Pipeline Pack automates these processing steps in any pipeline, optimizing log data without manual overhead.
How Edge Delta’s Syslog Pack Works
Our Syslog Pack is a pre-configured collection of processing steps built to parse, structure, and enrich Syslog messages in the RFC5424 format (with the option to handle the legacy RFC3164 format, as well). It enables teams to optimize the volume and quality of Syslog messages sent into their downstream destinations of choice, helping reduce ingestion and retention costs without sacrificing insight quality.
Here’s what a sample Syslog message in the RFC5424 format looks like pre- and post-pack processing:
Before:
"<165>1 2003-10-11T22:14:15.003Z myhostname myapp 1234 ID47 - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry..."
After:
{
"_type": "log",
"timestamp": 1065910455003,
"body": "<165>1 2003-10-11T22:14:15.003Z myhostname myapp 1234 ID47 - [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] An application event log entry...",
"resource": {
"ed.conf.id": "46aa874e-b2b8-46ff-b998-42af434d1b78",
"ed.org.id": "db3e3bbc-7c92-49c8-b17b-0d918a7da1e7",
"ed.source.name": "CloudTrail Logs",
"ed.source.type": "http_input",
"ed.tag": "prod-pipeline-cloud",
"host.ip": "10.51.174.43",
"host.name": "default-deployment-79cdc799d6-jvz89",
"http.method": "POST",
"http.route": "/",
"http.scheme": "http",
"server.port": 80,
"service.name": "http-80",
"src_type": "http"
},
"attributes": {
"appname": "myapp",
"ed.env.id": "ce8f420e-4889-4bd6-91ee-6d2377c0eda5",
"host": "myhostname",
"index": "index-k",
"msgid": "ID47",
"pri": null,
"procid": null,
"source": "linux",
"sourcetype": "linux-syslog",
"timestamp": "2003-10-11T22:14:15.003Z",
"version": null
}
}
As you can see, the pack converts a simple plaintext string into a structured log item, which includes rich metadata that’s critical for later monitoring and analysis. Let’s walk through the pack’s processing steps to cover how it accomplishes this transformation:
Format Standardization
The Syslog Pack begins processing Syslog messages with Mask nodes, which:
- Convert all tab character representations “#011” into actual spaces
- Condense multiple contiguous whitespace characters into single spaces
- Remove all leading white spaces
Though small, each step plays an important role in ensuring the log messages are uniformly aligned before structuring and enrichment takes place.
- name: 'Replace #011 with space'
type: mask
pattern: '#011'
mask: ' '
- name: Replace multiple whitespace with space
type: mask
pattern: \s\s+
mask: ' '
- name: Remove leading space
type: mask
pattern: ^\s+
mask: ""
Log Parsing and Timestamp Normalization
By default, the standardized logs flow into the Parse RFC5424 Format Grok node, which parses the log messages and structures each component in the attributes portion of the log item in accordance with the RFC5424 format. Once structured, teams can run queries on the log attributes much faster.
- name: Parse RFC5424 format
type: grok
pattern: <%{POSINT:pri}>%{POSINT:version}%{SPACE}%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{SYSLOG5424PRINTASCII:appname}%{SPACE}%{SYSLOG5424PRINTASCII:procid}%{SPACE}%{SYSLOG5424PRINTASCII:msgid}%{SPACE}(?:-|(?<structuredData>(\[.*?[^\\]\])+))%{SPACE}%{GREEDYDATA:message}
Next, the pack uses an OTTL Transform node to upsert the ISO8601-formatted timestamp attribute into the log item’s top-level timestamp field. This ensures the log’s timestamp accurately reflects when the log was originally generated (as opposed to when it was ingested by Edge Delta), which is critical for accurately mapping system behavior during troubleshooting.
- name: Extract Timestamp ISO8601
type: ottl_transform
statements: set(timestamp, UnixMilli(Time(attributes["timestamp"], "%Y-%m-%dT%H:%M:%SZ")))
Alternatively, the pack can easily be modified to run parallel log parsing and timestamp normalization operations built specifically for RFC3164-formatted data.
- name: Parse RFC3164 and Linux format
type: grok
pattern: (<%{POSINT:pri}>)?%{SPACE}%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{SYSLOGHOST:host}%{SPACE}%{DATA:appname}(\[%{POSINT:procid}\])?:%{GREEDYDATA:message}
- name: Extract Timestamp Syslog
type: ottl_transform
statements: set(timestamp, UnixMilli(Time(attributes["timestamp"], "%b %d %H:%M:%S")))
Log Enrichment
The logs are then routed to a Lookup processor node, which initializes a lookup table used to perform host-based log enrichments. It first scans the log’s resources and attributes for information about the associated host, locates the corresponding key in the lookup table, and adds the appropriate index, source, and sourcetype values as attributes.
- name: Lookup by host
type: lookup
location_path: ed://syslog_lookup.csv
reload_period: 5m0s
match_mode: exact
regex_option: first
key_fields:
- event_field: item["attributes"]["host"]
lookup_field: host
out_fields:
- event_field: item["attributes"]["index"]
lookup_field: index
- event_field: item["attributes"]["sourcetype"]
lookup_field: sourcetype
- event_field: item["attributes"]["source"]
lookup_field: source
- name: Lookup by source host
type: lookup
location_path: ed://syslog_lookup.csv
reload_period: 5m0s
match_mode: exact
regex_option: first
key_fields:
- event_field: item["resource"]["host.ip"]
lookup_field: host
out_fields:
- event_field: item["attributes"]["source_host"]["source"]
lookup_field: source
- event_field: item["attributes"]["source_host"]["sourcetype"]
lookup_field: sourcetype
- event_field: item["attributes"]["source_host"]["index"]
lookup_field: index
- name: Fill missing metadata
type: ottl_transform
statements: |-
set(attributes["source"], attributes["source_host"]["source"]) where attributes["source"] == nil or attributes["source"] == ""
set(attributes["source"], resource["host.ip"]) where attributes["source"] == nil or attributes["source"] == ""
set(attributes["index"], attributes["source_host"]["index"]) where attributes["index"] == nil or attributes["index"] == ""
set(attributes["index"], "syslog") where attributes["index"] == nil or attributes["index"] == ""
set(attributes["sourcetype"], attributes["source_host"]["sourcetype"]) where attributes["sourcetype"] == nil or attributes["sourcetype"] == ""
set(attributes["sourcetype"], "syslog") where attributes["sourcetype"] == nil or attributes["sourcetype"] == ""
delete_key(attributes, "source_host")
Once processing is complete, the logs exit the pack through the Processed output path. From there, they can be processed further if necessary, filtered, and routed to any downstream destination.
See It in Action
To start using the Syslog Pack, you’ll first need to create a new pipeline and configure an input node (most likely a TCP listener) to receive Syslog messages. After the pipeline is configured, navigate to the Knowledge tab in the Pipelines menu and select Packs. Search for the Syslog Pack and hit Add Pack. This will add the pack to your local library, where you can easily add it into the pipeline.
To add the pack to an existing pipeline, navigate to your Pipelines dashboard, select the pipeline where you want to apply the Syslog Pack, and enter Edit Mode. From there, click Add Node, and search for the Syslog Pack.
Once added, you can configure the incoming Syslog messages to flow through the pack for processing, before being routed to downstream destinations of your choice. For example, you might want to ship the processed logs to your Splunk backend for further investigation, and send log patterns into Edge Delta for a real-time, cost-effective overview of your log data:
Getting Started With the Syslog Pack
The Syslog Pack is just one of our many pre-built packs that simplify data processing through automated parsing, optimization, and enrichment — and we’re adding more to our library every day.
Want to get your hands on our next-generation Telemetry Pipelines and play around with packs? Check out our free playground. You can also schedule a demo with one of our experts to learn more about how Edge Delta can help you strengthen analysis and cut costs.
