Seamless Observability on OpenShift

1. Custom Security Context and Pod Configuration

OpenShift enforces strict security policies through SCCs, which can prevent containers from running with elevated privileges. To resolve this, we:

Implemented a Custom SCC: We enabled a configuration flag in our Helm chart to create the necessary security context automatically. For example:

helm upgrade edgedelta edgedelta/edgedelta -i --version v1.29.0 \
  --set podSecurity.securityContextConstraints.create=true \
  -n edgedelta --create-namespace

Provided a Dedicated Values File: Our documentation includes an openshift-specific values YAML that sets the appropriate security parameters, reducing the need for manual tweaks. Here are the full details on the security context needs for each feature: https://github.com/edgedelta/charts/blob/master/charts/edgedelta/templates/security_context_constraint.yaml

{{- if .Values.podSecurity.securityContextConstraints.create }}
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: {{ include "edgedelta.fullname" . }}
  labels:
    {{- include "edgedelta.labels" . | nindent 4 }}
users:
- system:serviceaccount:{{ .Release.Namespace }}:{{ include "edgedelta.fullname" . }}
# Allow host ports when any host port being used
{{ $usePorts := gt (len .Values.ports) 0 }}
allowHostPorts: {{ or .Values.storePort .Values.profilerPort $usePorts }}
# Allow host PID when eBPF tracer is enabled
allowHostPID: {{ .Values.tracerProps.enabled }}
# Allow host PID when eBPF tracer is enabled
allowHostNetwork: {{ .Values.tracerProps.enabled }}
# Allow hostPath for eBPF and container logs
volumes:
{{ toYaml .Values.podSecurity.volumes | indent 2 }}
# Use the `spc_t` selinux type to access the
# docker/cri socket + proc and cgroup stats
seLinuxContext:
{{- if .Values.podSecurity.seLinuxContext }}
{{ toYaml .Values.podSecurity.seLinuxContext | indent 2 }}
{{- end }}
# eBPF requires some specific seccomp and capabilities
seccompProfiles:
{{ toYaml .Values.podSecurity.seccompProfiles | indent 2 }}
allowedCapabilities:
{{ toYaml .Values.podSecurity.capabilities | indent 2 }}
#
# The rest is copied from default SCCs
#
allowHostDirVolumePlugin: true
allowHostIPC: false
allowPrivilegedContainer: {{ or .Values.podSecurity.privileged .Values.tracerProps.enabled }}
allowedFlexVolumes: []
defaultAddCapabilities: []
fsGroup:
  type: MustRunAs
readOnlyRootFilesystem: false
runAsUser:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
# If your environment restricts user access to the Docker socket or journald (for logging)
# create or use an existing group that has access and add the GID to
# the lines below (also remove the previous line, `type: RunAsAny`)
#   type: MustRunAs
#   ranges:
#   - min: <min-group-ID>
#   - max: <max-group-ID>
requiredDropCapabilities: []
{{- end }}

2. Handling DNS and Network Overrides

OpenShift’s network policies can complicate DNS resolution for your observability tools. Our solution allows for:

• DNS Override Configuration: By incorporating DNS selector overrides into our Helm commands, we ensure that our agents correctly resolve network endpoints even in restricted environments.

// Check for custom DNS service override from environment variables.
// For example, set ED_DNS_SERVICE_NAMESPACE and ED_DNS_SERVICE_NAME to override the defaults.
customDNSNamespace := env.Get(env.EDDnsServiceNamespace)
customDNSName := env.Get(env.EDDnsServiceName)
// Helper function to retrieve a service by namespace/name.
getService := func(namespace, name string) (*corev1.Service, error) {
svc, err := apiClient.CoreV1().Services(namespace).Get(ctx, name, metav1.GetOptions{})
if err != nil {
return nil, err
}
return svc, nil
}
var svc *corev1.Service
var err error
var dnsServiceInfo string
// 1. Try custom override if provided.
if customDNSNamespace != "" && customDNSName != "" {
svc, err = getService(customDNSNamespace, customDNSName)
if err == nil {
dnsServiceInfo = fmt.Sprintf("%s/%s", customDNSNamespace, customDNSName)
}
}
// 2. Try kube-system/kube-dns (common in many Kubernetes clusters).
if svc == nil {
svc, err = getService("kube-system", "kube-dns")
if err == nil {
dnsServiceInfo = "kube-system/kube-dns"
}
}
// 3. Fallback: try openshift-dns/dns-default (common in OpenShift).
if svc == nil {
svc, err = getService("openshift-dns", "dns-default")
if err == nil {
dnsServiceInfo = "openshift-dns/dns-default"
}
}

3. Addressing eBPF and Metrics Collection

For customers using advanced observability features like eBPF for tracing and metrics:

• eBPF Considerations:

eBPF programs run in a restricted kernel environment, and one key limitation is the small 512-byte stack size available per program. When using uprobes — which attach eBPF programs to user-level functions — the typical in-function stack usage may exceed this limit. To work around it, some implementations “offload” larger stack allocations to BPF maps.

• Kube-State Metrics:

Integration with kube-state metrics is supported out-of-the-box, provided the metrics service is installed in your OpenShift cluster. Our team continuously monitors and adjusts the integration to account for different deployment scenarios.