Kubernetes Service Not Reachable: Kube-Proxy, Endpoints & DNS

A Service fails when the chain between the client and backend breaks. That chain depends on EndpointSlices to list healthy pods, kube-proxy to program kernel rules, and cluster DNS to resolve names to ClusterIPs. This guide covers the gap between healthy backend pods and an unreachable Service, focusing on kube-proxy data-plane programming, endpoint state, and DNS dependencies. It does not cover application-level bugs inside the pod.

What This Means

Reachability is a control-loop problem. kube-proxy watches Services and EndpointSlices, then programs iptables, IPVS, or nftables rules to DNAT traffic to healthy endpoints. These rules persist in the kernel if kube-proxy crashes, but updates stop until it resumes. DNS resolution targets the CoreDNS ClusterIP, so a kube-proxy failure often appears as a DNS outage before a direct Service timeout.

Failures fall into four areas:

• Empty EndpointSlices (no ready addresses).
• Stale or missing kube-proxy rules (sync lag, API disconnect, lock contention).
• Conntrack table exhaustion (silent drops).
• DNS misconfiguration or CoreDNS unavailability.

Common Causes

Quick Checks

Run these read-only checks to narrow the blast radius.

# Verify the Service has endpoints
kubectl get endpoints <service-name> -n <namespace>
kubectl get endpointslices -n <namespace> | grep <service-name>

# Check if backend pods are ready and labels match the selector
kubectl get pods -n <namespace> -l <service-selector-key>=<value>

# Test DNS resolution from a client pod
kubectl run -it --rm debug --image=busybox:1.36 --restart=Never -- nslookup <service-name>.<namespace>.svc.cluster.local

# Check kube-proxy health and readiness on the node
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:10256/healthz
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:10256/readyz

# Check how recently kube-proxy completed a successful sync
curl -s http://127.0.0.1:10249/metrics | grep kubeproxy_sync_proxy_rules_last_timestamp_seconds

# Inspect programmed iptables rules for a Service (iptables mode)
sudo iptables -t nat -L KUBE-SERVICES -n | grep <service-cluster-ip>
sudo iptables -t nat -L -n | grep -c "KUBE-SEP-"

# Inspect IPVS state (IPVS mode)
sudo ipvsadm -Ln | grep <service-cluster-ip>

# Inspect nftables state (nftables mode)
sudo nft list table ip kube-proxy

# Check conntrack utilization
awk 'NR==FNR{c=$1;next} {printf "%.0f%%\n", c*100/$1}' /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max

# Check for conntrack drops
sudo conntrack -S | grep drop

How To Diagnose It

Follow this sequence to isolate which layer is dropping traffic.

1. Confirm the Service has endpoints. Run kubectl get endpoints <svc> and kubectl get endpointslices. If the addresses list is empty, the problem is upstream of kube-proxy. Check that the selector matches pod labels and that the pods are ready. If endpoints exist, continue.
2. Test DNS from the client namespace. DNS queries without a namespace qualifier resolve only within the pod’s own namespace. Run nslookup <service>.<namespace>.svc.cluster.local from the client pod. If this fails, check whether CoreDNS pods are running. Because CoreDNS is usually exposed as a ClusterIP Service, a kube-proxy failure can break DNS before it breaks your application Service. Inspect the client pod’s /etc/resolv.conf to ensure the nameserver is the cluster DNS IP and not a loopback stub like 127.0.0.53 from systemd-resolved.
3. Check kube-proxy health and sync recency. On the node where the client or backend runs, query http://127.0.0.1:10256/healthz and http://127.0.0.1:10256/readyz. A 503 on readyz means initial sync never completed. Pull the metrics from :10249/metrics and compare kubeproxy_sync_proxy_rules_last_timestamp_seconds against the current time. If the gap exceeds twice the sync period, the node is operating on stale rules.
4. Compare kernel rules to EndpointSlice state. For iptables mode, list the KUBE-SVC-* and KUBE-SEP-* chains and verify that the endpoint IPs match the addresses in the EndpointSlice. For IPVS mode, use ipvsadm -Ln to confirm virtual servers and real servers align with the Service and EndpointSlice. For nftables mode, use nft list table ip kube-proxy. If the rules are missing or point to old pod IPs, kube-proxy is either failing to sync or lagging behind.
5. Check conntrack saturation on the node. Read nf_conntrack_count and nf_conntrack_max. If utilization is above 90%, new connections are at risk of being dropped silently. Run conntrack -S and look for a non-zero drop counter. If drops are incrementing, conntrack exhaustion is the immediate cause. Use conntrack -L to see which protocols and states dominate the table.
6. Verify external traffic path for NodePort and LoadBalancer Services. Check ss -tlnp on the node to confirm the NodePort is listening and not held by another process. For Services with externalTrafficPolicy: Local, query the health check node port (curl http://<node-ip>:<healthCheckNodePort>/healthz). A 503 here means the node has no local endpoints, which is expected behavior for that policy, but external load balancers will stop sending traffic to that node.

flowchart TD
    A[Client cannot reach Service] --> B{Endpoints exist?}
    B -->|No| C[Check pod labels and readiness]
    B -->|Yes| D{DNS resolves?}
    D -->|No| E[Check CoreDNS and resolv.conf]
    D -->|Yes| F{Kernel rules match endpoints?}
    F -->|No| G[Check kube-proxy sync and API connectivity]
    F -->|Yes| H{Conntrack table full?}
    H -->|Yes| I[Check drops and raise limit]
    H -->|No| J[Check NodePort conflicts and traffic policy]

Metrics & Signals To Monitor

Fixes

Apply fixes that match the layer you isolated in diagnosis.

If The Cause Is Missing Or Unready Endpoints

Correct the Service selector to match pod labels. Fix readiness probes so they reflect actual serving capacity. If a rolling update left zero ready replicas, pause or roll back the deployment. A Service with zero endpoints causes kube-proxy to install a REJECT rule. This is correct behavior, not a kube-proxy bug.

If The Cause Is Kube-Proxy Sync Lag Or Stale Rules

If sync duration is high due to iptables rule bloat in a large cluster, temporarily increase the sync period to allow full syncs to complete. Plan a migration to nftables mode, which scales better than iptables; IPVS mode is deprecated as of Kubernetes 1.35, so nftables is the recommended target. If the sync is stuck because of API server disconnect, check network paths to the control plane and verify RBAC for the system:node-proxier ClusterRole. Restarting the kube-proxy pod on the affected node forces a full re-sync, but treat this as a temporary recovery step, not a root cause fix.

If The Cause Is Conntrack Exhaustion

Immediate relief: raise nf_conntrack_max. This takes effect instantly but consumes kernel memory (roughly 300 bytes per entry).

# Warning: state-changing command
sudo sysctl -w net.netfilter.nf_conntrack_max=262144

Then investigate the source. High TIME_WAIT counts suggest short-lived HTTP connections without pooling. High UDP counts suggest aggressive DNS or metrics traffic. Tune protocol-specific timeouts if your workload allows, and fix application connection leaks.

If The Cause Is DNS Resolution

Ensure CoreDNS pods are running and that their Service has endpoints. Verify that client pods use the cluster DNS IP in /etc/resolv.conf. If the node uses systemd-resolved, ensure kubelet is configured with --resolv-conf=/run/systemd/resolve/resolv.conf to avoid a stub resolver loop. If the upstream resolv.conf passed to pods contains more than 3 nameservers, reduce the list or use a local dnsmasq sidecar, because glibc limits nameservers to 3.

If The Cause Is NodePort Conflict Or External Traffic Policy

Identify the conflicting process with sudo ss -tlnp | grep <node-port> and either stop it or reallocate the NodePort. For externalTrafficPolicy: Local, either change the policy to Cluster (which permits cross-node forwarding but obscures source IP) or ensure the workload is scheduled so that every node receiving traffic has at least one local ready endpoint.

Prevention

Monitor conntrack utilization per node and alert at 70%, not at 100%. Track the trend of kubeproxy_sync_proxy_rules_duration_seconds against your sync period. When P99 duration crosses 50% of the sync period, start planning a mode migration or Service consolidation. Size kube-proxy memory limits with at least 2x headroom above steady-state RSS to accommodate re-list spikes after watch disconnects. Require readiness probes that are precise, and validate them before deployments. Restrict NodePort allocations to an approved range and audit ExternalIP usage via RBAC. Periodically spot-check that kernel rules match EndpointSlice state, especially after cluster upgrades or CNI changes.

How Netdata Helps

Use per-node kernel and container metrics to narrow the scope:

• Correlate nf_conntrack_count with connection error rates to confirm conntrack exhaustion before drops appear.
• Track kube-proxy sync duration and API latency to catch sync lag or watch disconnect.
• Alert on conntrack saturation and kube-proxy process restarts alongside Service timeouts.
• Check CPU softirq time and network drops to distinguish kube-proxy overhead from physical network issues.

Related Guides

• Kubernetes API server slow or unresponsive: causes and fixes
• Kubernetes kubelet not responding: PLEG, runtime, and certificate issues
• Kubernetes monitoring checklist: the signals every production cluster needs
• Kubernetes node NotReady: kubelet, runtime, and network diagnosis
• Kubernetes pod stuck ContainerCreating: volume, network, and image issues
• Kubernetes pod CrashLoopBackOff: causes, diagnosis, and fixes
• Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis
• Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes
• Kubernetes pod stuck Pending: scheduling failures explained