Kubernetes Pods Explained: Why Connecting to a Database Pod Is Hard

A developer SSHes into a Kubernetes node, runs kubectl exec -it postgres-7f4b9c-x2k4n -- psql -h localhost -U app, and the prompt drops them into a working PostgreSQL session. Five minutes later they try to connect from their laptop to the same database pod using the IP they saw in kubectl get pods -o wide, and the connection just hangs. Same database. Same credentials. Same network — or so it feels. The connection times out anyway. This is one of the most common rite-of-passage frustrations in Kubernetes, and the reason it happens is not a bug. It is the entire point of the design.

Why Kubernetes Exists in the First Place

Docker solved an honest problem — you can package an application with its dependencies and ship a single, reproducible image that runs the same on a laptop, a CI runner, or a production VM. If you have not yet internalized the container model, the Docker containers, from dev to production guide is the prerequisite for everything that follows in this post. Once you have a few dozen containers across a few dozen servers, though, Docker stops being enough. Which host should this container run on? What happens if that host dies at 3 a.m.? How do you roll out a new version without dropping requests? How do containers on host A find containers on host B? How much CPU and memory should each one be allowed to use?

Kubernetes is the answer that became consensus. Born inside Google as a re-implementation of Borg and open-sourced in 2014, it is a cluster operating system: you declare what you want running and how it should behave, and Kubernetes — through a chain of controllers that constantly compare desired state to actual state — makes that true. The unit it manages is not a container directly. It is a pod, which is a wrapper around one or more tightly-coupled containers that share a network identity and storage. Everything else — Deployments, Services, StatefulSets, Jobs, CronJobs — is a higher-level abstraction that ultimately tells Kubernetes which pods should exist, where they should run, and how to expose them.

The control plane is the brain. Worker nodes are the muscle. Each worker runs three things in its base layer: a kubelet (the agent that takes orders from the API server and makes them happen on that node), a container runtime (these days almost always containerd or CRI-O — Docker as a runtime was deprecated in 1.20 and removed in 1.24), and kube-proxy (the userspace process that programs the kernel’s iptables or IPVS rules so that Service IPs actually route somewhere). On top of those, you find pods.

The Pod: Smaller Than a VM, Bigger Than a Container

A pod is the smallest deployable unit in Kubernetes. The simplest possible pod runs a single container. But the abstraction exists precisely because the smallest unit you sometimes want to ship is more than one container — a primary application container plus a sidecar that handles logging, or TLS termination, or metric scraping, or as a database proxy. All containers in the same pod share the same network namespace (they can talk to each other over localhost) and can share filesystem volumes. They are always scheduled together onto the same node. They live and die together.

Here is the bare minimum pod manifest. Three things are worth noting: the apiVersion: v1 (pods are core API), the single container running an Nginx image, and the absence of any restart policy at the top level — bare pods are mortal. If the node dies, the pod dies with it. Nobody runs bare pods in production. They are useful for one-off tests and as a teaching object.

# pod.yaml — the simplest possible pod
apiVersion: v1
kind: Pod
metadata:
  name: hello-pod
  labels:
    app: hello
spec:
  containers:
  - name: web
    image: nginx:1.27
    ports:
    - containerPort: 80
    resources:
      requests:
        cpu: "50m"
        memory: "64Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"

What you actually deploy is a Deployment: a controller that maintains a desired number of identical pods, handles rolling updates, and recreates pods when nodes fail. The Deployment owns a ReplicaSet, the ReplicaSet owns the pods. You almost never reference pods directly — you reference the Deployment, and Kubernetes manages the pods underneath.

# deployment.yaml — a real workload
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: ghcr.io/acme/api:1.4.2
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet: { path: /healthz, port: 8000 }
          initialDelaySeconds: 5
        livenessProbe:
          httpGet: { path: /livez, port: 8000 }
          initialDelaySeconds: 15
        env:
        - name: DB_HOST
          value: "postgres.db.svc.cluster.local"
        - name: DB_PORT
          value: "5432"

The Flat Networking Model That Nobody Warns You About

Kubernetes has exactly four networking rules, and they are deceptively simple:

1. Every pod gets its own IP address, drawn from a cluster-wide CIDR range that does not overlap with the node IPs.
2. Pods on the same node can communicate without NAT.
3. Pods on different nodes can communicate without NAT.
4. The IP a pod sees as its own is the same IP that other pods see when they talk to it.

That last point is the kicker. In a typical Docker-on-one-host setup, a container has a private IP on a bridge network, and outside traffic gets NATed through the host. Kubernetes refuses this trick. Every pod is a first-class citizen of one flat network, regardless of which physical machine it happens to be sitting on. The plumbing that makes this true is a CNI plugin (Container Network Interface) — Calico, Cilium, Flannel, Weave, AWS VPC CNI, GKE’s native plugin, and so on. They all implement the same contract; how they implement it (overlays with VXLAN tunnels, BGP route advertisement, native cloud routing, eBPF dataplane) is where they differ.

This flatness is glorious for application code — you just connect to an IP and it works — and brutal for operators trying to reason about traffic. Every pod is mutually addressable inside the cluster, which means without policies, every pod can hit every database, every cache, every internal API. We will return to that fact when we discuss why directly addressing a database pod is more fragile than it looks.

Services: How Pods Actually Find Each Other

If pod IPs change on every restart, you cannot put one in a connection string. The fix is a Service: a stable virtual IP and DNS name that load-balances traffic to a set of pods identified by labels. The pods come and go. The Service stays. Inside the cluster, every Service automatically gets a DNS name of the form service-name.namespace.svc.cluster.local, served by CoreDNS, the cluster’s built-in DNS resolver.

Here is a ClusterIP Service for a Postgres pod. Note the selector field: the Service matches by labels, not by name. Any pod with app: postgres in the same namespace becomes a backend.

# service.yaml — ClusterIP for Postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
spec:
  type: ClusterIP            # default, can omit
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432              # the Service port
    targetPort: 5432        # the container port
    protocol: TCP

Inside any other pod in the cluster, you can now write psql -h postgres.db.svc.cluster.local -p 5432 and it will work. From your laptop, the exact same command will hang forever. That gap is what the next section is about.

Why Connecting Directly to a Database Pod Falls Apart

The mental model that breaks people coming from plain Docker is the assumption that “a container is a container, and if I know its IP and port, I can connect.” In Kubernetes, almost everything in that sentence is wrong. The pod IP is real but private, ephemeral, and on a network that exists only between the nodes of one cluster. The port is open inside the container but not exposed at any layer above it. There is no host-level port-publishing equivalent to docker run -p 5432:5432 — hostPort exists but is discouraged. Let me walk through the failure modes one by one.

Pod IPs are ephemeral. The moment a pod restarts — whether because the node rebooted, the liveness probe failed, you ran kubectl rollout restart, or the scheduler evicted it — the new pod gets a new IP from the CNI’s pool. Anything that referenced the old IP is now talking to nothing, or worse, to whatever recycled into that slot. This is why you never write a pod IP into a config file. You write a Service DNS name and let CoreDNS handle the lookup.

ClusterIP is invisible from outside. The Service IP that kubectl get svc shows you — 10.96.42.7 in our example — is a virtual IP. It does not belong to any network interface. It exists only as a row in kube-proxy’s iptables tables on each node. From a laptop outside the cluster, there is no route to 10.96.0.0/12, and even if you added one statically, no kernel outside the cluster has the rules to translate it.

Pods are not on the host network by default. You can set hostNetwork: true on a pod, and then the container does share the node’s network namespace, and yes, a container port maps directly to a node port. This is how things like CNI agents and node-exporter run. Doing it for a database is a terrible idea: you lose IP isolation, port collisions become possible, and any node failure takes the database with it on an IP you cannot move.

NetworkPolicies can deny traffic explicitly. Once you install a CNI that supports NetworkPolicy (most modern ones do), you can write rules like “only pods labeled role: api in the app namespace may connect to pods labeled app: postgres in the db namespace on port 5432.” Without such a rule and with a default-deny baseline, everything is blocked. Without any policies, everything is allowed — which is its own problem.

# networkpolicy.yaml — only the api can talk to postgres
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-api
  namespace: db
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: app
      podSelector:
        matchLabels:
          role: api
    ports:
    - protocol: TCP
      port: 5432

The container port is not automatically exposed at the node level. Docker users are used to -p 5432:5432 binding a host port to a container port. Kubernetes does not do this. The containerPort field in a pod spec is documentation — it tells humans and tools that the container intends to listen on this port. It does not punch a hole through any layer. To get external reachability, you need a Service of the right type plus, if you are on a cloud, a security group rule that allows traffic to whichever node port the cloud LB or NodePort uses.

Databases are stateful, and stateful pods need stateful controllers. A plain Deployment treats its pods as interchangeable cattle. The Deployment will happily reschedule postgres-0 from node 2 to node 5 if node 2 goes unhealthy, mounting whatever PersistentVolume is available, or worse, no volume at all if the PVC is gone. For a database you need a StatefulSet, which gives each pod a stable identity (postgres-0, postgres-1, etc.), a stable per-pod DNS name through a headless Service, and a stable PersistentVolumeClaim that stays attached to that ordinal across reschedules. Get this wrong on your first attempt and you will lose data.

The chain has many links and is only as strong as the weakest. For an external client to reach a pod, the request typically traverses: client → public DNS → cloud LB → node IP → iptables DNAT → pod IP → container port → Postgres listener → pg_hba.conf check → authentication. A misconfiguration anywhere — TLS cert wrong, SG blocking the LB health check, pg_hba.conf denying the source CIDR, Postgres bound to 127.0.0.1 inside the container instead of 0.0.0.0 — produces a connection failure that looks identical to a network problem from the client’s perspective.

Three Connection Patterns That Actually Work

There are essentially three legitimate ways to connect a client to a database running in a pod, depending on where the client lives. Picking the right one is mostly a question of who needs the connection and for how long.

Pattern A: In-cluster application to in-cluster database

This is the boring, correct, default pattern. Your application pod sets DB_HOST=postgres.db.svc.cluster.local as an environment variable. The application opens a connection. CoreDNS resolves the name, kube-proxy DNATs the virtual IP onto a real pod, and the connection lands. Pod restarts on either side are transparent because everything is named, not pinned. This is also the pattern that Airflow workloads use when they run via the Apache Airflow data pipeline orchestration guide’s KubernetesExecutor — each task spawns as a pod and reaches the database through a Service. The same is true for dbt jobs running on Kubernetes and for Kafka consumer workloads running in pods.

Pattern B: Local developer to in-cluster database

kubectl port-forward opens a tunnel from a local port on your laptop, through the Kubernetes API server, to a port on a pod. It is meant for development and one-off admin tasks. Here it is in action against the headless Service we will define next:

# forward localhost:5432 to the postgres-0 pod's port 5432
kubectl port-forward -n db pod/postgres-0 5432:5432

# Or forward through the headless Service to whichever endpoint is selected
kubectl port-forward -n db svc/postgres 5432:5432

# Now from another terminal, on your laptop:
psql -h localhost -p 5432 -U app -d production

And here is a Python client connecting through that forwarded port. Note that the connection string says localhost — that is true on the laptop. Inside the cluster, the same code would say postgres.db.svc.cluster.local.

# dev_query.py — assumes "kubectl port-forward" is running
import os
import psycopg2
from psycopg2.extras import RealDictCursor

# Local dev: connect through kubectl port-forward
# In production (in-cluster), DB_HOST would be postgres.db.svc.cluster.local
DB_HOST = os.environ.get("DB_HOST", "localhost")
DB_PORT = int(os.environ.get("DB_PORT", "5432"))
DB_NAME = os.environ.get("DB_NAME", "production")
DB_USER = os.environ.get("DB_USER", "app")
DB_PASS = os.environ["DB_PASS"]  # required, no default

def fetch_recent_orders(limit: int = 50):
    """Read the most recent orders — example dev-time query."""
    with psycopg2.connect(
        host=DB_HOST,
        port=DB_PORT,
        dbname=DB_NAME,
        user=DB_USER,
        password=DB_PASS,
        connect_timeout=5,
        sslmode="require",   # still enforce TLS even on port-forward
    ) as conn:
        with conn.cursor(cursor_factory=RealDictCursor) as cur:
            cur.execute(
                "SELECT id, customer_id, total_cents, created_at "
                "FROM orders ORDER BY created_at DESC LIMIT %s",
                (limit,),
            )
            return cur.fetchall()

if __name__ == "__main__":
    rows = fetch_recent_orders()
    for row in rows:
        print(row)

Pattern C: External application to in-cluster database

This is where most teams should hesitate. If you have an application outside the cluster that needs to read or write to the database, the right architecture is almost always to expose an API (HTTP/gRPC over Ingress with TLS and auth) and let the API talk to the database internally. That said, valid cases exist — analytics tools, BI dashboards, replication to another system — and the pattern looks like this: a type: LoadBalancer Service backed by the database pods, fronted by a cloud network load balancer, with the security group locked to specific client CIDRs, mandatory TLS, and rotated credentials. If you can substitute a managed database (RDS, Cloud SQL, Aurora) here, do it. Operating Postgres inside Kubernetes is doable but it is real work.

The StatefulSet plus headless Service pattern

A headless Service is what you get when you set clusterIP: None. Instead of a virtual IP, it produces DNS A records, one per pod backend. Combined with a StatefulSet, you get stable per-pod hostnames — postgres-0.postgres.db.svc.cluster.local, postgres-1.postgres.db.svc.cluster.local — which is exactly what a primary/replica database setup needs. The application points writes at the primary’s hostname and reads at any replica’s hostname.

# headless service + statefulset for postgres
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: db
  labels:
    app: postgres
spec:
  clusterIP: None          # headless — no virtual IP
  selector:
    app: postgres
  ports:
  - name: pg
    port: 5432
    targetPort: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: db
spec:
  serviceName: postgres    # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: postgres
        image: postgres:16.3
        ports:
        - containerPort: 5432
          name: pg
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        readinessProbe:
          exec:
            command: ["pg_isready", "-U", "postgres"]
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3-ssd
      resources:
        requests:
          storage: 200Gi

For real production databases you almost certainly want a purpose-built operator on top of this scaffolding — CloudNativePG, Zalando’s postgres-operator, or Crunchy PGO — that handles primary election, streaming replication, backups, point-in-time-recovery, and rolling minor-version upgrades. Choosing the right database backend is its own art; the database comparison for preprocessed time-series data is a good companion piece on that side of the decision.

Kubernetes 1.36 and What Changed in 2026

Kubernetes 1.36 is the latest minor release as of this writing in May 2026, and it doubles down on security defaults and AI workload support. According to the official release page (Source: kubernetes.io/releases, as of 2026-05-20), the project actively maintains release branches for the three most recent minor versions — currently 1.34, 1.35, and 1.36 — with 1.33 having entered maintenance on 2026-04-28 and reaching End of Life on 2026-06-28. The release tempo is fast enough that if you are running anything older than 1.33 today you are already off the supported track.

User Namespaces → GA is the headline security change in 1.36. With user namespaces enabled, the root user inside a container is mapped to a non-privileged user on the host. This dramatically reduces the blast radius of a container escape: even if an attacker compromises a container running as UID 0, they emerge on the host as something like UID 100000 with no special privileges. For database pods specifically, this means a Postgres container compromise no longer means immediate root on the node. Combined with seccomp and AppArmor profiles, this closes one of the long-standing gaps between Kubernetes security and traditional VM isolation.

Mutating Admission Policies → GA brings declarative, CEL-expression-based mutations to the admission chain, replacing many uses of webhook-based mutating admission controllers. You can now write policies that, say, automatically inject sidecar containers, add labels, set default resource requests, or enforce image-registry rules without operating a separate webhook server. Less moving infrastructure to maintain, fewer failure modes when the webhook goes down.

Fine-Grained Kubelet API Authorization → GA lets the kubelet enforce per-verb RBAC on its own API rather than treating all operations uniformly. This matters for hardening: tools that need nodes/proxy can be limited to read-only operations, and the kubelet can refuse risky combinations that previously required cluster-admin to fully restrict.

Beyond security, 1.36 leans further into AI workload support — refinements to Dynamic Resource Allocation (DRA) for GPU scheduling, support for accelerator partitioning, and improvements to the scheduler’s ability to handle long-running training jobs alongside short-lived inference pods. The trajectory is clear: the kubernetes-as-AI-platform pattern, which exploded in 2024-2025 as model-serving workloads moved off bespoke infrastructure, has now been first-class for two release cycles. For language and runtime choices when building operators or controllers around these new APIs, the Python vs Rust comparison is a useful framing — the controller-runtime ecosystem in Go remains dominant, but Rust operators are gaining ground for performance-sensitive components.

Frequently Asked Questions

Can a pod have more than one container?

Yes, and it is a common design pattern. The most frequent reason is the sidecar — a helper container that does logging, TLS termination, service-mesh proxying (Envoy in Istio or Linkerd), or connection pooling. All containers in a pod share a network namespace and can share volumes, but they remain separate processes with separate filesystems. Use multiple containers when their lifecycles are genuinely coupled. If the answer to “can these scale independently?” is yes, they belong in separate pods.

Why not just expose every database pod with a NodePort and connect directly?

NodePort opens the same port on every node in the cluster, in the 30000–32767 range, and routes it to whichever pod backs the Service. Three problems: the port numbers are non-standard so client tooling fights you, every node becomes an attack surface for the database, and you still need a cloud security group or firewall rule to control who can hit those ports. NodePort is fine for on-prem clusters without a cloud LB or for very specific debug scenarios. It is not a substitute for proper Service architecture.

Is kubectl port-forward safe to use in production?

It is safe to use, but it should not be how production traffic flows. The tunnel runs through the API server and consumes API-server resources. It bypasses NetworkPolicy — if you can port-forward, you can connect, regardless of how strict your in-cluster policies are. RBAC controls who can use it, and you should treat pods/portforward on a database namespace as a sensitive verb subject to audit. For production traffic, use a real Service.

What is the difference between a StatefulSet and a Deployment?

A Deployment treats pods as interchangeable. It will scale up by spinning up new pods with random suffix names, scale down by killing any of them, and roll updates in parallel. A StatefulSet maintains ordered, named pods (name-0, name-1, name-2) that always come up in order, always shut down in reverse order, and each get their own stable PersistentVolumeClaim. Use Deployment for stateless apps. Use StatefulSet for anything that has identity — databases, message brokers, ZooKeeper, distributed coordination services. Kafka brokers running in Kubernetes are a textbook StatefulSet workload.

Should I actually run my database in Kubernetes, or use a managed service?

For most teams below the scale of needing a database engineer on the org chart, managed (RDS, Cloud SQL, Aurora, AlloyDB, Spanner) is the right answer. Operating a stateful workload well — backups, point-in-time recovery, minor-version upgrades, failover, performance tuning, observability — is a continuous engineering investment that managed services amortize across thousands of customers. Run databases in your cluster when you have a real reason: cost at scale, regulatory data residency, latency requirements that make a separate database tier unworkable, or a database that managed offerings do not provide. The operator ecosystem (CloudNativePG and friends) makes this much more tractable than it was five years ago, but it is still real work.

Related Reading

References

• Kubernetes releases — official release and support timeline
• Kubernetes cluster networking concepts
• Kubernetes 1.36 sneak peek (official blog, March 2026)
• Connecting applications with Services (official tutorial)
• InfoQ: Kubernetes 1.36 released (May 2026)

Conclusion

The reason connecting to a database in a Kubernetes pod feels harder than it should is that Kubernetes is solving a different problem than the one you may think it is solving. It is not a fancy docker run. It is a cluster operating system whose entire networking model is designed around the assumption that pods talk to other pods through stable abstractions, and external clients talk to applications through carefully chosen entry points. The pod IP that kubectl get pods -o wide reveals is a debugging convenience, not an address. The ClusterIP that kubectl get svc shows is a fiction held together by iptables. The right address for production traffic from inside the cluster is a DNS name, served by CoreDNS, backed by a Service whose membership the controllers maintain for you. The right address from outside is whatever your LoadBalancer, Ingress, or jump-host configuration says it is — never a pod IP.

If you remember three things from this post, remember these. First: kubectl port-forward is your friend for development, and not your friend for production. Second: stateful workloads need StatefulSet plus PVC plus a headless Service, or you will lose data. Third: in Kubernetes 1.36 and beyond, the security defaults are tightening (User Namespaces GA being the most consequential), which is good news for anyone running databases in pods — but the surface area of what can go wrong between an external client and your database remains large enough that “expose Postgres directly to the internet” almost always loses to “put an API in front of it.” Build the boring, layered version first. Save the clever shortcuts for things that genuinely warrant them.