HOST_A: Welcome to Clawd Talks. I'm Emma, and today we're doing something a little different — a full mock interview for Kubernetes roles. All three levels: junior, mid, and senior. HOST_B: And I'm Ryan, playing the candidate. Which means I get to answer the questions and occasionally make the mistakes most people make in real interviews. HOST_A: Exactly. And I'll be the interviewer — probing, pushing back, and adding nuance where the textbook answer isn't quite the full picture. HOST_B: Before we dive in, I want to address something. A lot of people prep for K8s interviews by memorising YAML syntax and kubectl flags. And I think that's mostly the wrong approach. HOST_A: Completely agree. The best K8s interviews I've ever conducted — and the best ones I've sat in on — are testing three things. Conceptual clarity, operational judgment, and trade-off thinking. HOST_B: Not whether you can remember the exact flag for `kubectl rollout undo`. HOST_A: Right. You can look that up in thirty seconds. What I can't look up is how you reason through an incident at two in the morning. That's what separates candidates. HOST_B: So if you're preparing for a K8s interview, use this episode as a simulation. Pause, answer out loud, then compare your answer to what we say. That kind of active recall is worth ten times more than reading documentation passively. HOST_A: Alright. Let's start at the beginning. Junior level. And Ryan — I'm going to treat you like a real candidate, so expect me to push on your answers. HOST_B: Bring it on. HOST_A: First question. What is a Pod, and how is it different from a container? HOST_B: A Pod is the smallest deployable unit in Kubernetes. It wraps one or more containers that share the same network namespace — meaning they share the same IP address and can talk to each other over localhost. They can also share volumes if you set that up. HOST_A: Good. So why not just use containers directly? Why does the Pod abstraction exist? HOST_B: Kubernetes needs a unit that maps to "run these things together on the same host." Docker runs a single container. A Pod says, "these containers are a logical unit — they go together." The classic example is a main application container and a sidecar that handles logging or proxying. HOST_A: Exactly. The sidecar pattern. A service mesh like Istio injects a proxy sidecar into every Pod — that proxy shares the same network namespace as your app, so it can intercept all traffic without changing your application at all. HOST_B: And the key thing to understand is that Kubernetes doesn't schedule containers — it schedules Pods. The container runtime runs what's inside. HOST_A: Nice. Next: what's the difference between a Deployment and a StatefulSet? HOST_B: Deployments are for stateless applications. The pods are interchangeable — you can kill any of them, replace them in any order, scale up and down freely. They don't have a persistent identity. HOST_A: And StatefulSets? HOST_B: StatefulSets are for stateful applications — databases, Kafka, anything that needs to remember who it is. Each pod gets a stable hostname: pod-zero, pod-one, pod-two. They're created and deleted in order. And each pod gets its own persistent volume that survives restarts — even if the pod gets rescheduled to a different node, it reattaches to the same volume. HOST_A: Good. One common mistake candidates make: they say StatefulSets are for databases, and they stop there. What's the operational implication that people often miss? HOST_B: Hmm. I'd say the ordering guarantee. If you have a StatefulSet with three replicas and you're doing a rolling update, it goes pod-two, pod-one, pod-zero — in reverse order. And it waits for each pod to be ready before moving to the next. That matters a lot for primary-replica setups where you can't bring down the primary first. HOST_A: Exactly right. Rolling updates in StatefulSets are sequential and ordered, not parallel like Deployments. A lot of people don't know that. HOST_B: Something I've seen bite teams: they use a StatefulSet thinking it's just "Deployment with storage," and they're surprised when the upgrade takes forever because it's waiting on each pod individually. HOST_A: Alright. Services. What is a Service, and what are the types? HOST_B: A Service gives you a stable network endpoint — a fixed IP and a DNS name — for a dynamic set of pods. Pods come and go; their IPs change. The Service is the stable front door. HOST_A: Types? HOST_B: ClusterIP is the default — only accessible inside the cluster. NodePort exposes the service on a static port on every node in the cluster. LoadBalancer provisions an actual cloud load balancer — that's how you expose something to the internet on managed Kubernetes. And ExternalName is essentially a DNS alias to an external service outside the cluster. HOST_A: Good. And what selects which pods a Service routes to? HOST_B: Label selectors. You define a selector on the Service — like `app: my-api` — and it automatically picks up any pod with that label in the same namespace. The Endpoints object tracks the actual pod IPs behind the scenes. HOST_A: Let's talk probes. Liveness versus readiness — what's the difference? HOST_B: Liveness probe asks: is the container still alive? If it fails, Kubernetes restarts the container. You use it for things like detecting deadlocks — situations where the process is running but has completely seized up and will never recover on its own. HOST_A: And readiness? HOST_B: Readiness asks: is this container ready to serve traffic right now? If it fails, the pod is removed from the Service's endpoint list — so no traffic hits it — but the container is NOT restarted. You use it during warmup, when the app is loading caches, or when a circuit breaker has opened and the app is temporarily not accepting requests. HOST_A: There's a third probe that candidates often forget entirely. HOST_B: Startup probe. For containers that take a long time to start — like legacy apps with slow JVM startup. The startup probe runs first and gives the container extra time. Once it passes, liveness and readiness take over. Without it, your liveness probe might kill a slow-starting container before it even gets going. HOST_A: Perfect. I've seen that exact production incident — someone removed the startup probe thinking it was redundant, and their app started restart-looping during deployments because the liveness probe fired too early. HOST_B: That's a fun one to explain to management at three AM. HOST_A: Next: ConfigMaps and Secrets. What's the security difference? HOST_B: ConfigMap stores non-sensitive configuration — environment variables, config files, that kind of thing. Secret is for sensitive data like passwords and tokens. HOST_A: But here's where I want to push you. Someone junior might say "Secrets are encrypted." Are they? HOST_B: Not by default. Both ConfigMaps and Secrets are stored in etcd, and etcd stores them as base64-encoded strings — not encrypted. Base64 is encoding, not encryption. Anyone with read access to etcd can decode it trivially. HOST_A: So what makes Secrets actually secure? HOST_B: A few things working together. First, you can enable encryption at rest in etcd — that actually encrypts the data. Second, RBAC — you can give a service account access to specific Secrets without giving it access to ConfigMaps or other Secrets. Third, Secrets can be excluded from logs and audit trails more easily. And fourth, for production workloads, you really want an external secret manager — HashiCorp Vault, AWS Secrets Manager, the External Secrets Operator — so the actual secret value never even lives in etcd. HOST_A: That last point is what separates a mid-level answer from a senior one, honestly. HOST_B: The base64 confusion trips up so many people. I've seen it in actual production clusters — teams thinking their secrets were encrypted when they absolutely weren't. HOST_A: Last junior question. Walk me through what happens when you run `kubectl apply -f deployment.yaml`. HOST_B: Right. So kubectl serialises the YAML and sends it as an API request to the Kubernetes API server. The API server validates it, authenticates the request, runs admission controllers, and then persists the object to etcd. HOST_A: Then what? HOST_B: The Deployment controller — which is part of the controller manager — is watching etcd for changes. It notices the new or modified Deployment and reconciles: it creates or updates a ReplicaSet to match the desired state. Then the ReplicaSet controller notices it needs more pods and creates Pod objects in etcd. The scheduler picks up the unscheduled pods, evaluates which nodes have capacity, and writes the node assignment back to the Pod object. HOST_A: And then? HOST_B: The kubelet on the assigned node is watching for pods assigned to it. It pulls the container image, starts the container via the container runtime — containerd these days, not Docker. The container starts, the readiness probe eventually passes, and at that point the Endpoints controller adds the pod's IP to the Service endpoints. Traffic starts flowing. HOST_A: That reconciliation loop is the heart of Kubernetes. Every controller in the system is doing the same thing: watch desired state, compare to actual state, take action to close the gap. Understanding that model deeply explains almost every Kubernetes behaviour you'll encounter. HOST_B: It's turtles all the way down — controllers watching controllers watching controllers. HOST_A: Alright, let's move up. Mid-level. These questions get into operations and architecture. Rolling updates — how do they work and how do you configure them? HOST_B: Kubernetes replaces old pods with new ones incrementally. You configure it through `strategy.rollingUpdate` on the Deployment. Two key parameters: `maxUnavailable` — how many pods can be unavailable at once, defaults to twenty-five percent — and `maxSurge` — how many extra pods above your desired count can exist during the update, also defaults to twenty-five percent. HOST_A: What's the actual sequence? HOST_B: Kubernetes creates the surge pods first — new version, up to maxSurge above desired count. It waits for them to pass readiness probes. Once a new pod is ready, it terminates an old pod. It repeats until all old pods are replaced. HOST_A: And what happens if the new pods never pass readiness? HOST_B: The rollout stalls. It won't proceed to kill the old pods because that would violate maxUnavailable. So you end up in a half-and-half state — some old pods still running, some new pods stuck in a not-ready state. Which is actually the safe behaviour. Your service is degraded but not down. HOST_A: And you can roll back with? HOST_B: `kubectl rollout undo deployment my-app`. Or `kubectl rollout undo deployment my-app --to-revision=3` if you want a specific version. The rollout history is stored in ReplicaSets — each revision maps to an old ReplicaSet that Kubernetes keeps around. HOST_A: Good. Let's talk about etcd. What is it and why does it matter so much? HOST_B: etcd is the distributed key-value store that holds all Kubernetes cluster state. Every object — pods, deployments, services, secrets, everything — is persisted in etcd. It's the single source of truth for the entire cluster. HOST_A: Why distributed? HOST_B: Because it needs to be highly available. etcd runs as a cluster of typically three or five nodes and uses the Raft consensus algorithm — so you need a quorum of nodes to agree before a write is committed. This gives you consistency even if a node fails. HOST_A: What's the disaster scenario? HOST_B: You lose etcd without a backup. All cluster state is gone. Your workloads might still be running on nodes — kubelet keeps things running even if the control plane is down — but you've lost all the metadata. You can't do any management. You can't deploy anything. You've essentially lost your cluster configuration entirely. HOST_A: For self-managed clusters, etcd backup is non-negotiable. Managed Kubernetes — EKS, GKE, AKS — handles etcd for you. But if you're running your own control plane, automated backups are day-zero work. HOST_B: And the disk space gotcha. etcd is not meant for large data — it has a default storage limit of eight gigabytes. If your cluster has a lot of churn — lots of objects being created and deleted — the etcd database can fill up. Compaction and defragmentation are operational tasks people often forget about. HOST_A: That is a great mid-to-senior point. I've seen clusters go read-only because etcd hit its storage limit. Not fun. HOST_B: PodDisruptionBudgets — what are they and why do you need them? HOST_A: A PDB is a guarantee that a minimum number of your pods will remain available during voluntary disruptions. Things like node drains during cluster upgrades or maintenance. HOST_B: You define it with `minAvailable` — say two — meaning Kubernetes will refuse to drain a node if doing so would leave fewer than two pods running. Or `maxUnavailable` to express the same thing differently. HOST_A: Why is this critical for production? HOST_B: Without a PDB, if you have a three-replica deployment spread across three nodes and someone drains all three nodes simultaneously — say during a cluster upgrade — all three pods could be evicted at once. Service goes down. PDBs are the guardrail. HOST_A: And a fun real-world failure mode: a PDB that's set too aggressively. `minAvailable` equal to all your replicas, for example. Then a node drain will block forever because you can't evict any pods without violating the budget. HOST_B: That's blocked a cluster upgrade for hours in my experience. The drain hangs, you don't know why, you eventually check the PDB and realise it's preventing any evictions at all. HOST_A: Resource requests and limits. Explain the difference, and what happens if you set limits without requests? HOST_B: Requests are what the scheduler uses for placement — "this pod needs at least this much CPU and memory." Limits are the hard cap — "this pod cannot use more than this." The key insight is that a pod can temporarily use more CPU than its request if there's idle capacity on the node. But at the limit, CPU gets throttled. Memory is different — exceed the memory limit and the container gets OOMKilled immediately. HOST_A: And if you set limits without requests? HOST_B: Kubernetes defaults the request to equal the limit. So a pod with only limits specified becomes Guaranteed QoS — which sounds good but means you're telling the scheduler the pod always needs its full limit, which can over-constrain scheduling. HOST_A: The classic disaster scenario? HOST_B: You set a very high memory limit — say, four gigabytes — without a request, thinking it's a maximum. The scheduler sees the request as four gigs and places fewer pods per node accordingly. But actually, most pods only use five hundred megs in practice. You've over-reserved massively. The reverse is also bad: no limits at all, one pod goes rogue and consumes all node memory, OOMKiller starts evicting other pods on the same node. HOST_A: The QoS classes matter here. Guaranteed — requests equal limits. Burstable — requests less than limits. BestEffort — no requests or limits. BestEffort pods are the first to be evicted when the node is under memory pressure. HOST_B: Service discovery. How does it work in Kubernetes? HOST_A: Two mechanisms. Environment variables and DNS. Environment variables are injected at pod startup — every service in the namespace gets a set of env vars with its IP and port. The problem is they're stale immediately if services change after the pod starts. HOST_B: DNS is the right answer. CoreDNS runs in every cluster and every Service gets a fully qualified domain name: service-name dot namespace dot svc dot cluster dot local. You can use just the service name within the same namespace and DNS resolution handles the rest. HOST_A: StatefulSets get even more granular DNS — each pod gets its own record. pod-zero dot service-name dot namespace dot svc dot cluster dot local. That's how database clients can connect to specific replicas directly. HOST_B: And for StatefulSets you typically use a headless service — ClusterIP set to None — which means DNS resolves to the individual pod IPs rather than a single virtual IP. Clients get all the IPs back and decide which to use. HOST_A: Ingress. What is it, how does it work? HOST_B: Ingress is a Kubernetes object that defines HTTP and HTTPS routing rules — this host and path combination routes to this backend service. But an Ingress object by itself does nothing. You need an Ingress Controller to actually implement the rules. HOST_A: Common controllers? HOST_B: nginx-ingress is the most common. Traefik is popular especially with cert-manager integration. On AWS you'd use the ALB Ingress Controller which provisions an Application Load Balancer. On GKE there's the native GCE Ingress. HOST_A: The economic argument for Ingress? HOST_B: One LoadBalancer for the entire cluster instead of one per service. A LoadBalancer service on AWS or GCP costs real money. With Ingress, one load balancer fronts the Ingress Controller, which then routes internally to many different services based on rules. Far cheaper at scale. HOST_A: Alright. Senior level. These are the questions that actually separate people. Ryan, you're the senior now — I'm going to be more demanding. HOST_B: Understood. HOST_A: You're on-call. A service is degraded. Walk me through your debugging process. HOST_B: First thing — observability. I want Prometheus and Grafana up immediately. Is this a latency spike? An error rate spike? CPU or memory issue? What changed? The dashboard gives me the shape of the problem before I start poking around. HOST_A: Good. Then what? HOST_B: Assuming it's not immediately obvious from metrics — `kubectl get pods -n my-namespace`. I'm looking for any pods in a bad state: CrashLoopBackOff, OOMKilled, Pending, Init Error. That narrows it down fast. HOST_A: Say everything looks Running. HOST_B: Then I check `kubectl describe pod` on a few pods — specifically the Events section at the bottom. That tells me if there are failed liveness probes, image pull errors, volume mount failures, anything the pod tried and failed at recently. Then `kubectl logs pod-name --previous` for any containers that recently crashed, even if they've recovered. HOST_A: Still nothing obvious. HOST_B: I check the endpoints: `kubectl get endpoints my-service`. Are there actually healthy pods behind the service? Sometimes pods are Running but failing readiness probes — they're excluded from endpoints and traffic is going nowhere. Then I check the HPA if there is one — is autoscaling working? Did it try to scale up and fail to schedule new pods? `kubectl describe hpa`. HOST_A: Good. What else? HOST_B: Node-level: `kubectl describe node` on the affected nodes. I'm looking for resource pressure conditions — DiskPressure, MemoryPressure, PIDPressure. If a node is under pressure, kubelet starts evicting pods. And finally — did anything deploy recently? `kubectl rollout history deployment my-service`. Someone might have pushed a bad image ten minutes ago and that's the whole story. HOST_A: The network troubleshooting step? HOST_B: `kubectl exec -it pod-name -- sh`, then curl the service directly from inside a pod. Bypasses any external networking layers. You're testing the internal cluster networking. If that works but external access is broken, it's an Ingress or load balancer issue, not an application issue. HOST_A: Database migration. Zero downtime. How do you approach it? HOST_B: It depends entirely on what kind of migration. The easy case is purely additive — adding a new table, adding a nullable column. Both old and new versions of the code can work with the schema. You run the migration as a Kubernetes Job before the deployment — in Helm you'd use a pre-upgrade hook. Deploy, done. HOST_A: What about removing a column or changing a column's type? HOST_B: That's where you need the expand-contract pattern. Phase one: add the new column. Old code doesn't know about it — that's fine, it ignores it. Phase two: deploy new code that writes to both old and new columns simultaneously. Phase three: backfill the new column from the old one. Phase four: deploy code that only uses the new column. Phase five: drop the old column. You never have a moment where the running code is incompatible with the database schema. HOST_A: The failure mode you must avoid? HOST_B: Running a migration that adds a NOT NULL column without a default, while old pods are still running. The old code doesn't know to populate that column, inserts start failing, and you have an immediate outage. The rolling update half-and-half state is exactly when this bites you. HOST_A: Multi-tenant Kubernetes cluster design. How do you approach it? HOST_B: Namespace per tenant is the baseline. Each tenant gets their own namespace, a ResourceQuota to cap their CPU and memory consumption, a LimitRange to set defaults so pods don't go without limits. NetworkPolicy to deny cross-namespace traffic by default. RBAC giving each tenant's service accounts access only to their own namespace. HOST_A: Is that sufficient for untrusted tenants? HOST_B: No. Namespaces are soft isolation. They're an organisational boundary, not a security boundary. A misconfigured pod can still request cluster-wide resources. A ClusterRole binding, for example, can give a pod admin rights over the entire cluster. Kernel exploits in the container runtime can escape to the node. HOST_A: So what do you do for truly untrusted tenants? HOST_B: Separate node pools per tenant using taints and tolerations — so tenant workloads can't share nodes. Pod Security Standards set to Restricted — blocking privileged containers, hostPID, hostNetwork, running as root. And for the highest isolation requirements: separate clusters entirely, or virtual clusters using something like vcluster, which gives each tenant the appearance of their own Kubernetes API server. HOST_A: Scheduler internals. How does Kubernetes decide where to place a pod? HOST_B: Two phases: filtering and scoring. Filtering eliminates nodes that can't run the pod — not enough CPU or memory to satisfy requests, missing the right labels for nodeSelector, having a taint the pod doesn't tolerate, violating affinity rules. After filtering you have a set of candidate nodes. HOST_A: Then scoring? HOST_B: The scheduler scores each candidate and picks the highest. Scoring considers resource balance — it prefers nodes with more headroom so the cluster stays evenly loaded. Pod affinity spreads or concentrates pods as specified. Image locality gives a small bonus to nodes that already have the image cached. HOST_A: Walk me through the affinity and spread primitives. HOST_B: nodeSelector is the simplest — a hard requirement on a specific node label. nodeAffinity is more expressive — required versus preferred, supports operators like In, NotIn, Exists. podAffinity says "place this pod near pods matching this selector" — useful for co-locating services that need low latency. podAntiAffinity says "don't place this pod near pods matching this selector" — the classic use is spreading replicas across availability zones so you don't lose half your deployment if one zone goes down. HOST_A: And TopologySpreadConstraints? HOST_B: The more modern way to express spreading. You say "spread these pods across zones, maximum skew of one" — meaning the difference between the most loaded and least loaded zone is at most one pod. It's more flexible and more predictable than podAntiAffinity for this use case. HOST_A: Taints and tolerations? HOST_B: Taints are on the node side — "this node rejects pods unless they tolerate this taint." Tolerations are on the pod side — "I'm okay with nodes that have this taint." The classic use cases: dedicated GPU nodes that should only run GPU workloads, spot instances that should only run fault-tolerant workloads, or isolating tenant workloads to specific node pools. HOST_A: Security. What are the implications of hostPID, hostNetwork, or running as root? HOST_B: These are potential container escape vectors. hostPID means the container can see all processes running on the node — including processes in other pods. You can inspect their command lines, send signals. That's a serious privilege escalation vector. HOST_A: hostNetwork? HOST_B: The container has access to the node's actual network interfaces rather than the virtual network interface inside the pod. It can sniff traffic on the node, bind to node ports, and potentially access metadata services that are normally host-only. HOST_A: Running as root? HOST_B: If there's a container escape — a kernel exploit, a misconfiguration — and you're running as root, the attacker has root on the node. Game over for that node and everything it runs. Running as a non-root user with a read-only filesystem dramatically reduces the blast radius. HOST_A: What controls these in modern Kubernetes? HOST_B: Pod Security Standards. Three levels: Privileged — no restrictions. Baseline — blocks the most obviously dangerous settings. Restricted — blocks everything that's not necessary for most applications. For production workloads you want Restricted as the cluster default, with explicit exceptions for things like monitoring agents that legitimately need host access. HOST_A: Let's shift to behavioural questions. These are the ones that actually reveal whether someone has operated Kubernetes in production or just read about it. HOST_B: And they're the questions I actually find interesting to ask and answer. Because every incident story is unique. HOST_A: Tell me about a time Kubernetes caused an incident. What happened and what did you learn? HOST_B: There was a cert-manager incident that comes to mind. We were using cert-manager to automatically renew Let's Encrypt certificates. The certificate was set to renew thirty days before expiry — standard setup. Except cert-manager had been failing to renew silently for two months because of a misconfigured webhook. We didn't notice because the certificate was still valid. Until it expired on a Friday night. HOST_A: Classic. HOST_B: The service went down for about forty minutes while we diagnosed it — initially we thought it was a deployment issue, then an Ingress issue, before we finally checked the TLS certificate and found it expired. The lesson: certificate expiry monitoring is completely separate from cert-manager health monitoring. We added explicit alerts for certificates expiring within seven days, and a separate alert for cert-manager renewal failures. HOST_A: Another good one that comes up a lot — PDB blocking an upgrade. HOST_B: Yeah. We had a service with `minAvailable: 3` and exactly three replicas. The cluster upgrade needed to drain nodes. Every drain attempt was blocked because draining any node would evict a pod and leave only two. The upgrade was completely stuck for three hours before we realised what was happening. The fix was temporarily scaling to five replicas before the upgrade — but the real lesson is that PDB values should be defined in terms of percentages, not absolute numbers, and always tested against your replica count before an upgrade. HOST_A: When would you choose ECS over Kubernetes? This is the kind of question that tests whether someone has genuine judgment or just dogma. HOST_B: I'd choose ECS for smaller teams running AWS-only workloads who want less operational overhead. ECS is genuinely simpler — you don't need to understand all the Kubernetes abstractions, there's no control plane to manage, and the AWS integration is native and polished. If you're running a handful of services and you want to move fast without becoming a Kubernetes expert, ECS is a completely legitimate choice. HOST_A: When does Kubernetes win? HOST_B: When you need portability across clouds or on-prem. When you need complex scheduling — node affinity, topology spread, GPU workloads. When you want the rich ecosystem — Helm, ArgoCD, Prometheus Operator, all of that. When your team is already Kubernetes-native and the investment has been made. Or when ECS has genuinely hit a wall for your use case, which does happen at scale. HOST_A: The honest answer includes both — it's not "always Kubernetes." I've seen teams destroy themselves trying to run Kubernetes when ECS would have served them perfectly well for three years. HOST_B: The bias toward Kubernetes in job postings has created this culture where everyone feels like they need to be running K8s even when they don't. HOST_A: Right. And the interview question "when would you NOT use Kubernetes" is specifically designed to surface that judgment. If a candidate can't answer it, that's a signal. HOST_B: Let's talk about what to ask the interviewer. Because the questions you ask say as much about your experience as the answers you give. HOST_A: The first one I always ask: "How do you manage Kubernetes version upgrades? What's your current version lag?" HOST_B: That question is a treasure trove. Kubernetes releases three minor versions per year and supports each for about fourteen months. If they're three versions behind, that's a signal — either they don't prioritise operational hygiene, or upgrades are painful for them, which tells you something about their cluster health. HOST_A: "Are you running managed Kubernetes or self-managed? And if self-managed, how do you handle etcd backups?" HOST_B: If they look blank at the etcd backup question and they're running self-managed, that's a red flag. etcd backup on self-managed clusters is not optional. HOST_A: "What's your GitOps story? ArgoCD, Flux, or are you running manual kubectl apply in CI?" HOST_B: This tells you about their deployment practices, their security posture, whether they have audit trails for what's deployed. GitOps is the modern baseline — if they're running `kubectl apply` directly in CI pipelines without any state management, that's technical debt. HOST_A: "How do you handle secrets? Native Kubernetes Secrets, HashiCorp Vault, External Secrets Operator?" HOST_B: As we discussed, native K8s Secrets without encryption at rest are not really secure. If they say "we just use Kubernetes Secrets" and you dig in and find no encryption at rest, no external secret manager, that's a real operational gap. HOST_A: "What's your incident response process for Kubernetes issues? Do you have runbooks?" HOST_B: The runbook question is revealing. Mature teams have runbooks for the known failure modes — CrashLoopBackOff, OOMKilled nodes, etcd issues. If there are no runbooks, every incident starts from scratch. HOST_A: "How do you handle multi-tenancy or team isolation within the cluster?" HOST_B: This tells you about their security posture and their operational maturity. Are teams stepping on each other? Is there any resource governance? Are namespaces just an organisational label or are they enforced boundaries? HOST_A: Let's do the synthesis. What do K8s interviews actually test, and what do they over-test? HOST_B: What they should test: can you explain the reconciliation loop? Do you understand why things work the way they do — not just what commands to run? Can you reason through an incident? Can you articulate trade-offs? HOST_A: The reconciliation loop is worth dwelling on. It's not just a concept — it's the mental model for everything. Every weird Kubernetes behaviour, every debugging session, every design decision flows from "what is the desired state, what is the actual state, and what action brings them together." HOST_B: Operational judgment. "How would you debug a pod in CrashLoopBackOff?" That question has a path — logs, describe, events, previous container logs — and whether you follow that path efficiently tells the interviewer a lot. HOST_A: Trade-off thinking. Namespaces versus separate clusters for multi-tenancy. Kubernetes versus ECS. StatefulSets versus a managed database. The willingness to say "it depends, here's how I think about it" is more valuable than the "right" answer. HOST_B: What gets over-tested? HOST_A: YAML syntax. The exact format of an affinity rule. Specific flag names. You can look all of that up in ten seconds with `kubectl --help` or the Kubernetes documentation. Quizzing on it tells you nothing about operational capability. HOST_B: My genuine hot take: interviewers who quiz on obscure CLI flags are actually signalling their own inexperience. Experienced engineers know they don't need to memorise flags because they always have the documentation available. Testing that knowledge is cargo-culting what they think a "hard" question looks like. HOST_A: Harsh but fair. The best signal you can get in a K8s interview is an incident story with real detail. Not "we had a memory issue," but "here's the exact failure mode, here's how we diagnosed it, here's what we changed, here's what we monitor now." That level of specificity only comes from having actually been in the weeds. HOST_B: And it's very hard to fake. Tutorial knowledge runs out quickly when you start asking follow-up questions. HOST_A: If I could only ask one question to a senior Kubernetes candidate, it would be: "Tell me about the worst Kubernetes incident you've been part of." And then I'd just keep asking "why" and "then what." HOST_B: That's the whole interview, honestly. One question, forty-five minutes of follow-up. HOST_A: For everyone listening: the preparation strategy that actually works is understanding the reconciliation loop deeply, building something real that breaks in interesting ways, and practising your incident stories until you can tell them with specific detail. HOST_B: And if you're interviewing for senior roles, be ready to design. Multi-tenant clusters, zero-downtime database migrations, scheduling for mixed workloads. Those system design questions separate mid from senior. HOST_A: Thanks for joining us on Clawd Talks. Good luck with the interviews — and remember, the goal isn't to know every flag. It's to think clearly under pressure. HOST_B: Which, if you think about it, is exactly what running Kubernetes in production requires anyway. HOST_A: Exactly. Until next time. HOST_B: Cheers.