Liveness probes are preventing scaling to zero

[adamchalat]

I've a problem with scalling down to zero application when using liveness probe as below (if I remove this manifest section, everything works fine).

livenessProbe:
    failureThreshold: 3
    httpGet:
      path: /
      port: http
      scheme: HTTP
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1

Annotations:

  template:
    metadata:
      annotations:
        zeropod.ctrox.dev/disable-checkpointing: 'true'
        zeropod.ctrox.dev/scaledown-duration: 60s
      labels:
        istio.io/dataplane-mode: none

[ctrox]

Just to be sure, are you on the latest version v0.7.0? Can you share some more information about your environment (k8s version, CNI etc.)?

[adamchalat]

I'm using latest version: image: ghcr.io/ctrox/zeropod-manager:v0.7.0 deployed using
kubectl apply -k https://github.com/ctrox/zeropod/config/production without any additional changes.
Kubernetes: AWS EKS 1.33 with Amazon VPC CNI 1.19.6

I've tried increasing zeropod.ctrox.dev/probe-buffer-size and setting zeropod.ctrox.dev/disable-probe-detection strict to false without success.

We are using Istio, but I've disabled it for this specific service and Karpenter.

[ctrox]

Thanks for the details, I have a hunch why this might not work and it probably has to do with some EKS specifics. To detect kubelet probes zeropod looks for traffic from the kubelet process but if that process name differs or if there's another proxy or anything like that in between it won't work. For example, for k3s the manager binary is started with the flag -probe-binary-name=k3s as it bundles kubelet into the k3s process. EKS might be doing something similar but I'll have to setup a test cluster to debug this.

[ctrox]

So I setup an EKS cluster to try and reproduce this but everything worked as expected for me. EKS Auto Nodes (bottlerocket) don't work right now because of the locked-down file system but that's another topic.

With normal node group running Amazon Linux 2023.8.20250818, I get normal scaling behaviour and liveness probes are being intercepted.

Sep 11 18:05:42 ip-172-31-20-134.eu-central-1.compute.internal containerd[16342]: time="2025-09-11T18:05:42.074165428Z" level=debug msg="detected HTTP kube-probe request, responding" runtime=io.containerd.zeropod.v2

This is the containerd debug log on the node. Can you get the containerd logs of the node where your pod is running? It should be useful even without debug logging enabled.

One more thing you can try, set zeropod.ctrox.dev/scaledown-duration: 5s on the pod and then add initialDelaySeconds: 10 to the livenessProbe. This should result in the container scaling down before the first probe hits. I want to see if the container stays in scaled-down state as long as no other traffic comes in or if the probe also wakes it.

[adamchalat]

When I set zeropod.ctrox.dev/scaledown-duration: 5s on the pod and then added initialDelaySeconds: 10 to the livenessProbe eveything is working as expected - service is scalled to zero until I reach ingress endpoint.

I checked containerd for pod:

Sep 12 07:09:48 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:48.186916576Z" level=info msg="last activity was 8.001098576s ago" runtime=io.containerd.zeropod.v2
Sep 12 07:09:48 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:48.186947968Z" level=info msg="delaying scale down by 11.998855224s" runtime=io.containerd.zeropod.v2
Sep 12 07:09:53 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:53.045387234Z" level=debug msg="ExecSync for \"d6a2690042f0da0d0605fada1d77508eb97836574fc2bf044d687134a93014fc\" with command [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] and timeout 10 (s)"
Sep 12 07:09:53 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:53.045917101Z" level=debug msg="Generated exec id \"a4753df0ae0a09305839f841bf73436a9d6f9dc68a2ed1f4ffbfdb547a9a2412\" for container \"d6a2690042f0da0d0605fada1d77508eb97836574fc2bf044d687134a93014fc\""
Sep 12 07:09:53 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:53.136789251Z" level=debug msg="Finish piping \"stderr\" of container exec \"a4753df0ae0a09305839f841bf73436a9d6f9dc68a2ed1f4ffbfdb547a9a2412\""
Sep 12 07:09:53 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:53.136809146Z" level=debug msg="Finish piping \"stdout\" of container exec \"a4753df0ae0a09305839f841bf73436a9d6f9dc68a2ed1f4ffbfdb547a9a2412\""
Sep 12 07:09:53 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:53.137076815Z" level=debug msg="Exec process \"a4753df0ae0a09305839f841bf73436a9d6f9dc68a2ed1f4ffbfdb547a9a2412\" exits withexit code 0 and error <nil>"
Sep 12 07:09:56 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:56.512851801Z" level=debug msg="ExecSync for \"d6a2690042f0da0d0605fada1d77508eb97836574fc2bf044d687134a93014fc\" with command [/app/grpc-health-probe -addr=:50051 -connect-timeout=5s -rpc-timeout=5s] and timeout 10 (s)"
Sep 12 07:09:56 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:56.513518124Z" level=debug msg="Generated exec id \"ef8abf62211473b4101255be09880f5f479a798e69eb7e945584f32b063777d0\" for container \"d6a2690042f0da0d0605fada1d77508eb97836574fc2bf044d687134a93014fc\""
Sep 12 07:09:56 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:56.604643510Z" level=debug msg="Finish piping \"stdout\" of container exec \"ef8abf62211473b4101255be09880f5f479a798e69eb7e945584f32b063777d0\""
Sep 12 07:09:56 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:56.604651480Z" level=debug msg="Finish piping \"stderr\" of container exec \"ef8abf62211473b4101255be09880f5f479a798e69eb7e945584f32b063777d0\""
Sep 12 07:09:56 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:09:56.604788132Z" level=debug msg="Exec process \"ef8abf62211473b4101255be09880f5f479a798e69eb7e945584f32b063777d0\" exits withexit code 0 and error <nil>"
Sep 12 07:10:00 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:10:00.192955538Z" level=info msg="last activity was 8.006861114s ago" runtime=io.containerd.zeropod.v2
Sep 12 07:10:00 ip-10-0-27-XXX.ec2.internal containerd[47343]: time="2025-09-12T07:10:00.192984456Z" level=info msg="delaying scale down by 11.993101106s" runtime=io.containerd.zeropod.v2

The above containerd ID is coming from kube-system and aws-node.

Application is logging:

search-ui-6944fd5cf7-jbj9t search-ui ::ffff:10.0.24.130 - - [12/Sep/2025:06:34:43 +0000] "GET / HTTP/1.1" 200 605 "-" "kube-probe/1.33+"

Zeropod logs:

zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.660573223Z","level":"INFO","msg":"status event","component":"podlabeller","container":"search-ui","pod":"search-ui-74dbf8896-s9jqr","namespace":"default","phase":1}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.660670485Z","level":"INFO","msg":"status event","component":"podscaler","container":"search-ui","pod":"search-ui-74dbf8896-s9jqr","namespace":"default","phase":1}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.660729778Z","level":"DEBUG","msg":"ignoring container without resources","component":"podscaler","container":"search-ui","pod":"search-ui-74dbf8896-s9jqr","namespace":"default","phase":1}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.660753586Z","level":"INFO","msg":"status event","component":"event_creator","container":"search-ui","pod":"search-ui-74dbf8896-s9jqr","namespace":"default","phase":1}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.662260817Z","level":"INFO","msg":"attaching redirector for sandbox","pid":53766,"links":["eth0","lo"]}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.666027564Z","level":"INFO","msg":"tracking sandbox IP","addr":"10.0.27.255"}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.76775118Z","level":"INFO","msg":"cleaning up redirector","pid":52523}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.768161596Z","level":"INFO","msg":"deleting","path":"/sys/fs/bpf/zeropod_maps/52523"}
zeropod-node-pkvxq manager {"time":"2025-09-12T07:24:18.86864891Z","level":"INFO","msg":"subscribe closed","sock":"/run/zeropod/s/0f6758a3c5ae786fa3ace99f0ff7b44b61018bfab7b32c5ce087f003134c939e.sock"}
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installed criu binaries from ghcr.io/ctrox/zeropod-criu:v4.1
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installing runtime for containerd
zeropod-node-pkvxq installer 2025/09/12 07:20:57 configuring containerd 1.7.27
zeropod-node-pkvxq installer 2025/09/12 07:20:57 runtime already configured, no need to restart containerd
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installed runtime
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installed runtimeClass
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installed ca cert
zeropod-node-pkvxq installer 2025/09/12 07:20:57 installer completed

I don't see anything related to HTTP probes in zeropod or containerd logs for my app

Container probes:

    Liveness:             http-get http://:http/ delay=0s timeout=13s period=12s #success=1 #failure=10
    Startup:              http-get http://:http/ delay=0s timeout=1s period=5s #success=1 #failure=18

Maybe the problem is caused by AWS ALB/TargetGoup?

[ctrox]

I think I found the issue. What tipped me off was the IP address of kubelet from your log line: ::ffff:10.0.24.130. This is an IPv4 address embedded in a v6 one. I was able to reproduce your issue by simply running a container which opens an IPv6 socket (but uses an IPv4 pod address). I just pushed an attempt to fix this, can you please switch out the zeropod-manager image to ghcr.io/ctrox/zeropod-manager:dev-979fc12 to test the fix?