Skip to content

Add k8s settings: prefer-closest-numa-nodes and max-allowable-numa-nodes#848

Open
piyush-jena wants to merge 3 commits intobottlerocket-os:developfrom
piyush-jena:add-k8s-settings
Open

Add k8s settings: prefer-closest-numa-nodes and max-allowable-numa-nodes#848
piyush-jena wants to merge 3 commits intobottlerocket-os:developfrom
piyush-jena:add-k8s-settings

Conversation

@piyush-jena
Copy link
Contributor

@piyush-jena piyush-jena commented Mar 3, 2026

Issue number:

Closes # bottlerocket-os/bottlerocket#4750

Description of changes:
Add 2 topology manager policy options:

  1. max-allowable-numa-nodes - GA k8s-1.35+
  2. prefer-closest-numa-nodes - GA k8s-1.32+

Testing done:
Enabled the following settings and rebooted the instance:

apiclient set \
  kubernetes.cpu-manager-policy="static" \
  kubernetes.topology-manager-policy="best-effort" \
  kubernetes.topology-manager-policy-options.prefer-closest-numa-nodes="true"

Then ran a pod with the following spec:

apiVersion: v1
kind: Pod
metadata:
  name: numa-test
spec:
  containers:
  - name: numa-test
    image: registry.k8s.io/pause:3.9
    resources:
      requests:
        cpu: "4"
        memory: "100Mi"
      limits:
        cpu: "4"
        memory: "100Mi"

Its important that requests and limits are equal for both CPU and memory to ensure it is in the Guaranteed QoS class for the CPU pinning as a result of the static policy to work properly.

The assigned CPUs for this pod was 1,2,65,66.
The NUMA node ranges for this instances are:
node0: 0-31, 64-95
node1: 32-63, 96-127

All 4 CPUs belong to NUMA node0. The prefer-closest-numa-nodes policy worked as expected.

For max-allowable-numa-nodes I show the limits enforced by kubernetes are also applied here.

[ssm-user@control]$ apiclient set settings.kubernetes.topology-manager-policy-options.max-allowable-numa-nodes="4"
Failed to change settings: Failed PATCH request to '/settings/keypair?tx=apiclient-set-Z6sWPcf69TAMvS9h': Status 400 when PATCHing /settings/keypair?tx=apiclient-set-Z6sWPcf69TAMvS9h: Unable to match your input to the data model.  We may not have enough type information.  Please try the --json input form.  Cause: Error during deserialization: Invalid Kubernetes max-allowable-numa-nodes value '4': must be >= 8 at line 1 column 79
[ssm-user@control]$ apiclient set settings.kubernetes.topology-manager-policy-options.max-allowable-numa-nodes="8"

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Update settings-models dependency to pick up the new
topology-manager-policy-options setting type.
Add topologyManagerPolicyOptions rendering to kubelet-config templates,
version-gated by GA status verified from Kubernetes source code:
- prefer-closest-numa-nodes (bool): GA in k8s 1.32+ (added to 1.32-1.35)
- max-allowable-numa-nodes (int, >= 8): GA in k8s 1.35 (added to 1.35 only)

Source: pkg/kubelet/cm/topologymanager/policy_options.go stableOptions set
Signed-off-by: Piyush Jena <jepiyush@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant