Part 1 – Scale From Zero Node Pools in NKP: How It Works and How to Configure It

Introduction

Kubernetes clusters are often overprovisioned, with dedicated node pools created for batch workloads, data processing, CI pipelines, GPU jobs, seasonal traffic, or specialized application tiers. These pools offer isolation and flexibility, but they often remain underutilized for extended periods, quietly accumulating infrastructure costs without delivering value.

The Nutanix Kubernetes Platform (NKP) supports scaling node pools from zero. Instead of keeping capacity provisioned by default, NKP allows node pools to be defined with a minimum size of zero. When a workload demands infrastructure resources, NKP automatically provisions additional capacity. This shifts the cluster from static, pre-allocated infrastructure to elastic capacity, aligning operational costs directly with actual workload demand.

This post is Part 1 of a two-part series on Scale From Zero in NKP. Here we explain how it works and how to configure it. In Part 2, we’ll focus on testing and validating the setup.

---

How Scale From Zero Works in NKP

NKP leverages upstream Cluster Autoscaler on Cluster API. Autoscaling from zero is an opt-in enhancement that infrastructure providers can implement. The method described in this blog is agnostic to the infrastructure provider, meaning it should work with any provider included in NKP.

Disclaimer: This method has only been tested with the Cluster API provider for Nutanix AHV (CAPX)

At a high level, the flow is:

1. You configure:
   • A node pool with a minimum of zero and a maximum size.
   • You add extra configuration to the generated MachineDeployment:
     1. Capacity annotations.
     2. Labels.
     3. Taints (anti-affinity).
2. When you deploy a workload matching the node pool labels and taints, Cluster Autoscaler:
   • Uses capacity annotations to simulate node resources, since no real nodes exist yet.
   • Validates that the pod would fit
   • Scales the MachineDeployment
   • Triggers node creation
3. When workloads disappear:
   • Nodes become “unneeded.”
   • After cooldown timers:
     1. Machines are drained and deleted
     2. Replicas return to 0

---

Step-by-Step: Creating a Scale From Zero Pool in NKP

Prerequisites

• NKP cluster (any edition) running on Nutanix AHV
• NKP CLI
• yq (version 4 – install guide)

Step 1 — Create a Node Pool

Let’s start by creating a .env file with all the required variables and values for the different steps you’ll be walking through.

Note: All the commands must be executed against the NKP Management cluster, which hosts the Cluster API resources.

# .env file

# If you don't know the cluster name run: kubectl get cluster -A
# NAMESPACE   NAME          CLUSTERCLASS          PHASE         AGE   VERSION
# default     nkp-2-nodes   nkp-nutanix-v2.17.0   Provisioned   34h   v1.34.1
export CLUSTER=<cluster_name>

# Namespace for your NKP cluster
export NAMESPACE=<cluster_namespace>

# Nutanix AHV cluster name
export NUTANIX_CLUSTER=<prism_element_cluster_name>

# Name for the node pool. Ex.: scale-from-zero
export NODEPOOL_NAME=<nodepool_name>

# Infrastructure resources
export NODEPOOL_VCPU=8
export NODEPOOL_MEMORY=32
export NODEPOOL_SUBNET=<subnet_name>

# Must match the name with the image already uploaded in Prism Central. Ex.: nkp-rocky-9.6-release-cis-1.34.1-20251206060914.qcow2
export NODEPOOL_VM_IMAGE=<vm_image>

# Generated manifest file
export FILE=$CLUSTER-scale-from-zero-nodepool.yaml

With the environment variables file ready, let’s generate an updated Cluster manifest that includes the new node pool. We are doing a --dry-run because we need to make some tweaks to the manifest before we can apply it.

source .env

nkp create nodepool nutanix $NODEPOOL_NAME \
  --namespace $NAMESPACE \
  --cluster-name $CLUSTER \
  --prism-element-cluster $NUTANIX_CLUSTER \
  --vcpus $NODEPOOL_VCPU \
  --memory $NODEPOOL_MEMORY \
  --subnets $NODEPOOL_SUBNET \
  --vm-image $NODEPOOL_VM_IMAGE \
  --replicas 0 \
  --dry-run \
  --output yaml > $FILE

If you take a look at the generated manifest file, you’ll see at the bottom of it the addition of the new node pool.

Step 2 — Add Extra Configuration

Before applying it, we must add the following information to our MachineDeployment (new node pool):

• Max size configuration
• Autoscaler capacity annotations
• Label
• Taint

These are required for scale-from-zero to work correctly.

To ensure the changes are applied correctly, we’ll be using the following yq snippet. It will:

• Filter by the new MachineDeployment
• Add the following capacity annotations (values will be different in your case):
  • capacity.cluster-autoscaler.kubernetes.io/cpu: 8
  • capacity.cluster-autoscaler.kubernetes.io/memory: 32G
  • capacity.cluster-autoscaler.kubernetes.io/labels: node-role.kubernetes.io/worker=demo
  • capacity.cluster-autoscaler.kubernetes.io/taints: dedicated=demo:NoSchedule
• Update the annotation cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size
• Add a metadata label to the MachineDeployment with the API node-role.kubernetes.io to ensure it propagates to the nodes when deployed
• Add the taint dedicated=demo:NoSchedule to the workerConfig variables

source .env

yq -i '
  (.spec.topology.workers.machineDeployments[]
    | select(.name == env(NODEPOOL_NAME))
  ) |= (
    # --- autoscaler annotations ---
    .metadata.annotations |= (
      . + {
        "capacity.cluster-autoscaler.kubernetes.io/cpu": strenv(NODEPOOL_VCPU),
        "capacity.cluster-autoscaler.kubernetes.io/memory": "\(env(NODEPOOL_MEMORY))G",
        "capacity.cluster-autoscaler.kubernetes.io/labels": "node-role.kubernetes.io/worker=\(env(NODEPOOL_NAME))",
        "capacity.cluster-autoscaler.kubernetes.io/taints": "dedicated=\(env(NODEPOOL_NAME)):NoSchedule"
      }
      | .["cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size"] = strenv(NODEPOOL_MAX_SIZE)
    )
    |
    # --- labels ---
    .metadata.labels |= (
      . + {
        "node-role.kubernetes.io/worker": env(NODEPOOL_NAME)
      }
    )
    |
    # --- persistent taints ---
    (.variables.overrides[]
      | select(.name == "workerConfig")
      | .value.taints
    ) = [
      {
        "effect": "NoSchedule",
        "key": "dedicated",
        "value": env(NODEPOOL_NAME)
      }
    ]
  )
' "$FILE"

Before applying the new changes, confirm that the manifest file has the correct annotations, label, and taint.

cat $FILE

If everything looks alright, now it’s time to apply the manifest.

kubectl apply -f $FILE

If you check the MachineDeployments, you’ll see the new node pool created with no values.

kubectl get machinedeployment --namespace $NAMESPACE

Why Capacity Annotations Matter

When replicas = 0, no nodes exist. Autoscaler must simulate scheduling against a virtual template node. Without CPU and memory capacity annotations, scale-from-zero will fail.

---

Final Thoughts

Scale from zero node pools is one of the most effective infrastructure optimizations available in Kubernetes today. When implemented correctly, they allow platforms to align infrastructure with real workload demand and eliminate idle compute resources.

However, this capability depends on understanding how the autoscaler evaluates node capacity, how scaling from zero works, and how to correctly configure node pools so the scheduler can make safe scaling decisions.

With the fundamentals and configuration in place, the next step is to validate that scaling behaves as expected in real-world scenarios.

In Part 2, we will explain how to test scale from zero behavior and verify that workloads correctly trigger node creation.