Easing Kubernetes Scalability: Server-Side Sharding for List and Watch in v1.36

Introduction

As Kubernetes clusters expand to accommodate tens of thousands of nodes, controllers that depend on monitoring high-cardinality resources—like Pods—encounter a significant scaling bottleneck. Each instance of a horizontally scaled controller receives the complete stream of events from the API server, consuming CPU, memory, and network bandwidth just to deserialize everything, only to discard objects that fall outside its responsibility. Simply adding more replicas exacerbates the cost rather than easing it. Kubernetes v1.36 introduces an alpha feature (KEP-5866) that moves event filtering upstream into the API server, so each replica receives only the events it truly needs.

The Problem with Client-Side Sharding

Some controllers, such as kube-state-metrics, already implement horizontal sharding on the client side. Each replica is assigned a specific slice of the key space and discards objects not in its designated range. While this enables workload distribution, it does nothing to reduce the data flowing from the API server. Every replica still has to handle the entire event stream, meaning:

• N replicas × full event stream – Each instance deserializes and processes every event, then throws away what it doesn’t own.
• Network bandwidth scales with replicas, not with shard size.
• CPU spent on deserialization is wasted for the discarded fraction.

The fundamental issue is that filtering happens too late, after the data has already been transmitted and parsed. Server-side sharding solves this by pushing the filter into the API server itself.

How Server-Side Sharding Works

With the new feature, each controller replica tells the API server which hash range it owns. The API server then sends only the matching events, both in list responses and during watch streams.

The shardSelector Field

A new field, shardSelector, is added to ListOptions. Clients specify a hash range using the shardRange() function. For example:

shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')

The API server computes a deterministic 64-bit FNV-1a hash of the specified field path and returns only objects whose hash falls within the inclusive start to exclusive end range ([start, end)). This ensures consistent results across all API server replicas, making the feature safe in multi-instance API server environments.

Supported Fields and Hash Function

Currently, the supported field paths are object.metadata.uid and object.metadata.namespace. The FNV-1a hash provides a fast, uniformly distributed mapping that splits the key space into equal-sized shards. Controllers can use any range that partitions the entire 64-bit space.

Implementing Sharded Watches in Controllers

Controllers that rely on informers to list and watch resources can easily adopt server-side sharding by injecting the shardSelector into the ListOptions when creating a shared informer factory.

Using Informers with WithTweakListOptions

The informers.NewSharedInformerFactoryWithOptions function accepts a WithTweakListOptions callback. Inside that callback, set the ShardSelector field. For example:

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/informers"
)

shardSelector := "shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')"

factory := informers.NewSharedInformerFactoryWithOptions(client, resyncPeriod,
    informers.WithTweakListOptions(func(opts *metav1.ListOptions) {
        opts.ShardSelector = shardSelector
    }),
)

Each replica uses a different hash range. For a two-replica deployment, you split the key space in half:

• Replica 0: 0x0000000000000000 to 0x8000000000000000
• Replica 1: 0x8000000000000000 to 0xFFFFFFFFFFFFFFFF

Note that the ranges are exclusive at the end, so the full space is covered by these two intervals.

Benefits and Considerations

Server-side sharding dramatically reduces the volume of data transmitted and processed. Controllers no longer need to deserialize events they will discard, lowering CPU and memory usage. Network bandwidth scales with shard size rather than replica count, making horizontal scaling truly efficient.

However, there are a few things to keep in mind:

• This is an alpha feature in v1.36; it may change or be removed in future releases.
• Currently only uid and namespace are supported field paths—other fields like name or labels are not yet available.
• Controllers must be aware of the sharding scheme to ensure all objects are covered across replicas.
• Resharding (adding or removing replicas) requires updating the sets of ranges, which may cause temporary duplication or missed events during transition.

Despite these limitations, server-side sharding represents a significant step forward for large-scale Kubernetes deployments, enabling controllers to maintain responsiveness without wasting resources.

Conclusion

Kubernetes v1.36’s server-side sharded list and watch feature addresses a long-standing scaling challenge for controllers monitoring high-cardinality resources. By moving event filtering into the API server, it reduces the per-replica processing overhead and network traffic. Early adopters can test this alpha feature with informers and custom controllers, paving the way for more efficient resource utilization in massive clusters. As the feature matures, it promises to become an essential tool for Kubernetes administrators striving for optimal performance at scale.