Configure admission fair sharing in Red Hat build of Kueue 1.4

In shared AI and high-performance computing clusters, OpenShift platform engineers face constant challenges managing tightly constrained hardware resources like GPUs and high-spec CPUs. When multiple tenants submit workloads that compete for a limited resource quota, the first-in, first-out (FIFO) admission rule can cause long-term starvation of workloads submitted later and reduce service predictability. Red Hat build of Kueue supports 2 ways of enforcing fairness: fair sharing-based preemption and admission fair sharing.

Admission fair sharing helps workloads from different tenants competing for quota from the same shared resource pool get admitted fairly by prioritizing workloads based on each tenant's historical resource use; workloads from tenants that used fewer resources over time are admitted first. Admission fair sharing doesn't evict running workloads, whereas fair sharing-based preemption does. Instead, it leads to fair sharing at admission time.

In this guide, you will configure a shared CPU quota in Red Hat build of Kueue 1.4 and observe how Kueue uses historical usage to decide which tenant's workload gets admitted next.

Configure Red Hat build of Kueue to enable admission fair sharing

To enable admission fair sharing, edit the Kueue custom resource (CR) using either the Default or Custom configuration. The Default configuration enforces the default settings for the admissionFairSharing configuration, which are: 30 minutes for usageHalfLifeTimeSeconds, 5 minutes for usageSamplingIntervalSeconds, and a weight of 1 for resourceWeights for each resource. All these values are subject to change.

Specify the following to use the default:

oc create -f - <<EOF
apiVersion: kueue.openshift.io/v1   
kind: Kueue
metadata:
  name: cluster
  namespace: openshift-kueue-operator
spec:
  managementState: Managed
  config:
    integrations:
      frameworks:
      - BatchJob
    admissionFairSharing:
      configuration: Default
EOF

The Custom configuration allows you to specify user-provided settings, such as:

oc create -f - <<EOF
apiVersion: kueue.openshift.io/v1   
kind: Kueue
metadata:
  name: cluster
  namespace: openshift-kueue-operator
spec:
  managementState: Managed
  config:
    integrations:
      frameworks:
      - BatchJob
    admissionFairSharing:
      configuration: Custom
      custom:
        usageHalfLifeTimeSeconds: 120
        usageSamplingIntervalSeconds: 60
        resourceWeights:
          - name: cpu
            weight: "2.0"
EOF

For more details on each setting, refer to the official documentation.

Following that, create an empty ResourceFlavor that tells Kueue the cluster has a uniform resource pool.

oc create -f - <<EOF
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: default-flavor
EOF

Create a ClusterQueue, which manages the overall cluster quota with an admissionScope set for UsageBasedAdmissionFairSharing.

oc create -f - <<EOF
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: afs-default-cq
spec:
  namespaceSelector: {}
  admissionScope:
    admissionMode: UsageBasedAdmissionFairSharing
  resourceGroups:
  - coveredResources: ["cpu"]
    flavors:
    - name: default-flavor
      resources:
      - name: cpu
        nominalQuota: "1"
EOF

Create the namespace and LocalQueue instances, where each will contain workloads closely related to each other. Optionally, you can define the weight of the LocalQueue. If the weight is higher, the LocalQueue appears to have consumed fewer resources than it has. You can find more details in the official documentation.

oc create namespace afs-test && oc label namespace afs-test kueue.openshift.io/managed=true;
for lq in lq-a lq-b; do
  oc create -f - <<EOF
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: $lq
  namespace: afs-test
spec:
  clusterQueue: afs-default-cq
EOF
done

Create the first 2 jobs targeting the LocalQueue lq-a, then 1 job targeting the LocalQueue lq-b.

for queue in lq-a lq-a lq-b; do
  name="job$((++i))"
  oc create -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: $name
  namespace: afs-test
  labels:
    kueue.x-k8s.io/queue-name: $queue
spec:
  suspend: true
  template:
    spec:
      containers:
      - name: example-job
        image: busybox
        command: ["sh", "-c", "sleep infinity"]
        resources:
          requests:
            cpu: "1"
      restartPolicy: Never
EOF
done

Observe that the first submitted job is admitted and subsequent jobs remain pending because the first admitted job uses all the quota.

When the job from lq-a is admitted, Kueue adds a temporary value—a penalty—to the lq-a usage tracking before usage statistics are updated. This prevents Kueue from admitting many workloads submitted at once from the same tenant.

Observe that consumedResources on the LocalQueue updates with historical resource consumption.

oc get lq lq-a -n afs-test -o jsonpath={.status.fairSharing}

The output is similar to the following:

{"admissionFairSharingStatus":{"consumedResources":{"cpu":"256m"},"lastUpdate":"2026-07-23T19:41:21Z"},"weightedShare":0}

Eventually, when quota becomes available, the scheduler decides which pending workload to admit next. It computes resource use for each LocalQueue by combining consumedResources with the penalty and dividing by the LocalQueue's weight. The job from the lowest-use LocalQueue is admitted first.

Delete the admitted job from lq-a and observe that the job from lq-b is admitted.

oc delete job job1 -n afs-test

On the following usageSamplingIntervalSeconds tick, consumedResources from lq-a start to decrease. The rate is controlled by usageHalfLifeTimeSeconds, and consumedResources from lq-b start to increase. Observe lq-b increase with the following command:

oc get lq lq-b -n afs-test -o jsonpath={.status.fairSharing}

To disable the admission fair sharing feature, remove the admissionFairSharing field specified in the Kueue CR.

Additional considerations

When admission fair sharing is enabled, usage-based ordering across LocalQueue instances takes precedence over priority classes. Priority classes still apply as a tiebreaker when LocalQueue instances have equal usage.

Additionally, note that admission fair sharing takes effect only when multiple tenants share the same ClusterQueue. If each tenant uses a dedicated ClusterQueue, there's no cross-tenant contention within a single ClusterQueue, and the feature has no impact on admission order.

Conclusion

Admission fair sharing prevents long-term workload starvation across competing teams without needing to terminate active jobs. It reorders workloads and helps tenants get a proportional share of workload admission over time.

Explore the Red Hat build of Kueue documentation or deploy the operator on an OpenShift staging cluster to start balancing batch workloads today.