Kueue Admission Control

Hawk can install Kueue to admit runner Jobs and sandbox services against shared cluster capacity. Installation and enrollment are separate controls: installing the controller does not change existing submissions. A Job becomes Kueue-managed only when Hawk creates it with a queue label in a namespace explicitly labelled for admission. Changing enrollment affects new and resumed submissions; it does not retroactively enrol existing Jobs.

Set hawk:kueueAdmissionEnabled: "true" to enroll a deployment's scan, eval-set, and human-eval runners together with their standard-chart sandbox services. Admission is disabled by default.

Validate the deployment's budgets and resource shapes before expanding enrollment; see validation scope and rollout.

Ownership and queues

The stack that owns the EKS cluster owns the Kueue installation and its cluster-scoped resources. Hawk pins the controller Helm chart to Kueue 0.19.4. The same stack owns two fixed, shared ClusterQueues:

Queue	Namespace role	Purpose
hawk-runners	runner	Runner Job admission
hawk-sandboxes	sandbox	Standard-chart sandbox services

These are independent quota partitions. They are not a shared pool, do not borrow from each other, and are never recreated for each Hawk deployment. Every enrolled job release creates a default LocalQueue in its own namespace, pointing to the appropriate shared ClusterQueue. Users cannot choose another queue.

Dev environments share the staging EKS cluster, controller, and ClusterQueues. A dev stack reads the owning staging stack's exported queue destinations and controls only its own hawk:kueueAdmissionEnabled setting. Do not put hawk:kueueEnabled or hawk:kueueQueues in a dev stack to create another controller or another set of ClusterQueues.

Configuration

The relevant Pulumi settings are:

Setting	Default	Meaning
hawk:kueueEnabled	false	Install Kueue in a cluster-owning stack. Requires hawk:createEks: "true".
hawk:kueueQueues	{}	Resource budgets for both runners and sandboxes. Both partitions must be present before enrollment.
hawk:kueueAdmissionEnabled	false	Enroll runners and standard-chart sandbox services. Requires the controller and both queue budgets.

Define both quota partitions explicitly, using quantities selected from your own capacity validation:

hawk:kueueQueues:
  runners:
    cpu: <measured-runner-cpu-budget>
    memory: <measured-runner-memory-budget>
  sandboxes:
    cpu: <measured-sandbox-cpu-budget>
    memory: <measured-sandbox-memory-budget>
    ephemeral-storage: <measured-sandbox-ephemeral-storage-budget>

Each queue partition requires positive Kubernetes quantities for cpu and memory. Kueue also needs quota coverage for every other resource requested by its managed Pods, including sandbox ephemeral-storage and GPU resources. Missing resource quota leaves a Workload queued. Hawk accepts additional resource entries; nvidia.com/gpu quantities must be integral. Include sidecars, init containers, and Pod overhead when measuring requests. Select budgets from measured cluster and account capacity. Kueue quotas control admission reservations, while AWS service quotas, subnet addresses, Karpenter NodePool requirements, instance availability, and image architecture still determine whether an admitted Pod can start. A Karpenter upgrade is separate from enabling Kueue.

The installed policy deliberately stays close to native Kueue behavior:

• ClusterQueues use BestEffortFIFO, with no cohort borrowing and no preemption.
• Unlabelled Jobs remain unmanaged (manageJobsWithoutQueueName: false).
• Only namespaces labelled hawk-admission-managed: "true" are managed, and the hawk-admission-role label restricts each namespace to its runner or sandbox queue.
• Kueue's WaitForPodsReady behavior is disabled. Admission is the reservation boundary; ordinary Kubernetes scheduling and Hawk's existing startup deadlines remain responsible for Pod startup.

BestEffortFIFO considers priority and age, but can admit a lower-priority Workload that fits while an older or higher-priority Workload cannot fit. The priority classes are hawk-low (-100), hawk-medium (0), and hawk-high (100). These numeric values are Hawk's convention: their ordering matters, and the spacing has no special scheduling effect. The shared submission option --priority defaults to medium for evals and low for scans. Every launch and explicit resume uses its current option or the job-type default. The API carries that value in InfraConfig.priority; automatic retries reuse it. Within each shared queue, these classes order waiting, enrolled Workloads from every user and Hawk deployment in the cluster. Priority grants no resource entitlement or permission, does not preempt running work, and provides no reservation or start-time guarantee.

Native sandbox admission

With admission enabled, eval-set and human-eval submissions label both the runner and sandbox namespaces and create a default LocalQueue in each, pointing to their separate shared ClusterQueues. Scans still enroll only their runner namespace. The pinned inspect-k8s-sandbox package uses native StatefulSet admission for standard-chart services. Custom charts, extra workload resources, and resources Hawk cannot classify are explicitly rejected when sandbox admission is enabled. Supported static noncompute resources, such as ConfigMaps and Services, remain allowed.

Hawk serializes the queue, submission priority, and trusted runner namespace and Job name into the runner's infrastructure config. User config cannot choose that source. A Role and RoleBinding grant the runner service account only get on its exact runner Job for this lookup. Each new sandbox release reads the Job's current priority label. A failed source read or missing label fails the installation. Existing sandbox releases retain their priority. Relabelling the runner therefore affects later sandbox installations without changing services already installed.

Services reserve quota independently. Two multi-service environments can each hold quota for one service while waiting for another, stalling until an existing timeout or external cancellation releases capacity. This is an accepted limitation of native service admission; Hawk does not make a reservation across services. A quota-specific timeout is a proposal only, neither implemented nor required for enrollment.

Cancelling sandbox initialization

hawk stop EVAL_SET_ID sends a graceful command to the runner. A runner waiting for sandbox initialization may not process that command, so a successful stop request does not prove that its queued sandbox has been cancelled. This initialization path differs from an evaluation already executing in an admitted sandbox.

To cancel a queued initializer, delete the exact eval-set ID:

hawk delete EVAL_SET_ID --yes

Then verify that the eval set's runner and sandbox resources, namespaces, and native Workloads are gone and its quota reservations have drained. Shared queues can still hold reservations for other evaluations. This uses existing deletion behavior; Hawk adds no initialization timeout or recovery controller.

Completed or orphaned runner releases are also eligible for the existing janitor after its one-hour grace period. The janitor uses Helm uninstall, with cleanup permissions for the release's LocalQueues and priority-source Role.

Runner process failure

A killed Inspect process can leave its sandbox releases running. The sandbox library already documents this behavior: cleanup hooks and release tracking belong to the process that created them. Kueue continues reserving quota for those running resources, including after a replacement runner finishes successfully. A retained sandbox can also leave too little quota for the replacement sandbox to start.

After a crashed evaluation has finished, inspect its remaining sandbox resources and use hawk delete EVAL_SET_ID --yes to reclaim them, then verify quota drain. If retained quota blocks recovery, cancel that eval with exact-ID deletion before using hawk eval-set resume EVAL_SET_ID to resume it from its saved state. Cleaning up orphaned sandboxes during an active eval remains an upstream follow-up.

Validation scope

Staging checks covered real scan/eval execution and result import, priority changes, sandbox replacement/retry, checkpoint recovery, explicit resume, and exact-ID cleanup. The janitor's unchanged Helm uninstall removed two LocalQueues and the priority-source Role under its service account. This validated permissions and uninstall, not the elapsed one-hour CronJob eligibility path or automatic crash-orphan reclamation.

Fresh CPU provisioning passed on amd64 and arm64 with default and gVisor runtimes. A staging sizing test held 10,000 queued Workloads for twenty minutes and passed delayed service checks: 100 suspended Jobs and 9,900 gated sandbox Pods across 200 namespaces and LocalQueues, under two zero-quota test ClusterQueues. The controller stayed Ready with no restarts or OOMs. Working-set memory peaked at 1,044 MiB during the 10,000 stage and 1,065 MiB during cleanup; the separate container-lifetime memory high-water mark reached 1,155 MiB. The test used a 512 MiB memory request and a 2 GiB limit. Hawk now sets a 1,536 MiB request and 2 GiB limit based on these observations; CPU remains at the chart defaults of a 500m request and 2 CPU limit.

All test resources, including LocalQueues and sandbox children, were removed and temporary database scaling was restored. Kubernetes API 429/503 errors interrupted two post-cleanup recovery windows; a later continuous ten-minute window passed. These results cover a warm queued backlog. They do not qualify 10,000 running workloads, sustained churn, heterogeneous scheduling costs, or controller restart and cold-cache behavior, nor establish a maximum supported backlog or production budgets. GPU/P-family capacity and live optional metric export remain unvalidated. Multi-service partial admission and retained sandbox quota after runner death remain the accepted limitations described above.

Submission burst guard

Kueue can hold a large backlog, so Hawk also provides an independent submission guard. It is disabled by default:

hawk:submissionGuardEnabled: "false"
hawk:maxOutstandingJobsPerUser: "128"

When enabled, Hawk takes a fresh Kubernetes count immediately before each runner release install. New Jobs carry the deployment label and a stable hashed subject in the submitter label. Pending, queued, and running Jobs count. A Job stops counting when Kubernetes reports a Complete=True or Failed=True condition. Until older Jobs drain, Hawk also counts Jobs that lack the submitter label when their legacy sanitized created-by value matches the submitting subject and they are in the exact expected deployment runner namespace. Sanitization can make two legacy subjects collide, so this fallback may conservatively overcount; Hawk does not relabel the old Jobs.

The guarded CLI paths are hawk eval-set / hawk eval-set run, hawk eval-set resume, hawk scan run, hawk scan resume, and hawk human eval start. --force bypasses only the count threshold for one request. It still performs the fresh count, and it does not bypass authentication, configuration validation, or native admission. A threshold refusal is an HTTP 429 Problem Details response with code outstanding_job_limit, observed_count, and limit. A count failure is HTTP 503.

The guard serializes concurrent submissions only within one API process. Because API replicas and workers do not share that lock, it is an accidental-burst guard rather than an exact distributed quota. Coordinate purposeful large submissions with other cluster users before using --force.

Status and diagnosis

The optional admission field reports runner admission as pending, queued, admitted, or unknown, preserving the existing JobStatus lifecycle. It includes the queue, priority class, current numeric Workload priority, reason, message, and known timestamps. The numeric priority on the current Workload is authoritative.

Pending sandbox samples and pod scheduling diagnostics remain visible. For sandbox admission waits, inspect native Workload and LocalQueue conditions with kubectl; Hawk does not aggregate sandbox queue counts.

Admission snapshots use an optional shared Valkey cache with a five-second lifetime per job, scoped by deployment, namespace prefix, and Job.updated_at. Launch, explicit resume, and deletion advance that version so their next status read bypasses stale entries, including an earlier read that finishes after the change. When Valkey is not configured or a cache operation fails, Hawk falls back to native reads. The admission reader queries Workloads in batches of at most 100 runner UIDs, excluding the sandbox backlog even for large job lists.

A queued Workload is not yet admitted; it can be wholly unreserved or have a QuotaReserved condition while waiting for the separate Admitted condition. Inspect the native Workload conditions and resource usage when that distinction matters. An admitted Workload whose Pod is Unschedulable has passed Kueue but is waiting on Kubernetes or infrastructure capacity. A missing or mismatched LocalQueue, inactive ClusterQueue, controller fault, or permanent scheduling error is an operational fault rather than ordinary quota waiting. If a Kubernetes or Kueue read fails, the corresponding state is reported as unknown/unavailable. A scheduling gate or a generic Pending Pod does not prove quota waiting, because permanent scheduling faults can produce the same surface state.

When admission is unknown and the job lifecycle status is pending, the CLI and viewer display Status unavailable. During a read outage, hawk watch keeps polling until reads recover; it conservatively avoids inferring completion from an earlier attempt's results, which may remain visible after a resume.

Use native resources for detailed diagnosis:

kubectl --context CONTEXT -n RUNNER_NAMESPACE get jobs,workloads,localqueues
kubectl --context CONTEXT -n SANDBOX_NAMESPACE get statefulsets,workloads,localqueues,pods
kubectl --context CONTEXT -n RUNNER_NAMESPACE describe workload WORKLOAD_NAME
kubectl --context CONTEXT describe clusterqueue RUNNER_CLUSTER_QUEUE SANDBOX_CLUSTER_QUEUE

Hawk adds no queue-expiry policy. A high-priority smoke submission still has no reservation or startup guarantee and continues to use the smoke framework's existing startup deadline with force=false.

For a queued runner, a cluster operator can change the current Workload priority by relabelling its Job:

kubectl --context CONTEXT -n RUNNER_NAMESPACE label job JOB_NAME \
  kueue.x-k8s.io/priority-class=hawk-high --overwrite

Kueue updates the same queued Workload. Relabelling an already admitted Job updates priority while preserving its running Pod. This does not preempt other work, change existing sandboxes, or promise that a priority survives Workload deletion and recreation. These operator changes affect cluster admission only; inference requests retain the priority chosen when the job was launched or explicitly resumed.

To change a current sandbox Workload's numeric priority, use its exact name and recorded UID:

kubectl --context CONTEXT -n SANDBOX_NAMESPACE patch workload WORKLOAD_NAME \
  --type=json -p='[
    {"op":"test","path":"/metadata/uid","value":"WORKLOAD_UID"},
    {"op":"replace","path":"/spec/priority","value":37}
  ]'

The patched value belongs to that Workload and does not persist across Workload recreation. A later sandbox installation reads the runner Job's current priority label again.

Telemetry

With hawk:enableDatadog, Kueue's authenticated native OpenMetrics endpoint supplies pending/admitted counts, quota usage, admission wait histograms, and eviction events. The existing annotations filter these to Hawk's queues, flavor, priorities, and supported resource names. Metric export is optional and does not affect admission.

When hawk:cloudwatchDashboardsEnabled is on, the API log group also has a kueue_submission_guard_decisions metric filter. The guard emits structured hawk_submission_guard events for rejected and forced decisions; the filter counts them with bounded Decision and Deployment dimensions. Ordinary accepted submissions do not emit this event.

Rollout and rollback

1. Install the pinned controller with hawk:kueueAdmissionEnabled: "false". Confirm the controller, admission webhooks, and both budgeted ClusterQueues.
2. Validate quotas against AWS limits, Karpenter provisioning, subnet capacity, and the runner and sandbox resource shapes your deployment uses.
3. Set hawk:kueueAdmissionEnabled: "true" in one non-production deployment. Exercise scan and eval execution, sandbox startup, priority ordering, queueing, cancellation, result import, and native cleanup. Check controller unavailability and admitted-but-unschedulable diagnosis, including any configured telemetry.
4. Account for the accepted multi-service partial-admission and crash-cleanup limitations, and measure controller capacity before enabling other deployments.

To roll back, set hawk:kueueAdmissionEnabled: "false" to stop enrolling new submissions. Leave the controller and shared queues in place while existing managed work drains or is cancelled normally; monitoring remains available. Uninstall Kueue only when no managed resources remain. Do not bulk unsuspend Jobs, strip finalizers, delete CRDs, or delete shared ClusterQueues during rollback.