Scale OpenShift Dev Spaces in air-gapped environments

Centralizing developer environments helps platform teams deliver consistent tooling while eliminating laptop configuration drift. To deploy Red Hat OpenShift Dev Spaces at enterprise scale, addressing sidecar conflicts and security contexts early prevents silent failures before developers ever encounter them.

The following sections resolve these friction points, detailing how to remediate runtime conflicts, drive configurations through declarative CI/CD pipelines, track capacity with Prometheus ServiceMonitors, and verify cluster readiness for disconnected environments.

Production caveats and technical resolutions

At enterprise scale, OpenShift Dev Spaces often interacts unexpectedly with third-party monitoring agents, security policies, and resource controllers. Addressing these issues during initial design prevents silent failures and emergency troubleshooting after rollout.

Managing cluster integrations and runtime interactions

Platform teams must account for 4 primary friction points during rollout: third-party monitoring agents, rootless container runtimes, custom install namespaces, and admission webhooks.

Dynatrace OneAgent sidecar conflicts

When you configure Dynatrace OneAgent for automated namespace-level injection or cluster-wide DaemonSet monitoring, its admission webhook injects an init container into every newly scheduled pod. This injected init container alters file permissions and volume mounts in ways that conflict with the unique security context and lifecycle of OpenShift Dev Spaces workspace pods. As a result, workspace pods fail to start or crash during container initialization. 

Resolution: Exclude all OpenShift Dev Spaces namespaces (openshift-devspaces, devspaces-checluster, devspaces-openvsx, and every -devspaces sandbox) by applying the dynatrace.com/inject: "false" label.

Rootless Podman and security context constraints (SCCs)

To let developers run podman build or podman run inside their workspace pods without full cluster root privileges, OpenShift Dev Spaces uses Linux kernel user namespaces.

# CheCluster configuration for container build/run capabilities
spec:
  devEnvironments:
    disableContainerRunCapabilities: false  # Enables nested container execution
    containerBuildConfiguration:
      openShiftSecurityContextConstraint: container-build  # Grants SETUID/SETGID for builds
    containerRunConfiguration:
      openShiftSecurityContextConstraint: container-run    # Grants runtime execution permissions
      containerSecurityContext:
        allowPrivilegeEscalation: true  # Required for user namespace mapping
        capabilities:
          add:
            - SETGID  # Maps group IDs inside container user namespaces
            - SETUID  # Maps user IDs inside container user namespaces

Podman socket mapping: A version-specific regression

Build tools such as Gradle or Maven that build container images from inside a workspace need an active Podman UNIX socket to communicate with. In OpenShift Dev Spaces 3.25, the working approach was a manual script that assigned a custom UID/GID subordinate range before starting the socket:

#!/bin/sh
USER=$(whoami)
START_ID=$(( $(id -u)+1 ))
echo "${USER}:${START_ID}:2147483646" > /etc/subuid
echo "${USER}:${START_ID}:2147483646" > /etc/subgid
podman system migrate
podman system service --time=0 unix:///tmp/podman.sock &

After upgrading to 3.26, this same script started failing:

ERRO[0000] running `/usr/bin/newuidmap 7340 0 1000 1 1 1001 2147483646`:
newuidmap: write to uid_map failed: Operation not permitted
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

The root cause—a Linux user namespace has a hard ceiling of 65,536 UIDs, and the old script's range (2,147,483,646) blew straight past it. This was always a latent problem—it fails on 3.25 too once disableContainerRunCapabilities: false is set—but 3.26's stricter enforcement is what surfaced it. The fix depends on which Universal Developer Image (UDI) you're running:

• udi-rhel9:3.22: Still needs a manual script, but with a range that respects the 65,536 ceiling.

  #!/bin/sh
    USER_NAME=$(whoami)
    START_ID=$(( $(id -u)+1 ))
    COUNT=$(( 65536 - ${START_ID} ))
    IDS_RANGE="${USER_NAME}:${START_ID}:${COUNT}"
    echo "${IDS_RANGE}" > /etc/subgid
    echo "${IDS_RANGE}" > /etc/subuid
    podman system migrate
    podman system service --time=0 unix:///tmp/podman.sock &

• udi-rhel9:3.26: Needs no /etc/subuid or /etc/subgid script at all; the image's entrypoint now handles user namespace mapping automatically.

  #!/bin/sh
    podman system service --time=0 unix:///tmp/podman.sock &

Upgrades often surface subtle permissions quirks. If your team uses custom workspace scripts touching /etc/subuid or /etc/subgid, test these mappings in a staging cluster first to catch unexpected behaviors before your developers do.

Stale webhooks after changing the operator's install namespace

If you install the OpenShift Dev Spaces Operator into a non-default namespace and later reinstall it into a different namespace, the operator's admission webhooks from the original install aren't automatically cleaned up. Both the old and new webhook configurations remain registered on the cluster simultaneously, and they conflict:

{"level":"error","logger":"setup","msg":"failed to setup webhooks","error":"conflicting webhook definition found on cluster, webhook mutate.devworkspace-controller.svc clientConfig points at namespace rh-devspaces-operator"}

Resolution: Before reinstalling the operator into a new namespace, explicitly delete the ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects left behind by the previous install. If you're still deciding on a namespace layout, settle on it early—moving the operator's install namespace later is disruptive enough that it's worth avoiding rather than fixing after the fact.

The following table summarizes common production symptoms, root causes, and technical resolutions for Red Hat OpenShift Dev Spaces:

CI/CD pipeline automation and observability

When platform teams scale OpenShift Dev Spaces, automated updates protect developers from unexpected downtime and keep development environments consistent. Managing cluster configuration through ad-hoc oc patch commands or manual web console edits inevitably leads to environment drift across clusters. To maintain platform stability, manage every configuration manifest through an automated CI/CD pipeline alongside real-time cluster monitoring.

Pipeline design principles

A production-grade deployment pipeline for OpenShift Dev Spaces should be built around 3 principles, regardless of which CI/CD tool you're using:

• Idempotency: Every deployment step should be safe to run twice. Applying the same manifest a second time should produce no change, not an error—this is what lets a pipeline recover cleanly from a partial failure or a re-run.
• Change-scoped execution: Rather than reapplying every manifest on every commit, trigger each deployment step only when its specific manifest actually changed. This keeps pipeline runs fast and—more importantly—keeps the blast radius of any single change small: a CheCluster update shouldn't have the chance to also touch your PrometheusRule resources.
• Environment safety: Gate every stage behind an explicit environment check (non-production vs. production), so a change merged for a test cluster can never silently apply itself against production.

In practice, this usually maps to 1 pipeline stage per manifest, each scoped to watch only its own file:

The specific implementation—GitLab CI, GitHub Actions, Jenkins, Tekton, or a GitOps operator such as Argo CD—matters less than making sure each of these 4 concerns stays independently triggerable.

Authentication and provisioning guidelines

Regardless of which CI/CD tool you use, two operational patterns will save your team from major operational headaches down the road:

• Don't store long-lived, highly privileged credentials as static CI/CD variables. A pipeline authenticating with a permanent kubeconfig or service account token sitting in a secrets store is a standing risk. Exchange credentials for a short-lived, narrowly scoped token at the start of each job run, so a leaked pipeline log or compromised runner doesn't hand over a permanent credential.
• Make namespace and resource creation idempotent by default. Before creating a namespace or resource, check whether it already exists, and treat "already exists" as success rather than failure. Most CI tools and oc/kubectl support a client-side dry-run pattern for exactly this—generate the manifest, then apply it unconditionally, rather than branching your pipeline logic on existence checks. It's also worth considering whether a GitOps controller (Argo CD, Flux) is a better fit than an imperative pipeline altogether—they handle idempotency and drift detection as a first-class feature rather than something you have to hand-roll.

Real-time observability with Prometheus monitoring and ServiceMonitors

Monitoring OpenShift Dev Spaces requires visibility at 2 layers: the operator control plane and the individual workspace containers running on top of it.

DevWorkspace controller ServiceMonitor

Scrapes internal controller metrics directly from the openshift-devspaces namespace every 10 seconds:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: devworkspace-controller
  namespace: openshift-devspaces
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 10s
      port: metrics
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
  namespaceSelector:
    matchNames:
      - openshift-devspaces
  selector:
    matchLabels:
      app.kubernetes.io/name: devworkspace-controller

Prerequisite

For cluster-wide Prometheus to discover and scrape targets inside the operator namespace, that namespace must be explicitly labeled as shown here:

oc label namespace openshift-devspaces openshift.io/cluster-monitoring=true

Dev Spaces metrics exporter ServiceMonitor

The operator deploys an additional exporter (openshift-devspaces-metrics-exporter) tracking workspace-level statistics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: openshift-devspaces-metrics-exporter
  namespace: openshift-devspaces
  labels:
    olm.managed: 'true'
spec:
  endpoints:
    - interval: 30s
      port: metrics
      scheme: http
  namespaceSelector:
    matchNames:
      - openshift-devspaces
  selector:
    matchLabels:
      app.kubernetes.io/component: devspaces

Proactive capacity alerting

Because workspaces can consume significant compute when multiple developers run concurrent workloads, custom PrometheusRule resources give platform teams an early warning before compute pools reach saturation:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cluster-capacity-alerts
  namespace: openshift-monitoring
spec:
  groups:
    - name: cluster-capacity
      rules:
        # Alert when total worker node CPU requests exceed 80%
        - alert: OpenShiftClusterCPUCapacityHigh
          expr: |
            100 * sum(kube_pod_container_resource_requests{resource="cpu"})
            /
            sum(kube_node_status_allocatable{resource="cpu", node=~"compute-.*"})
            > 80
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Cluster CPU capacity above 80%"
            description: "CPU requests exceed 80% of allocatable capacity on compute nodes. Consider scaling compute or cleaning up stale spaces."

Recommended capacity thresholds

Monitor cluster resource usage against these recommended capacity thresholds to prevent scheduling delays and node evictions:

Air-gapped and restricted-network considerations

Deploying software in an air-gapped or network-restricted OpenShift cluster introduces operational constraints that are easy to overlook during early proof-of-concept planning. In a fully disconnected environment, outbound calls to public registries, vendor distribution servers, and external certificate authorities don't fail loudly—they fail silently or surface as workspace pod startup timeouts, which makes them much harder to diagnose after the fact. Operationalizing OpenShift Dev Spaces in a restricted environment comes down to resolving 5 external dependencies before onboarding your first developer cohort.

Container image mirroring

An enterprise-approved internal registry (such as Quay, JFrog Artifactory, or Sonatype Nexus) must contain every container image OpenShift Dev Spaces needs before it can be pulled inside the cluster:

• Operator catalogs: The OpenShift Dev Spaces Operator bundle and the DevWorkspace Operator index image
• Core operands: The dashboard, the gateway (che-gateway), the OpenShift Dev Spaces server, and the internal plug-in registry proxy
• Developer workspace images: The Universal Developer Image (UDI) and any specialized runtime images your teams use (such as custom Java, Python, or Go bases)

Rather than editing every devfile and manifest to point at your internal registry directly, apply an ImageDigestMirrorSet (or the legacy ImageContentSourcePolicy on older clusters) at the cluster level. This tells OpenShift to transparently redirect any pull request for a given public source (such as registry.redhat.io or quay.io) to your internal mirror instead, so devfiles and operator manifests can keep referencing the public image names as written, while the cluster silently serves them from your internal copy.

Eliminating outbound dependencies

Out of the box, OpenShift Dev Spaces reaches out to several public endpoints for devfile samples, integrated development environment (IDE) extensions, and runtime binaries. Re-route each of these internally or explicitly disable them:

• Disable the public devfile registry. By default, OpenShift Dev Spaces pulls sample projects from public GitHub repositories. Disable this in the CheCluster CR and instead host your own curated, enterprise-approved sample devfiles on an internal Git repository.
• Point extension distribution internally. Configure the plug-in registry to use your internal Open VSX server rather than the public marketplace, as detailed in the air-gapped readiness checklist later in this article.
• Host IDE binaries internally. Mirror JetBrains tarballs and other remote editor binaries to an internal artifact repository.

Internal PKI and certificate trust

In restricted networks, internal Git repositories, container registries, artifact managers, and Open VSX instances rely on certificates signed by a private certificate authority (CA). To establish Transport Layer Security (TLS)—the encrypted connection between workspace containers and internal services—you must explicitly import your corporate CA bundle into the cluster. Left unaddressed, missing trust chains cause TLS handshake failures during repository clones or extension pulls. Disabling the default cluster-wide CA bundle mount and supplying your corporate CA bundle through the che-git-custom-cert ConfigMap resolves these errors.

In restricted networks, internal Git repositories rely on certificates signed by a private CA. To establish TLS—the encrypted connection between workspace containers and internal services—you must explicitly import your corporate CA bundle into the cluster.

Air-gapped readiness checklist

Before declaring an air-gapped OpenShift Dev Spaces deployment ready for developer onboarding, verify each of the following:

Outcomes and next steps

Moving developers to OpenShift Dev Spaces shifts workspace management off personal laptops into standardized, repeatable cloud platforms. By standardizing runtime environments as code, platform engineering teams can drastically reduce configuration drift, proxy headaches, and local dependency conflicts.

Core benefits

Deploying OpenShift Dev Spaces with automated lifecycle guardrails, internal extension registries, and air-gapped trust chains prevents several common cluster failure modes:

• Faster developer onboarding. New engineers move from zero access to an active, fully configured, and policy-compliant  workspace in minutes rather than days.
• Guaranteed environmental parity. Devfiles help every developer, tester, and CI runner work from an identical base configuration, preventing late-stage "works on my machine" bugs.
• Centralized security governance. Platform teams vet, host, and audit extensions, runtime base images, and container tools internally. Dynamic Open Authorization (OAuth) authentication tokens replace local secrets and static source code management (SCM) credentials.
• Predictable platform costs. Enforcing per-container resource caps, inactive session idling, and automated monthly workspace pruning prevents resource hoarding and keeps cluster etcd footprints within healthy operational bounds.

Strategic roadmap for platform teams

To get the most out of your OpenShift Dev Spaces deployment long term, focus on expanding the platform across 3 main areas:

• Automated self-service portals. Connect namespace provisioning directly to your enterprise developer portal (such as Red Hat Developer Hub or ServiceNow), automating chargeback labeling, quota assignments, and group authorization mappings in a single workflow.
• Advanced observability analytics. Build custom Grafana dashboards using the metrics scraped by the devworkspace-controller and openshift-devspaces-metrics-exporter ServiceMonitor objects to track team-by-team platform adoption, active concurrent sessions, and compute resource trends over time.
• Automated supply chain security. Advance your internal Open VSX publishing pipeline from manual review to automated static analysis, vulnerability scanning, and license validation for all requested Visual Studio Extension (VSIX) packages.

With automated pipelines, internal registries, and strict security controls, platform teams can deliver fast, ready-to-use environments while maintaining cluster security. Standardize your development platform and try Red Hat OpenShift Dev Spaces today.

References and further reading

For the latest documentation and configuration limits, refer to the following resources:

• Red Hat OpenShift Dev Spaces Administration Guide
• Devfile v2 Specification & Schema Reference
• Red Hat Guidance: Running Dev Spaces at Scale
• Scaling DevSpaces & etcd Best Practices (Charro Gruver)
• Eclipse Che Issue #22198: CPU Limits on Gateway Container