On-demand screen context in Red Hat Developer Hub intelligent assistant

As developers, we spend a lot of time copying and pasting context. When you run into a confusing deployment error or need help understanding a complex plugin configuration in your developer portal, asking an AI assistant usually requires manual toil. You highlight text, inspect the Document Object Model (DOM), copy the markup, paste it into a chat window, and then explain what went wrong.

To eliminate this friction and cut down debugging time, we added on-demand screen and DOM reading to Red Hat Developer Hub intelligent assistant. Currently available in developer preview, this feature allows the assistant to inspect your active browser view in docked and overlay modes. In a single step, you can ask questions like "What is wrong with this resource configuration?" without manually gathering screen data.

Here's how we designed the pipeline to grab visual context without exposing sensitive user data.

Built for privacy: On-demand screen context

We designed context capture around explicit user consent and data hygiene. In addition to stripping sensitive fields and credentials from the DOM tree before dispatch, context collection is strictly event-driven rather than continuous:

• Screen context is captured only for the currently active page.
• Capture triggers strictly when you select Submit.
• Continuous and background screenshot capturing is completely disabled.

By making context capture explicit and instantaneous, the assistant only receives the exact data you actively choose to share. Figure 1 shows the intelligent assistant analyzing the active Learning Paths page after an on-demand screen context query.

How screen context extraction works on the front end

Screenshots capture what developers see immediately—broken layouts, misaligned widgets, or truncated error notifications. Paired with cleaned DOM markup, the model gets both the visual cues and the exact code tokens needed to diagnose the issue.

On the front end, we divided the pipeline between 2 core modules: ScreenContextChip.tsx manages the UI trigger in the message bar, while screen-context-utils.ts handles canvas rendering and DOM extraction. When you enable the feature using the context chip in the chat bar, the front end attaches a context reader to your submission event.

When you select Send, the application generates a WebP image of the current page, excludes the chat overlay from the capture to avoid visual recursion, and extracts the cleaned DOM tree.

In screen-context-utils.ts, the extraction pipeline captures the viewport and strips unnecessary DOM nodes:

export async function captureScreenContext(): Promise<{ screenshot: string; domText: string }> {
  const canvas = await html2canvas(document.body, {
    ignoreElements: (element) => element.classList.contains('intelligent-assistant-chat-overlay')
  });

  const screenshot = canvas.toDataURL('image/webp', 0.8);
  const domText = extractCleanDomText(document.body);

  return { screenshot, domText };
}

This pipeline converts the rendered DOM into a compressed WebP data URL and pairs it with a structured text representation of the active document. The user interface displays a pulsing animation on the ScreenContextChip component while this extraction runs, giving you immediate feedback that context is being attached.

We also updated the model selection interface in MessageBarModelSelector.tsx. Models capable of processing image attachments display a vision indicator, making it clear which models can evaluate both text and images. Red Hat Developer Hub offers 3 chat layouts: docked alongside your portal view, floating as an overlay, or expanded to full screen. Context capture is disabled in full-screen mode because the expanded assistant obscures the active workspace view it needs to inspect.

Optimizing back-end latency with background vision probing

Processing multimodal attachments introduces performance challenges on the back end. In early iterations, the back end validated model capabilities synchronously during query handling, which increased latency on every user prompt.

To resolve this bottleneck in intelligent-assistant-backend, we moved vision capability detection out of the query execution path. The back end now returns a supportsVision property directly on the GET /v1/models endpoint.

When the back end initializes, it runs a non-blocking background probe using probeModelVisionSupport to test each configured model. The result is stored in ModelCapabilitiesCache, an in-memory cache with a 24-hour expiration per pod.

The following TypeScript snippet demonstrates how the models endpoint attaches cached vision capabilities:

export async function getModelsWithCapabilities(
  models: Model[]
): Promise<ModelWithCapabilities[]> {
  return Promise.all(
    models.map(async (model) => {
      const supportsVision = await modelCapabilitiesCache.getOrProbe(
        model.id,
        () => probeModelVisionSupport(model)
      );
      return { ...model, supportsVision };
    })
  );
}

With cached model capabilities, the front end immediately knows whether to gate image attachments without waiting for an individual validation round-trip. Additionally, the POST /v1/query route accepts multipart requests containing the compressed WebP buffer and sanitized DOM text, forwarding both directly to the inference service.

Enabling screen context in Red Hat Developer Hub

Because screen context inspection handles visual workspace data, administrators must explicitly opt in to enable the capability across an organization. The feature is disabled by default.

If your team is running Red Hat Developer Hub, ask your platform administrator to enable the feature in your instance's configuration file:

intelligent-assistant:
  screen-context:
    enabled: true

Setting the enabled key to true activates the opt-in toggle in the chat interface for all users while keeping actual capture strictly on demand.  Once enabled by an administrator, users can manage screen context sharing in the assistant options menu, as shown in Figure 2.

Caching vision capabilities at initialization and compressing screen captures to WebP keep the assistant fast while ensuring private data stays under user control. Test the developer preview in your Red Hat Developer Hub instance today, enable the context chip on your next debugging task, and let us know how it improves your daily workflow.