Unsurprisingly, developers are looking for ways to include powerful new technologies like AI assistants to improve their workflow and productivity. However, many companies are reluctant to allow such technology due to privacy, security, and IP law concerns.
This activity addresses the concerns about privacy and security and describes how to deploy and integrate a private AI assistant in the emulated air-gapped on-premise environment. We will guide you through setting up a CDE (cloud development environment) using Ollama, Continue, Llama3, and Starcoder2 large language models (LLMs) with Red Hat OpenShift Dev Spaces, empowering you to code faster and more efficiently.
Ready to streamline your cloud development workflow and bring some AI into it? Grab your favorite beverage, and let's embark on this journey to unlock the full potential of the cloud development experience!
Prerequisites
Access Red Hat OpenShift Dev Spaces on Developer Sandbox
Once you have registered for a Developer Sandbox account, you can access Red Hat OpenShift Dev Spaces by navigating to https://workspaces.openshift.com. This redirects you to the Red Hat OpenShift Dev Spaces user dashboard, as shown in Figure 1.
Start the cloud development environment
On the User Dashboard, navigate to the Create Workspace tab and provide the URL to the repository that we will use for this activity, as shown in Figure 2: https://github.com/redhat-developer-demos/cde-ollama-continue. Then, click the Create & Open button.
During the workspace startup, you will be asked to authorize the GitHub OAuth app (Figure 3).
This allows users to have full Git access from the workspaces and execute commands like git push without any setup. Once the permissions are granted, the git-credentials-secret is created in the user namespace which stores the token that is used by Red Hat OpenShift Dev Spaces.
Note
You can always revoke the access at any time on the User Dashboard via User Preferences → Git Services or directly from the GitHub settings.
Once the workspace is started, you will be asked if you trust the authors of the files in the workspace (see Figure 4). Opt in by clicking the Yes, I trust the authors button.
After some seconds, the Continue extension will then be automatically installed.
Note
Continue is the leading open source AI code assistant. Learn more about the extension from the official documentation.
When installation is complete, you can click on the new symbol on the left in the sidebar, where a Welcome to Continue screen shows up. Because the Continue extension has already been preconfigured, you can scroll to the bottom of this page and click the Skip button, as shown in Figure 5.
Now you are ready to use the personal AI assistant (Figure 6).
The devfile and how it works
Under the hood, Red Hat OpenShift Dev Spaces uses the devfile from the root of the repository to create the CDE that contains not only the source code but also the runtime, together with predefined commands for instant development (Figure 7).
Note
Devfile is a CNCF sandbox project that provides an open standard defining containerized development environments. Learn more about Devfile from the official documentation.
By using the devfile for creating a new workspace, the following two containers are being started as part of the CDE:
udi: A container based on the Universal Developer Image, which hosts the Continue server and is used for main development activities.
ollama: A container based on the official Ollama image that comprises the Ollama web server.
Tip
Additionally, it is possible to leverage GPUs by setting nvidia.com/gpu: 1 in the container’s resource request on the devfile level, which would tremendously improve the performance of the personal AI assistant. Due to that configuration, the ollama container (and the entire pod) will be deployed on an OpenShift worker node that hosts a GPU, which significantly accelerates the inference step of the local LLM and hence tremendously improves the performance of the personal AI assistant for developers. Developer Sandbox clusters currently do not have worker nodes with GPU available hence nvidia.com/gpu: 1 configuration is commented out in the devfile. If you have a cluster available with GPU nodes, feel free to uncomment the lines to enable the nvidia.com/gpu: 1, and run the activity there instead of Developer Sandbox. This configuration would tremendously improve the performance of the personal AI assistant for developers.
At the bottom of the devfile, a set of postStart commands are defined; these commands are executed just after the cloud development environment starts up:
events:
postStart:
- pullmodel
- pullautocompletemodel
- copyconfigpullmodel: Pulls the llama3 LLM to the CDE.pullautocompletemodel: Pulls the starcoder2 LLM to the CDE.copyconfig: Configures the AI assistant Continue to use the local LLMs by copying the continue-config.json file.
What can you do with a personal AI assistant?
Now that we've covered devfile basics, let’s get back to the cloud development environment (CDE). Once everything is set up, we'll demonstrate the use of a private personal AI assistant for developers using some common development use cases.
Inside the CDE, after clicking on the new "Continue" symbol on the left in the sidebar, a dialog shows up that you can use to communicate with the AI model. Inside the text box, enter Write a hello world program in Python. You will get output similar to Figure 8.
The AI model will remember your inputs; you can also ask it to modify the answer based on additional needs using the ask a follow-up prompt underneath the response. Insert Let the user input a string, which then is also printed to the screen in the text box and press Enter. The result is something like Figure 9.
Besides this pure chat functionality, the personal AI assistant for developers can also directly manipulate code, make code suggestions, write documentation or tests, and analyze the code for known issues. In the next example, we'll use it to write a program that checks a given date for proper format.
In the Continue extension, create a new session by pressing the plus sign. Now enter the text “Write the code in Python that validates a date to be of a proper format” into the text box and observe the output.
Then, create a new file named validate_date.py and add the AI-generated code by hovering over the code snippet in the Continue extension and clicking the Insert at cursor button.
Finally, click Menu → Terminal → New Terminal and execute the newly generated file by entering python validate_date.py. The output will look similar to Figure 10.
Select the entire code in the Python file validate_date.py and press Ctrl+L (or Cmd+L). One can see that the selected code is added to the Continue extension, i.e. a context is provided to the AI model. Next, type into the text box /comment , as shown in Figure 11.
Pressing enter after having typed /comment into the text box tells the AI assistant to write documentation for the selected code lines and add it directly to the code in the Python file validate_date.py. Then a Continue Diff tab opens and one can see the differences, in this case, the added lines of documentation. To accept the changes, you can press Ctrl+Enter (or Shift + Cmd + Enter), and the code is inserted into the file, as shown in Figure 12.
Tip
Hot keys might vary depending on the underlying OS; you can find them using F1 → Continue.
These are just a few examples of how to use the AI assistant for a developer’s everyday life and make it more productive.
Tip
More information about the usage of the Continue extension within the cloud development environment can be found on the extension’s homepage. Additional models are available on the Ollama website, and more information for configuring development environments using devfile can be found in the official devfile.io documentation.
Privacy and security
The pervasive challenge with most large language models is their availability predominantly as cloud-based services. This setup necessitates sending potentially sensitive data to external servers for processing. For developers, this raises significant privacy and security concerns, particularly when dealing with proprietary or sensitive codebases. The requirement of sending data to a remote server not only poses a risk of data exposure but can also introduce latency and dependence on internet connectivity for real-time assistance. This architecture inherently limits the use of such LLMs in environments where data governance and compliance standards restrict the transfer of data off-premises or where developers prioritize complete control over their data and intellectual property.
Addressing the challenge of data privacy and security when using cloud-based LLMs, the Continue extension emerges as a compelling solution. The extension is marketed as an “open-source autopilot for software development” and uniquely, it enables the utilization of local LLMs.
In this activity, we emulated the on-premises environment by using the on-premises Llama3-8b model. While using the personal AI assistant, you can open the Network tab in the browser window and make sure that no request is sent outside of the cluster (Figure 13).
Conclusion
Artificial intelligence (AI) assistants have the potential to revolutionize application development by enhancing productivity and streamlining workflows. For developers, an AI sidekick can act as a coding companion, offering real-time optimizations, automating routine tasks, and debugging on the fly. By running a local instance of an LLM on an air-gapped, on-premise OpenShift cluster, developers can benefit from AI intelligence without the need to transmit data externally.
When integrated within Red Hat OpenShift Dev Spaces the solution offers a seamless and secure development experience right within Visual Studio Code - Open Source (Code - OSS). This setup ensures that sensitive data never leaves the confines of the local infrastructure, all the while providing the sophisticated assistance of an AI using the Continue extension. It is a solution that not only helps mitigate privacy concerns but also empowers developers to harness AI’s capabilities in a more controlled and compliant environment.
Read more about using Red Hat OpenShift Dev Spaces in an air-gapped, on-premise environment in this success story. Happy coding!
