Working with AI models locally can be tricky. You often have to deal with many different software pieces and settings. Podman AI Lab and RamaLama are two projects that make running AI models on your own computer simple and straightforward.
RamaLama helps you run AI models in two main ways: directly on your computer (if you have a local inference engine like llama.cpp installed, for example) or inside containers. Podman AI Lab always uses containers to run AI models. Both projects use a way to let the AI models inside containers use your computer's powerful graphics card (GPU) to run faster.
RamaLama and AI Lab: Making local AI simple
RamaLama was created to make running AI models on your computer easy. It tries to use your computer's own software first if it can. For example, if you have a local inference engine like llama.cpp installed, RamaLama can use it directly. This can make the models run very quickly. RamaLama can also run models inside containers if needed.
Podman AI Lab is an extension of Podman Desktop, a graphical tool for managing containers and Kubernetes clusters. It gives you a simple way to try out AI models by running them inside containers. A big part of the AI Lab is making sure the AI models in containers can use your computer's GPU to speed things up.
Working together: Using the same containers
Previously, both RamaLama and the Podman AI Lab used their own sets of containers for running AI models. Now Podman AI Lab uses containers built by the RamaLama project. This has several key advantages:
- Less duplication: You no longer need to build and keep track of two different sets of containers.
- Consistent experience: Whether you use RamaLama in a container or the AI Lab, you use the same well-made containers.
- Easier updates: When we improve the containers or introduce a new GPU support, both projects benefit.
- Simpler GPU use: The RamaLama containers are set up to easily use your computer's GPU, and the AI Lab now uses this setup. This makes it easier to get your AI models running fast within the AI Lab.
What you need: Podman and GPU support
To use the GPU with the containers, you need to have Podman installed on your computer. Podman is a tool for running containers. The way GPU support works depends on your operating system:
- On macOS, you typically need krunkit and libkrun for efficient GPU sharing.
- On Windows, GPU passthrough is generally facilitated through WSL2 (Windows Subsystem for Linux 2).
For detailed instructions and further benefits of GPU integration with Podman, refer to the Podman Desktop documentation. This resource provides comprehensive guidance on configuring GPU support across different platforms, ensuring you can harness the full power of your GPU for demanding containerized applications.
What this means for you
By working together and using the same containers, RamaLama and the Podman AI Lab are making it much simpler to run AI models on your own computer. Spend less time setting things up and more time working with AI.
Both Podman AI Lab and RamaLama are part of the same GitHub organization to facilitate closer collaboration.
Learn more and get involved
Check out the following resources to explore the projects in detail and contribute to their development.
Podman AI Lab:
- Website: https://podman-desktop.io/extensions/ai-lab
- Project code: https://github.com/containers/podman-desktop-extension-ai-lab
- Report issues: https://github.com/containers/podman-desktop-extension-ai-lab/issues
RamaLama:
- Website: https://ramalama.ai/
- Project code: https://github.com/containers/ramalama
- Report issues: https://github.com/containers/ramalama/issues
