Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Configure and run a QEMU-based VM outside of libvirt with virt-manager

March 6, 2020
Kevin Buettner
Related topics:
DevOpsLinuxVirtualization
Related products:
Developer Tools

Share:

    I recently needed to run a virtual machine (VM) created using virt-manager outside of libvirt. I was investigating an issue that required running QEMU with the machine option dump-guest-core=on. By default, libvirt runs with that option off, so I decided to set up a standalone QEMU environment. I found the process of configuring the test VM and writing the boot script more involved than expected, so I decided to document the steps I took.

    I hope this article makes it easier for you to configure and run your own QEMU-based VM for similar investigations. Note that I do not recommend the approach described here for a VM running in production (at least, not without backup).

    Create the test VM

    This article assumes that you created a VM with virt-manager or another tool in the libvirt framework. I advise against using a VM that you care about, so we'll start by using virt-manager to create a test VM from scratch. Another option would be to use a clone of an existing VM.

    The virtualization host that I used for my investigations was also a VM. If you want to do the same, you'll need to configure your virtualization environment to allow running nested VMs. (Follow the link for instructions for enabling nested virtualization on Fedora.) Running your virtualization host as a VM will make it easier to debug the virtualization host kernel if that is something you need to be able to do.

    Step 1: Install a guest OS using virt-manager's New VM wizard

    The New VM wizard breaks down installation to five steps:

    1. Name the guest VM and choose the installation type.
    2. Locate and configure the installation media.
    3. Choose memory and CPU settings.
    4. Configure the virtual machine's storage.
    5. Configure networking, architecture, and other hardware settings.

    I won't describe all the steps in detail because that has been done elsewhere. Instead, I offer the following tips:

    • In step 3 of 5, Choose memory and CPU settings, I usually choose the defaults. If you're going for a minimal install, or if your virtualization host doesn't have much memory (RAM), you might decrease the amount of memory from the default value.
    • In step 4 of 5, I leave the box labeled Enable storage for this virtual machine checked, but decrease the storage space to 10GB. As noted earlier, my virtualization host is also a VM, and I do not usually allocate more than 60GB of storage for that host. If I want to install several VMs, I need to keep each one relatively small.
    • In step 5 of 5, on the Ready to begin the installation screen, check the box labeled Customize configuration before install. This action allows you to configure the chipset and firmware to your liking. As an example, if you want your new VM to boot using UEFI, you can change the Firmware setting from BIOS to UEFI x86_64: /usr/share/edk2/ovmf/OVMF_CODE.fd. You might also want to change the Chipset setting to Q35 (from i440FX), assuming this value is not already the default.

    Note: If you want to change your VM's chipset or firmware, do so before installing the OS.

    I also find it useful to note the network interface's MAC address so that I can assign it a suitable name on the system that provides DHCP services.

    Step 2: Install the OS

    My only advice here is to make sure that you install whatever packages you need for testing purposes. Otherwise, I advise keeping things fairly minimal.

    Step 3: Boot and test the OS under virt-manager

    Make sure that your newly installed OS boots and that you can log in. If you want to test a particular facility using a standalone QEMU environment, make sure that this facility is also working.

    In my case, I made sure that the network device was working and that I could log in via SSH. That way, I could make the VM headless and still log in to see that the setup still worked. In retrospect, this step might have been unnecessary due to turning on options that ultimately provided a console.

    Step 4: Reduce the size of the virt-manager configuration

    Next, it's time to remove the clutter from the typical virt-manager configuration. Doing this reduces the size of the QEMU command that we have to work with later on.

    Figure 1 shows the hardware details for a VM that I named F31-OVMF.

    Virt-manager window showing the hardware details for VM F31-OVMF

    I removed the following devices:

    • SATA CDROM 1
    • Tablet
    • Display spice
    • Sound ich9
    • Serial 1
    • Channel qemu-ga
    • Channel spice
    • Video QXL
    • USB Redirector 1
    • USB Redirector 2
    • RNG /dev/urandom

    Note that not all devices can be removed. As an example, I clicked the Remove button on Controller VirtIO Serial 0, but virt-manager added it back.

    Figure 2 shows the resulting hardware details for F31-OVMF once I finished removing devices.

    VM hardware configuration

    Boot the VM again and make sure that it still works. If you made the changes outlined above, the VM will boot headless and you will need to connect to it via SSH. Do not skip this step. You still need to start the VM again in order to write virt-manager's (now reduced) QEMU command to a log file.

    Create a boot script from the QEMU command

    With the test VM set up, the next thing to do is create the script you will use to start the VM in a standalone QEMU environment. You can use the QEMU command as the basis for your script, but you'll have to make a number of edits first. The biggest hurdle is identifying and removing any file descriptors that reference resources that are unavailable outside of libvirt. Once again, I'll guide you through the steps.

    Step 1: Locate the QEMU log file on your virtualization host

    Log files for QEMU running under libvirt or virt-manager are found in /var/log/libvirt/qemu. For my VM, this file is named F31-OVMF.log. Locate this file and prepare to open it using your favorite text editor.

    Step 2: Copy lines from the log file to your new script

    Open the log file and scroll through it until you find the line that looks something like this:

    /usr/bin/qemu-system-x86_64 \

    Look above this line and identify all the environment variables that are set for running this command. These lines all end with a backslash character (\). In my log file, the first environment variable line starts with LC_ALL=C \.

    Copy the lines starting from the first environment variable and ending at the bottom of the file. Paste these lines into a new file, skipping any obvious cruft from the bottom of the log file. This will be the script that you run to start the VM as a standalone QEMU environment, so I suggest naming it something like start-<VM>. I named my file start-F31-OVMF.

    This is what my soon-to-be script looks like, so far:

    LC_ALL=C \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
    HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF \
    XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.local/share \
    XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.cache \
    XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.config \
    QEMU_AUDIO_DRV=none \
    /usr/bin/qemu-system-x86_64 \
    -name guest=F31-OVMF,debug-threads=on \
    -S \
    -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-F31-OVMF/master-key.aes \
    -machine pc-q35-4.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
    -cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,cmp-legacy=on,perfctr-core=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,monitor=off \
    -drive file=/usr/share/edk2/ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
    -drive file=/var/lib/libvirt/qemu/nvram/F31-OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
    -m 2048 \
    -overcommit mem-lock=off \
    -smp 2,sockets=2,cores=1,threads=1 \
    -uuid cbfc504c-69e7-4955-9e42-c96acfcd4f20 \
    -display none \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,fd=37,server,nowait \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=utc,driftfix=slew \
    -global kvm-pit.lost_tick_policy=delay \
    -no-hpet \
    -no-shutdown \
    -global ICH9-LPC.disable_s3=1 \
    -global ICH9-LPC.disable_s4=1 \
    -boot strict=on \
    -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
    -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
    -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
    -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
    -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
    -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
    -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
    -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
    -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
    -drive file=/var/lib/libvirt/images/F31-OVMF.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
    -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
    -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:2d:4f,bus=pci.1,addr=0x0 \
    -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
    -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
    -msg timestamp=on

    Step 3: Create the shell script

    Next, edit this log file into a workable shell script by doing the following:

    • Add #!/bin/sh as the file's first line. I also add a blank line after this one, but that's not strictly necessary.
    • Adjust the environment variable section:
      • Remove the backslash (\) character from the end of each environment variable line.
      • Place the word export followed by a space at the beginning of each of these lines.
      • Add a blank line at the end of this section.
    • Use a leading # character to comment out the export PATH= and export HOME= lines. (It might be possible to disable other lines that set environment variables, but I have not tested doing this.)

    The beginning of my script now looks like this:

    #!/bin/sh
    
    export LC_ALL=C
    #export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
    #export HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF
    export XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.local/share
    export XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.cache
    export XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.config
    export QEMU_AUDIO_DRV=none
    
    /usr/bin/qemu-system-x86_64 \

    Now scroll down the file and edit the following from the qemu-system-x86_64 command:

    • Delete the line starting with -object secret.
    • Delete the line starting with -chardev socket. (This line references a file descriptor that is not accessible outside of the libvirt environment.)
    • Delete the line starting with -sandbox.
    • Delete the line starting with -mon.
    • Change the line -display none to -display gtk. Make sure you leave the trailing backslash at the end of this line.
    • Change the -netdev tap line to netdev bridge,id=ID, where ID is the same as the id in the original -netdev tap line. In this case, it's hostnet0, so the line in question changes from:
    -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
    

    to:

    -netdev bridge,id=hostnet0 \

    Note that this change removes another fd= argument from the overall command. Again, this line references a file descriptor that is not available from the standalone environment outside of libvirt.

    Also note that, for this change to work, it's necessary to have a bridge named br0 configured on the virtualization host. For a bridge named something else, like br1, use the following instead:

    -netdev bridge,id=hostnet0,bridge=br1

    Consult the QEMU network options documentation for further information.

    You're nearly done editing, but you have a few more options for simplifying the script. First, you can search the file for fd=. Hopefully, there won't be any occurrences, but if there are, you need to figure out how to safely remove these references to file descriptors. You might try deleting the lines in question. If that doesn't work, consult the QEMU documentation.

    Second, you could remove the -nodefaults line. (With this line gone, you probably could simplify the script even more, but I've haven't explored that option.)

    Now, make the script executable. For my example script, I used:

    # chmod a+x start-F31-OVMF

    Finally, with all of the edits in place, this is what my script looks like:

    #!/bin/sh
    
    export LC_ALL=C
    #export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
    #export HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF
    export XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.local/share
    export XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.cache
    export XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-F31-OVMF/.config
    export QEMU_AUDIO_DRV=none
    
    /usr/bin/qemu-system-x86_64 \
    -name guest=F31-OVMF,debug-threads=on \
    -S \
    -machine pc-q35-4.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off \
    -cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,cmp-legacy=on,perfctr-core=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,monitor=off \
    -drive file=/usr/share/edk2/ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
    -drive file=/var/lib/libvirt/qemu/nvram/F31-OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
    -m 2048 \
    -overcommit mem-lock=off \
    -smp 2,sockets=2,cores=1,threads=1 \
    -uuid cbfc504c-69e7-4955-9e42-c96acfcd4f20 \
    -display gtk \
    -no-user-config \
    -rtc base=utc,driftfix=slew \
    -global kvm-pit.lost_tick_policy=delay \
    -no-hpet \
    -no-shutdown \
    -global ICH9-LPC.disable_s3=1 \
    -global ICH9-LPC.disable_s4=1 \
    -boot strict=on \
    -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
    -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
    -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
    -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
    -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
    -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
    -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 \
    -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
    -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
    -drive file=/var/lib/libvirt/images/F31-OVMF.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
    -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
    -netdev bridge,id=hostnet0 \
    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c9:2d:4f,bus=pci.1,addr=0x0 \
    -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
    -msg timestamp=on

    Step 4: Run the VM using your newly created script

    As root, run your new script. I ran mine as follows:

    [root@f31-2 ~]# ./start-F31-OVMF

    If it works, a window similar to Figure 3 pops up.

    virt-manager VM window paused at the display warning

    Select Machine -> Pause to cancel the pause and continue the boot process. At this point, the machine should boot and, ultimately, you should see a login prompt as shown in Figure 4.

    virt-manager window reaching the VM's login

    Log in and make sure that the VM functions as expected.

    Conclusion

    You should now have a workable script for booting your VM in a standalone QEMU environment. The next steps depend on your use case. In my case, I modified the dump-guest-core option, changing the value from off to on, and then proceeded with my investigations.

    Last updated: October 8, 2024

    Related Posts

    • Creating a VMDK using image mode for Red Hat Enterprise Linux

    • OpenShift Virtualization for vSphere admins: A change in the traditional storage paradigm

    • OpenShift Virtualization for VMware vSphere admins: Disaster and site recovery

    • Scaling virtio-blk disk I/O with IOThread Virtqueue Mapping

    • A self-service approach to building virtual machines at scale

    • Minimize downtime when migrating from VMware to OpenShift Virtualization

    Recent Posts

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    • How to encrypt RHEL images for Azure confidential VMs

    • How to manage RHEL virtual machines with Podman Desktop

    • Speech-to-text with Whisper and Red Hat AI Inference Server

    What’s up next?

    Learn how to create and manage virtual machines using Red Hat OpenShift and the Developer Sandbox in this hands-on activity.

    Start the activity
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue