Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qm sub-package Kernel-based Virtual Machine (KVM) #608

Merged
merged 1 commit into from
Oct 8, 2024
+187 −5
Merged

qm sub-package Kernel-based Virtual Machine (KVM) #608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ rpm: clean dist ## - Creates a local RPM package, useful for develop
--define="enable_qm_mount_bind_tty7 0" \
--define="enable_qm_mount_bind_ttyUSB0 0" \
--define="enable_qm_mount_bind_sound 0" \
--define="enable_qm_mount_bind_kvm 0" \
--define="enable_qm_mount_bind_input 0" \
--define="_topdir ${RPM_TOPDIR}" \
--define="version ${VERSION}" \
Expand All @@ -76,27 +77,32 @@ rpm: clean dist ## - Creates a local RPM package, useful for develop
ostree: qm_dropin_img_tempdir ## - A helper for creating QM packages for ostree based distros

.PHONY: qm_dropin_img_tempdir
qm_dropin_img_tempdir: ## - Creates a QM RPM sub-package qm_dropin_img_tempdir
qm_dropin_img_tempdir: ## - QM RPM sub-package qm_dropin_img_tempdir
sed -i 's/%define enable_qm_dropin_img_tempdir 0/%define enable_qm_dropin_img_tempdir 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

.PHONY: qm_dropin_mount_bind_ttyUSB0
qm_dropin_mount_bind_ttyUSB0: ## - Creates a QM RPM sub-package to mount bind /dev/ttyUSB0 in the nested containers
qm_dropin_mount_bind_ttyUSB0: ## - QM RPM sub-package to mount bind /dev/ttyUSB0 in the nested containers
sed -i 's/%define enable_qm_mount_bind_ttyUSB0 0/%define enable_qm_mount_bind_ttyUSB0 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

.PHONY: qm_dropin_mount_bind_kvm
qm_dropin_mount_bind_kvm: ## - QM RPM sub-package to mount bind /dev/kvm in the nested containers
sed -i 's/%define enable_qm_mount_bind_kvm 0/%define enable_qm_mount_bind_kvm 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

.PHONY: qm_dropin_mount_bind_sound
qm_dropin_mount_bind_sound: ## - Creates a QM RPM sub-package to mount bind /dev/snd in the nested containers
qm_dropin_mount_bind_sound: ## - QM RPM sub-package to mount bind /dev/snd in the nested containers
sed -i 's/%define enable_qm_mount_bind_sound 0/%define enable_qm_mount_bind_sound 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

.PHONY: qm_dropin_mount_bind_tty7
qm_dropin_mount_bind_tty7: ## - Creates a QM RPM sub-package to mount bind /dev/tty7 in the nested containers
qm_dropin_mount_bind_tty7: ## - QM RPM sub-package to mount bind /dev/tty7 in the nested containers
sed -i 's/%define enable_qm_mount_bind_tty7 0/%define enable_qm_mount_bind_tty7 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

.PHONY: qm_dropin_mount_bind_input
qm_dropin_mount_bind_input: ## - Creates a QM RPM sub-package to mount bind /dev/input in the nested containers
qm_dropin_mount_bind_input: ## - QM RPM sub-package to mount bind /dev/input in the nested containers
sed -i 's/%define enable_qm_mount_bind_input 0/%define enable_qm_mount_bind_input 1/' ${SPECFILE}
$(MAKE) VERSION=${VERSION} rpm

Expand Down
129 changes: 129 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- [Installing QM sub-packages](Installing-qm-sub-packages)
- [Removing QM sub-packages](Removing-qm-sub-packages)
- [Creating your own drop-in QM sub-package](Creating-your-own-drop-in-QM-sub-package)
- [QM sub-package kvm](QM-sub-package-kvm)
- [QM sub-package Sound](QM-sub-package-Sound)
3. [SELinux Policy](#selinux-policy)
4. [BlueChi](#bluechi)
Expand Down Expand Up @@ -288,6 +289,134 @@ Example changing the spec and triggering the build via make (feel free to automa
$ make clean && VERSION=YOURVERSIONHERE make rpm
```

## QM sub-package KVM

The QM sub-package KVM includes drop-in configuration that enables the integration of Kernel-based Virtual Machine (KVM) management into the QM (Quality Management) container environment. This configuration allows users to easily configure and manage KVM virtual machines within the QM system, streamlining virtualization tasks in containerized setups.

Below example step by step:

Step 1: clone QM repo, install libvirt packages, prepare some files inside QM and start the libvirt daemon.

```bash
$host> git clone https://github.com/containers/qm.git && cd qm
$host> make qm_dropin_mount_bind_kvm
$host> sudo dnf install rpmbuild/RPMS/noarch/qm_mount_bind_kvm-0.6.7-1.fc40.noarch.rpm
$host> sudo podman restart qm # if you have qm already running
$host> sudo dnf --installroot /usr/lib/qm/rootfs/ install virt-install libvirt-daemon libvirt-daemon-qemu libvirt-daemon-kvm -y
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer to use this one

sudo dnf install qemu

qemu-system-x86_64 --version

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, qemu should be containerized. This is nice for testing that the kvm device has been mounted correctly, but it is not the way to run a VM inside the QM.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aesteve-rh ah, yes, the whole things should be containerized,
Step 1:
is running qemu in qm
Step 2:
qemu containerized in qm
Is it?

One more,
Do you think we should use virt-* packages?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 2 should be supported already, so we can aim for that directly.

Do you think we should use virt-* packages?

You mean for this documented example or in general? Probably, as a response to both, it should not be necessary to use them. I agree with you in that qemu would be preferred.

Copy link
Collaborator Author

@dougsland dougsland Oct 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Yarboa yes, should work like that. @aesteve-rh can you share more about your vision so all of us are in the same page?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am only saying that workloads inside the qm are supposed to be containerized. Like, the image should be stored inside the qm and launched through a quadlet file. That allows to set rules to ensure FFI, control the system's surface that the process has access to, and also allows applying SELinux labels to allow only a correct subset of operations.

For developers, it may suffice to install all those packages directly on the qm container and test it there. But imo, this is not how virtualization should be documented for users.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aesteve-rh completed agreed. It's just a documentation.


# Copy default network settings to /root dir inside QM (/usr/lib/qm/rootfs/root)
$host> sudo cp /usr/share/libvirt/networks/default.xml /usr/lib/qm/rootfs/root

Step 2: Preparing cloudinit files inside QM (/usr/lib/qm/rootfs/root)

# Cloud-init files
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you using cloud-init? i assume it is for the guest?
Is it only for the password?

I prefer to use this command after that one

virt-customize --uninstall cloud-init --root-password password:Aa1234 -a ./<fedora-image>.qcow2

https://github.com/containers/qm/pull/608/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R321

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds a good plan. If we merge can you test and improve it for us?

------------------------------
$host> cd /usr/lib/qm/rootfs/root/
$host> cat meta-data
instance-id: fedora-cloud
local-hostname: fedora-vm

# We are setting to user fedora the default password as fedora
$host> cd /usr/lib/qm/rootfs/root/
$host> cat user-data
#cloud-config
password: fedora
chpasswd: { expire: False }
ssh_pwauth: True

# Download the Fedora Cloud image for tests and save it /usr/lib/qm/rootfs/var/lib/libvirt/images/
$ wget -O /usr/lib/qm/rootfs/root/Fedora-Cloud-Base-Generic.qcow2 https://download.fedoraproject.org/pub/fedora/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-Generic.x86_64-40-1.14.qcow2

# Generate the cloud-init.iso and move it to /usr/lib/qm/rootfs/var/lib/libvirt/images/
$host> cloud-localds cloud-init.iso user-data meta-data
$host> mv cloud-init.iso /usr/lib/qm/rootfs/var/lib/libvirt/images/

# Change permission to qemu:qemu
$host> chown qemu:qemu /usr/lib/qm/rootfs/var/lib/libvirt/*

Step 3: Starting libvirtd and checking if it's active inside QM
Comment on lines +335 to +337
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you hit selinux issues?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, must be addressed in a next patch.


##################################################################
# Keep in mind for the next steps:
# Depending on the distro you are running SELinux might complain
# about libvirtd running on QM / udev errors
##################################################################

# Going inside QM
$ sudo podman exec -it qm bash

# Starting libvirtd
bash-5.2# systemctl start libvirt

# Check if it's running:
bash-5.2# systemctl is-active libvirtd
active
```

Step 4: Creating a script inside QM and running the VM

```bash
$host> cd /usr/lib/qm/rootfs/root/
$host> vi run
##### START SCRIPT ############
# Set .cache to /tmp
export XDG_CACHE_HOME=/tmp/.cache

# Remove previous instance
virsh destroy fedora-cloud-vm 2> /dev/null
virsh undefine fedora-cloud-vm 2> /dev/null

# Network
virsh net-define ./default.xml 2> /dev/null
virsh net-start default 2> /dev/null
virsh net-autostart default 2> /dev/null

# Install
virt-install \
--name fedora-cloud-vm \
--memory 20048 \
--vcpus 4 \
--disk path=/var/lib/libvirt/images/Fedora-Cloud-Base-Generic.qcow2,format=qcow2 \
--disk path=/var/lib/libvirt/images/cloud-init.iso,device=cdrom \
--os-variant fedora-unknown \
--network network=default \
--import \
--graphics none \
--console pty,target_type=serial \
--noautoconsole

##### END SCRIPT ############
```

Step 5: Running the script

```bash
qm$ sudo podman exec -it qm bash
bash-5.2# cd /root
bash-5.2# ./run
Domain 'fedora-cloud-vm' destroyed

Domain 'fedora-cloud-vm' has been undefined

Network default marked as autostarted

Starting install...
Creating domain... | 0 B 00:00:00
Domain creation completed.
bash-5.2# virsh list
Id Name State
---------------------------------
4 fedora-cloud-vm running

bash-5.2# virsh console fedora-cloud-vm

fedora-vm login: fedora
Password:

Last login: Tue Oct 8 06:01:18 on ttyS0
[fedora@fedora-vm ~]$
```

## RPM building dependencies

In order to build qm package on CentOS Stream 9 you'll need Code Ready Builder
Expand Down
7 changes: 7 additions & 0 deletions etc/qm/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Drop-in configuration for Podman to mount bind /dev/kvm from host to container
#
[containers]
devices = [
"/dev/kvm:/dev/kvm",
"/dev/net/tun:/dev/net/tun"
]
40 changes: 40 additions & 0 deletions rpm/qm.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@
###########################################
%define enable_qm_mount_bind_ttyUSB0 0

###########################################
# subpackage QM - mount bind /dev/kvm #
###########################################
%define enable_qm_mount_bind_kvm 0

###########################################
# subpackage QM - input devices #
###########################################
Expand Down Expand Up @@ -141,6 +146,22 @@ install -d %{buildroot}%{_sysconfdir}/qm/containers/containers.conf.d
# END - qm dropin sub-package - img tempdir #
########################################################

########################################################
# START - qm dropin sub-package - mount kvm #
########################################################
%if %{enable_qm_mount_bind_kvm}
# first step - add drop-in file in /etc/containers/containers.d.conf/qm_dropin_mount_bind_kvm.conf
# to QM container mount bind /dev/kvm
install -m 644 %{_builddir}/qm-%{version}/etc/qm/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf %{buildroot}%{_sysconfdir}/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf

# second step - add drop-in file in /etc/qm/containers/containers.d.conf/qm_dropin/mount_bind_kvm.conf
# to nested containers in QM env mount bind it in /dev/kvm
install -m 644 %{_builddir}/qm-%{version}/etc/qm/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf %{buildroot}%{_sysconfdir}/qm/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf
%endif
########################################################
# END - qm dropin sub-package - mount kvm #
########################################################

########################################################
# START - qm dropin sub-package - mount sound #
########################################################
Expand Down Expand Up @@ -355,6 +376,25 @@ additional drop-in configurations.
%{_sysconfdir}/qm/containers/containers.conf.d/qm_dropin_mount_bind_snd.conf
%endif

#######################################
# sub-package QM Mount Bind /dev/kvm #
#######################################
%if %{enable_qm_mount_bind_kvm}
%package -n qm_mount_bind_kvm
Summary: Drop-in configuration for QM containers to mount bind /dev/kvm
Requires: %{name} = %{version}-%{release}
BuildArch: noarch

%description -n qm_mount_bind_kvm
This sub-package installs a drop-in configurations for the QM.
It creates the `/etc/qm/containers/containers.conf.d/` directory for adding
additional drop-in configurations.

%files -n qm_mount_bind_kvm
%{_sysconfdir}/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf
%{_sysconfdir}/qm/containers/containers.conf.d/qm_dropin_mount_bind_kvm.conf
%endif

#######################################

%changelog
Expand Down
Loading