212 lines
8.2 KiB
Markdown
212 lines
8.2 KiB
Markdown
|
|
Spin up libvrt instances from a cloud-init capable base that can be
|
|
maintained using chroot on qcow2, and by ansible using libvirt_qemu.
|
|
|
|
This builds on two projects:
|
|
1) bash scripts to take a Gentoo qcow2 nightly box and use it as a base layer
|
|
for another qcow image that you spin up in libvirt, overlaying the base layer
|
|
for testing. You can then throw away that image for each test (if you like);
|
|
it takes less than a minute to make the base qcow2 layer, and other minute
|
|
to make the overlay layer and spin it up.
|
|
https://github.com/earlruby/create-vm/
|
|
|
|
2) python scripts to take a Gentoo stage3 and portage to build a qcow2 box and
|
|
use it as a base layer for a cloud-init capable qcow image.
|
|
https://github.com/NucleaPeon/gentooimgr/
|
|
|
|
The huge advantage of this is that you can maintain the base image
|
|
under chroot, so you can build test instances that have no incoming
|
|
network and still maintain them using tools like ansible, which
|
|
support maintaing chroot instances. The problem of using 1) gentooimgr
|
|
is that the configuration code of the install step is written in
|
|
Python, rather than handled by a more capable tool like ansible.
|
|
|
|
The problem of using 2) gentooimgr is that the base box may be old (it
|
|
is) and you may want to use an existing gentoo kernel and initramfs,
|
|
and the base box may be missing qemu-quest-agent (it is). *Much* worse
|
|
is that to use cloud-init you have to load rust and 344 crates from
|
|
God only knows where that you'll never audit, just to get oauth: no
|
|
thanks Google.
|
|
|
|
You can use ansible to maintain the base layer using chroot, and use
|
|
it again to create and spin up the test instance. And as we install
|
|
qemu-quest-agent in the base layer, you can manage the test instance
|
|
with ansible using libvirt guest-agent, even if the test instance
|
|
allows no incoming network.
|
|
|
|
For now, the code is written to build Gentoo base images, but it could
|
|
be extended to other bases. It can build a Gentoo base image from a
|
|
Gentoo system. or another system with a Gentoo system mounted on some
|
|
directory, that can be chrooted into, and files copied from. It may be
|
|
able to build from a Debian without a mounted Gentoo filesystem, but this
|
|
is currently untested.
|
|
|
|
## Workflow
|
|
|
|
1) We build the qcow2 base image that we can maintain by chroot
|
|
mounting the disk, so we can sort out problems conveniently. We started
|
|
doing that by using EarlRuby's python approach, but later rewrote an
|
|
old ansible role by https://github.com/agaffney/ansible-gentoo_install/
|
|
That ansible role is in roles/ansible-gentoo_install/tasks/ which is
|
|
executed as an included role by the toxcore role.
|
|
|
|
It's a very basic piece of coding that works on a local connection
|
|
and is run on the host by the build_base Makefile target. It starts
|
|
out as a local connection play, and run chroot internally when it needs it.
|
|
You must set these variable in a host in hosts.yml in the linux_chroot_group:
|
|
BOX_NBD_DEV: nbd1
|
|
BOX_NBD_MP: /mnt/gentoo
|
|
BOX_NBD_FILES: "/i/data/Agile/tmp/Topics/GentooImgr"
|
|
BOX_NBD_BASE_QCOW: "/g/Agile/tmp/Topics/GentooImgr/gentoo.qcow2"
|
|
This role is slow and may take an hour or more;
|
|
It will build the BOX_NBD_BASE_QCOW.
|
|
|
|
As a safety feature you must create and open the qcow base image before
|
|
running the roles: the roles do not use qemu-nbd -c or qemu-nbd -d by
|
|
design. You may also choose to download the gentoo latest stage3 and
|
|
portage files to the directory specified in hosts.ynl as BOX_NBD_FILES
|
|
These
|
|
|
|
2) We build the qcow2 overlay image that we can maintain by libvirt.
|
|
It is run on the host by the build_overlay Makefile target which runs
|
|
/usr/local/bin/toxcore_build_overlay_qcow.bash. It gets its parameters from
|
|
the hosts.yml file from the host called gentoo1 in the linux_libvirt_group.
|
|
|
|
## Roles
|
|
|
|
There are 3 ansible roles:
|
|
|
|
1. base : The base role sets up the basics and is required to be run.
|
|
It sets up the essential parameters to run roles on the host or client.
|
|
Check the settings in roles/base/defaults/main.yml before running the role.
|
|
|
|
2. proxy : The proxy role sets up the networking with proxies,
|
|
and is required to be run, even if you don't use a proxy.
|
|
It sets proxying and installs basic packages on the host or client.
|
|
Check the settings in roles/proxy/defaults/main.yml before running the role.
|
|
|
|
3. toxcore :
|
|
This role sets up the software to run libvirt on the host.
|
|
Check the settings in roles/toxcore/defaults/main.yml before running the role.
|
|
|
|
In addition, toxcore calls an included role ansible-gentoo_install.
|
|
This is an updated version of the abandonned
|
|
https://github.com/agaffney/ansible-gentoo_install/ This role,
|
|
when run on the host, builds the Gentoo base qcow image. As a safety
|
|
feature, you must create the qcow2 image and activate it with:
|
|
|
|
The host creates the base qcow2 image and then creates the overlay
|
|
image. When both are created, it install Tox software on the host and
|
|
client.
|
|
|
|
modprobe nbd
|
|
qemu-img $BOX_NBD_BASE_QCOW 20G
|
|
qemu-nbd -c $BOX_NBD_DEV $BOX_NBD_BASE_QCOW
|
|
|
|
and put these values into the hosts.yml file in the pentoo or devuan
|
|
target, depending on your host operating system. The filesytem that
|
|
holds base qcow2 $BOX_NBD_BASE_QCOW must have at least 12G available,
|
|
and may grow to almost 20G.
|
|
|
|
After you have finished building the base qcow2 image, you will want
|
|
to dismount it with qemu-nbd -d $BOX_NBD_DEV. Be careful and look
|
|
in /proc/partitions to see if it is still there after you dismount it:
|
|
it the partition is busy the dismount will fail silently, and you can
|
|
get into trouble if partprobe complains. You may have to reboot,
|
|
or it may resolve itself. Wierd.
|
|
|
|
Each role has been conditionalized to run with different connections.
|
|
|
|
## Connection Types
|
|
|
|
There are 3 ansible connection types:
|
|
|
|
1. localhost : ( ansible_connection == local )
|
|
Running the roles with a local connection will setup the host to
|
|
be able to run the software. The toxcore role will build the
|
|
toxcore software on the localhost and runs tests on it. It will also
|
|
build the base qcow2 image that will underly the overlay box.
|
|
|
|
2. chroot : ( ansible_connection == chroot )
|
|
When you have built the base box, you can chroot mount the qcow2 image
|
|
with qemu-nbd, so all of the configuring of the base can be done without
|
|
a network. A local bash script then builds the overlay qcow2 instance,
|
|
in less than a minute.
|
|
|
|
3. remote : ( ansible_connection == libvirt_qemu )
|
|
The base box provides the libvirt_qemu connection that is be used
|
|
to run ansible roles on the overlay image. The toxcore role will
|
|
build the toxcore software in the overlay and runs tests on it.
|
|
|
|
All of the 3 roles can all be run with all 3 connection types.
|
|
|
|
## Stages
|
|
|
|
There are 4 stages to building an instance:
|
|
|
|
1. Setup the localhost :
|
|
set up the host up with the software and settings needed to build boxes.
|
|
|
|
2. Build the base qcow2 base box :
|
|
|
|
3. Build the overlay qcow2 instance :
|
|
|
|
4. Test the overlay instance :
|
|
|
|
## Hosts.yml targets
|
|
|
|
## Makefile targets
|
|
|
|
all: install lint build check run test
|
|
|
|
1. install
|
|
|
|
2. lint
|
|
|
|
3. build
|
|
|
|
4. check
|
|
|
|
5. run
|
|
|
|
6. test
|
|
|
|
## Simplest usage
|
|
|
|
On Ansibles from 2.10 and later, you will need the community plugins installed.
|
|
|
|
### Downloaded base qcow
|
|
|
|
### Created and copuied base qcow on a Gentoo system
|
|
|
|
### Created and copied base qcow on a non-Gentoo system with a Gentoo mounted
|
|
|
|
## Advanced Usage
|
|
|
|
### GentooImgr: Gentoo Image Builder for Cloud and Turnkey ISO installers
|
|
|
|
There is a modified version of https://github.com/NucleaPeon/gentooimgr/
|
|
where we've modified the code a little to do use Python logging. We can
|
|
still use it for the build stage, but we think the install stage is better
|
|
done using ansible, hence the libvirt_cloud playbook.
|
|
|
|
The code is in src/ansible_gentooimgr The code is being supported as
|
|
an ansible module using library/ansible_gentooimgr.py which is a work
|
|
in progress; the idea is to use it for the build and status actions,
|
|
but handle the install tasks using ansible.
|
|
|
|
### ansible_local.bash
|
|
|
|
We have a script that calls ansible to run our play: ansible_local.yml
|
|
but sets some defaults
|
|
|
|
[ -l limit ]
|
|
[ -c connection ]
|
|
[ --skip comma,separated]
|
|
[ --tags comma,separated]
|
|
[ --check]
|
|
[ --diff]
|
|
[ --step]
|
|
[ --verbose 0-3] higher number, more debugging
|
|
roles - base and proxy roles will always be run.
|