Docker container
To run docker on proxmox we will create a new container based on the rocky linux 9.5 lxc template.
Resources:
- 2 GB memory
- 512 MB swap
- 2 cores
- 20 GB diskspace.
Ensure the lxc container is connected to a physical network (one that is connected to a vmbr[x] network), there can be issues with docker lxc contianers, connected to software defined networks. These issues can be very subtle, so test everything before you install lxc on SDN in production.
It looks like overkill, but we will add multiple services to this docker instance.
Docker needs a special configuration parameter to run nested in a lxc container.
After creation, do not start the container yet, adapt the settings under "Options":
- Unprivilleged container = no
- Features = nesting=1, FUSE=1
After changing these options, start the container.
The option FUSE=1 is very important, if you want to run podman builds in a container (pipeline), this will only work if this option is set.
Once the container has been started, log in on the console.
On the proxmox host where the lxc lives, edit the configuration file for the lxc:
vi /etc/pve/lxc/
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.apparmor.profile: unconfined
this will reduce security, but when running a gitlab runner in docker, this will ensure an errorfree environment.
Installing docker
To install docker on this rocky linux container, we run the following ansible playbook:
---
- name: Install docker
hosts: "{{ instances | default('dummy') }}"
tasks:
- name: Write some extra options into the configuration of the LXC host
ansible.builtin.blockinfile:
path: "/etc/pve/lxc/{{ hostvars[inventory_hostname]['id'] }}.conf"
insertafter: EOF
mode: '0650'
owner: root
group: www-data
block: |
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
become: true
delegate_to: "{{ hostvars[inventory_hostname]['proxmox_node'] }}.homelab"
failed_when: false
- name: Part 1 install docker
become: true
block:
- name: Add docker repos to hosts
ansible.builtin.yum_repository:
file: docker-ce
name: docker-ce-stable
baseurl: https://download.docker.com/linux/rhel/9/x86_64/stable
gpgcheck: false
enabled: true
description: docker-ce
- name: Install packages
ansible.builtin.package:
name:
- docker-ce
- docker-ce-cli
- containerd.io
- docker-buildx-plugin
- docker-compose-plugin
- nfs-utils
- python3.11
- name: Enable the docker service
ansible.builtin.service:
name: docker
enabled: true
state: started
- name: Load modules by default
ansible.builtin.copy:
dest: /etc/modules-load.d/docker.conf
content: |
ip_tables
ip_conntrack
iptable_filter
ipt_state
mode: '0655'
owner: root
group: root
- name: Write docker daemon.json
ansible.builtin.template:
src: daemon.json.j2
dest: /etc/docker/daemon.json
owner: root
group: root
mode: '0644'
- name: Set the volume dir writable by ansible
ansible.builtin.file:
path: /var/lib/docker/volumes
recurse: true
mode: '0775'
owner: root
group: ansible
- name: Reboot the machine
ansible.builtin.reboot:
Upon reading this ansible playbook, you will see it uses a template file, this templates the earlier mentioned daemon.json file.
Testing your docker installation.
docker run hello-world
sudo aa-status
docker ps -a
docker image ls
docker rmi hello-world:latest --force
docker rm 7642a4a6b9c9
When your Hello World container runs without errors your docker installation is complete and functional.
Ready for work....
Adding a local registry
To be able to pull images from your local registry, we will add a registry to our docker installation.
We will use this registry lateron.
docker run -d --restart unless-stopped -p 5000:5000 --name registry registry:latest
Simple, but effective.
To be able to pull images from this registry with the correct hostname(we do not use https yet),
add the following file to the docker host:
/etc/docker/daemon.json
{
"insecure-registries" : ["<your-docker-host-fqdn>:5000"],
"min-api-version": "1.43"
}
Reload the system daemon files systemctl daemon-reload
Restart docker systemctl restart docker
After restarting docker, your pipelines can pull images from this registry, based on hostname.
The min-api-version setting, mitigates a bug in the interface betwween the runner and the newest docker version.
What will run on this docker container
Some enterprise services we need to make this installation mimmick an enterprise
can be run in a docker container. We will run the following services in docker containers:
- ldap authentication
- gitlab pipeline runner
- pve-prometheus-exporter
- prometheus
- grafana
We might add more, for the moment is this enough.
Creating images for pipeline use
When we add a gitlab-runner container to this docker instance, we will need images which can be run by
pipelines. What we do not want, is that one image runs all. This is a security risk, create small images
for specific tasks.