ocp-customizations

This repository contains various scripts and YAMLs to perform several very specific customizations to an OCP4.x cluster. They all in some form of another require the use of a custom Red Hat CoreOS installer image.

Building the image

The custom installer image can be built with the following steps:

git clone https://github.com/RHsyseng/coreos-installer-custom-partitions -b legacy
cd coreos-installer
curl https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/latest/rhcos-installer-initramfs.x86_64.img -o rhcosinstall-initramfs.img # or the latest currently available
./combine.sh

The resulting file will be called rhcos-install-new.img and can be used as the installer image for PXE booting OCP nodes.

NetworkManager tweaks

To use a MAC address naming scheme in NetworkManager, create a file called 99-default.link:

[Link]
NamePolicy=mac
MACAddressPolicy=persistent

To disable additional NICs, and use dhclient for DHCP instead of the NetworkManager internal mechanism, create a file called 10-dhcp-config.conf:

[main]
no-auto-default=*
dhcp=dhclient

These files must be base64 encoded and included in a custom ignition configuration.

First create a template for the ignition template:

{
  "ignition": {
    "config": {},
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "2.2.0"
  },
  "networkd": {},
  "passwd": {},
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "overwrite": false,
        "path": "/etc/systemd/network/99-default.link",
        "contents": {
          "source": "data:text/plain;charset=utf-8;base64,${DEFAULT_LINK}",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "overwrite": false,
        "path": "/etc/NetworkManager/conf.d/10-dhcp-config.conf",
        "contents": {
          "source": "data:text/plain;charset=utf-8;base64,${DISABLE_NICS}",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "systemd": {}
}

Then base64 encode the contents of the two NetworkManager configuration files:

export DEFAULT_LINK=$(base64 -w 0 99-default.link)
export DISABLE_NICS=$(base64 -w 0 10-dhcp-config.conf)

Then generate the final ignition file:

envsubst '${DEFAULT_LINK} ${DISABLE_NICS}' < configure-nics.ign.tmpl > configure-nics.ign

The configure-nics.ign file can then be merged with ignition files generated by the openshift-install command:

assets_dir=/path/to/ignition/configs
ignition_extra=configure-nics.ign

openshift-install --dir ${assets_dir} create ignition-configs
mv ${assets_dir}/master.ign{,.orig}
jq -s '.[0] * .[1]' ${ignition_extra} ${assets_dir}/master.ign.orig | tee ${assets_dir}/master.ign
mv ${assets_dir}/worker.ign{,.orig}
jq -s '.[0] * .[1]' ${ignition_extra} ${assets_dir}/worker.ign.orig | tee ${assets_dir}/worker.ign
openshift-install --dir ${assets_dir} create ignition-configs
mv ${assets_dir}/bootstrap.ign{,.orig}
jq '.storage.files[.storage.files | length] |= .+ {"filesystem":"root","overwrite":false,"path":"/etc/systemd/network/99-default.link","contents":{"source":"data:text/plain;charset=utf-8;base64,W0xpbmtdCk5hbWVQb2xpY3k9bWFjCk1BQ0FkZHJlc3NQb2xpY3k9cGVyc2lzdGVudAo=","verification":{}},"mode":420}' < ${assets_dir}/bootstrap.ign.orig > ${assets_dir}/bootstrap.ign.tmp
jq '.storage.files[.storage.files | length] |= .+ {"filesystem":"root","overwrite":false,"path":"/etc/NetworkManager/conf.d/10-dhcp-config.conf","contents":{"source":"data:text/plain;charset=utf-8;base64,W21haW5dCm5vLWF1dG8tZGVmYXVsdD0qCmRoY3A9ZGhjbGllbnQK","verification":{}},"mode":420}' < ${assets_dir}/bootstrap.ign.tmp > ${assets_dir}/bootstrap.ign

The newly generated bootstrap, master and worker ignition files can now be used for deploy the OCP4 cluster.

Configure an OVS Bond + Bridge

Make any necessary modifications to setup-ovs.sh and mco_ovs.yml.tmp and run:

export SCRIPT_BASE64=$(base64 -w 0 setup-ovs.sh)
envsubst '${SCRIPT_BASE64}' < mco_ovs.yml.tmpl > mco_ovs.yml

Then apply the MachineConfig to the cluster:

for node in $(oc get nodes -l node-role.kubernetes.io/worker --no-headers=true -o name | awk -F/ '{print $2}'); do
  oc label node $node network.operator.openshift.io/external-openvswitch=true
done
oc apply -f mco_ovs.yml

Storage

mco_storage.yml will mount an extra 5th partition in the specified location. Modify mco_storage.yml if necessary (e.g. to change the path) and apply it to the cluster:

oc apply -f mco_storage.yml

Combined MCO

To save on reboots, all customizations can be combined in to one MachineConfig object.

export SCRIPT_BASE64=$(base64 -w 0 setup-ovs.sh)
envsubst '${SCRIPT_BASE64}' < mco_all.yml.tmpl > mco_all.yml

And apply it to the cluster:

oc apply -f moc_all.yml