OpenShift UPI install: install-config, ignition, bootstrap and CSRs

This is the install flow we follow for OpenShift 4 on user-provisioned infrastructure after the prerequisites and load balancers are ready. The commands are written for a RHEL 9 bastion that also serves ignition files. Names and addresses are documentation-only values.

This procedure was validated for the described release family and UPI architecture, not every OpenShift 4 release or RHCOS image. Before each build or version-family change, pin one supported release, use its matching installer/client/RHCOS artifacts, verify checksums and signatures through the approved supply chain, and revalidate install-config, Ignition schema and coreos-installer options against that release's documentation.

The sequence in one view#

1. Prepare the bastion and download the matching installer and oc client.
2. Write install-config.yaml and back it up.
3. Create manifests, apply any extra MachineConfigs, create ignition configs.
4. Publish ignition files from a temporary, access-controlled endpoint.
5. Boot bootstrap, then control plane nodes, then workers.
6. Wait for bootstrap-complete, remove bootstrap from the load balancers.
7. Approve CSRs, wait for install-complete, validate.

1. Bastion preparation#

Create a working directory and keep the installer, client and generated assets together.

umask 077
install_root="$HOME/ocp-install"
backup_root="$HOME/ocp-backup"
installer_archive="$HOME/downloads/openshift-install-linux-approved.tar.gz"
client_archive="$HOME/downloads/openshift-client-linux-approved.tar.gz"
install -d -m 0700 "$install_root" "$backup_root"
test -r "$installer_archive" && test -r "$client_archive"
sudo tar --no-same-owner -xzf "$installer_archive" -C /usr/local/bin openshift-install
sudo tar --no-same-owner -xzf "$client_archive" -C /usr/local/bin oc kubectl
sudo chown root:root /usr/local/bin/openshift-install /usr/local/bin/oc /usr/local/bin/kubectl
sudo chmod 0755 /usr/local/bin/openshift-install /usr/local/bin/oc /usr/local/bin/kubectl
openshift-install version && oc version --client

Installer and client versions must match the release you approved. Inspect archive members with tar tzf and verify vendor-published integrity/signature data before extraction; the generic archive names above deliberately avoid inventing a minor version.

2. install-config.yaml#

apiVersion: v1
baseDomain: example.com
metadata:
  name: ocp-prod
compute:
- name: worker
  replicas: 0            # UPI: you provision workers yourself
  architecture: amd64
  hyperthreading: Enabled
controlPlane:
  name: master
  replicas: 3
  architecture: amd64
  hyperthreading: Enabled
platform:
  none: {}
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 10.42.0.0/16
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
# proxy:                 # only if required
#   httpProxy: http://proxy.example.com:8080
#   httpsProxy: http://proxy.example.com:8080
#   noProxy: localhost,127.0.0.1,10.42.0.0/16,10.128.0.0/14,172.30.0.0/16,.example.com
pullSecret: 'REPLACE_WITH_VAULT_RETRIEVED_PULL_SECRET'
sshKey: 'REPLACE_WITH_APPROVED_SSH_PUBLIC_KEY'

Verify the exact fields against Red Hat's documentation for your version. The installer consumes this file, so copy it first:

umask 077
install_root="$HOME/ocp-install"
backup_root="$HOME/ocp-backup"
test -f "$install_root/install-config.yaml"
cp --preserve=mode,timestamps "$install_root/install-config.yaml" "$backup_root/install-config.yaml.$(date +%Y%m%dT%H%M%S%z)"
chmod 0600 "$install_root/install-config.yaml" "$backup_root"/install-config.yaml.*

3. Manifests, chrony and ignition#

openshift-install --dir ~/ocp-install create manifests

Because workers are not created by the installer, check scheduling of the control plane:

grep mastersSchedulable ~/ocp-install/manifests/cluster-scheduler-02-config.yml
# set to false if you want dedicated control plane nodes

To force your own NTP servers on every node, add MachineConfigs before generating ignition. Encode your chrony.conf:

base64 -w0 chrony.conf

# ~/ocp-install/openshift/99-worker-chrony.yaml  (repeat with role: master)
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-chrony
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - path: /etc/chrony.conf
        mode: 420
        overwrite: true
        contents:
          source: data:text/plain;charset=utf-8;base64,REPLACE_WITH_REVIEWED_BASE64_DATA

Then generate ignition:

openshift-install --dir ~/ocp-install create ignition-configs
ls -lh ~/ocp-install/*.ign ~/ocp-install/auth/

Re-running manifest creation on the same directory overwrites generated files. Do not generate ignition until the real pull secret is in place, and never boot nodes with dummy or old ignition files.

4. Publish ignition#

Use a temporary web endpoint that is reachable only from the provisioning and node networks. The exact TLS and trust-bootstrap options differ by RHCOS release, so select the supported method from the installation documentation for the release image being installed. For an isolated HTTP provisioning network, a minimal example is:

sudo dnf install --assumeno httpd
# After repository, package, dependency and change approval:
sudo dnf install -y httpd
umask 077
install_root="$HOME/ocp-install"
publish_root=/var/www/html/ocp4
web_group=apache
getent group "$web_group"
sudo install -d -o root -g "$web_group" -m 0750 "$publish_root"
for role in bootstrap master worker; do
  test -s "$install_root/$role.ign" || exit 1
  sudo install -o root -g "$web_group" -m 0640 "$install_root/$role.ign" "$publish_root/$role.ign"
done
sudo restorecon -Rv "$publish_root"
# Restrict the listener and firewall to the approved source networks.
sudo systemctl enable --now httpd
ignition_url=http://10.42.10.31/ocp4/master.ign
curl --fail --silent --show-error --noproxy '*' --head "$ignition_url"

web_group must be changed to the actual unprivileged group used by the installed server. Do not make the files world-readable merely to fix a permissions error.

Treat ignition files as secrets. They contain credentials and configuration that can grant access to the cluster. Bind only to the provisioning interface where possible, allow only expected node source addresses, disable directory listing and logging of content or query strings, and stop the service and remove the published copies after installation. If HTTPS is used, configure the RHCOS-supported CA verification mechanism; do not bypass verification as a permanent design. Using an IP address can reduce DNS dependencies, but it does not provide confidentiality or server authentication.

5. Boot order and RHCOS install#

Boot bootstrap first, then the three control plane nodes, then workers. Do not power on physical nodes until the ignition URLs return HTTP 200.

# From the RHCOS live environment, control plane example
# Verify the target disk and the options supported by this RHCOS image first.
boot_disk=/dev/disk/by-id/REPLACE_WITH_VERIFIED_LOCAL_BOOT_DISK
ignition_url=https://provisioning.example.com/ocp4/master.ign
test -b "$boot_disk"
lsblk -d -o NAME,SIZE,MODEL,SERIAL "$boot_disk"
# Stop for serial/capacity review, then run the release-reviewed command and trust options.
sudo coreos-installer install "$boot_disk" --ignition-url "$ignition_url"
reboot

Use worker.ign for workers and the matching ignition for the bootstrap machine. coreos-installer install destroys data on the target disk. Confirm its by-ID path, serial number and capacity against the build record before running it. Do not copy --insecure-ignition from an example without understanding the transport and verification consequences for your RHCOS release.

Per-node check before reboot: hostname, IP, DNS, correct ignition URL, and a matching backend line in the load balancer.

If a node does not appear:

journalctl -b -u kubelet --no-pager | tail -200
journalctl -b -u crio --no-pager | tail -100
ip addr; ip route; cat /etc/resolv.conf
curl -I http://10.42.10.31/ocp4/master.ign
cluster_ca=/secure/path/cluster-ca-bundle.pem
test -r "$cluster_ca"
curl --fail --show-error --cacert "$cluster_ca" -I https://api-int.ocp-prod.example.com:22623/config/master
KUBECONFIG="$HOME/ocp-install/auth/kubeconfig" oc get --raw='/readyz?verbose'

Use the release-documented trust bundle and endpoint test. A TLS or hostname error is a failed check; do not add -k or disable certificate verification to hide it.

6. Bootstrap complete#

openshift-install --dir ~/ocp-install wait-for bootstrap-complete --log-level=info

When it reports success, remove the bootstrap server lines from the API and MCS backends on both load balancers and reload HAProxy. Then shut down the bootstrap machine.

7. Approve CSRs carefully#

Nodes can request client and serving certificates during installation. Do not approve every Pending CSR or assume there will always be exactly two per node. For each request, compare its signer, requester, requested subject/SANs and usages with the expected node and with Red Hat's CSR procedure for the installed release.

export KUBECONFIG=~/ocp-install/auth/kubeconfig
oc get csr
csr_name=REPLACE_WITH_INSPECTED_CSR_NAME
oc describe csr "$csr_name"
# Approve only this inspected, expected request.
oc adm certificate approve "$csr_name"
watch -n 5 'oc get nodes -o wide; echo; oc get csr | tail -20'

Record the CSR name and the matching inventory node before approval. Leave unexpected requests Pending while investigating them; never use an unfiltered loop or pipeline to approve all Pending CSRs. Repeat the inspect-and-approve step until every expected node is Ready.

8. Install complete and first checks#

openshift-install --dir ~/ocp-install wait-for install-complete --log-level=info
oc get nodes -o wide
oc get clusterversion
oc get co          # Available=True, Progressing=False, Degraded=False
oc get mcp         # updated, not degraded

Save the console URL and the initial kubeadmin credentials from auth/ in your vault. Stop HTTP serving, remove published Ignition copies, and verify they return 404 after every node is provisioned; retain the protected original install directory only under the approved credential-retention policy. Next, set up enterprise login and retire that account: OpenShift LDAP/AD authentication and removing kubeadmin.

If bootstrap or install completion fails, preserve logs and stop rather than regenerating assets over the working directory. Rollback means powering off unaccepted nodes, removing bootstrap from load-balancer pools if it was added incorrectly, restoring the reviewed load-balancer configuration, revoking temporary publication access, and rebuilding affected RHCOS nodes from newly reviewed assets. Ignition is first-boot state and is not safely "rolled back" in place.

Examples use documentation-only names and addresses. Validate against the Red Hat documentation for your exact OpenShift version.

Need a second pair of experienced hands for your cluster build? See Kubernetes and OpenShift or start a project.