# Cinc Omnibus Toolchain Long-Term Maintenance Plan

## Background

`omnibus-toolchain` is the foundation of every Cinc omnibus build: it
ships the bash, git, ruby, gem, bundle, make, curl, and GNU userland
that downstream projects (cinc-workstation, cinc-server, cinc-auditor,
cinc-client, …) source via `/home/omnibus/load-omnibus-toolchain.sh`
inside their build containers.

Historically Cinc consumed two artifacts:

1. The `chef/omnibus-toolchain` source on GitHub, lightly patched on a
   `stable/cinc` branch hosted at
   `gitlab.com/cinc-project/upstream/omnibus-toolchain`.
2. A separate downstream-build wrapper at
   `gitlab.com/cinc-project/distribution/toolchain` that cloned the
   above, merged the patches, and built packages for the architectures
   Chef did not cover (ppc64le, s390x, riscv64).

Chef's omnibus pipeline is being wound down as part of the move to
Chef 360 / Habitat-only, so Cinc now maintains `omnibus-toolchain` as
a **proper independent fork** in the same shape as cinc-workstation:
source, build pipeline, CI, and tests in a single repo at
`gitlab.com/cinc-project/distribution/omnibus-toolchain`.

## Current State

### Architecture

| Component             | Description                                            |
|-----------------------|--------------------------------------------------------|
| `config/projects/`    | Omnibus project def (`omnibus-toolchain.rb`)           |
| `config/software/`    | Cinc-specific software defs on top of omnibus-software |
| `build/build*.sh`     | Per-platform CI build entry points                     |
| `ci/`                 | GitLab CI templates                                    |
| `test/integration/`   | InSpec validation profiles                             |

### Codebase Metrics

- Single omnibus project (`omnibus-toolchain`).
- 2 Cinc-specific software defs (`helper-gems`, `omnibus-toolchain`),
  everything else pulled from the omnibus-software gem.
- Build matrix: 28 jobs covering 9 Linux distros across x86_64,
  aarch64, ppc64le, s390x, riscv64; macOS 14/15/26 (x86_64 + aarch64);
  FreeBSD 13/14; Windows 2016.

### Cinc Patches Carried From Chef

These predate the proper-fork migration and remain on the `main`
branch:

- `Use the latest libffi globally to fix compiler issues`
- `Point to our mirror`
- `Use Cinc S3 omnibus bucket`
- `Add support for Alpine Linux`
- `Don't exclude arm from making berkshelf`
- `Set default_version for helper-gems so caching works`
- `Update to latest version to work around run time issue`
- `Use berkshelf-no-depselector to make things easier to build on odd archs`
- `Point omnibus-software to Cinc repo`

---

## Phase 1: Initial Fork Consolidation (May 2026)

### 1.1 Repository Migration

- [x] Create `cinc-project/distribution/omnibus-toolchain` GitLab project
- [x] Push current `stable/cinc` from `cinc-project/upstream/omnibus-toolchain`
      as `main`
- [x] Open `fork/proper-migration` MR with the cleanup commits
- [ ] Retire the `cinc-project/upstream/omnibus-toolchain` project once
      the `cinc-project/distribution/toolchain` build-wrapper's
      `patch.sh` no longer references it
- [ ] Retire the `cinc-project/distribution/toolchain` build-wrapper repo
      once this repo produces all required artifacts

### 1.2 Strip Chef Infrastructure

- [x] Remove `.expeditor/`, `.github/`, `.buildkite`-style scripts
- [x] Remove Test-Kitchen suites and `omnibus-test.{sh,ps1}`

### 1.3 Rebrand

- [x] Maintainer / homepage / copyright moved to Cinc Project
- [x] Drop signing identities (Apple, Windows code-signing, RPM passphrase)
- [x] Generate fresh MSI `upgrade_code`
- [x] `install_dir` on Windows moved to `c:/cinc-project/`
- [x] Reset `VERSION` to `26.0.0`; archive history as `CHANGELOG.pre-cinc.md`
- [x] New `README.md` describing Cinc consumption flow and platform matrix

### 1.4 CI / Release Flow

- [x] `build/{build,build-macos,build-freebsd,build-windows,build-source,deploy}.sh`
- [x] `ci/{lint,package,test,rules}.yml` modeled on cinc-workstation
- [x] `.gitlab-ci.yml` orchestrating lint -> package -> test -> deploy
      -> publish -> release -> tag
- [x] `cinc-project/ci-tools` changelog-flow components
      (`changelog-release-mr` + `changelog-tag-release`) for label-driven
      SemVer releases, paired with `cliff.toml`

### 1.5 Build Validation

- [ ] Successful Linux build for at least one x86_64 distro
- [ ] Successful Linux build across the full matrix
- [ ] Successful macOS build on macos-14/15/26 (both archs)
- [ ] Successful FreeBSD build on freebsd-13/14
- [ ] Successful Windows build on the existing windows-2016 runner
      once the operator cuts over from the Chef seed toolchain
      (see [§3.1](#31-windows-builder))
- [ ] First publish to `current` channel on the OSUOSL mirror
- [ ] Downstream `cinc-omnibus-cookbook` install path verified end-to-end
      (toolchain installs cleanly into a fresh `cincproject/omnibus-debian:13`
      rebuild)

---

## Phase 2: Ongoing Maintenance (Q3 2026+)

### 2.1 Upstream Tracking

Chef is winding down `chef/omnibus-toolchain`, so we no longer
cherry-pick from upstream as a routine. We do still:

1. Watch upstream for security fixes to bundled software
2. Pull software def changes from the omnibus-software gem
   (`gitlab.com/cinc-project/distribution/omnibus-software`). The gem
   is sourced from `rubygems.cinc.sh` in `Gemfile`; we do not track a
   `Gemfile.lock`, so each CI build resolves to the latest published
   version automatically.

### 2.2 Dependency Update Strategy

We carry no local version pins for OpenSSL, Ruby, libffi, cmake,
gtar, or git. All software defaults are inherited from the
omnibus-software gem at build time, so the upgrade loop is:

| Component        | Update Frequency  | Notes                         |
|------------------|-------------------|-------------------------------|
| omnibus-software | As-needed         | No local pin; the gem is sourced from `rubygems.cinc.sh` in `Gemfile` and each CI build resolves the latest published version, pulling in the latest defaults for OpenSSL, Ruby, libffi, cmake, gtar, etc. |
| cinc-auditor     | As-needed         | Bumped in `cinc-auditor.rb` over in omnibus-software, then propagates here on the next gem bump. |
| helper-gems      | As-needed         | Local def in `config/software/helper-gems.rb`; bump pinned gem versions here. |

### 2.3 Platform Support

#### Current Matrix

| Family         | Versions                | Architectures                       |
|----------------|-------------------------|-------------------------------------|
| Amazon Linux   | 2023                    | x86_64                              |
| AlmaLinux/RHEL | 8, 9, 10                | x86_64, aarch64 (8/9/10), ppc64le (8/9/10), s390x (9/10) |
| Debian         | 12, 13                  | x86_64, riscv64 (13)                |
| Rocky Linux    | 8, 9, 10                | x86_64                              |
| Ubuntu         | 20.04, 22.04, 24.04, 26.04 | x86_64, riscv64 (24.04)          |
| macOS          | 14, 15, 26              | x86_64, aarch64                     |
| FreeBSD        | 13, 14                  | x86_64                              |
| Windows        | 2016+                   | x64                                 |

#### Lifecycle

- **Add** new platforms when downstream Cinc products start consuming them.
- **Drop** EOL platforms 6 months after the downstream project drops them.
- **Test** new platforms in MR pipelines before adding to release builds.

### 2.4 Version Management

- Version file: `VERSION` in repo root (currently `26.0.0`).
- Push to `main` runs the `cinc-project/ci-tools` `changelog-release-mr`
  component, which opens (or refreshes) an MR that bumps `VERSION` and
  prepends a new `CHANGELOG.md` entry. Bump magnitude is label-driven on
  the merged source MR: `version::patch` default / `version::minor` /
  `version::major`.
- When the release MR merges, the `changelog-tag-release` component
  creates and pushes the matching tag. The release pipeline is then
  triggered manually from the GitLab UI on the resulting tag.

---

## Phase 3: Long-Term Sustainability

### 3.1 Windows Builder

The Windows package and test jobs run on the existing
`windows-2016` runner shared with cinc-workstation. No dedicated
Windows VM is required: the Cinc toolchain installs to
`c:\cinc-project\omnibus-toolchain\` while the seed Chef toolchain
sits at `c:\opscode\omnibus-toolchain\`, so the first Cinc build
on this runner has no path collision. Subsequent builds (after the
operator installs the Cinc-built MSI) are protected by the same
rename-aside pattern as macOS — see §3.2.

#### Bootstrap Ruby for the build

`build/build-windows.sh` needs a Ruby on PATH that does **not** live
under `c:\cinc-project\omnibus-toolchain` (which the script renames
aside — see §3.2 for the rationale; omnibus binaries hard-code
install_dir in their wrapper scripts and stop working after a rename).

**`build/build-windows.sh` auto-installs
[RubyInstaller-DevKit](https://rubyinstaller.org/downloads/) for Ruby
3.4 at `C:\Ruby34-x64-toolchain` if it is missing** (silent Inno Setup
install, pinned version in the script). This is distinct from the
standard `C:\Ruby34-x64` path that omnibus-software builds on the same
runner own and routinely remove. The dedicated path stops the two
projects from clobbering each other's interpreter — and the auto-
install means a wipe by another job self-heals on the next build.

The installer .exe is cached in the GitLab CI cache (`cache/`),
restored from S3 on every run, so the network round-trip to GitHub
releases only happens on a cold cache. To bump the pinned version,
edit `RUBYINSTALLER_VERSION` at the top of `build/build-windows.sh`.

We use the `-devkit` installer because omnibus pulls in gems with
native C extensions (ffi-yajl, ohai, parts of aws-sdk) that need
MSYS2 + MinGW to compile when a precompiled fat binary is not
published for the target Ruby. The silent install bundles MSYS2 under
`C:\Ruby34-x64-toolchain\msys64\` without an interactive `ridk`
prompt.

**Do NOT add `C:\Ruby34-x64-toolchain\bin` to the runner's system PATH.**
The runner is shared with cinc-workstation / cinc-server / etc. builds
that source `load-omnibus-toolchain.ps1` and expect the
omnibus-toolchain Ruby to win on PATH. System-wide RubyInstaller would
shadow that.

Instead, `build/build-windows.sh` prepends `RUBYINSTALLER_BIN` to PATH
only for its own invocation. The default is `/c/Ruby34-x64-toolchain/bin`;
override via a GitLab CI/CD variable named `RUBYINSTALLER_BIN`
(msys2-style path) if you installed RubyInstaller elsewhere.

We do **not** call `C:\omnibus\load-omnibus-toolchain.ps1` from the
package job — that shim adds a toolchain install to PATH, and the
build is about to rename that toolchain aside.

#### First-time cutover from Chef seed to Cinc toolchain

1. (Optional, faster first build) Pre-install **RubyInstaller-DevKit**
   for Ruby 3.4 at `C:\Ruby34-x64-toolchain`. The build script will
   auto-install it on a fresh runner anyway, but operator pre-install
   skips a ~1-minute download+install step on the first build. Use a
   non-default path only if you also set the `RUBYINSTALLER_BIN`
   CI/CD variable to the msys2-style bin path. **Do not add it to the
   system PATH**, and **do not use `C:\Ruby34-x64`** (the
   omnibus-software builds own that path).
2. Trigger a release pipeline. The first build auto-installs
   RubyInstaller if needed, prepends `RUBYINSTALLER_BIN` to its own
   PATH, drives omnibus, writes a fresh artifact to
   `c:\cinc-project\omnibus-toolchain\`, and produces an MSI. The
   rename-aside path is skipped on this first run because
   `c:\cinc-project\omnibus-toolchain` does not yet exist.
3. Install the Cinc-built MSI on the runner. Verify
   `c:\cinc-project\omnibus-toolchain\embedded\bin\ruby.exe --version`
   works.
4. The runner is now in the steady state: every build renames
   `c:\cinc-project\omnibus-toolchain` to `.bootstrap`, drives omnibus
   from the locally-prepended RubyInstaller, builds into a fresh copy,
   packages an MSI, and the trap restores the bootstrap on exit.

#### Ongoing operation

- After every release: uninstall the previous MSI, install the new
  one into `c:\cinc-project\omnibus-toolchain\`. The build script
  handles the bootstrap-rename automatically, so no separate
  snapshot maintenance is needed.
- If something on the runner removes `C:\Ruby34-x64-toolchain`, the
  next build will auto-reinstall it (download from GitHub if the
  cached installer is also gone, otherwise from the cached .exe in
  `cache/rubyinstaller/`). Operator intervention is not required.

#### Why we might still want a dedicated Windows VM (future)

Sharing the `windows-2016` runner with cinc-workstation means a
broken toolchain build can affect cinc-workstation's build queue.
If that becomes a problem, stand up a dedicated `windows-toolchain`
runner with the same software prerequisites (Visual Studio Build
Tools with `Desktop development with C++`, WiX 3.11+, Git, 7-Zip,
seed toolchain MSI) and retag the Windows jobs accordingly. Not
required to ship the fork.

### 3.2 macOS / FreeBSD Builder Upgrade Runbook

The macOS runners (`macos-14`, `macos-14-aarch64`, `macos-15`,
`macos-15-aarch64`, `macos-26`, `macos-26-aarch64`) and the FreeBSD
runners (`freebsd-13`, `freebsd-14`) come pre-loaded with a previous
`omnibus-toolchain` install at `/opt/omnibus-toolchain`. They share
the same bootstrap pattern, build-script structure, and operator
runbook — the only difference is the package format (.pkg/.dmg on
macOS, self-extracting .sh on FreeBSD).

#### Runner prerequisites

`build/build-macos.sh` and `build/build-freebsd.sh` install a fresh
Ruby 3.4 via RVM, compiled from source. RVM's autolibs is disabled in
the scripts (it tries to `brew update` mid-build, which is fragile),
so the runner must already have the libraries needed to compile
Ruby's bundled C extensions — most importantly `psych` (depends on
libyaml; without it `bundle install` dies with `cannot load such file
-- psych`).

On macOS, install once per runner (any version, both arches):

```bash
HOMEBREW_NO_AUTO_UPDATE=1 brew install libyaml openssl@3 readline libffi
```

`HOMEBREW_NO_AUTO_UPDATE=1` skips the auto-update step that can fail
on self-hosted runners with stale brew installs.

On FreeBSD, the equivalent:

```bash
sudo pkg install -y libyaml openssl readline libffi
```

Why each:
- `libyaml` — Ruby's `psych` extension (YAML parser) Bundler needs.
- `openssl` — Ruby's `openssl` extension; without it Bundler cannot
  talk to https gem sources.
- `readline` — Ruby's `readline` extension (irb history, debugging).
- `libffi` — Ruby's `fiddle` stdlib and the `ffi` gem (pulled in
  transitively by `ffi-yajl` → `omnibus`).

Xcode Command Line Tools (macOS) / the base FreeBSD `pkg install -y
gcc make autoconf automake libtool pkgconf` toolchain must also be
present — those are usually already installed as part of the
original omnibus-toolchain bootstrap.

The Linux builds install these themselves in `build/build.sh`'s
`dnf install` / `apt install` lines, so no operator action is
required on Docker runners.

#### How a build avoids overwriting the running toolchain

Two collaborating mechanisms:

1. **Rename the live install aside for rollback safety.** Before any
   build work, `build/build-macos.sh` and `build/build-freebsd.sh`
   move the live install out of the way and set a trap that restores
   it on script exit, success or failure:

   ```bash
   sudo mv /opt/omnibus-toolchain /opt/omnibus-toolchain.bootstrap
   trap 'sudo rm -rf /opt/omnibus-toolchain; \
         sudo mv /opt/omnibus-toolchain.bootstrap /opt/omnibus-toolchain' EXIT
   ```

   This guarantees the runner ends every build — even a failed one —
   in the exact state it started. `.bootstrap` is ephemeral, not a
   long-lived snapshot.

2. **Bootstrap Ruby via RVM in the repo workspace.** Once the install
   is renamed, the build cannot use the renamed copy's `ruby`, `gem`,
   or `bundle` even though those binaries still exist on disk: every
   omnibus-built binary's shebang hard-codes `install_dir`
   (`#!/opt/omnibus-toolchain/embedded/bin/ruby`), so they all
   reference a path that no longer exists. The scripts sidestep this
   by installing a fresh Ruby 3.4 under `${TOP_DIR}/rvm` and driving
   omnibus from there. The build then writes to a clean
   `/opt/omnibus-toolchain` without touching the renamed bootstrap.

If a previous build was killed before its trap could run, the next
build's startup checks repair the state: if `.bootstrap` exists and
the live install looks broken, the bootstrap is moved back; if both
look healthy, the bootstrap is treated as stale leftover and removed.

#### Post-release upgrade (operator-driven)

After a successful release the runner still has the *previous*
toolchain installed (the trap restores it). To pick up the
just-built version, for each macOS / FreeBSD runner:

1. SSH into the builder.
2. Optionally back up the current install for rollback:
   `sudo mv /opt/omnibus-toolchain /opt/omnibus-toolchain.prev`
3. Install the new package:
   - macOS: `sudo installer -pkg ~/Downloads/omnibus-toolchain-*.pkg -target /`
   - FreeBSD: `sudo sh ~/Downloads/omnibus-toolchain-*.sh`
4. Verify: `/opt/omnibus-toolchain/embedded/bin/ruby --version`

The next build will start its rename-aside dance against the *new*
install, so future packages are produced against the upgraded
toolchain.

#### Rollback

If the new toolchain breaks downstream consumers and you preserved
the previous install at step 2:

```bash
sudo rm -rf /opt/omnibus-toolchain
sudo mv /opt/omnibus-toolchain.prev /opt/omnibus-toolchain
```

### 3.3 What We Maintain

- Omnibus packaging for all platforms in the matrix.
- Security updates for bundled dependencies (carried via omnibus-software).
- Platform support for current OS versions consumed by downstream
  Cinc products.
- Cinc branding consistency.

### 3.4 What We Do **Not** Do

- Major feature development (toolchain is a foundation, not a product).
- Chef Habitat packaging.
- AIX, Solaris support (Cinc does not ship for these).
- Package signing (Cinc does not have signing certificates).

---

## CI/CD Pipeline

### Pipeline Architecture

```
Stages: lint -> package -> test -> deploy -> publish -> release -> tag

MR Pipeline:
  lint (auto) -> debian-13:x86_64 single platform (manual, optional)

Release Pipeline (web trigger):
  lint -> all platforms build -> all platforms test -> deploy (manual) -> publish

Push to main:
  changelog-release-mr (opens/refreshes a release MR with VERSION + CHANGELOG bumps)

Release MR merge:
  changelog-tag-release (creates and pushes the matching tag)
```

### Infrastructure

| Component        | Provider                                       |
|------------------|------------------------------------------------|
| Linux x86_64 builds   | OSUOSL Docker runners (`docker-x86_64`)   |
| Linux aarch64 builds  | OSUOSL Docker runners (`docker-aarch64`)  |
| Linux ppc64le builds  | OSUOSL Docker runners (`docker-ppc64le` on EL8, `docker-ppc64le-p9` on EL9+) |
| Linux s390x builds    | OSUOSL Docker runners (`docker-s390x`)    |
| Linux riscv64 builds  | OSUOSL Docker runners (`docker-riscv64`)  |
| macOS builds          | Self-hosted (`macos-{14,15,26}[-aarch64]`)|
| FreeBSD builds        | Self-hosted (`freebsd-{13,14}`)           |
| Windows builds        | Shared `windows-2016` runner (see §3.1)   |
| S3 cache              | OSUOSL (`s3.osuosl.org/cinc-omnibus-cache`) |
| Package mirror        | OSUOSL (`ftp-osl.osuosl.org`)             |
| Source mirror         | OSUOSL                                    |

### Build Images

Bare upstream images (NOT `cincproject/omnibus-*` — see Guardrails):

- `debian:{12,13}`
- `ubuntu:{20.04,22.04,24.04,26.04}`
- `almalinux:{8,9,10}`
- `rockylinux/rockylinux:{8,9,10}`
- `amazonlinux:2023`
- `riscv64/debian:trixie`
- `cincproject/docker-auditor` (for source tarball validation only)

---

## Risk Assessment

| Risk                                                | Impact | Likelihood | Mitigation |
|-----------------------------------------------------|--------|-----------|------------|
| Chef removes their omnibus-toolchain MSI            | High   | Already happening | Stand up Cinc Windows builder (§3.1) |
| macOS builder upgrade breaks next build             | Medium | Medium    | Keep `/opt/omnibus-toolchain.prev` backup |
| omnibus-software gem regression                     | High   | Low       | InSpec test:* jobs catch on every release |
| Cross-arch builder unavailability (ppc/s390/riscv)  | Medium | Low       | Skip platform temporarily; Cinc Server/Auditor users degrade gracefully |
| Bare distro image bootstrap deps drift              | Medium | Medium    | `build/build.sh` pins explicit dnf/apt package list |
| RVM upstream changes break `rvm install ruby-3.4`   | Medium | Low       | Pin specific RVM tag if it breaks |

---

## References

- [Cinc Project](https://cinc.sh/)
- [Cinc Omnibus Toolchain GitLab](https://gitlab.com/cinc-project/distribution/omnibus-toolchain)
- [Cinc Workstation](https://gitlab.com/cinc-project/distribution/cinc-workstation)
- [Cinc omnibus-software](https://gitlab.com/cinc-project/distribution/omnibus-software)
- [cinc-omnibus-cookbook](https://gitlab.com/cinc-project/cinc-omnibus-cookbook)
- [docker-images](https://gitlab.com/cinc-project/docker-images)
- [Upstream Chef omnibus-toolchain](https://github.com/chef/omnibus-toolchain)
- [OSUOSL](https://osuosl.org/) — Infrastructure sponsor
