There were many reasons people came to use CentOS as an alternative Linux platform to RedHat Enterprise Linux (RHEL). CentOS was originally built as a downstream release of RHEL, which was free to use without support. CentOS became the de facto standard for many organizations that did not want to use RHEL for production workload, since it’s basically the same thing, just rebranded.
While CentOS was originally a community project, over time RedHat began contributing substantially and eventually took over maintenance of the linux Variant. This allowed them to shift the direction of the CentOS project, much to the dismay of its users. Especially those who rely on it for their production workloads.
With the shift to “Stream”, RedHat has moved CentOS 8 from being a downstream release (built after RHEL) to an upstream release (built before RHEL). This means that instead of a stabilized release, CentOS Stream is now somewhat of a development/QA release, which is unacceptable for production system use across most organizations.
With all the vulnerabilities in the wild (and with numbers growing rapidly by the day), it’s simply not an option to leave these systems as is. There are several options available including re-installation to a new OS, updating to CentOS Stream 8, or migration to another downstream RHEL variant (of which there are many available today).
Updates not available, the process is broken.
Not only did CentOS 8 go end of life, but the latest updates have been archived to the CentOS vault, removing the packages from mirrors. This caused a breaking change in the update manager for any systems using the internet based mirrors. If you’re using an internal proxy system to stage the updates, you may not even realize that you are no longer receiving updates for CentOS 8.
This means that a lot of CentOS 8 hosts can’t even update to the most recent packages that are available, at least not without modifications.
This brings us to the first task…
The first task in the example module addresses this issue at scale. Systems should be updated to the latest version of CentOS 8 that is available in the CentOS 8 vault, prior to migration to any other Linux variant.
In the attached example puppet module, the `to_centos_vault` task will update the yum repos to point to the CentOS 8 vault so you can get the most recent updates that are available. While there will be no future updates, this will at least get you to the latest that were available up to the point of EOL.
Note: If you’re using any desired state (i.e. Puppet Code) to manage the current yum repos on your systems, you may (likely will) need to disable that prior to migrations and make modifications for the new target OS to be applied after migration.
What migration options do you have?
While this is not an exhaustive list, I will cover some of the most common options. The example module has tasks to convert to each of the following, which can be used at scale.
CentOS 8 Stream
If you’re fine using an upstream release to RHEL, then CentOS Stream 8 is the path of least resistance. It’s still EL8. It’s close to RHEL, but it is a few steps ahead, so maybe not the best choice for production workloads.
EL8 downstream alternatives (AlmaLinux 8 & Rocky Linux 8)
Both Rocky and Alma are nearly identical alternatives to CentOS8. They are both downstream, rebranded, community supported variants of RHEL.
Automate your migration with Puppet
Why use Puppet?
Scale, consistency, speed & reduced risk.
Simply put, if you’re doing these migrations manually for a lot of systems, you may accidentally skip steps or end up with other inconsistencies, which could lead to your systems being left in an unstable, or even worse, an unrepairable state.
Using Puppet Tasks allows for an efficient and consistent way to perform these migration tasks at scale. It will also save the output from the Tasks in a centralized manner for reporting and audit tracking purposes. If you’re running these migrations individually from a shell, the output is typically lost as soon as you close the terminal, which can be disastrous for large migration projects that need to be tracked or migrations that are performed by a team of many individuals.
If you have hundreds or even up to hundreds of thousands of systems to migrate, manual migration or using scripts on each host is nearly impossible. At the very least, it will take a significant amount of time and you may be exposed to multiple vulnerabilities before you can complete the project. Tasks can help significantly reduce the time it takes to migrate at scale.
Not supported, but it works.
While this example module is fully functional and should work without issue, there are a few caveats and you are completely on your own for support.
That said, If you’re using CentOS 8 today, you’re likely used to rolling on your own and are fine without official support anyway.
Use at your own risk
There is no support for this workflow or module from Puppet, nor myself. You are on your own.
Applications may not support the target OS
Check your applications. This will convert the underlying OS. While it’s still technically EL8, you may have issues with applications that don’t support the target OS.
Have a backout plan
Use snapshots and backups before migration if possible. This is a potentially destructive operation, so act accordingly.
The “centos8migrate” Puppet Module
The example Puppet module can be found on GitHub here:
https://github.com/psreed/centos8migrate
With Puppet Enterprise, you can simply add this module to your Puppetfile in your current control repository to allow the tasks to become available in the PE Tasks interface:
Puppetfile:
## centos8migratemod 'psreed-centos8migrate', :git => 'https://github.com/psreed/centos8migrate.git', :branch => 'main' |
This module has been tested starting with a default CentOS 8.3 installation “Server with GUI”.
These Puppet Tasks are simple, but functional. They should probably be expanded into Puppet Plans for use in production, which could include better error handling or adding workflows for your snapshot/backup processes. At the very least, these can be used as building blocks for a more robust solution.
The tasks include the following:
centos8migrate::to_centos_vault
This task will update YUM repos to point to the CentOS 8 vault so you can update to the last available updates. After running this task, followed by a `yum update`, you should have a system that’s running CentOS 8.5 v2111
centos8migrate::to_centos_stream
This task will convert your system to CentOS Stream 8. It’s the least impactful of the options, but if Stream is not a production option for you, consider one of the following tasks to convert to a different Linux variant.
centos8migrate::to_rocky
This task will convert CentOS 8 systems to Rocky Linux 8
centos8migrate::to_alma_linux
This task will convert CentOS 8 systems to AlmaLinux 8
Note: As a caution, you may want to temporarily disable the periodic Puppet run while these migration tasks are in progress. Yum will typically lock, however you could get into a race condition. Simply stop the puppet agent service before running these tasks, but don’t stop the pxp-agent! You will still be able to execute the tasks using the pxp-agent and the puppet agent should be restarted after the migration’s reboot (as part of the default process).
Shameless Plug: Puppet Solutions & Services
Disclaimer: I currently work for Puppet Inc.
If your project is too big to handle on your own, or you would like to know how to manage your infrastructure more consistently and at scale, please reach out to Puppet at sales@puppet.com
Puppet Enterprise and other Puppet Products and Services can help drastically reduce your technical debt and overhead through automated workflows. Let’s discuss how to better get you to your goals of a stable, compliant, automated infrastructure whether it’s on premise or in the cloud!
Good Luck!
Hopefully this will help if you have a CentOS 8 migration project upcoming or already underway!
Feel free to reach out to us at Puppet if you have any questions on how our automation can help you!
