OpenStack upstream CI/CD tests the things on defined LTS or stable distribution versions. OpenStack Technical Committee defines each cycle testing runtime. As per OpenStack Victoria testing runtime, defined versions are:
- Ubuntu 20.04
- CentOS 8
- openSUSE Leap 15
Ubuntu Focal (Ubuntu LTS 20.04) was released on April 23, 2020, and in OpenStack Victoria (released on 14th Oct 2020), we migrated the upstream CI/CD testing on the above-defined testing runtime. This work is done as one of the community-wide goals “Migrate CI/CD jobs to new Ubuntu LTS Focal“.
What is this migration:
OpenStack CI/CD is implemented with Zuul jobs prepare the node to deploy the OpenStack using Devstack and run tests (Tempest or its plugins, project in-tree tests, rally tests etc). Base OS installed on the node is where OpenStack will be deployed by DevStack.
Till OpenStack Ussuri release, the base OS on the majority of the job’s node was Ubuntu Bionic (18.04). So DevStack used to deploy OpenStack on Ubuntu Bionic and then run tests.
With the new version of Ubuntu Focal (20.04), the node’s base OS has been moved from Ubuntu Bionic -> Ubuntu Focal. On every code change, it will make sure OpenStack work properly on Ubuntu Focal.
NOTE: This migration target only zuulv3 native jobs. Legacy jobs are left to keep running on Bionic and plan to be migrated on Focal while they migrate to zuulv3 native jobs . We had another community-goal to migrate all the legacy jobs to zuulv3 native.
Perform migration testing in advance:
We started the work in June and prepared the devstack, tempest, and tox-based base jobs on Focal so that all projects gate can be tested and fixed in advance before devstack and Tempest base jobs merge. The idea behind the advance testing is to avoid or minimize the gate failure in any of the repo under any projects. This advance testing includes integration as well as tox based unit, functional, doc, pep8, and lower-constraint testing.
Bugs & fixes:
This migration had more things to fix compared to the previous migration from Ubuntu Xenial to Bionic. One reason for that was Ubuntu Focal and python dependencies dropping python2.7 support and MySQL 8.0 compatibility. OpenStack already dropped the Python2.7 in the Ussuri release but OpenStack dependencies lower constraints were not updated to their python-3 only version because many of them were not python3-only at that time. So in Ubuntu Focal, those dependencies versions are python3-only which caused many failures in our lower constraints jobs.
A few of the key issues we had to fix for this migrations are:
- Failing device detachments on Focal: Bug#1882521
- Few of the lower constraints are not compatible with python3.8: Bug#1886298.
- Migrations broken on MySQL 8.x: story/2007732
- Duplicate check constraint name ‘ck_started_before_ended’: story/2008121
- No implicit user creation with GRANT syntax in MySQL 8.0: Bug#1885825
- pyflakes till 2.1.0 not compatible with py3.8: Bug#1886296.
Fixing these bugs took a lot of time for us and that is the reason this migration was late and missed the initial deadlines.
- Tox based job migration happened on Sept 9th.
- devstack and Tempest base jobs migration happened on Sept 24th.
All the work for this migration are tracked on: https://storyboard.openstack.org/#!/story/2007865
All changes are: https://review.opendev.org/q/topic:%2522migrate-to-focal%2522+(status:open+OR+status:merged)
How to migrate the third-party CI:
If your 3rd party CI jobs are still not migrated to zuulv3 then you need to first migrate legacy jobs to zuulv3 native. Refer to this community-goal for details.
For zuulv3 native jobs, like upstream jobs you need to switch the jobs nodeset from ubuntu Bionic to ubuntu Focal.
Below Diagram gives a quick glance of changing the nodeset to Ubuntu Focal:
If you want to verify the nodeset used in your zuul jobs, you can see the hostname and label in job-output.txt
In the same way, you can migrate your third-party CI also to Focal. If third-party job is using the base job without overriding the ‘nodeset’ then the job is automatically switched to Focal. If the job overrides the ‘nodeset’ then, you need to switch to Focal nodeset like shown above. All the Ubuntu Focal nodeset for a single node to multinode jobs are defined in devstack.
We encourage all the 3rd party jobs to migrate to Focal asap as devstack will not support the Bionic related fixes from Victoria onwards.
Dependencies compatibility versions for upgrades to OpenStack Victoria:
There are many dependencies constraints that need to be bumped to upgrade the OpenStack cloud to the Victoria release. To know all those compatible versions, check the project-specific patches merged from here or from this Bug#1886298. This can help to have prepared the smooth upgrades.
Completion Summary Report:
- A lot of work was involved in this. There are many changes required for this migration as compare to previous migration from Xenial to Bionic.
- Many dependencies with lower constraints compatible with py3.8 and Focal distro had to be updated. Almost in all the repos.
- Mysql 8.0 caused many incompatible DB issues.
- I missed the original deadline of m-2 to complete this goal due to falling gates and to avoid any gate block for any projects.
- More than 300 repo had been fixed or tested in advance before migration happened. This helped a lot to keep the gate green during this migration.
I would like to convey special Thanks to everyone who helped in this goal and made it possible to complete it in the Victoria cycle itself.