OpenStack CI/CD migration from Ubuntu Xenial -> Bionic (Ubuntu LTS 18.04)

admin Open Source, OpenStack

Ubuntu Bionic (Ubuntu LTS 18.04) was released on April 26, 2018 but OpenStack CI/CD and its all gate jobs were running on Ubuntu Xenial version. We have to migrate the OpenStack to Ubuntu Bionic at this time and make sure we test all ur gate jobs with Ubuntu Bionic image.

Jens Harbott (frickler) and I started this task in December 2018 in two phases. Phase 1st to migrate the zuulv3 devstack based jobs and 2nd phase to migrate the legacy zuulv2 jobs.

    What is the meaning of migration:

OpenStack CI/CD is implemented with Zuul jobs prepare the node to deploy the OpenStack using Devstack and run tests (Tempest or its plugins, project in-tree tests, rally tests etc). Base OS installed on node is what where OpenStack will be deployed by DevStack.

Till OpenStack Rocky release, base OS on node was Ubuntu Xenial. So DevStack will deploy OpenStack on Ubuntu Xenial and then running tests to make sure every project governed by OpenStack work properly on Ubuntu Xenial.

With the new version of Ubuntu Bionic, the node base OS has been moved from Ubuntu Xenial -> Ubuntu Bionic. The same way it will make sure the OpenStack CI/CD with Ubuntu Bionic. On every code change, it will make sure OpenStack work properly on Ubuntu Bionic.

    Goal:

The end goal is to migrate all the OpenStack projects gate jobs testing from Ubuntu Xenial to Ubuntu Bionic and make sure OpenStack works fine on Bionic.

As OpenStack support stable branch also, all the jobs running till OpenStack Rocky gate will be on Ubuntu Xenial and all jobs running from OpenStack Stein onwards will be on Ubuntu Bionic.

    Phase1:

It started with devstack based zuul v3 jobs first

  • https://etherpad.openstack.org/p/devstack-bionic

devstack base job nodeset has been switched to Bionic and before we merge the devstack patch, we had DNM testing

patches on each project side to make sure we do not break anyone. Few projects faced the issue and fixed before devstack patch merge.

You can check more details on ML.

    Phase2:

After finishing the devstack zuul v3 native jobs, we have to move all legacy jobs also to Bionic. Most of the projects are still using legacy jobs and as per Stein PTI we need to move to next python version (py3.6 and py3.7 in Train).

Work is tracked in https://etherpad.openstack.org/p/legacy-job-bionic

legacy-base and legacy-dsvm-base job nodeset was moved to Bionic and before it merge I have pushed DNM testing patch for all the projects.

Step 1. Push the testing DNM patch on your project owned repos with:

– Depends-On: https://review.openstack.org/#/c/641886/

-Remove or change the nodeset to Bionic from your repo owned legacy jobs (if any of the legacy jobs has overridden the nodeset from parent job) Example: https://review.openstack.org/#/c/639017

Step 2. If you have any legacy job not derived from base jobs then, you need to migrate their nodeset to bionic which are defined in – https://review.openstack.org/#/c/639018/ Example – https://review.openstack.org/#/c/639361/

Step3. if any of the jobs start failing on bionic then, either fix it before the deadline of March 13th or make failing job as n-v and fix later.

You can check more details on ML.

Below Diagram gives a quick glance of base image nodeset using Bionic

If you want to verify the nodeset used in your zuul jobs, you can see the hostname and label in job-output.txt

 

    Migrate the third-party CI to Bionic:

The same way you can migrate your third-party CI also to Bionic. If third-party job is using the base job without overriding the ‘nodeset’ then job is automatically switched to Bionic. If job overrides the ‘nodeset’ then, you can use the Bionic node to test on Bionic. Third-party CI jobs are not migrated as part of upstream migration.

    Completion Summary Report:

– Started this migration very late in stein (in Dec 2018)

– Finished migration in 2 part. 1. zuulv3 native job migration, 2. legacy jobs migration.

– networking-midonet bionic jobs are n-v because it is not yet ready on bionic.

– ~50 patches merged to migrate the gate jobs

– ~ 60 DNM testing patches done before migration happens

– We managed almost zero downtime in the gate. Except for Cinder which was for 1 day.

I sent the summary of this work on OpenStack ML also- here with all references.

You May Also Like..

OpenStack QA tooling & How to use it for Production Cloud testing

OpenStack Days + CloudNative Japan 2019 Conference Day 1, July 22nd, Room F, 6.00 PM & OpenInfra Day Vietnam, 2019 […]

Open Infrastructure Summit: QA Summary for Summit & PTG

Open Infrastructure Summit followed by OpenStack PTG held in Denver, USA on 29th April 2019 till 4th May 2019. We […]

OpenStack QA Stein PTG Summary

  We had another great PTG in Denver for Stein cycle. It was good time to meet all co-developers and […]

Leave a Reply

Your email address will not be published. Required fields are marked *