OpenStack CI/CD migration from Ubuntu Xenial -> Bionic (Ubuntu LTS 18.04)

admin Open Source, OpenStack

Ubuntu Bionic (Ubuntu LTS 18.04) was released on April 26, 2018 but OpenStack CI/CD and its all gate jobs were running on Ubuntu Xenial version. We have to migrate the OpenStack to Ubuntu Bionic at this time and make sure we test all ur gate jobs with Ubuntu Bionic image.

Jens Harbott (frickler) and I started this task in December 2018 in two phases. Phase 1st to migrate the zuulv3 devstack based jobs and 2nd phase to migrate the legacy zuulv2 jobs.

    What is the meaning of migration:

OpenStack CI/CD is implemented with Zuul jobs prepare the node to deploy the OpenStack using Devstack and run tests (Tempest or its plugins, project in-tree tests, rally tests etc). Base OS installed on node is what where OpenStack will be deployed by DevStack.

Till OpenStack Rocky release, base OS on node was Ubuntu Xenial. So DevStack will deploy OpenStack on Ubuntu Xenial and then running tests to make sure every project governed by OpenStack work properly on Ubuntu Xenial.

With the new version of Ubuntu Bionic, the node base OS has been moved from Ubuntu Xenial -> Ubuntu Bionic. The same way it will make sure the OpenStack CI/CD with Ubuntu Bionic. On every code change, it will make sure OpenStack work properly on Ubuntu Bionic.

    Goal:

The end goal is to migrate all the OpenStack projects gate jobs testing from Ubuntu Xenial to Ubuntu Bionic and make sure OpenStack works fine on Bionic.

As OpenStack support stable branch also, all the jobs running till OpenStack Rocky gate will be on Ubuntu Xenial and all jobs running from OpenStack Stein onwards will be on Ubuntu Bionic.

    Phase1:

It started with devstack based zuul v3 jobs first

  • https://etherpad.openstack.org/p/devstack-bionic

devstack base job nodeset has been switched to Bionic and before we merge the devstack patch, we had DNM testing

patches on each project side to make sure we do not break anyone. Few projects faced the issue and fixed before devstack patch merge.

You can check more details on ML.

    Phase2:

After finishing the devstack zuul v3 native jobs, we have to move all legacy jobs also to Bionic. Most of the projects are still using legacy jobs and as per Stein PTI we need to move to next python version (py3.6 and py3.7 in Train).

Work is tracked in https://etherpad.openstack.org/p/legacy-job-bionic

legacy-base and legacy-dsvm-base job nodeset was moved to Bionic and before it merge I have pushed DNM testing patch for all the projects.

Step 1. Push the testing DNM patch on your project owned repos with:

– Depends-On: https://review.openstack.org/#/c/641886/

-Remove or change the nodeset to Bionic from your repo owned legacy jobs (if any of the legacy jobs has overridden the nodeset from parent job) Example: https://review.openstack.org/#/c/639017

Step 2. If you have any legacy job not derived from base jobs then, you need to migrate their nodeset to bionic which are defined in – https://review.openstack.org/#/c/639018/ Example – https://review.openstack.org/#/c/639361/

Step3. if any of the jobs start failing on bionic then, either fix it before the deadline of March 13th or make failing job as n-v and fix later.

You can check more details on ML.

Below Diagram gives a quick glance of base image nodeset using Bionic

If you want to verify the nodeset used in your zuul jobs, you can see the hostname and label in job-output.txt

 

    Migrate the third-party CI to Bionic:

The same way you can migrate your third-party CI also to Bionic. If third-party job is using the base job without overriding the ‘nodeset’ then job is automatically switched to Bionic. If job overrides the ‘nodeset’ then, you can use the Bionic node to test on Bionic. Third-party CI jobs are not migrated as part of upstream migration.

    Completion Summary Report:

– Started this migration very late in stein (in Dec 2018)

– Finished migration in 2 part. 1. zuulv3 native job migration, 2. legacy jobs migration.

– networking-midonet bionic jobs are n-v because it is not yet ready on bionic.

– ~50 patches merged to migrate the gate jobs

– ~ 60 DNM testing patches done before migration happens

– We managed almost zero downtime in the gate. Except for Cinder which was for 1 day.

I sent the summary of this work on OpenStack ML also- here with all references.

You May Also Like..

OpenInfra Day Veitnam: Technical Committee highlights and different missions teams in Upstream

Open Infra day Vietnam was held in Hanoi in Aug 2019. I and Rico lin talked about the current activities […]

Recap of Open Infrastructure Summit & PTG, Shanghai 2019

Open Infrastructure Summit, Shanghai 2019 Open Infrastructure Summit followed by OpenStack PTG was held in Shanghai, China: 4th Nov 2019 […]

Open Infrastructure Summit, Shanghai 2019: QA Summit & PTG Summary

OpenStack Quality Assurance Summit & PTG Summary Open Infrastructure Summit, Shanghai 2019 Open Infrastructure Summit followed by OpenStack PTG was […]

Leave a Reply

Your email address will not be published. Required fields are marked *