Ghanshyam Mann

OpenStack PTG for Xena development planning and discussions are held virtually from 19 – 23, April 2021. Overall we had good discussions and another successful virtual PTG though most of us missed the face-to-face interaction. This blog covers the OpenStack Quality Assurance discussion that happened on Monday and Tuesday.

 Etherpad

    Wallaby Retrospective

First, we discussed whether the QA office hour timing (14:00 UTC) is good for everyone? And we agreed to keep the same time 14:00 UTC on Tuesday.

We discussed all the good things we did or things we need to improve. Tempest scenario manager is a stable interface now and devstack provides a way to run in parallel mode. Glance testing is improved in devstack side support as well as in Tempest. QA team played a key role in keeping the CI/CD up and running especially for the stable branches with a mix of different distro/python versions supported.

On the improvement side, we still need to improve the review rate and patch merge time. Also keep continuing triaging the bugs during weekly office hours.

Action Items:

   Migration of Devstack and Tempest tests to new secure RBAC

As secure RBAC is the priority work for OpenStack, this is one of the priority things for us to do in Devstack and tempest. The main challenge is not all services are ready with the new RBAC. We can handle that via the configuration option in Devstack as well as in Tempest. For Example Tempest, DevStack.  But sometimes, Tempest integration tests involving the services and not all of them are ready with new RBAC will be challenging. We might need more work to fix such tests or have a workaround until all the required services by the test are ready with RBAC.

But we agree to work towards migrating the Devstack and Tempest tests to the new RBAC.

Action Items: All the work we need to do on Devstack and Tempest side.
Assignee: gmann, paras333, kopecmartin

   Patrole stable release and what all things are pending

Patrole which is an RBAC testing tool in QA needs to have a stable release. Stable release means we come to the point where the project starts adding it in their gate testing. One of the key things we need to improve in that is the performance of Patrole. Currently, it takes too much time because it performs the complete API operation to check the RBAC permissions.

We do not have active contributors in this project which is the main challenge we are facing to improve it. If you or anyone around you in your company or university would like to help us in this project, feel free to ping me.

Action Items: Improve the total run time of Patrole tests.
Assignee: Volunteer needed.

   What is Feature freeze means for QA

In the Wallaby cycle, to keep CI/CD green during the release time, we introduce the concept of ‘Feature Freeze’ in three of the QA projects Tempest, Devstack, and Grenade. It was not so clear that what ‘Feature Freeze’ means in QA (It is a clear concept in other OpenStack projects). We discussed the tasks and what we can freeze during the release time so that we can deliver the non-breaking CI/CD.

• Tempest: Week R-3 (Hard StringFreeze) of cycle release schedule. Example Victoria Release Schedule
  • The following is the subject of Feature Freeze:
    • New tests
      • New API tests are OK to merge after seeing a green gate
      • Scenario tests need to be discussed during QA office hour and will be decided based on their complexity
    • New dependencies/dependency bumps
      • Not to be merged unless necessary for the release.
    • Non-stable/stable interface
      • Any removal of deprecated interfaces or variable or any change which has possibility of breaking the plugins users.
      • If non-deprecated interface then we need to postpone to the next cycle.
    • Any framework change will be checked dynamically and decided whether to postpone to the next release or not.

• Devstack: Week R-3 (Hard StringFreeze) of cycle release schedule.
  • The following is the subject of Feature Freeze:
    • Changing the default behavior/configuration.
    • New backup/driver support if it’s not isolated.

• Grenade: Week R-3 (Hard StringFreeze) of cycle release schedule.
  • The following is the subject of Feature Freeze:
    • Changing the default behavior/configuration.

• Patrole: Week R-3 (Hard StringFreeze) of cycle release schedule.
  • The following is the subject of Feature Freeze:
    • Same as for Tempest.

We also agreed on below points:

Action Items: Update QA release wiki so that it mentions the points we raised above.
Assignee: kopecmartin

   Testing Tempest plugins jobs using Tempest master version

Tempest Plugins’ check/gate jobs aren’t using tempest from master, they use the latest tag instead, which is potentially dangerous as a code not compatible with tempest master might get merged, more info: https://bugs.launchpad.net/devstack/+bug/1916771

We can’t remove tempest from plugins’ requirements.txt. But one of the solutions is to remove tempest from master’s upper-constraints.txt and have it pinned only in the upper-contraints.txt of a stable release on every release.

Action Items: Try removing the Tempest from master’s u-c and see if that work
Assignee: gmann

   Plan on Tempest plugins to use Tempest’s scenario.manager

Tempest scenario manager is a stable interface for tempest plugins now. This means it is ready to use on the plugin side. But to remove their own copy we need to wait more due to the stable branch testing.

AGREED:

   Fix backward compatibility of run-tempest role

We deprecated a couple of tempest’s CLI args and in parallel introduced the new ones in Tempest 26.1.0, however, as it turns out the run-tempest role from Tempest master on stable branches creates the backward incompatibility. We will try to create a stable job variant to be used in jobs with older tempest versions.

Action Items: Create the job variant of run-temepst for old and new interface.
Assignee: gmann

At the end, we discussed the Xena cycle priority which are mentioned in this etherpad