OpenStack PTG for Xena development planning and discussions are held virtually from 19 – 23, April 2021. Overall we had good discussions and another successful virtual PTG though most of us missed the face-to-face interaction. This blog covers the OpenStack Quality Assurance discussion that happened on Monday and Tuesday.
Wallaby Retrospective
First, we discussed whether the QA office hour
we agreed to keep the same time 14:00 UTC on Tuesday.We discussed all the good things we did or things we need to improve. Tempest scenario manager is a stable interface now and devstack provides a way to run in parallel mode. Glance testing is improved in devstack side support as well as in Tempest. QA team played a key role in keeping the CI/CD up and running especially for the stable branches with a mix of different distro/python versions supported.
On the improvement side, we still need to improve the review rate and patch merge time. Also keep continuing triaging the bugs during weekly office hours.
Action Items:
As secure RBAC is the priority work for OpenStack, this is one of the priority things for us to do in Devstack and tempest. The main challenge is not all services are ready with the new RBAC. We can handle that via the configuration option in Devstack as well as in Tempest. For Example Tempest, DevStack. But sometimes, Tempest integration tests involving the services and not all of them are ready with new RBAC will be challenging. We might need more work to fix such tests or have a workaround until all the required services by the test are ready with RBAC.
But we agree to work towards migrating the Devstack and Tempest tests to the new RBAC.
Action Items: All the work we need to do on Devstack and Tempest side.
Assignee: gmann, paras333, kopecmartin
Patrole stable release and what all things are pending
Patrole which is an RBAC testing tool in QA needs to have a stable release. Stable release means we come to the point where the project starts adding it in their gate testing. One of the key things we need to improve in that is the performance of Patrole. Currently, it takes too much time because it performs the complete API operation to check the RBAC permissions.
We do not have active contributors in this project which is the main challenge we are facing to improve it. If you or anyone around you in your company or university would like to help us in this project, feel free to ping me.
Action Items: Improve the total run time of Patrole tests.
Assignee: Volunteer needed.
What is Feature freeze means for QA
In the Wallaby cycle, to keep CI/CD green during the release time, we introduce the concept of ‘Feature Freeze’ in three of the QA projects Tempest, Devstack, and Grenade. It was not so clear that what ‘Feature Freeze’ means in QA (It is a clear concept in other OpenStack projects). We discussed the tasks and what we can freeze during the release time so that we can deliver the non-breaking CI/CD.
- Tempest: Week R-3 (Hard StringFreeze) of cycle release schedule. Example Victoria Release Schedule
- The following is the subject of Feature Freeze:
- New tests
- New API tests are OK to merge after seeing a green gate
- Scenario tests need to be discussed during QA office hour and will be decided based on their complexity
- New dependencies/dependency bumps
- Not to be merged unless necessary for the release.
- Non-stable/stable interface
- Any removal of deprecated interfaces or variable or any change which has possibility of breaking the plugins users.
- If non-deprecated interface then we need to postpone to the next cycle.
- Any framework change will be checked dynamically and decided whether to postpone to the next release or not.
- New tests
- The following is the subject of Feature Freeze:
- Devstack: Week R-3 (Hard StringFreeze) of cycle release schedule.
- The following is the subject of Feature Freeze:
- Changing the default behavior/configuration.
- New backup/driver support if it’s not isolated.
- The following is the subject of Feature Freeze:
- Grenade: Week R-3 (Hard StringFreeze) of cycle release schedule.
- The following is the subject of Feature Freeze:
- Changing the default behavior/configuration.
- The following is the subject of Feature Freeze:
- Patrole: Week R-3 (Hard StringFreeze) of cycle release schedule.
- The following is the subject of Feature Freeze:
- Same as for Tempest.
- The following is the subject of Feature Freeze:
We also agreed on below points:
Action Items: U
Assignee:
Testing Tempest plugins jobs using Tempest master version
We can’t remove tempest from plugins’ requirements.txt. But one of the solutions is to
Action Items: T
Assignee: gmann
Plan on Tempest plugins to use Tempest’s scenario.manager
Tempest scenario manager is a stable interface for tempest plugins now. This means it is ready to use on the plugin side. But to remove their own copy we need to wait more due to the stable branch testing.
AGREED:
Fix backward compatibility of run-tempest role
We deprecated a couple of tempest’s CLI args and in parallel introduced the new ones in Tempest 26.1.0, however, as it turns out the run-tempest role from Tempest master on stable branches creates the backward incompatibility. We will try to create a stable job variant to be used in jobs with older tempest versions.
Action Items: Create the job variant of run-temepst for old and new interface.
Assignee: gmann
At the end, we discussed the Xena cycle priority which are mentioned in this etherpad