We had another great PTG in Denver for Stein cycle. It was good time to meet all co-developers and discuss the stein cycle plan. I am writing the QA PTG summary report. Detailed discussion can be found in respective topic Etherpad or in main Etherpad.
QA Help Room
QA team was present in Help Room on Monday. We were happy to help few queries from Octavia multinode job and Kuryr-kubernetes testing part. Other than that, there was not much that day except few other random queries.
We discussed the Rocky Retrospective as first thing on Tuesday. We went through 1. what went well and 2. what needs to improve and gather some concrete action items.
Patrole has good progress in Rocky cycle with code as well as documentation. Also we were able to fill the compute microversion gap almost all till Rocky.
- Need to add Tempest CLI documentation and other usefull stuff from tripleo Doc to Tempest Doc – chandankumar
- Run all tests in tempest-full-parallel job and move it to periodic job pipeline – afazekas
- Need to merge the QA office hour, check with andrea for 17 UTC office hour and if ok then, close that and modify the current office hour from 9 UTC to 8 UTC . – gmann
- Need to ask chandankumar or manik for bug triage volunteer. – gmann
- Create the low hanging items list and publish for new contributors – gmann
We will be tracking the above AI in our QA office hour to finish them on time.
Stable interfaces from Tempest Plugins
We discussed about having stable interface from Tempest plugins like Tempest so that other plugins can consume those. Service client is good example of those which are required to do cross project testing. For example: congress tempest plugin needs to use mistral service clients for integration testing of congress+mistral. Similarly Patrole need to use neutron tempest plugin service client(for n/n-1/n-2).
Idea here is to have lib or stable interface in Tempest plugins side like Tempest so that other plugins can use them. We will start with some documentation about use case and benefits and then work with neutron-tempest-plugin team to make their service client expose as stable interface. Once that is done then, we can suggest the same to other plugins.
- Need some documentation and guidance with use case and example, benefits for plugins. – felipemonteiro
- mailing list discussions on making specific plugins stable that are consumed by other plugins – felipemonteiro
- check with requirement team to add the tempest plugin in g-r and then those can be added on other plugins requirement.txt – gmann
Tempest Plugins CI setup & Plugins release and tagging clarification
We discussed about how other projects or Plugins can setup the CI to cover the stable branches testing on their master changes. Solution can be simple to define the supported stable branches and run them on master gate (same way Tempest does). QA team will start the guidelines on this.
Other part we need to cover is release and tagging guidelines. There were lot of confusion about release of Tempest plugins in Rocky. To make it better, QA team will write guidelines and document the clear process.
- move/update documentation on branchless considerations in tempest to somewhere more global so that it covers plugins documentation too – gmann
- Add tagging and release clarification for plugins.
- talk with neutron team about moving in-tree tempest plugins of stadium projects to neutron-tempest-plugin or separate tempest-plugins repositories – slaweq
- Add config options to disable the plugins load – gmann
Tempest Cleanup Feature
Current Tempest CLI for cleanup the test resource is not so good. It does cleanup the resources based on saved_state.json file which save the resources difference before and after Tempest run. This can end up cleaning up the other non-test resources which got created during time period of tempest run.
There is a QA spec which proposing the different approach for cleanup. After discussing all those approach, we decided to go with resource_prefix. We will bring back the resource_prefix approach (which got removed after deprecation) and modify the “tempest cleanup” cli to cleanup resource based on resource_prefix. Complete discussion can be found in etherpad. As of felipemonteiro is the owner but he will check with nicholashelgeson or AT&T folks to further work on this.
- Update spec with idea from 0.0.2 (because it’s relatively easy to implement) – get merged – felipemonteiro/nicholashelgeson
- Add back resource_prefix config option and add back to data_utils.rand_name – felipemonteiro/nicholashelgeson
- Go through all tempest tests and make sure data_utils.rand_name is used for resources – felipemonteiro/nicholashelgeson
- Update tempest cleanup – felipemonteiro/nicholashelgeson
- Update documentation – felipemonteiro/nicholashelgeson
Tempest conf Plugin Discovery process
This topic is about generating the Tempest conf from plugins config options too. This idea is more for python-tempestconf not for QA as such. But there were python-tempestconf folks present in QA room and discussed that this is doable in python-tempestconf itself.
There is nothing from QA side on this so I would like to drop this item from QA tracking.
Proper handling of interface/volume/pci device attach/detach hotplug/unplug
Tempest tests not handling the hotplug/unplug events properly. The guest does not check for button press event at early boot time.
The hotplug events sent before the kernel fully initialized can be lost. test_stamp_pattern.py could be unskipped, if we would try to ssh vm before hot plug event (volume attach). Also there are API tests which knows nothing about the guest state, therefore it cannot know when the guest is ready for checking the button press. Details problem can be found here.
Idea is to perform the ssh validation step before we consider the test server ready to use in test.
- Adding ssh validation steps for api/scenrio tests where is required – afazekas
- making the run_validation default true – afazekas
- soft reboot , is nova event tells was it soft reboot (check) , is some special register on the machine can tell it ? – afazekas
Shared network handling
Attila observed few tests failing when using shard network. But looks like the only 100% reproducible issue is test_create_list_show_delete_interfaces_by_fixed_ip
There should not be issue for shared network also and as of now we will just fix the failing tests.
- Fix the failing tests – afazekas
- Try to run tempest in parallel with shared network create/delete parallel testS to search for other incidents locally) – afazekas
Planning for Patrole Stable release
We continue the Patrole stable release discussion in PTG. We prepared the concrete list of items we need to release it stable and targeted for Stein cycle.
Along with multi-policy, system scope support, we will check the framework stability also. Documentation is already in better shape.
TODO items before stable release:
- system scope support:
- Better support for generic check policy rules (e.g. user_id:%(user.id))
- Iterate through all the patrole interface/framework which needs to be used outside of patrole
- let’s finish the above planned items in Stein. – felipemonteiro
Proposal for New QA project: Harbinger
OpenStack QA currently lacks a dataplane testing framework that can be easily consumed, configured and maintained. To fill that gap, there is a new project proposal called “Harbinger”. Harbinger allows execution of various OpenStack data plane testing frameworks while maintaining a single standardized interface and input format.
Currently it cover Shaker and Yardstick and Kloudbuster is WIP. This can be useful to consume in Eris (extreme testing idea). There are few points which need more clarification like Standardization of output, can it cover Control plane testing etc. IMO, this is good project to start and can be consumed in Eris and cross community testing. Author or this project was not present in PTG and felipemonteiro proxy this too. We would like to extend this discussion on ML and with Extreme testing stackholders and also start the QA spec.
- There are many item we planned as AI but first step will to start the ML and spec.
Clean up the tempest documentation
This is always outstanding item :-). We discussed about more improvement in documentation like better doc structure, CLI doc, consuming the Tempest related docs at central place which is Tempest. We had list of item to cover with different assignee.
- Complete all the documentation points written in etherpad. – tosky, gmann, masayukig
Consuming all Tempest CLI in gate
Tempest has many CLI and due to lack of unit tests, there are chance where we can/did break the CLI. Idea is to consume all the CLI on gate job so that we can improve their testing coverage. Few CLI will be covered in main gate job and other as functional testing.
- Continue this patch on zuul v3 – https://review.openstack.org/#/c/355666/ – masayukig
- add funcional tests and new functionl job – gmann
Migration from Launchpad to storyboard
We discussed about possibility of migration to storyboard. Patrole is first QA projects we are trying and based on that we can proceed on other projects.
- Wait for BP script from storyboard team and then finish the Patrole projects migration. – gmann
- Feedback or request for Storyboard:
- Create a story for adding some filed to indicative user interest (heat / vote / points ) – afazekas
- Gerrit automation work to story about automatic assignee etc or adding the Topic field on gerrit.
- Have a way to not diverge too much the set of used tags – possible solution: sort proposed tags by popularity
QA Stein Priority:
We discussed about the priority items for Stein and listed the items and owner of each items in etherpads.