Open Infrastructure Summit: QA Summary for Summit & PTG

admin Open Source, OpenStack

Open Infrastructure Summit followed by OpenStack PTG held in Denver, USA on 29th April 2019 till 4th May 2019.

We had good discussions at QA forum sessions in Summit and PTG. I am summarizing the QA related discussions.

    Summit: QA Forum sessions:

    1. OpenStack QA – Project Update: Tuesday, April 30, 2:35pm-2:55pm

We gave the updates on what we finished on Stein and draft plan for Train cycle.
The good thing to note is we still have a lot of activity going on in QA.
As overall QA projects, we did >3000 reviews and 750 commits. Video is not up still so I am copying the slide link below. Slides:

    2. OpenStack QA – Project Onboarding : Wednesday, May 1, 9:00am-9:40am

We did host the QA onboarding sessions but there were only 3 attendees and no new contributor. I think it is hard to see any new contributor in summits now so I am thinking whether we should host the onboarding sessions from next time.


    3. Users / Operators adoption of QA tools / plugins : Wednesday, May 1, 10:50am-11:30am

As usual, we had more attendees in this session and useful feedback. There are few tooling is being shared by attendees:

1. Python hardware module for bare metal detailed hardware inspection & anomaly detection

2. Workload testing: Another good idea from Doug was plugin feature in openstack-health dashboard.

That is something we discussed in PTG. For more details on this, refer the PTG ” OpenStack-health improvement” section.

    QA PTG: 2nd – 3rd May:

We were 3-4 attendee in the room always and others attended per topics. Good discussions and few good improvement ideas about gate stability and dashboard etc.

    1. Topic: Stein Retrospective:

We collect good and need improvement things in this session. In term of good things, we completed the OpenStack gate migration fro Xenial to Bionic, lot of reviews and code. Doug from AT&T mentioned about to add tempest and patrole to gates and check in their production deployment process, “Thank you for all of the hard work from the QA team!!!” Slow reviews are a concern as we have a good number of the incoming request. This is something we should improve in Train.

Action items:

gmann: start the plan for backlogs especially for review and doc cleanup.

masayukig: plan to have resource leakage check in gate.

ds6901:will work with his team to clean up leaks and submit bugs

    2. Topic: Keystone system-scope testing:

QA and Keystone team gathered together in this cross-project session about next steps on system scope testing. We talked on multiple points about how to cover all new roles for system scope and how to keep the backward compatibility testing for stable branches still testing the without system scope. We decided to move forward for system_admin as of now and fall back the system_admin to project scope if there is no system_scope testing flag is true on Tempest side (this will cover the stable branch testing unaffected).

We agreed :

– To move forward with system admin –

– Add tempest job to test system scope –

– Then add to tempest full – gmann – Then add testing for system reader

– Investigate more advanced RBAC testing with Patrole – gmann


    3. Topic: new whitebox plugin for tempest:

This is a new idea from artom about testing things outside of Tempest’s scope (currently mostly used to check instance XML for NFV use case tests). Currently, this tool does ssh into VM and fetch the xml for further verification etc. We agreed on point to avoid any duplicate test verification from the Tempest or nova functional tests This is good to tests from more extra verification by going inside VM like after migration data, CPU pinning etc. As next step artom to propose the QA spec with details and proposal of this plugin under QA program.

    4. Topic: Document the QA process or TODO things for releases, stable branch cut:

Idea is to start a centralized doc page for QA activities and process etc. we want to use the qa-specs repo to publish the content to This can be not so easy and need few tweaks on doc jobs. I will get into the details and then discuss with infra team. This is a low priority for now.

    5. Topic: Plugin sanity check:

Current tempest-plugins-sanity job is not stable and so it is n-v. We want to make it voting by only installing the active plugins. many plugins are failing either they are dead or not so active. We agreed on: – make faulty plugins as blacklist with bug/patch link and notify the same on ML every time we detect any failure – Publish the blakclist on plugins-registry doc. – After that make this job voting, make the process of fixing and removing the faulty plugin which unblocks the tempest gate with author self-approve. – Make sanity job running on plugins which are dependent on each other. For example, congress-tempest-plugin use neutron-tempest-plugin, mistral-tempest-plugin etc so all these plugins should have a sanity job which can install and list these plugins tests only not all the plugins.

    6. Topic: Planning for Patrole Stable release:

We had a good amount of discussions for Patrole improvements area to release it stable. Refer the below ML thread for details and further discussions on this topic: –

    7. Topic: How to make tempest-full stable:

Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We discussed the few ideas to improve it. Refer the below ML thread for details and further discussions on this topic :


    8. Topic: OpenStack-Health Improvement:

Doug from AT&T has few improvement ideas for health dashboard which has been discussed in PTG:

– Test Grouping – Define groups

– Assigned test to groups – filter by groups

– Compare 2 runs = Look into push AQuA report to subunit2SQL as a tool

Action Items:

– Doug is going to write the spec for plugin approach. All other ideas can be done after we have the plugin approach ready.

– filter

– presentation

    9. Topic: Stein Backlogs & Train priorities & Planning:

We collected the Train items in below mentioned etherpad with the assignee. Anyone would like to help on any of the item, ping me on IRC or reply here.


    10. Topic: grenade zuulv3 jobs review/discussions:

We did not get the chance to review these. Let’s continue it after PTG.

    Full Detail discussion:


You May Also Like..

10 Years of OpenStack – Ghanshyam Mann at NEC

While OpenStack celebrated its 10th Anniversary, here is my favorite memory from the last 10 years of OpenStack 10 Years […]

Recap of OpenStack Quality Assurance Xena Virtual PTG, 2021

OpenStack PTG for Xena development planning and discussions are held virtually from 19 – 23, April 2021. Overall we had […]

Recap of OpenStack Technical Committee Xena Virtual PTG, 2021

OpenStack PTG for Xena development planning and discussions are held virtually from 19 – 23, April 2021. Overall we had […]

Leave a Reply

Your email address will not be published. Required fields are marked *