Neutron Bugs

Neutron (client, core, FwaaS, LBaaS, VPNaaS) maintains all of its bugs in the following Launchpad projects:

Neutron Bugs Team In Launchpad

The Neutron Bugs team in Launchpad is used to allow access to the projects above. Members of the above group have the ability to set bug priorities, target bugs to releases, and other administrative tasks around bugs. The administrators of this group are the members of the neutron-drivers-core gerrit group. Non administrators of this group include anyone who is involved with the Neutron project and has a desire to assist with bug triage.

If you would like to join this Launchpad group, it’s best to reach out to a member of the above mentioned neutron-drivers-core team in #openstack-neutron on Freenode and let them know why you would like to be a member. The team is more than happy to add additional bug triage capability, but it helps to know who is requesting access, and IRC is a quick way to make the connection.

As outlined below the bug deputy is a volunteer who wants to help with defect management. Permissions will have to be granted assuming that people sign up on the deputy role. The permission won’t be given freely, a person must show some degree of prior involvement.

Neutron Bug Deputy

Neutron maintains the notion of a “bug deputy”. The bug deputy plays an important role in the Neutron community. As a large project, Neutron is routinely fielding many bug reports. The bug deputy is responsible for acting as a “first contact” for these bug reports and performing initial screening/triaging. The bug deputy is expected to communicate with the various Neutron teams when a bug has been triaged. In addition, the bug deputy should be reporting “High” and “Critical” priority bugs.

To avoid burnout, and to give a chance to everyone to gain experience in defect management, the Neutron bug deputy is a rotating role. The rotation will be set on a period (typically one or two weeks) determined by the team during the weekly Neutron IRC meeting and/or according to holidays. During the Neutron IRC meeting we will expect a volunteer to step up for the period. Members of the Neutron core team are invited to fill in the role, however non-core Neutron contributors who are interested are also encouraged to take up the role.

This contributor is going to be the bug deputy for the period, and he/she will be asked to report to the team during the subsequent IRC meeting. The PTL will also work with the team to assess that everyone gets his/her fair share at fulfilling this duty. It is reasonable to expect some imbalance from time to time, and the team will work together to resolve it to ensure that everyone is 100% effective and well rounded in their role as _custodian_ of Neutron quality. Should the duty load be too much in busy times of the release, the PTL and the team will work together to assess whether more than one deputy is necessary in a given period.

The presence of a bug deputy does not mean the rest of the team is simply off the hook for the period, in fact the bug deputy will have to actively work with the Lieutenants/Drivers, and these should help in getting the bug report moving down the resolution pipeline.

During the period a member acts as bug deputy, he/she is expected to watch bugs filed against the Neutron projects (as listed above) and do a first screening to determine potential severity, tagging, logstash queries, other affected projects, affected releases, etc.

From time to time bugs will be filed and auto-assigned by members of the core team to get them to a swift resolution. Obviously, the deputy is exempt from screening these.

Finally, the PTL will work with the deputy to produce a brief summary of the issues of the week to be shared with the larger team during the weekly IRC meeting and tracked in the meeting notes.

Getting Ready to Serve as the Neutron Bug Deputy

If you are interested in serving as the Neutron bug deputy, there are several steps you will need to follow in order to be prepared.

  • Request to be added to the neutron-bugs team in Launchpad. This request will be approved when you are assigned a bug deputy slot.
  • Read this page in full. Keep this document in mind at all times as it describes the duties of the bug deputy and how to triage bugs particularly around setting the importance and tags of bugs.
  • Sign up for neutron bug emails from LaunchPad.
    • Navigate to the LaunchPad Neutron bug list.
    • On the right hand side, click on “Subscribe to bug mail”.
    • In the pop-up that is displayed, keep the recipient as “Yourself”, and your subscription something useful like “Neutron Bugs”. You can choose either option for how much mail you get, but keep in mind that getting mail for all changes - while informative - will result in several dozen emails per day at least.
    • Do the same for the LaunchPad python-neutronclient bug list.
  • Configure the information you get from LaunchPad to make visible additional information, especially the ‘age’ of the bugs. You accomplish that by clicking the little gear on the left hand side of the screen at the top of the bugs list. This provides an overview of information for each bug on a single page.
  • Optional: Set up your mail client to highlight bug email that indicates a new bug has been filed, since those are the ones you will be wanting to triage. Filter based on email from “@bugs.launchpad.net” with “[NEW]” in the subject line.
  • Volunteer during the course of the Neutron team meeting, when volunteers to be bug deputy are requested (usually towards the beginning of the meeting).
  • View your scheduled week on the Neutron Meetings page.
  • During your shift, if it is feasible for your timezone, plan on attending the Neutron Drivers meeting. That way if you have tagged any bugs as RFE, you can be present to discuss them.

Plugin and Driver Repositories

Many plugins and drivers have backend code that exists in another repository. These repositories may have their own Launchpad projects for bugs. The teams working on the code in these repos assume full responsibility for bug handling in those projects. For this reason, bugs whose solution would exist solely in the plugin/driver repo should not have Neutron in the affected projects section. However, you should add Neutron (Or any other project) to that list only if you expect that a patch is needed to that repo in order to solve the bug.

It’s also worth adding that some of these projects are part of the so called Neutron stadium. Because of that, their release is managed centrally by the Neutron release team; requests for releases need to be funnelled and screened properly before they can happen. To this aim, the process to request a release is as follows:

  • Create a bug report to your Launchpad project: provide details as to what you would like to release;
    • If you provide an exact commit in the bug report then you need to be a bit careful. In most cases, you’ll want to tag the merge commit that merges your last commit in to the branch. This bug shows an instance where this mistake was caught. Notice the difference between the incorrect commit and the correct one which is the merge commit. git log 6191994..22dd683 --oneline shows that the first one misses a handful of important commits that the second one catches. This is the nature of merging to master.
  • Add Neutron to the list of affected projects.
  • Add ‘release-subproject’ tag to the list of tags for the bug report.
  • The Neutron release management team will watch these bugs, and work with you to have the request fulfilled by following the instructions found here.

Bug Screening Best Practices

When screening bug reports, the first step for the bug deputy is to assess how well written the bug report is, and whether there is enough information for anyone else besides the bug submitter to reproduce the bug and come up with a fix. There is plenty of information on the OpenStack wiki on how to write a good bug report and to learn how to tell a good bug report from a bad one. Should the bug report not adhere to these best practices, the bug deputy’s first step would be to redirect the submitter to this section, invite him/her to supply the missing information, and mark the bug report as ‘Incomplete’. For future submissions, the reporter can then use the template provided below to ensure speedy triaging. Done often enough, this practice should (ideally) ensure that in the long run, only ‘good’ bug reports are going to be filed.

Bug Report Template

The more information you provide, the higher the chance of speedy triaging and resolution: identifying the problem is half the solution. To this aim, when writing a bug report, please consider supplying the following details and following these suggestions:

  • Summary (Bug title): keep it small, possibly one line. If you cannot describe the issue in less than 100 characters, you are probably submitting more than one bug at once.
  • Further information (Bug description): conversely from other bug trackers, Launchpad does not provide a structured way of submitting bug-related information, but everything goes in this section. Therefore, you are invited to break down the description in the following fields:
    • High level description: provide a brief sentence (a couple of lines) of what are you trying to accomplish, or would like to accomplish differently; the ‘why’ is important, but can be omitted if obvious (not to you of course).
    • Pre-conditions: what is the initial state of your system? Please consider enumerating resources available in the system, if useful in diagnosing the problem. Who are you? A regular tenant or a super-user? Are you describing service-to-service interaction?
    • Step-by-step reproduction steps: these can be actual neutron client commands or raw API requests; Grab the output if you think it is useful. Please, consider using paste.o.o for long outputs as Launchpad poorly format the description field, making the reading experience somewhat painful.
    • Expected output: what did you hope to see? How would you have expected the system to behave? A specific error/success code? The output in a specific format? Or more than a user was supposed to see, or less?
    • Actual output: did the system silently fail (in this case log traces are useful)? Did you get a different response from what you expected?
    • Version:
      • OpenStack version (Specific stable branch, or git hash if from trunk);
      • Linux distro, kernel. For a distro, it’s also worth knowing specific versions of client and server, not just major release;
      • Relevant underlying processes such as openvswitch, iproute etc;
      • DevStack or other _deployment_ mechanism?
    • Environment: what services are you running (core services like DB and AMQP broker, as well as Nova/hypervisor if it matters), and which type of deployment (clustered servers); if you are running DevStack, is it a single node? Is it multi-node? Are you reporting an issue in your own environment or something you encountered in the OpenStack CI Infrastructure, aka the Gate?
    • Perceived severity: what would you consider the importance to be?
  • Tags (Affected component): try to use the existing tags by relying on auto-completion. Please, refrain from creating new ones, if you need new “official” tags, please reach out to the PTL. If you would like a fix to be backported, please add a backport-potential tag. This does not mean you are gonna get the backport, as the stable team needs to follow the stable branch policy for merging fixes to stable branches.
  • Attachments: consider attaching logs, truncated log snippets are rarely useful. Be proactive, and consider attaching redacted configuration files if you can, as that will speed up the resolution process greatly.

Bug Triage Process

The process of bug triaging consists of the following steps:

  • Check if a bug was filed for a correct component (project). If not, either change the project or mark it as “Invalid”.
  • For bugs that affect documenation (including autogenerated DocImpact bugs) proceed like this. If documentation affects
    • the ReST API, add “openstack-api-site” to the affected projects.
    • the OpenStack manuals, like the Networking Guide or the Configuration Reference, add “openstack-manuals” to the affected projects
    • developer documentation (devref), set the bug to “Confirmed” for the project Neutron, otherwise set it to “Invalid”.
  • Check if a similar bug was filed before. Rely on your memory if Launchpad is not clever enough to spot a duplicate upon submission. You may also check already verified bugs for Neutron and python-neutronclient to see if the bug has been reported. If so, mark it as a duplicate of the previous bug.
  • Check if the bug meets the requirements of a good bug report, by checking that the guidelines are being followed. Omitted information is still acceptable if the issue is clear nonetheless; use your good judgement and your experience. Consult another core member/PTL if in doubt. If the bug report needs some love, mark the bug as ‘Incomplete’, point the submitter to this document and hope he/she turns around quickly with the missing information.

If the bug report is sound, move next:

  • Revise tags as recommended by the submitter. Ensure they are ‘official’ tags. If the bug report talks about deprecating features or config variables, add a deprecation tag to the list.
  • As deputy one is usually excused not to process RFE bugs which are the responsibility of the drivers team members.
  • Depending on ease of reproduction (or if the issue can be spotted in the code), mark it as ‘Confirmed’. If you are unable to assess/triage the issue because you do not have access to a repro environment, consider reaching out the Lieutenant, go-to person for the affected component; he/she may be able to help: assign the bug to him/her for further screening. If the bug already has an assignee, check that a patch is in progress. Sometimes more than one patch is required to address an issue, make sure that there is at least one patch that ‘Closes’ the bug or document/question what it takes to mark the bug as fixed.
  • If the bug indicates test or gate failure, look at the failures for that test over time using OpenStack Health or OpenStack Logstash. This can help to validate whether the bug identifies an issue that is occurring all of the time, some of the time, or only for the bug submitter.
  • If the bug is the result of a misuse of the system, mark the bug either as ‘Won’t fix’, or ‘Opinion’ if you are still on the fence and need other people’s input.
  • Assign the importance after reviewing the proposed severity. Bugs that obviously break core and widely used functionality should get assigned as “High” or “Critical” importance. The same applies to bugs that were filed for gate failures.
  • Choose a milestone, if you can. Targeted bugs are especially important close to the end of the release.
  • (Optional). Add comments explaining the issue and possible strategy of fixing/working around the bug. Also, as good as some are at adding all thoughts to bugs, it is still helpful to share the in-progress items that might not be captured in a bug description or during our weekly meeting. In order to provide some guidance and reduce ramp up time as we rotate, tagging bugs with ‘needs-attention’ can be useful to quickly identify what reports need further screening/eyes on.

You are done! Iterate.

Bug Expiration Policy and Bug Squashing

More can be found at this Launchpad page. In a nutshell, in order to make a bug report expire automatically, it needs to be unassigned, untargeted, and marked as Incomplete.

The OpenStack community has had Bug Days but they have not been wildly successful. In order to keep the list of open bugs set to a manageable number (more like <100+, rather than closer to 1000+), at the end of each release (in feature freeze and/or during less busy times), the PTL with the help of team will go through the list of open (namely new, opinion, in progress, confirmed, triaged) bugs, and do a major sweep to have the Launchpad Janitor pick them up. This gives 60 days grace period to reporters/assignees to come back and revive the bug. Assuming that at regime, bugs are properly reported, acknowledged and fix-proposed, losing unaddressed issues is not going to be a major issue, but brief stats will be collected to assess how the team is doing over time.

Tagging Bugs

Launchpad’s Bug Tracker allows you to create ad-hoc groups of bugs with tagging.

In the Neutron team, we have a list of agreed tags that we may apply to bugs reported against various aspects of Neutron itself. The list of approved tags used to be available on the wiki, however the section has been moved here, to improve collaborative editing, and keep the information more current. By using a standard set of tags, each explained on this page, we can avoid confusion. A bug report can have more than one tag at any given time.

Proposing New Tags

New tags, or changes in the meaning of existing tags (or deletion), are to be proposed via patch to this section. After discussion, and approval, a member of the bug team will create/delete the tag in Launchpad. Each tag covers an area with an identified go-to contact or Lieutenant, who can provide further insight. Bug queries are provided below for convenience, more will be added over time if needed.

Tag Description Contact
access-control A bug affecting RBAC and policy.json Kevin Benton
api A bug affecting the API layer Salvatore Orlando
auto-allocated-topology A bug affecting get-me-a-network Henry Gessau
baremetal A bug affecting Ironic support Sukhdev Kapur
db A bug affecting the DB layer Henry Gessau
deprecation To track config/feature deprecations Neutron PTL/drivers
dns A bug affecting DNS integration Miguel Lavalle
doc A bug affecting in-tree doc Edgar Magana
fullstack A bug in the fullstack subtree Assaf Muller
functional-tests A bug in the functional tests subtree Assaf Muller
fwaas A bug affecting neutron-fwass Sean Collins
gate-failure A bug affecting gate stability Armando Migliaccio
ipv6 A bug affecting IPv6 support Henry Gessau
l2-pop A bug in L2 Population mech driver Kevin Benton
l3-bgp A bug affecting BGP service plugin Vikram Choudhary
l3-dvr-backlog A bug affecting distributed routing Ryan Moats
l3-ha A bug affecting L3 HA (vrrp) Assaf Muller
l3-ipam-dhcp A bug affecting L3/DHCP/metadata Miguel Lavalle
lbaas A bug affecting neutron-lbaas Brandon Logan
linuxbridge A bug affecting ML2/linuxbridge Sean Collins
loadimpact Performance penalty/improvements Ryan Moats
logging An issue with logging guidelines Matt Riedemann
low-hanging-fruit Starter bugs for new contributors N/A
metering A bug affecting the metering layer ?
needs-attention A bug that needs further screening PTL/Bug Deputy
opnfv Reported by/affecting OPNFV initiative Drivers team
ops Reported by or affecting operators Drivers Team
oslo An interop/cross-project issue Ihar Hrachyshka
ovs A bug affecting ML2/OVS Kevin Benton
ovs-lib A bug affecting OVS Lib Terry Wilson
py34 Issues affecting the Python 3 porting Cedric Brandily
qos A bug affecting ML2/QoS Miguel Ajo
released-neutronclient A bug affecting released clients Kyle Mestery
release-subproject A request to release a subproject Kyle Mestery
rfe Feature enhancements being screened Drivers Team
rfe-approved Approved feature enhancements Drivers Team
sg-fw A bug affecting security groups Kevin Benton
sriov-pci-pt A bug affecting Sriov/PCI PassThrough Moshe Levi
troubleshooting An issue affecting ease of debugging Assaf Muller
unittest A bug affecting the unit test subtree Cedric Brandily
usability UX, interoperability, feature parity PTL/Drivers Team
vpnaas A bug affecting neutron-vpnaas Paul Michali
xxx-backport-potential Cherry-pick request for stable team Ihar Hrachyshka

Operators/Operations (ops)

Table Of Contents

Previous topic

Blueprints and Specs

Next topic

Contributor Onboarding

Project Source

This Page

Navigation

© Copyright 2011-present, OpenStack Foundation.. Last updated on Apr 19, 2017. Created using Sphinx 1.2.3.