..
 This work is licensed under a Creative Commons Attribution 3.0 Unported
 License.

 http://creativecommons.org/licenses/by/3.0/legalcode

==========================================
Neutron Core and Vendor code decomposition
==========================================

https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

Contributing and reviewing existing and/or new vendor code in Neutron is painful
for a number of reasons. We are proposing changes to the existing structure
of the project to address these pain points. More precisely we are promoting
changes in the following areas:

* Code structure;
* Contribution process: this extends to the following areas:

  * Design and Development;
  * Testing and Continuous Integration;
  * Defect management;
  * Documentation.

Problem Description
===================

Vendor and community concerns are competing for scarce resources
because our process is currently one-size-fits-all. Holding
plugins/drivers close made sense in the early days, to build the community, but
Neutron has matured as a project.  We have the opportunity to evolve the
relationship between plugin/driver maintainers and the core to ensure that as
a community we can grow ever larger without incurring the bottlenecks and
issues we are currently facing.

Currently, we have 20+ plugins and 12+ ML2 mechanism drivers. There are
plenty of new plugin/drivers in the pipeline whose specifications and
code contributions need to go through the contribution process where:

* a developer submits a blueprint specification to the neutron-specs repo;
* a developer submits the code patch(es) to the neutron repo;

* at least two members from the core team need to (iteratively) review the spec;
* at least one member of the drivers team needs to approve the spec;
* at least two core reviewers need to (iteratively) review/approve the patch(es);

This model clearly does not scale because each core can only have limited
knowledge of some parts of the codebase and the sheer amount of vendor-specific
contributions that Neutron as a project has seen so far meant that no single
core reviewer can know it all. This model has also the following issues:

* Requires an artificial oversight from the core team, as core reviewers do
  not have the ability to fully understand and judge quality of vendor-specific
  source, sometimes in the form of interrelated patches made of many
  thousands lines of code; granted, a good static code review can help
  address issues like styling, use of data structures, algorithms,
  modularity, etc, but without the ability to fully understand all the moving
  parts it is sometime difficult to assess possible failure modes, potential
  bottlenecks, and even why certain fixes after the fact end up being necessary;
* Core review may sometimes require playing with the patch, which is made
  difficult, if not impossible, by the lack of access to proprietary and/or
  external systems;
* Prevents core reviewer resources from being solely focused on targeting core
  concerns of Neutron as a management service;
* Prevents vendors from having more control over the development and release of
  their own code;
* Prevents vendors from having different levels of engagement with the Neutron
  community: the existing model imposes a high acceptance bar. Vendor
  developers are invited to actively participate in the day-to-day management
  of the project, maintaining their code etc, even though they sometimes fail
  to do so. If the code contribution is substantial, it is only customary for
  these developers to participate in the cross-cutting issues that may arise
  (e.g. gate failures, oslo libraries refreshes, change in style guidelines,
  etc); in other words, the current process requires that a vendor generate
  goodwill through contribution to non-vendor initiatives such that cores are
  willing to review or address vendor's non-core contributions in a timely
  fashion. In an ideal world this would foster community development, but it
  presupposes that all vendors have similar resources to devote to Neutron and
  that the core team can scale non-linearly (neither of which is the case
  today);
* Does not promote clear separation of concerns, as the code is in one big
  bucket;
* The maintainance cost of the entire codebase is unevenly distributed across
  the Neutron team;
* Refactoring effort is hindered by the difficulty of handling code that the
  developer is not familiar with;
* Negatively impacts unit test feedback runtime, as a lot more tests need to run
  at once to for a specific change;

All these issues ultimately affects the pace at which the project can evolve.

It is worth noting that this problem, and potential solutions, may affect
different people in the OpenStack community in different ways. For instance:

* OpenStack Infra: what tooling would the team need to provide to help the
  project? Would they be impacted on a regular basis because of a change
  in the way the project decides to organize itself?
* Vendor Developers: how can a developer get his/her changes merged faster
  and with minimal overhead? Writing better code and participating more in
  core reviews certainly helps, but this is a bar that not every contributor
  can or is willing to set for himself/herself;
* Vendor Marketeers: how can a marketeer claim that his/her vendor plugin is
  OpenStack friendly? In this particular regard, the fact that vendor plugins
  are shipped as 'compatible' with OpenStack is largely independent from the
  code's origin. The perception that things in openstack.org trunk are
  'good' is in reality a misconception, and as such it is a task for the
  marketing team(s) to fix it.
* Distros: how can a distro package Neutron by capturing the code necessary,
  their external depedencies, and provide configuration tools that allow for
  easy consumption of the solution?
* Neutron developers: how can a developer support (non)vendor code
  contributions without burning out?
* Operators: do they even care so long as they can get access to a stable,
  upgradable and well documented product?
* OpenStack Technical Committee: would the TC members promote this effort,
  so long as OpenStack ideals like Openness, Transparency, Commonality,
  Integration and Quality are still enforced? As far as openness is concerned,
  any new project arrangement must guarantee that the four opens are still
  in place (open source, open design, open development, open community [17]);
  TC members also represent the Active Technical Contributor, so it is of
  paramount importance for them to alleviate the pain in contributing to
  OpenStack;
* OpenStack Foundation: so long as the job of promoting the development,
  distribution and adoption of the OpenStack cloud operating system is not
  hindered by any attempt at making a project functioning better, they should
  actually welcome the effort. The Foundation has also the duty, with the
  marketplace on openstack.org, to promote drivers compatibiility and soon
  there will also be RefStack [14];

Having said that, the proposed change will primarily address the development
challenges that the Neutron community faces today, leaving social or marketing
perceptions and concerns thereof, a solution to be sought with guidance of
the OpenStack Foundation.

Proposed Change
===============

The proposal being made here is an initial step, and something it has been
deemed attainable in the span of a single release. It is not by all means
the end goal, but a good balance that will let us find the golden mean [5]
in iterations. It is also worth noting that this effort is about the core
plugins, rather than advanced services like Load Balancer, VPN and Firewall.
Spinning off those services will be tracked with a different blueprint [8].

High Level Codebase Structure
-----------------------------

We propose that:

* The *monolithic plugins*, *ML2 MechanismDrivers*, and *L3 service plugins*
  become integration-only to code that lives outside the tree (more details
  provided in the section below); the same applies for any vendor-specific
  agents: the only part that will remain in the tree is the agent 'main' (a
  small python file that imports agent code from the vendor library and
  starts it). L3 being an integral part of Neutron core will go through the
  same process as core plugins and ML2 drivers. The 'outside the tree' can be
  anything which the vendor is comfortable with: it may be a stackforge repo
  for instance, a tarball, a pypi package, etc. It is then important that this
  vendor library be publicly accessible for a number of reasons:

  * Potential licensing conflicts, that might be solved only by getting access
    to the vendor library, but more importantly under no circumstances should
    a key OpenStack project like Neutron be encouraging closed source
    development of a critical component such a plugin or mechanism driver;
  * Ease of packaging support from distros; even though in most cases, distros
    will have a relationship with vendors they want to package and validate,
    there may still be pure open source players, that may be interested in
    getting to use the undelying technology, should that be open source itself.

  A plugin/drivers maintainer team self-governs in order to promote sharing,
  reuse, innovation, and release of the 'out-of-tree' backbone. It should not
  be required for any member of the core team to be involved with this process,
  although core members of the Neutron team can be in whichever capacity
  necessary for out-of-tree development.

Development Strategy
--------------------

* The following elements are suggested to remain in the tree for all plugins
  and drivers (called vendor integration hereinafter) - at least in the first
  iteration of this effort:

  * Data models;
  * Extension definitions;
  * Configuration files;
  * Requirements file targeting vendor code;

* Things that do not remain in the tree (called vendor library hereinafter):

  * Vendor specific logic;
  * Associated unit tests;

The idea here would be to keep in-tree the plugin/driver code that implements
an API, but have it delegate to out-of-tree code for backend-specific
interactions; the vendor integration will then typically involve minor
passthrough/parsing of parameters, minor handling of DB objects as well as
handling of responses, whereas the vendor library will do the heavylifting and
implement the vendor-specific logic. The boundary between the in-tree layer and
the out-of-tree one should be defined by the maintainer while asking these
types of questions:

  * If something changes in my backend, do I need to alter the integration
    layer drastically? Clearly, the least impact there is, the better the
    separation being achieved;
  * If I expose vendor details (e.g. protocols, auth, e.g.), can easily swap
    and replace (e.g. a hardware with a newer version being supplied) without
    affecting the integration too much? Clearly, the more reusable the
    integration the better separation.

Furthermore, we can observe that:

  * Regarding db models, we would like to keep backward compatibility
    (for already deployed systems) at the present time. There are a number of
    technical points to be addressed before moving db models out of tree
    completely, e.g., FK, joined query, db migration for existing deployments
    and so on, as currently alembic requires extra work to support multiple db
    migration paths onto a single database [16]. This technical challenge can
    be dealt with in due course, once the major parts of this proposal have
    been completed;
  * Moving config files elsewhere (or even relying on config autogeneration
    capabilities) could be explored as a next step. We have been tracking
    config changes with DocImpact flags to ensure that OpenStack Manuals are
    kept in sync, and we should not break this model;
  * The vendor code *must* be publicly available via pypi, or public
    source repository compatible with pip requirements (if packaging is
    desired); this requirements file should not be confused with the Neutron
    requirements file that lists all common dependencies); instead it is a
    file 'requirements.txt' that is located in neutron/plugins/pluginXXX/,
    whose content is something along the lines of
    'my_plugin_xxx_library>=X.Y.Z'. Vendors should be responsible
    for ensuring that their library did not depend on libraries conflicting
    with global requirements, but could include libraries not included
    in the global requirements. If the vendor library depended on a python
    package not captured in the global requirements, this is no different
    by what already happens today and we are not trying to address this
    in the scope of this effort.
  * Versioning can be used to pin vendor code to specific versions;

* For the reference implementation: they will still follow the same model, but
  the implementation will remain in tree, due to gating requirements; the
  reason being that there is no way to externalize the reference implementation
  until we have backwards compatible interfaces in the mix. Having to
  coordinate the landing of a breaking change in core neutron repo and a fix
  in the reference implementation repo so as to avoid breaking openstack
  integration jobs would be an extremely tricky proposition.
  The proposal documented here is intended to encourage plugins/drivers to
  separate themselves into a part that integrates with Neutron and a part that
  integrates with a backend. Given that most breakage is going to be in the
  integration with Neutron (which remains in the tree) - and that breakage will
  only be allowed when fixes can reasonably be expected to be landed before the
  end of cycle (e.g. before milestone2) there should substantially less risk
  for non-gating plugins/drivers. Trunk will always work: for breaking changes
  that can be detected, we can ensure that at least the in-tree part of plugin
  drivers are fixed before merge. Plugin/driver-specific breakage will have to
  be fixed by the maintainer of that plugin/driver, but as a community we
  should address the potential for that kind of breakage separately.

* No DevStack changes required: Devstack already has the right hooks [6,7,10];
  moreover, the vendor is in full control of the 3rd party CI environment;
  they could just install the needed dependencies out of band, while the CI
  is setting up the environment and before stacking. Some of them already do
  all sorts of stuff.

For instance a vendor integration module can become as simple as one that
contains only the following:

* Registering config options;
* Registering the plugin class;
* Registering the models;
* Registering the extensions.

For instance:

::

>    class DummyPlugin(db_base_plugin_v2.NeutronDbPluginV2,
>                      your_mixin_here):
>
>    supported_extension_aliases = ["the_extension_you_implement"]
>
>    __native_bulk_support = True
>    __native_pagination_support = True
>    __native_sorting_support = True
>
>    def __init__(self):
>        super(DummyPlugin, self).__init__()
>        config.register_config_opts()
>        self.vendor_plugin_init()
>
>    db_base_plugin_v2.NeutronDbPluginV2.register_dict_extend_funcs(...)


The Nuage plugin [2, 3] follows a slightly similar approach. Since unit test
must be purely testing the units that make up for the vendor library, they no
longer make sense to be part of the Neutron codebase. Having said that, the
vendor integration should be tested via functional testing. The VMware plugin
was also going down that direction [4], where the plugin definition becomes
purely declarative.

Testing Strategy
----------------

The testing process will change as follows:

* There will be no unit tests for plugins and drivers in the tree, except the
  reference implementation. The expectation is that vendors would run unit test
  in their own external library (e.g. in stackforge where Jenkins setup is for
  free); For unit tests that validate the vendor library, it is responsibility
  of the vendor to choose what CI system they see fit to run them; there is no
  need or requirement to use OpenStack CI resources if they do not want to;
  ultimately these tests can run as part of the 3rd party CI system [9] that is
  currently required, based on specific filters, if necessary. It is noteworthy
  that this effort may unveil areas of the core code that turn out to be barely
  tested. We should strive to identify those, and improve their coverage so
  that, in turn, the improvements can be beneficial for the various
  vendor integrations as well, as added/revised tests will exercise the shared
  framework;
* 3rd Party CI will continue to validate vendor integration with Neutron via
  Tempest (e.g. functional testing); 3rd Party CI is a communication mechanism.
  This objective of this mechanism  is threefold:

  * it communicates to plugin/driver maintainers when someone has contributed
    a change that is potentially breaking. It is then up to the a given
    maintainer to determine whether the failure is transient or real, and
    resolve the problem if it is real;
  * it communicates to a patch author that they may be breaking a plugin/driver.
    If they have the time/energy/relationship with the maintainer of the
    plugin/driver in question, then they can (at their discretion) work to
    resolve the breakage;
  * it communicates to the community at large whether a given plugin/driver
    is being actively maintained.

  Gating on 3rd party CI *is* a terrible idea, though. It introduces too much
  variability into the merge process, and we already have more chaos than we
  can handle. The proposal in question attempts to minimize some breakage
  by keeping plugin/driver interfaces in-tree. We could add sanity unit tests
  of interface integration, but it appears that integration failures should
  be more effectively caught at a functional and integration level rather
  than unit one. Having said that, nothing outlined here would prevent to
  adjust test coverage over time.

Review and Defect Management Strategies
---------------------------------------

We propose that:

* The same bug management process applies: bugs that affect vendor code can be
  filed against the Neutron integration, if the integration code is at fault;
  otherwise, the code maintainer may decide to fix a bug without oversight, and
  issue a new version pin against the requirements file for the code in question
  in case the vendor library is being pinned; it makes sense to require 3rd
  party CI for a given plugin/driver to pass when changing their dependency
  before merging to any branch (i.e. both master and stable branches);
* The same review process applies: vendor specific code follow the same review
  guidelines as any other code in the tree, however, the code being reviewed
  strictly applies to integration code or requirements' version bumps; as for
  the vendor library, it becomes an external repo; the vendor can choose anyone
  to approve/merge changes in this repo;
* The same release process applies: this does not change;
* The same security vulnerability process applies: this does not change. However,
  since the attack surface is a lot larger for the vendor library, it is most
  likely that the vulnerability is found there, and the maintainer will need to
  issue a fix according to his/her own procedures; if a vulnerability is found in
  the vendor integration, the vulnerability fix will need to follow the same
  procedures are defined by OpenStack. Vendor may decide to publish the
  vulnerability on LP or any other tracking system: completely up to them.

Blueprint Spec Submission Strategy
----------------------------------

We propose that:

* Provided vendors adhere to the limited development footprint laid out in the
  proposal, they should not be required to follow the spec process for changes
  that only affect their vendor integration and library.
* Proposal of new vendor plugins and drivers should no longer follow the
  spec submission process in place for any other Neutron contribution, so long
  as the vendor plugin/driver adopts the guidelines outlined in this proposal.
  New contributions could simply be submitted for code review, with the proviso
  that adequate documentation and 3rd CI party is supplied at the time of
  the code submission. For tracking purposes, the review itself can be tagged
  with a Launchpad bug report, marked as wishlist; design documents can still
  be supplied in form of RST documents, within the same vendor library repo,
  for documentation purposes;
* Blueprint specifications for new vendor plugin/drivers that target the
  current release cyle for Neutron will therefore be evaluated with the
  assumption that this proposal goes ahead, and hence marked for abandonment.
  Code submission can still go ahead, and evaluated according to the
  guidelines laid out in this proposal. Code inclusion will most likely
  considered in time for the third milestone of the cycle to allow enough
  time for the core team to adequately demonstrate this proposal in practice.

Documentation Strategies
------------------------

There is a sister proposal for docs referenced in [15]. The intention going
forward is that the documentation team will fully document the reference
plugins/drivers and add short sections for vendor plugins and drivers, in
the same spirit promoted by this proposal (about code decomposition). Existing
plugins and drivers will have to add references to additional docs, just like
new ones will, once their vendor integration merges in the Neutron repo.

Adoption And Deprecation Policy
-------------------------------

We propose that:

* All new (core or L3) plugins and drivers follow this model from Kilo;
* It is strongly recommended that existing plugins and drivers adopt
  this model in the Kilo cycle, and structure the code according the
  guidelines outlined above;
* No in-tree features can be added to existing plugins and drivers: this
  means that the feature of existing plugins/drivers code in the main
  Neutron tree will be frozen for the Kilo release; some may argue that the
  freeze might not be necessary and we could still rely on some review
  best effort. However, it is important to recognize that strict discipline
  is required to see this proposal through;
* Only critical bug fixes will be permitted to existing plugins and drivers;
  ML2 needs to be excluded, at least for now, because we gate on ML2+OVS
  which precludes splitting any part of it out in advance of backwards
  compatible interfaces being introduced. It is nonetheless proposed that we
  refactor the OVS ML2 driver such that the drivers themselves contain the
  same minimum footprint proposed for all drivers and that they are cleanly
  separated from the backend implementation.
* Plugins and drivers that are still 'as is' (i.e. not complying with this
  proposal) at the end of Kilo will be evaluated for deprecation and removal;
  maintainers will need to make a positive effort to start this process in
  the Kilo timeframe; process well underway but to complete in the Lxxx
  will still be acceptable; however, it is paramount that maintainers
  prioritize their resources so that the rewriting process is not deferred
  until Lxxx, to minimize the impact of the code freeze, and to mitigate the
  potential danger of deprecation and removal in time for Mxxx.

Having said that, the deprecation target and freeze will need to be assessed
at the various Kilo milestones, to establish how close to completion this
proposal is, and how far each vendor maintainer is in completing the tasks
outilined in the 'Work Items' section below. Should this proposal get the go
ahead, then the clock will be set and actions will be tracked for each plugin,
ML2 driver, as well as the exent of progress being made (e.g. setting up of
repos, patches addressing the relevant code, etc).

NOTE: No action or lack of engagement from the interested parties will need to
be tracked to ensure that, deprecation is considered a last resort measure.

Summary
-------

The approach being proposed here tackles all the issues (outlined above) that
Neutron developers have grown so tired of; it does so by delegating some
control to the vendor, whilst retaining some of degree of visibility into what
is considered part of the Neutron project.

FAQ
---

Add your question here and we will try to answer to best way we can:

* Q: If ML2 plugin refactoring is pursued by the core team. Code-changes for
  the same can affect the interfaces and behaviour of existing large set of
  vendor mech-drivers in the neutron tree. So, does the vendor mech-drivers
  continue to remain in core tree.
* A: The proposal is not to move drivers out of the tree. Rather, the
  intention is to encourage a separation of concerns such that plugins and
  drivers are as lightweight as possible, and serve primarily to integrate
  with libraries external to the tree that provide backend-specific
  implementation. This would have the advantage of allowing vendors greater
  control over their backend-specific code and and offer the possibility of
  allowing changes to plugins/drivers with a lighter-weight process (no spec
  process and faster merging).
* Q: Wouldn't it be easier if the three code artifacts (Model, Extensions,
  and config files) also lived in the vendor repositories? By leaving the
  above three in-tree, the proposal still puts significant burden on the
  Neutron cores to review vendor code artifacts.
* A: Short answer is No, as (hopefully) the artifacts are fairly small and
  easy to understand. The long answer is: the effort required for full
  separation is considerable. The suggested approach would allow plugin/driver
  maintainers to gain control over their efforts faster and with less risk. It
  will most likely take the entire cycle to get everyone on board and adjusted
  to the new model, which does not require any additional work, like new hooks,
  different DB timelines, etc. but just the 'mechanical' translation.
  Once that is cleared out, we can then focus on refining the model and pursue
  further delegation to vendor code.
  We also need to consider the migration path: we have a couple of things
  to consider on splitting db model: for data models, we need to consider the
  db migration path and FK relationship to Neutron core models (networks,
  ports, etc). Coming up with a concrete plan on how this can be handled ahead
  of this proposal is considered a risk worth deferring. API extensions and
  configuration files may be able to be moved out, but it has been considered
  useful documentation material, at least for the first iteration.
* Q: The DB models for each vendors will stay in the main repo. This probably
  avoid the alembic challenge for migrations, but it also confuses me a bit
  when it comes to synchronizing models in master and versioning how the plugin
  library. In other words, if you release a version of a library which requires
  changes in the data model, then you will have to wait for data model changes
  to merged before pinning Neutron to that version of the library. What should
  the process be in this case?
* A: Ideally, the model changes are backward compatible, but if they were not,
  then, as outlined, the process is still pretty straighforward.
* Q: What is the distro packaging strategy? For instance, do we need to define
  versioning policy for a vendor plugin module? For example, vendor plugin
  module for Kilo should have a version of 2015.1.N?
* A: We do not require openstack specific versioning of other python
  dependencies, and we should not require it of the plugin/driver dependency
  libraries either. We could rely on the per-plugin/driver requirements files
  in a given Neutron release to authoritatively determine the version of the
  library required for a given plugin/driver for that release, and the version
  could be updated on stable branches as required.
* Q: It seems that the plugin library maintainer is expected to provide
  packaging for the appropriate distros. Is this true?
* A: Just as distros are responsible for determining how to package
  plugin/driver dependencies today, they would continue to do so under this
  proposal. Their job would be made easier by the addition of a plugin/driver
  requirements file that could be used to determine plugin/driver dependencies
  that currently require manual discovery. The expectation here is that the
  plugin business logic will have its own lifecycle in a library which is not
  part of OpenStack. The relationship between Neutron and the vendor library
  would be for instance the same that there is between Neutron and sqlalchemy.
  Therefore versioning for the vendor library can be whatever one sees fit.
  The plugin itself however, stays in the main neutron repository, but it is
  minimised in a way that it could be just a stub. We will still have a Neutron
  "vendor X" plugin for openstack 2015.1
* Q: How do we deal with Oslo integrations?
* A: When oslo integrations occur, 3rd party CI (and a plugin/driver maintainers
  internal testing) should detect any breakage in the external dependencies of
  the plugins/drivers. We may want to announce when these kinds of changes are
  expected such that plugin/driver maintainers can respond in a timely fashion
  and minimize the duration of job failure. Only if plugin/driver maintainers
  were completely unresponsive to breaking changes in the Neutron tree would
  the possibility exist of those changes making it all the way to production.
  Were a maintainer to be that unresponsive, the inclusion of their plugin
  in the tree would likely be in question. The aim here is to relieve the core
  team of some of the maintainance burder in addressing these types of issues.
* Q: How can we ensure that API's will not be changed and versioned? Say a
  internal method of a mixin is updated - then a vendor overwriting or using
  this may miss this. In other words, we do not have stable API's, therefore
  in the current model breaking changes can be prevented by doing a one
  sweep-fix-all type of patch. With the model being proposed here, how are we
  going to deal with this type of issue?
* A: The proposal is to keep as strict a separation as possible between Neutron
  and vendor concerns, and continuing to develop our test suite to catch
  breakage (API or otherwise) when it occurs. Provided plugin/driver maintainer
  is diligent in following up on breakage, the only thing that will change
  under this proposal is how fast breakage will be fixed (same patch vs one
  patch to Neutron + one patch per external plugin dependency). The fact that
  the cost of breakage will be increased could actually be a positive thing if
  it can encourage the community to take more care in providing cleaner
  separation between concerns (via APIs or otherwise). Today both the
  integrated gate and non-gated plugins happen to break from time to time; so
  long as there is a prompt reaction to the issue, we do not see this model
  exacerbating this issue further. As a matter of fact, CI's are getting
  better and the team is getting better at catching these problems and fixing
  them swiftly; all we need is to be engaged!
* Q: Can you please elaborate a little more on the governane of new repos?
  There is at least 1 maintainer needed. How many minimum number of members?
  If there is 1 member, he/she will be committing, reviewing and merging
  his/her own code?
* A: This proposal does not mandate any kind of governance (or even distribution
  via a repo) of a given plugin/driver dependency library. We would continue to
  require an in-tree maintainer/point of contact for each plugin/driver, as we
  do today, and that maintainer would be responsible for coordinating with
  out-of-tree development.
* Q: Previously, if there was any change affecting plugins/drivers, it was done
  in all the code paths to-be-affected to make sure nothing breaks. However,
  with split repos, how to make third party CIs pass on changes which break
  vendor code and require a change across two or more repos? Looks like we
  will have to override third party CI vote either in Neutron or vendor
  plugin/driver repo? If the override is absolutely needed, where should it
  be done?
* A: 3rd party CI votes but does not gate, and this proposal in no way changes
  that. When breaking changes land in the Neutron tree, it will be up to the
  plugin/driver maintainers to resolve the breakage on their own timeline. In
  the short-term we are introducing coordination overhead by externalizing
  backend-specific code, but in the long-term this should encourage a decrease
  in coupling that minimizes such costs.
* Q: How do we define some vendor plugin is a part of OpenStack release? Do we
  define it? Defined by third party testing integration? It seems related to
  distro packaging policy to some extent.
* A: We can define whether a plugin is part of a given release is the answer to
  the question: 'does the plugin/driver exist in the Neutron tree?'. The
  question that we do need to answer is 'under what conditions is a
  plugin/driver allowed to be in the tree'? We are going to have to rely on a
  combination of relationship management (PTL <-> plugin/driver maintainer) and
  3rd party CI to answer that question, and this proposal does not need to
  answer this question. Some plugins already rely on a variation of the
  proposed model, and there does not seem to be any issue with distribution.
* Q: Should Third party CI result be posted to Neutron core change? We need to
  define third party testing requirements?
* A: They can choose to do so, if they believe the change may break their
  support, but this already happens today; 3rd party CI requirements need to
  be clarified regardless of this proposal.
* Q: Does this proposal make any effort into an ML2 Agent no longer feasible?
* A: No, effort into developing an ML2 agent would not be the best use of
  community resources. For historic reasons Neutron has grown to be not only a
  networking orchestration project but also a reference implemention that is
  resembling what some might call an SDN controller. We need to move away from
  this model, and for these reasons, pursuing an ML2 agent is no longer viable.
* Q: What about Linux Bridge?
* A: Linux Bridge is not currently the option we choose to focus both the
  development and testing; any effort in maintaining it at par with OVS would
  require resources that need to be identified and quantified. In lack of those
  resources, Linux Bridge has already fallen behind and will not be considered
  in the refactoring effort, unless someone else is willing to sponsor it.
* Q: How do we deal with imports of Neutron specifics (like utility functions,
  constants, base artifacts, etc) in the vendor library?
* A: External libraries that implement backend-specific interaction for
  plugins/drivers are expected to explicitly depend on Neutron to allow them
  to reference things like db models, utility functions and constants. Since
  the Neutron's dependency on the external libraries is not explicit
  (plugin/driver requirements file will not be considered by pip), a circular
  dependency will not result. The model would follow loosely what any OpenStack
  project already does when referencing Oslo. When it comes to pinning to a
  specific 'snapshot' of the core codebase, git hashes could be used. The model
  would need to ultimately evolve so that a better demarcartion will occur
  between core and vendor specifics, which makes the dependency less of a
  potential problem.
* Q: What if a vendor library uses an API defined in the core?
* A: Reuse of some plugin methods by an external library is possible through
  dependency injection, which are fancy words for 'pass a reference to the
  plugin to the code that needs to call its methods'.
* Q: With this approach, would packagers ignore some of the plugins in tree by
  not making a package for their library?
* A: Before the split, all plugins were 'magically' packaged by distributions,
  at least their source code. The packager then explicitely needed to add
  'control' rules to integrate the various parts together with the OS, and/or
  config management tools to reflect the configuration choices in the respective
  configuration files. This does not necessarily endorse validity of the
  vendor solution on the platform, but packagers were left with no choice but
  to follow this route in order to avoid dangling code being packaged and
  distributed. With this approach, consuming the actual source code becomes
  optional, and it is up to the vendor to decide to continue to preserve the
  existing packaging model by pulling from the various sources. In other words,
  this opens up the question whether all distributions should consider if it is
  worth the effort of shipping plugins that are not explicitely covered by
  relationship agreements with vendors. Every distro has different internal
  processes for tracking packages and releases, and it sensible that distros
  take the appropriate actions to ensure the overall quality of the Neutron
  solution they sell and support. We should emphasize that according to this
  proposal, long term, some plugins may loose packaging and distribution in the
  commercial editions of these distros, but at the same time, open source
  initiatives can still go ahead unaffected.
* Q: The fact that many vendor-specific plugins/drivers are currently in-tree,
  creates the impression that a new vendor has 'failed' in some sense if they
  do not get their new plugin/driver in tree as well - where 'failed' has
  various possible connotations including 'not being guaranteed to work with
  OpenStack, and to continue to work as OpenStack evolves', and 'not having
  demonstrated sufficient engagement with the OpenStack community'. Does this
  proposal help in addressing this aspect of code contribution?
* A: Absolutely! First of all, let us be clear of one thing: in-tree does not
  equate success, and out-of-tree does not equate failure. This is a perception
  that somewhat built over time into people's minds, because of past events
  that led to the removal of code of questionable quality or because of lack
  of testing (most notably Hyper-V in Nova). Today, the reality is completely
  different, OpenStack has evolved and 3rd party CI plays a fundamental role
  in establishing the sanity of all the components in motion in an OpenStack
  based cloud. Furthermore, with this proposal, the failure for not getting
  new vendor support in the tree is mitigated by the fact that it is a lot
  easier for the core team to go through the checklist of steps to be taken
  to consider a vendor integrated with Neutron, without being encumbered by
  the need of going through multiple review cycles to get the plugin/driver
  in the tree.

Data Model Impact
-----------------

N/A

REST API Impact
---------------

N/A

Security Impact
---------------

N/A

Notifications Impact
--------------------

N/A

Other End User Impact
---------------------

From an end-user standpoint (the tenant or admin that interact with the Neutron
API); this approach has no impact on them.

Performance Impact
------------------

N/A

IPv6 Impact
-----------

N/A

Other Deployer Impact
---------------------

Minimal. Depending on how Neutron is consumed, deployers need to pull an
extra dependency, which is the vendor library of the plugin of their choice.
This dependency can be pulled in in various ways, with or without the actual
support from the OS/Distro the deployer is using to consume Neutron.

It is worth mentioning that this approach empowers the distros (namely the
packagers) to choose where to focus resources more effectively. This means
that they can now choose to target packaging only for specific vendor
solutions more explicitely. Although this may sound like a setback for some,
the benefit of automatic packaging was only giving false reassurance that the
overall Neutron solution was going to be free of supportability issues, or
that it was implicitely validated by either the vendor or the distro. Any
customer should do the appropriate due diligence when adopting the combination
of Neutron+Backend, should they choose to access the overall Neutron solution
via two distinct channels. This proposal does hinder this process.

Developer Impact
----------------

As outlined in this proposal.

Community Impact
----------------

Very minimal, or nearly none in terms of diverging drastically from the way
we operate today. As a matter of fact, we believe there will be a positive
community impact:

* Cores able to iterate faster on the core;
* Better separation of responsibilities;
* Vendors able to iterate faster into their plugin;

Alternatives
------------

Alternatives could be explored with or without tooling assistance:

* Increase the number of Neutron cores (1)

  We could allow every vendor to have core(s) reviewers so that they can
  approve only their own code (i.e. subtree delegation). This poses an
  interesting set of challenges:

   * What do we do with new contributors?
   * What does core developer actually mean?
   * How can we ensure a coherent standard of code reviews?

* Rubberstamping from Neutron cores (2)

  Neutron cores can become paper pushers and blindly approve changes
  that pertains only to a specific vendor. This poses an interesting
  set of challenges:

  * How do we ensure that cross-cutting concerns are not overlooked?
  * How do we ensure that vendor maintainers are still held accountable?

* Adopt a different decomposition of the source tree

  The existing source tree could be decomposed in separate repositories:

  * One repo hosts core elements (DB, API, RPC, etc) and fully open source
    implementation of L2/L3 API that is currently gated;
  * One repo for each vendor, that has contributed one or more (monolithic)
    plugin;
  * One repo for all ML2 mechanism drivers; A vendor that contributed both
    a monolithic plugin and an ML2 driver should decide where the ML2 driver
    needs to belong;

  Ideally this is decomposition that makes the most sense, if it was not for
  the political and social rules that we are constrained by. In other words,
  from a pure software development standpoint, this decomposition makes the
  most sense, and any objection to this approach could be overcome with
  technical solutions. That said, it may be too hard to pursue this approach
  in the span of a single release. It might make sense to reassess this
  approach in a release from now, once the codebase has gone through this
  initial transformation and other efforts (e.g. advanced service spin-off,
  API refactoring, etc.) have been completed.

* Using git submodules to decompose the source tree

  This has been strongly discouraged by OpenStack Infra, because of the
  complications of managing git submodules:  not only does it add potential
  complications in CI tooling, but it is nearly indistinguishable from using
  separate repositories and as such does not really address any of the
  concerns raised with separate repositories.

* Using feature branches

  This could be achieved by using long lived branches (similar to the Linux
  kernel model where subteam maintainers have there own branch). Short lived
  branches do not really lend themselves to representing the lifespan of a
  plugin/driver. Even though one of the perks of feature branches is that
  they can own different ACLs so a different group of people can get +2/+W
  on them, the drawback to this approach is that it does not promote clear
  separation and responsibilities between the various parts of the system.
  In other words, if we had stable interfaces between these parties, using
  feature branches would not provide any benefits more than having these
  parties reside in different repositories.

Commentary

While approaches 1 and 2 may help address change velocity, they may
ultimately lead to an increased instability of the codebase, they do
not promote cleaner interfaces, and they weight down the Neutron team
that would need to deal with a much, much bigger codebase. In other
words, even though these approaches may make sense in the short term,
they may seriously affect the project in the long run.

Implementation
==============

Assignee(s)
-----------

For the reference implementation refactoring, code and test re-org:

* Armando Migliaccio
* Maru Newby

For examples of how this proposal develops in practice:

* Kevin Benton (Big Switch)
* Doug Wiegley (A10)
* Kyle Mestery (OpenDayLight ML2 Mech Driver)
* Yamamoto Takashi (ofagent)
* Gary Kotton (VMware)
* <add your name here>

For new plugin/driver that want to pioneer this model:

* Sandhya Dasu (Cisco UCS Manager ML2 mech driver)
* Vivekanandan Narasimhan (OVSvApp [12, 13])
* <add your name here>

Work Items
----------

* Announce new policy and deprecation timeline for existing plugins/drivers;
* Separate reference implementation in a similar manner but keep the code in
  tree (ML2 OVS mech driver + L3 service pluign with l3-agent, which is what
  it is currently gated on); This is of paramount importance in order to
  flesh out any remaining technical details and pave the way for vendor code
  maintaners to follow in the their own code.

  * the plugin becomes the integration point to the bulk of the code held in
    something like neutron/reference_code.
* Suggested work items for each vendor:

  * Setup a repo for hosting the vendor library (stackforge is useful because
    it provides jenkins integration); there is no naming convention forced
    down on vendors, however, should the stackforge route be pursued, vendor
    should avoid naming the repo with explicit reference to Neutron: this
    protects the vendor in case new naming issues arise (anyone remembers the
    Quantum naming debacle?), and it allows the vendor to have the freedom
    to target the same Python bindings for potentially different CMS;
  * Setup code in new repo: it is worth noting that if code is being copied
    out of neutron core and into these drivers, the git subtree tool can
    preserve history across this operation. OpenStack infra is doing something
    similar with puppet modules on [11];
  * Setup unit tests;
  * Publish code to pypi;
  * Change neutron driver/plugin to import pypi module and remove all vendor
    internal logic;

Dependencies
============

N/A

Testing
=======

For more details, see section on Testing Strategy.

Tempest Tests
-------------

N/A

Functional Tests
----------------

N/A

API Tests
---------

N/A

Documentation Impact
====================

For further details, see section on Documentation Strategies.

User Documentation
------------------

After this proposal, the overall Neutron system will not change in terms of how it
functions. From a user perspective, once the system has been deployed, configured
and up and running, no difference can be perceived. However, a sibling proposal
has been documented in [15]. Operators will be expected to follow documentation
and/or pointers to external references provided by the Upstream OpenStack docs
to understand how to install and configure the system to use specific plugins and
drivers.

Developer Documentation
-----------------------

No API change has been proposed here, so providing guidelines for contributing
a plugin/driver will suffice. This proposal already outlined quite extensively
some of the work involved, and will be used as base for the in-tree developer
documentation.

References
==========

* [1]  https://etherpad.openstack.org/p/aE7ydRU35m
* [2]  https://github.com/openstack/neutron/tree/master/neutron/plugins/nuage
* [3]  https://github.com/openstack/neutron/tree/master/neutron/tests/unit/nuage
* [4]  https://github.com/openstack/neutron/blob/master/neutron/plugins/vmware/plugin.py
* [5]  http://en.wikipedia.org/wiki/Golden_mean_(philosophy)
* [6]  https://github.com/openstack-dev/devstack/tree/master/lib/neutron_plugins
* [7]  https://github.com/openstack-dev/devstack/tree/master/lib/neutron_thirdparty
* [8]  https://review.openstack.org/#/c/136835/
* [9]  http://docs.openstack.org/infra/system-config/third_party.html
* [10] https://review.openstack.org/#/c/137054
* [11] http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html
* [12] https://review.openstack.org/#/c/136091/
* [13] https://review.openstack.org/#/c/104452/
* [14] https://wiki.openstack.org/wiki/RefStack
* [15] https://review.openstack.org/#/c/133372/
* [16] https://github.com/stackforge/group-based-policy/tree/master/gbp/neutron/db/migration/alembic_migrations
* [17] https://wiki.openstack.org/wiki/Open
