Pollsters No Transform¶
https://blueprints.launchpad.net/ceilometer/+spec/pollsters-no-transform
The ceilometer polling agents currently perform two roles: They poll various
services to generate samples and they optionally transform those samples, using
pipelines defined by pipeline.yaml, to compose other samples eventually
publishing them as measurements. This latter transformation functionality raises
the complexity of the pollster code. An alternative would be for the pollsters
to send notifications for the meters it creates and have them be processed
by the notification agent (which has a transformation pipeline of its own) and
then be published.
Problem description¶
Including the transformation pipeline in the pollsters increases complexity in the pollsters and duplicates functionality found in the notification agent.
Proposed change¶
One way to resolve this complexity is to not do transformations in the pollsters. Instead when new samples are polled, format them as notifications and push them onto the notification bus to be retrieved by the notification agent. The pipeline within that agent will then do any required transformations.
Besides clarifying the focus of the polling agents it is likely this change will make it easier to test and build performance improvements both in code and in deployments. This is murky speculation but it is challenging to test the theory without doing the work.
We also get an opportunity to review and audit the polling code, which is a useful thing to do on a regular basis.
Alternatives¶
The primary alternative is to leave things as they are. Obviously if we do this we lose the benefits described above.
Data model impact¶
The proposal does not impact the structure of the data, but it would impact the flow of the data. Testing will be required to see what kind of impact this will have on the timeliness of data acquisition.
REST API impact¶
No direct impact on the resources presented by the API.
Security impact¶
None beyond existing concerns.
Pipeline impact¶
Describing pipeline impact is somewhat complex because the word pipeline can be used:
to describe the overall process whereby a suite of samples is gathered and transformed
to identify the pairing of a
sourcewith asinkin thepipeline.yamlfileto point at the file
pipeline.yamlwhich configures many aspects of gathering, transforming and publishing samples as well as transforming and publishing samples that have been produced from notifications
In this change transformation and publishing will be localized in the notification agents. Gathering and producing of samples will be done by polling agents, the api server (via posting meters) and the notification agents.
In order to ease migration the polling agents will continue to use
the pipeline.yaml file to configure the sources they will poll
and the intervals at which that polling will be done. The same file
may be used, or a new file without any sinks. The idea is that both
should work.
The most significant difference is that the existing validation
which confirms that there is a pairing between source and
sink will be removed. The polling agents may poll whatever
they like. If the notification agent doesn’t want the samples that
are generated it can simply drop them on the floor. This provides
two benefits:
it makes the pipeline (on the polling side) less complex
it focuses authority in the notification agent
Other end user impact¶
None.
Performance/Scalability Impacts¶
There is hope that this will decrease the footprint of the pollsters making it easier to deploy many of them, thus increasing the overall performance of the polling system.
On the negative side there could be increased load on both the messaging and the notification agents but this could be compensated for by:
Having the pollsters publish their notifications on their own topic or even bus.
Deploying more notification agents (in a horizontal fashion).
Other deployer impact¶
Deployers will need to be aware of their options for deploying suitable numbers of notification agents.
Developer impact¶
Testing sample transformation from point of sample creation to point of storage will become more complex as it will traverse several services. With the advent of in-tree functional testing there is a place where this testing can and should be done. Having those tests will be very helpful in the long run despite the initial pain in setting them up.
Implementation¶
Assignee(s)¶
- Primary assignee:
chdent
- Other contributors:
you and all your friends
- Ongoing maintainer:
chdent
Work Items¶
Ensure existence of robust tests of both polling and notification agents.
Replace source and sink handling in the pollster code with a simpler read of the source descriptions in pipeline.yaml (leaving open the future option of using a different file). Create samples will be republished as notifications.
Update documenation to reflect new architecture.
Future lifecycle¶
The members of the Telemetry program will do any required ongoing maintenance for this feature.
Dependencies¶
No new dependencies.
Testing¶
Testing as described above.
Documentation Impact¶
Deployment documentation will need to be updated to reflect the connection between the polling and notification agents, the option to use a custom messaging bus and other opportunities for customization.
References¶
None at this time.
