TaskFlow

TaskFlow is a Python library that helps to make task execution easy, consistent and reliable. [1]

Note

If you are just getting started or looking for an overview please visit: http://wiki.openstack.org/wiki/TaskFlow which provides better introductory material, description of high level goals and related content.

Contents

• Atoms, tasks and retries
  • Atom
  • Task
  • Retry
  • Interfaces
  • Hierarchy
• Arguments and results
  • Arguments specification
  • Results specification
  • Revert arguments
  • Retry arguments
• Inputs and outputs
  • Flow inputs and outputs
  • Engine and storage
• Patterns
  • Linear flow
  • Unordered flow
  • Graph flow
  • Hierarchy
• Engines
  • Overview
  • Creating
  • Usage
  • Types
  • How they run
  • Special cases
  • Scoping
  • Interfaces
  • Implementations
  • Hierarchy
• Notifications and listeners
  • Overview
  • Receiving notifications with callbacks
  • Listeners
  • Interfaces
  • Implementations
  • Hierarchy
• Persistence
  • Overview
  • How it is used
  • Usage
  • Types
  • Interfaces
  • Models
  • Implementations
  • Storage
  • Hierarchy
• Resumption
  • Overview
  • Factories
  • Names
  • Scenarios
• Jobs
  • Overview
  • Definitions
  • High level architecture
  • Features
  • Usage
  • Types
  • Considerations
  • Interfaces
  • Implementations
  • Hierarchy
• Conductors
  • Overview
  • Considerations
  • Interfaces
  • Implementations
  • Hierarchy

Supplementary

Examples

While developing TaskFlow the team has worked hard to make sure the various concepts are explained by relevant examples. Here are a few selected examples to get started (ordered by perceived complexity):

• Hello world
• Passing values from and to tasks
• Using listeners
• Using listeners (to watch a phone call)
• Dumping a in-memory backend
• Making phone calls
• Making phone calls (automatically reverting)
• Building a car
• Iterating over the alphabet (using processes)
• Watching execution timing
• Distance calculator
• Table multiplier (in parallel)
• Linear equation solver (explicit dependencies)
• Linear equation solver (inferred dependencies)
• Linear equation solver (in parallel)
• Creating a volume (in parallel)
• Summation mapper(s) and reducer (in parallel)
• Sharing a thread pool executor (in parallel)
• Storing & emitting a bill
• Suspending a workflow & resuming
• Creating a virtual machine (resumable)
• Creating a volume (resumable)
• Running engines via iteration
• Controlling retries using a retry controller
• Distributed execution (simple)
• Distributed notification (simple)
• Distributed mandelbrot (complex)
  • Output
  • Code
• Jobboard producer/consumer (simple)
• Conductor simulating a CI pipeline
• Conductor running 99 bottles of beer song requests

To explore more of these examples please check out the examples directory in the TaskFlow source tree.

Note

If the examples provided are not satisfactory (or up to your standards) contributions are welcome and very much appreciated to help improve them. The higher the quality and the clearer the examples are the better and more useful they are for everyone.

Considerations

Things to consider before (and during) development and integration with TaskFlow into your project:

• Read over the paradigm shifts and engage the team in IRC (or via the openstack-dev mailing list) if these need more explanation (prefix [Oslo][TaskFlow] to your emails subject to get an even faster response).
• Follow (or at least attempt to follow) some of the established best practices (feel free to add your own suggested best practices).
• Keep in touch with the team (see above); we are all friendly and enjoy knowing your use cases and learning how we can help make your lives easier by adding or adjusting functionality in this library.

Miscellaneous

• Exceptions
• States
  • Engine
  • Flow
  • Task
  • Retry
  • Jobs
• Types
  • Entity
  • Failure
  • Graph
  • Notifier
  • Sets
  • Timing
  • Tree
• Utilities
  • Async
  • Deprecation
  • Eventlet
  • Iterators
  • Kazoo
  • Kombu
  • Miscellaneous
  • Mixins
  • Persistence
  • Redis
  • Schema
  • Threading

Bookshelf

A useful collection of links, documents, papers, similar projects, frameworks and libraries.

Note

Please feel free to submit your own additions and/or changes.

• Libraries & frameworks
• Languages
• Services
• Papers
• Related paradigms

Release notes

• CHANGES
  • 1.30.0
  • 1.29.0
  • 1.28.0
  • 1.27.0
  • 1.26.0
  • 1.25.0
  • 1.24.0
  • 1.23.0
  • 1.22.0
  • 1.21.0
  • 1.20.0
  • 1.19.0
  • 1.18.0
  • 1.17.0
  • 1.16.0
  • 1.15.0
  • 1.14.0
  • 0.13.0
  • 0.12.0
  • 0.11.0
  • 0.10.1
  • 0.10.0
  • 0.9.0
  • 0.8.1
  • 0.8.0
  • 0.7.1
  • 0.7.0
  • 0.6.1
  • 0.6.0
  • 0.5.0
  • 0.4.0
  • 0.3.21
  • 0.3
  • 0.2
  • 0.1.3
  • 0.1.2
  • 0.1.1
  • 0.1

Indices and tables

• Index
• Module Index
• Search Page

[1]	It should be noted that even though it is designed with OpenStack integration in mind, and that is where most of its current integration is it aims to be generally usable and useful in any project.