Rocky Series Release Notes

14.2.0-7

Upgrade Notes

  • [bug 1872737] Added a default TTL of 15 minutes for signed EC2 credential requests, where previously an EC2 signed token request was valid indefinitely. This change in behavior is needed to protect against replay attacks.

Critical Issues

  • [bug 1872733] Fixed a critical security issue in which an authenticated user could escalate their privileges by altering a valid EC2 credential.

  • [bug 1872735] Fixed a security issue in which a trustee or an application credential user could create an EC2 credential or an application credential that would permit them to get a token that elevated their role assignments beyond the subset delegated to them in the trust or application credential. A new attribute app_cred_id is now automatically added to the access blob of an EC2 credential and the role list in the trust or application credential is respected.

Security Issues

  • [bug 1872733] Fixed a critical security issue in which an authenticated user could escalate their privileges by altering a valid EC2 credential.

  • [bug 1872735] Fixed a security issue in which a trustee or an application credential user could create an EC2 credential or an application credential that would permit them to get a token that elevated their role assignments beyond the subset delegated to them in the trust or application credential. A new attribute app_cred_id is now automatically added to the access blob of an EC2 credential and the role list in the trust or application credential is respected.

  • [bug 1872737] Fixed an incorrect EC2 token validation implementation in which the timestamp of the signed request was ignored, which made EC2 and S3 token requests vulnerable to replay attacks. The default TTL is 15 minutes but is configurable.

  • [bug 1872755] Added validation to the EC2 credentials update API to ensure the metadata labels ‘trust_id’ and ‘app_cred_id’ are not altered by the user. These labels are used by keystone to determine the scope allowed by the credential, and altering these automatic labels could enable an EC2 credential holder to elevate their access beyond what is permitted by the application credential or trust that was used to create the EC2 credential.

  • [bug 1873290] [bug 1872735] Fixed the token model to respect the roles authorized OAuth1 access tokens. Previously, the list of roles authorized for an OAuth1 access token were ignored, so when an access token was used to request a keystone token, the keystone token would contain every role assignment the creator had for the project. This also fixed EC2 credentials to respect those roles as well.

Bug Fixes

  • [bug 1831918] Credentials now logs cadf audit messages.

  • [bug 1872733] Fixed a critical security issue in which an authenticated user could escalate their privileges by altering a valid EC2 credential.

  • [bug 1872735] Fixed a security issue in which a trustee or an application credential user could create an EC2 credential or an application credential that would permit them to get a token that elevated their role assignments beyond the subset delegated to them in the trust or application credential. A new attribute app_cred_id is now automatically added to the access blob of an EC2 credential and the role list in the trust or application credential is respected.

  • [bug 1872737] Fixed an incorrect EC2 token validation implementation in which the timestamp of the signed request was ignored, which made EC2 and S3 token requests vulnerable to replay attacks. The default TTL is 15 minutes but is configurable.

  • [bug 1872755] Added validation to the EC2 credentials update API to ensure the metadata labels ‘trust_id’ and ‘app_cred_id’ are not altered by the user. These labels are used by keystone to determine the scope allowed by the credential, and altering these automatic labels could enable an EC2 credential holder to elevate their access beyond what is permitted by the application credential or trust that was used to create the EC2 credential.

  • [bug 1873290] [bug 1872735] Fixed the token model to respect the roles authorized OAuth1 access tokens. Previously, the list of roles authorized for an OAuth1 access token were ignored, so when an access token was used to request a keystone token, the keystone token would contain every role assignment the creator had for the project. This also fixed EC2 credentials to respect those roles as well.

  • [bug 1878938] Previously when a user used to have system role assignment and tries to delete the same role, the system role assignments still existed in system_assignment table. This causes keystone to return HTTP 404 Not Found errors when listing role assignments with names (e.g., –names or ?include_names).

    If you are affected by this bug, you must remove stale role assignments manually. The following is an example SQL statement you can use to fix the issue, but you should verify it’s applicability to your deployment’s SQL implementation and version.

    SQL:

    • delete from system_assignment where role_id not in (select id from role);

  • [bug 1885753] Keystone’s SQL identity backend now retries update user requests to safely handle stale data when two clients update a user at the same time.

14.2.0

Bug Fixes

  • [bug 1773967] Fixes an issue where users who had role assignments only via a group membership and not via direct assignment could create but not use application credentials. It is important to note that federated users who only have role assignments via a mapped group membership still cannot create application credentials.

  • [bug 1782922] Fixed the problem where Keystone indiscriminately return the first RDN as the user ID, regardless whether it matches the configured ‘user_id_attribute’ or not. This will break deployments where ‘group_members_are_ids’ are set to False and ‘user_id_attribute’ is not in the DN. This patch will perform a lookup by DN if the first RND does not match the configured ‘user_id_attribute’.

  • [bug 1801873] This fixes an issue where an LDAP-backed domain could not be deleted due to the existence of shadow users in the SQL database.

  • [bug 1840291] Adds retries for delete_credential_for_user method to avoid DBDeadlocks when deleting large number of credentials concurrently.

  • [bug 1843609 <https://bugs.launchpad.net/keystone/+bug/1843609>] Fixed an issue where system-scoped tokens couldn’t be used to list users and groups (e.g., GET /v3/users or GET /v3/groups) if keystone.conf [identity] domain_specific_drivers_enabled=True and the API would return an HTTP 401 Unauthorized. These APIs now recognize system-scoped tokens when using domain-specific drivers.

  • [bug 1856881] keystone-manage bootstrap can be run in upgrade scenarios where pre-existing domain-specific roles exist named admin, member, and reader.

  • [Bug 1856904] The initiator object for CADF notifications now will always contain the username for the user who initated the action. Previously, the initator object only contained the user_id, which lead to issues mapping to users when using LDAP-backed identity providers. This also helps the initiator object better conform to the OpenStack standard for CADF.

  • [bug 1858012] Fixes a bug in the /v3/role_assignments filtering where the role.id query parameter didn’t properly filter role assignments by role in cases where there were multiple system role assignments.

14.1.0

Bug Fixes

  • [bug 1780503] The notification wrapper now sets the initiator’s id to the given user id. This fixes an issue where identity.authentication event would result in the initiator id being a random default UUID, rather than the user’s id when said user would authenticate against keystone.

  • [bug 1810393] Now when an identity provider protocol is deleted, the cache info for the related federated users will be invalidated as well.

  • [bug 1798184] [bug 1820333] In Python 3, python-ldap no longer allows bytes for some fields (DNs, RDNs, attribute names, queries). Instead, text values are represented as str, the Unicode text type. Compatibility support is provided for Python 2 by setting bytes_mode=False [1].

    The keystone LDAP backend is updated to adhere to this behavior by using bytes_mode=False for Python 2 and dropping UTF-8 encoding and decoding fields that are now represented as text in python-ldap.

    [1] More details about byte/str usage in python-ldap can be found at: http://www.python-ldap.org/en/latest/bytes_mode.html#bytes-mode

    Note that at a minimum python-ldappool 2.3.1 is required.

  • [bug 1810983] With the removal of KeystoneToken from the token model, we longer have the ability to use the token data syntax in the policy rules. This change broke backward compatibility for anyone deploying customized Keystone policies. Unfortunately, we can’t go back to KeystoneToken model as the change was tightly coupled with the other refactored authorization functionalities.

    Since the scope information is now available in the credential dictionary, we can just make use of it instead. Those who have custom policies must update their policy files accordingly.

14.0.1

Bug Fixes

  • [bug 1796887] Add caching on trust role validation to improve performance. Services relying heavily on trusts are impacted as the trusts are validated against the database. This adds caching on those operations to improve performance

14.0.0

Prelude

Keystone has historically used a custom rolled WSGI framework based loosely on [webob] which was in turn loaded by the [pythonpaste library]. The Keystone team has been planning to move away from the home-rolled solution and to a common framework for a number of release cycles. As of the Rocky release Keystone is moving to the Flask framework.

The token_auth middleware functionality has been merged into the main auth middleware (keystone.middleware.auth.AuthContextMiddleware). admin_token_auth must be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file. The [filter:token_auth] block will also need to be removed from your paste ini file. Failure to remove these elements from your paste ini file will result in keystone to no longer start/run when the token_auth is removed in the Stein release.

New Features

  • [bug 1754185] Registered limits and project limits now support an optional, nullable property called description. Users can create/update a registered limit or project limit with description now.

  • [blueprint basic-default-roles] Support has been added for deploying two new roles during the bootstrap process, reader and member, in addition to the admin role.

  • [blueprint strict-two-level-model] A new limit enforcement model called strict_two_level is added. Change the value of the option [unified_limit]/enforcement_model to strict_two_level to enable it. In this [model]: 1. The project depth is force limited to 2 level. 2. Any child project’s limit can not exceed the parent’s. Please ensure that the previous project and limit structure deployment in your Keystone won’t break this model before starting to use it. If a newly created project results in a project tree depth greater than 2, a 403 Forbidden error will be raised. When try to use this model but the project depth exceed 2 already, Keystone process will fail to start. Operators should choose another available model to fix the issue first.

  • [blueprint strict-two-level-model] The project_id filter is added for listing limits. This filter is used for system-scoped request only to fetch the specified project limits. Non system-scoped request will get empty response body instead.

  • [blueprint strict-two-level-model] The include_limits filter is added to GET /v3/projects/{project_id} API. This filter should be used together with parents_as_list or subtree_as_list filter to add parent/sub project’s limit information the response body.

  • [bug 1779903] When a project is deleted, the limits which belong to it will be deleted as well.

Upgrade Notes

  • [bug 1778945] The pluggable interface for token providers has changed. If you’re maintaining a custom token provider, you’re going to be affected by these interface changes. Implementing the new interface will be required before using your custom token provider with the Rocky release of keystone. The new interface is more clear about the relationship and responsibilities between the token API and pluggable token providers.

  • Keystone no longer is loaded via paste.deploy and instead directly loads the Flask based application. If a deployment is relying on the entry-point generated wsgi files, it is important to get the newest ones. These new files have minor changes to support the new loading mechanisms. The files will be auto-generated via PBR and setup. The paste.ini file will now be ignored, but will remain on disk until the Stein release to ensure deployment tools are not inadvertently broken. The paste.ini file will have a comment added to indicate it is ignored.

  • With the change to not load via paste.deploy it is no longer possible to remove any elements from the pipeline that keystone relies on. This includes former extensions (S3, EC2) or middleware. If these APIs must be disabled, it is recommended to utilize policy to deny access.

  • With the change to not load via paste.deploy it is no longer possible to inject custom middleware into the pipeline directly, it is recommended to wrap the entire stack if custom middleware is needed outside of what Keystone relies on. It is also possible to change/modify requests and responses via a smart proxy layer (e.g. HAProxy).

  • As a performance improvement, the base mapping driver’s method get_domain_mapping_list now accepts an optional named argument entity_type that can be used to get the mappings for a given entity type only. As this new call signature is already used in the identity.core module, authors/maintainers of out-of-tree custom mapping drivers are expected to update their implementations of get_domain_mapping_list method accordingly.

  • Remove token_auth from your keystone paste.ini file. Failure to remove these elements from your paste ini file will result in keystone to no longer start/run when the token_auth is removed in the Stein release.

  • The token provider API has removed the needs_persistence property from the abstract interface. Token providers are expected to handle persistence requirement if needed. This will require out-of-tree token providers to remove the unused property and handle token storage.

  • Keystone now relies on python-ldap instead of pyldap. The pyldap library is a deprecated fork from python-ldap. Starting with python-ldap 3.0 release this has been merged and is maintained there.

Deprecation Notes

  • The option [token] infer_roles=False is being deprecated in favor of always expanding role implications during token validation. Default roles depend on a chain of implied role assignments, ex: an admin user will also have the reader and member role. Therefore by ensuring that all these roles will always appear on the token validation response, we can improve the simplicity and readability of policy files.

  • The keystone.middleware.core:TokenAuthMiddleware is deprecated for removal.

Security Issues

  • It is no longer possible to, via the paste.ini file to inject middleware into the running keystone application. This reduces the attack surface area. While this is not a huge reduction in surface area, it is one less potential place that malicious code could be loaded. Malicious middleware historically could collect information and/or modify the requests and responses from Keystone.

Bug Fixes

  • [bug 1703666] Fixing multi-region support for the templated v3 catalog by making sure that the catalog contains only one definition per endpoint, and that each region is listed under that endpoint. Previously each region and endpoint would have had its own definition.

  • [bug 1728907] In some rare cases, an empty key file can get created within the fernet key repository. When keystone tries to load the keys from disk, it will fail with an invalid fernet key ValueError. Keystone now handles empty key files when loading and rotating keys. If an empty file exists, it will be ignored when loaded, reported as a warning in the log, and overwritten with a valid key upon rotation.

  • [bug 1746599] Fixes user email being set for federated shadow users, when the rule contains email in user.

  • [bug 1748970] A bug was introduced in Queens that resulted in system role assignments being returned when querying the role assignments API for a specific role. The issue is fixed and the list of roles returned from GET /v3/role_assignments?role.id={role_id} respects system role assignments.

  • [bug 1749264] A user’s system role assignment will be removed when the user is deleted.

  • [bug 1749267] A group’s system role assignments are removed when the group is deleted.

  • [bug 1750415] Fixes an implementation fault in application credentials where the application credential reference was not populated in the token data, causing problems with the token validation when caching was disabled.

  • [bug 1751045] It is now possible to clean up role assignments for groups that don’t exist in the identity backend. This is relevant to deployments that are backed by LDAP and groups are removed directly by LDAP and not through keystone.

  • [bug 1753584] Fix formatting of ImportError when using a driver not found in the list of token providers.

  • [bug 1755874] Users now can have the resource option lock_password set which prevents the user from utilizing the self-service password change API. Valid values are True, False, or “None” (where None clears the option).

  • [bug 1756190] When filtering projects based on tags, the filtering will now be performed by matching a subset containing the given tags against projects, rather than exact matching. Providing more tags when performing a search will yield more exact results while less will return any projects that match the given tags but could contain other tags as well.

  • [bug 1757022] In previous releases, keystone-manage mapping_purge --type {user,group} command would purge all mapping incorrectly instead of only purging the specified type mappings. keystone-manage mapping_purge --type {user,group} now purges only specified type mappings as expected.

  • [bug 1759289] The keystone-manage token_flush command no longer establishes a connection to a database, or persistence backend. It’s usage should be removed if you’re using a supported non-persistent token format. If you’re relying on external token providers that write tokens to disk and would like to maintain this functionality, please consider porting it to a separate tool.

  • [bug 1760205] When deleting a shadow user, the related cache info is not invalidated so that Keystone will raise 404 UserNotFound error when authenticating with the previous federation info. This bug has been fixed now.

  • [bug 1760521] Fixed the bug that the result count for domain list may lack one if the config option list_limit in [resource] is set.

  • [bug 1760809] Identity providers registered to domains will now be cleaned up when the domain is deleted.

  • [bug 1763824] JSON Schema implementation nullable in keystone.common.validation now properly adds None to the enum if the enum exists.

  • [bug 1765193] The unified limit API now exposes a deployment’s configured enforcement model via the GET /limits/model endpoint.

  • [bug 1774229] The API reference for token management now includes more specific examples for different token scopes.

  • [bug 1778109] Previously the token data for a trust-scoped token may have contained duplicate roles, when implied roles were present. This is no longer the case, for the sake of accuracy and to prevent the breaking of applications which may consume this role list.

  • [bug 1778945] There were several improvements made to the token provider API and interface that simplify what external developers need to do and understand in order to provide their own token provider implementation. Please see the linked bug report for more details as to why these changes were made and the benefits they provide for both upstream and downstream developers.

  • [bug 1780159] Revoke the role cache when creating a project. This removes the delay before making it appear in the list when a user has inherited role on it.

  • [bug 1782704] Checking for non-existant configuration files is more robust to ensure proper logging to users when passing configuration information to keystone-manage.

  • [bug 1785164] Setting resource limits on domains is explicitly unsupported. Previously, it was possible to set a limit on a domain and the response would include the domain ID as the project ID of the limit. This issue has been corrected by explicitly opting domains out of limit support. A later release may include functionality for domains to be associated to limit resources.

  • [bug 1754184] The unified limit APIs has been refactored to align with the following API-WG guidelines: 1. POST unified limits no longer returns all the limits during create operations. It now only returns the newly created limits. 2. Support for updating multiple limits in a single request has been removed by implementing PATCH instead of PUT. Please note that the unified limits APIs is still experimental making it possible to include these improvements.

Other Notes

  • A new interface called list_federated_users_info is added to shadow backend. It’s used to get the shadow user information internally. If you are maintaining any out-tree shadow backends, please implement this function for them as well.

  • [#openstack-tc IRC log] With Technical Comittee consensus the Keystone team is not wiring up the reminents of the V2.0 API that was maintained strictly due to a failure to copy/paste a direct deprecation notice on the controllers even though the V2.0 API was deprecated in its entirety. This should have no meaningful impact on any user as the APIs (ec2token) have a v3 equivalent

  • The implied roles API has been marked as stable. This API was originally implemented in Mitaka and marked as experimental. There haven’t been any backwards incompatible updates since then. As a result, the API is being marked as stable.

  • [blueprint removed-as-of-rocky] Removed support for direct import of authentication drivers. If you’re using full path names for authentication methods in configuration, please update your configuration to use the corresponding namespaces.

  • [blueprint removed-as-of-rocky] Removed support for token bind operations, which were supported by the uuid, pki, and pkiz token providers. Support for this feature was deprecated in Pike.