2023.1 Series Release Notes¶
17.0.0¶
Upgrade Notes¶
Python 3.6 & 3.7 support has been dropped. The minimum version of Python now supported is Python 3.8.
16.0.0.0rc1¶
New Features¶
Now healthcheck middleware is enabled by default. Applications like loadbalancer or monitoring tools can use
/healthcheckpath to monitor health of each API endpoints. Remove entries for healthcheck fromapi-paste.inito disable this functionality.
15.0.0¶
Upgrade Notes¶
The default value of
[oslo_policy] policy_fileconfig option has been changed frompolicy.jsontopolicy.yaml. Operators who are utilizing customized or previously generated static policy JSON files (which are not needed by default), should generate new policy files or convert them in YAML format. Use the oslopolicy-convert-json-to-yaml tool to convert a JSON to YAML formatted policy file in backward compatible way.
Deprecation Notes¶
Use of JSON policy files was deprecated by the
oslo.policylibrary during the Victoria development cycle. As a result, this deprecation is being noted in the Xena cycle with an anticipated future removal of support byoslo.policy. As such operators will need to convert to YAML policy files. Please see the upgrade notes for details on migration of any custom policy files.
12.0.0.0rc1¶
Upgrade Notes¶
Python 2.7 support has been dropped. Last release of sahara to support python 2.7 is OpenStack Train. The minimum version of Python now supported by sahara is Python 3.6.
10.0.0¶
Prelude¶
Added new tool sahara-status upgrade check.
Sahara’s APIv2 is now considered stable, and no longer experimental.
New Features¶
New framework for
sahara-status upgrade checkcommand is added. This framework allows adding various checks which can be run before a Sahara upgrade to ensure if the upgrade can be performed safely.
Adding the ability to change default timeout parameter for Ambari agent package installation
Users of Sahara’s APIv2 may request a microversion of that API, with “OpenStack-API-Version: data-processing [version]” in the request headers.
In Sahara APIv2, the type, availability zone, and locality of boot volumes may be expressed explicitly through the boot_volume_type, boot_volume_availability_zone, and boot_volume_local_to_instance parameters.
In an effort to improve Sahara’s usability and maintainability we are splitting the plugins from Sahara core into their own repositories.
Upgrade Notes¶
Operator can now use new CLI tool
sahara-status upgrade checkto check if Sahara deployment can be safely upgraded from N-1 to N release.
The default proxy role for Swift is now member instead of Member. Keystone now creates the former by default, even if the latter is recognised to be the same (case preserving).
Bug Fixes¶
With APIv2 we detected some inconsistencies on the policies. This patch updates the policy to fix those inconsistencies.
The command hdfs fs has been deprecated in favour of hdfs fs. This fixes will allow the use of Hbase service.
This fixes the issue with NTP configuration where a preferred server provided by the user is added to the end of the file and the defaults are not deleted. Here we add the preferred server to the top of the file.
Other Notes¶
As part of the APIv2 work we changed all tenant_id references to project_id on the return payload of REST calls.
All APIv2 policy names have been changed to the recommended format: specifically, changes to resource names (now _singular_, whereas previously they may have been _plural_, or otherwise inconsistent), action verbs (now fully independent of HTTP semantics) and overall formatting (hyphens replace underscores). Eventually, the remaining non-conforming policy names will be deprecated too.
Some polishings to APIv2 have been made in an effort to bring it from experimental (and therefore, evolving and unpredictable) to stable. More instances of tenant_id have been changed to project_id, in the cluster and job template APIs. job_id was changed to job_template_id in the job API. The newly-minted query string validation feature has been fixed to allow show_progress as a parameter on cluster GET; on a similar note some APIv2 endpoints which previously could be filtered by hadoop_version are now filtered by plugin_version instead. Also, the schema for cluster PATCH in APIv1.1 now no longer includes the key update_keypair; its prior inclusion was a mistake.
In APIv2 there is now strict checking of parameters in the query string. This means that unexpected values in the query string will give a 400 error (as opposed to previously being ignored, or causing a 500 error).
9.0.0.0rc1¶
Other Notes¶
The URL of an S3 data source may have s3:// or s3a://, equivalently.
9.0.0.0b3¶
Prelude¶
Sahara APIv2 is reaching a point of maturity. Therefore, new deployments should include an unversioned endpoint in the service catalogue for the “data-processing” service, for the purposes of more intuitive version discovery. Eventually existing deployments should switch to an unversioned endpoint, too, but only after time is given for the use of older clients to be less likely.
Every new release of Sahara we update our plugins list. Some new versions are added and some removed and other marked as deprecated. For Rocky we are deprecating CDH 5.7.0, Spark 1.6.0 and 2.1 as well as Storm 1.0.1. We are also removing CDH 5.5.0, MapR 5.1.0, Spark 1.3.1 and Storm 0.9.2.
New Features¶
Adding the ability to create Ambari 2.6 images on sahara-image-pack
Adding the ability to boot a Sahara cluster from volumes instead of images.
Adding support to CDH 5.13.0 in CDH plugin.
The experimental APIv2 supports simultaneous creation of multiple clusters only through POST /v2/clusters (using the count parameter). The POST /v2/clusters/multiple endpoint has been removed.
Operators can now update the running configuration of Sahara processes by sending the parent process a “HUP” signal. Note: The configuration option must support mutation.
The behaviour of force deletion of clusters (APIv2) has changed. Stack-abandon is no longer used. The response from the force-delete API call now includes the name of the stack which had underlain that deleted cluster.
Use a new keypair to access to the running cluster when the cluster’s keypair is deleted.
An EDP data source may reference a file stored in a S3-like object store.
Upgrade Notes¶
Adding Spark 2.3 to supported plugins list.
Adding new versions of Storm, 1.2.0 and 1.2.1. Both will exist under the same tag 1.2.
We are removing some plugins versions. Those are CDH 5.5.0, MapR 5.1.0, Spark 1.3.1 and Storm 0.9.2.
Deprecation Notes¶
The sahara-all entry point is now deprecated. Please use the Sahara-API and sahara-engine entry points instead.
We are deprecating CDH 5.7.0, Spark 1.6.0 and 2.1 and Storm 1.0.1.
Bug Fixes¶
Hadoop is now better configured to use the proper Keystone domain for interaction with Swift; previously the ‘default’ domain may have been incorrectly used.
Other Notes¶
A few responses in the experimental (but nearly-stable) APIv2 have been tweaked. To be specific, the key hadoop_version has been replaced with plugin_version, the key job has been replaced with job_template, the key job_execution has been replaced with job, and the key oozie_job_id has been replaced with engine_job_id. In fact, these changes were all previously partially implemented, and are now completely implemented.
9.0.0.0b2¶
New Features¶
Support deploy Hadoop 2.7.5 with vanilla plugin.
9.0.0.0b1¶
New Features¶
Implemented support of HDP 2.6 in the Ambari plugin.
Known Issues¶
Remove the step “upload httpclient to oozie/sharelib” in sahara code. User should use latest vanilla-2.8.2 image which is built on SIE “Change-ID: I3a25ee8c282849911089adf6c3593b1bb50fd067”.
8.0.0¶
New Features¶
Enables the creation and validation of Ambari 2.4 images using the new image generation process where libguestfs replaces the use of DIB.
Sahara’s APIv2 is now exposed by default (although its state is still experimental). It has feature parity with Sahara’s APIv1.1, but APIv2 brings better REST semantics, tweaks to some response payloads, and some other improvements. APIv2 will remain labelled experimental until it is stabilised following the addition of new features to it in the upcoming cycle(s).
Enables the creation and validation of CDH 5.11.0 images using the new image generation process where libguestfs replaces the use of DIB.
Enables the creation and validation of CDH 5.9.0 images using the new image generation process where libguestfs replaces the use of DIB.
The ability to force delete clusters is exposed in Sahara APIv2. The Heat service must support Stack Abandon for force delete to function properly.
Adding ability to create and validate MapR 5.2.0 images using the new image gen tool.
This feature allows the policy enforcement to be done in code thus facilitating better maintenance of the policy file. In code the default policies are set and the operator only needs to change the policy file if they wish to override the rule or role for a specific policy or operation. Also, a complete policy file can be generated using genconfig tool.
Adding Spark version 2.2 to Sahara.
An EDP job binary may reference a file stored in a S3-like object store.
Support deploy Hadoop 2.8.2 with vanilla plugin.
Upgrade Notes¶
Nova Network has been fully removed from the OpenStack codebase, remove all instances of switches on use_neutron and the configuration value.
Sample configuration files previously installed in share/Sahara will now be installed into etc/Sahara instead.
Deprecation Notes¶
The use of [keystone_authtoken] credentials for trust creation is now deprecated. Please use the new [trustee] config section. The options
username,password,project_name,user_domain_name,project_domain_name, andauth_url(with version) are obligatory within that section.
Other Notes¶
The presence of project ID in Sahara APIv1 paths is now optional.
7.0.0.0rc1¶
Deprecation Notes¶
Spark version 1.3.1 is deprecated.
The custom admin_user_domain_name and admin_project_domain_name configuration options have been removed; they are provided by keystone_authtoken as user_domain_name and project_domain_name respectively.
Other Notes¶
We are assuring that Ironic support was tested after latest updates to Nova and Sahara and it is fully functional.
7.0.0.0b3¶
New Features¶
CDH 5.11.0 is supported in CDH plugin.
Enables the creation and validation of CDH 5.7.0 images using the new image generation process where libguestfs replaces the use of DIB.
7.0.0.0b2¶
New Features¶
Storm 1.1.0 is supported in Storm plugin.
Known Issues¶
Ironic integration might be broken if Floating IPs are used, due to the use of pre-created ports by the Sahara engine. The status of Ironic support was untested for this release.
Deprecation Notes¶
Storm version 0.9.2 is deprecated.
Support for Nova-Network is removed, reflective of its removal from Nova itself and from python-novaclient. use_neutron=False is unsupported.
7.0.0.0b1¶
New Features¶
Version 5.5.0 of Cloudera plugin is deprecated.
Versions 5.0.0 5.3.0 and 5.4.0 of Cloudera plugin are removed.
6.0.0¶
New Features¶
Add Kafka to MapR plugin
MapR repositories now can be configured in general section of cluster template configs
CDH 5.9.0 is supported in CDH plugin.
Implemented support of HDP 2.5 in the Ambari plugin.
The following service versions were added to MapR 5.2.0 plugin - Pig 0.16 - Spark 2.0.1 - Hue 3.10 - Drill 1.8, 1.9
Upgrade Notes¶
Neutron is used by default now (use_neutron=True). Nova-Network is not functional for most use cases starting from Ocata.
Deprecation Notes¶
MapR 5.1.0.mrv2 is now deprecated and will be removed in the Ocata release. It is recommended to use MapR 5.2.0.mrv2 instead.
Removed support for the MapR 5.0.0 plugin.
The ‘enable’ option of the ‘oslo_messaging_notifications’ section has been removed. To enable notifications now please specify the ‘driver’ option in the same section.
5.0.0.0rc1¶
New Features¶
MapR 5.2.0 is supported in MapR plugin.
Add Sentry to MapR plugin
Added event log for HDP plugin
Kerberos support implemented for Cloudera and Ambari plugins. New Oozie client implemented to support authentication for Oozie in Kerberised cluster.
Custom health check is added to MapR plugin
The following service versions were added to MapR 5.2.0 plugin - HBase 1.1 - Drill 1.6 - Mahout 0.11 0.12 - Spark 1.6.1 - Impala 2.5
Added rack awareness feature for CDH 5.5 and CDH 5.7
Added rack awareness feature for HDP plugin
5.0.0.0b3¶
New Features¶
Kafka was added in CDH 5.5 and CDH 5.7
Add ability to automatically generate better configurations for Ambari cluster by using ‘ALWAYS_APPLY’ strategy
Versions 5.5.0 and 5.7.0 of Cloudera plugin are declared as stable.
CDH 5.7 is supported in CDH plugin.
Added integration of Designate for hostname resolution through DNS servers
MapR 5.1.0.mrv2 is now Enabled.
Remove support for Spark standalone in MapR plugin
Refactoring the logic on how Floating IPs are used by Sahara. This change will allow the coexistence of cluster using Floating IPs with cluster that do not.
Deprecation Notes¶
Versions 5, 5.3.0, 5.4.0 of Cloudera plugin are deprecated. It is no longer maintained and supposed to be removed in P release.
Option ‘infrastructure engine’ is removed from Sahara configs.
MapR 5.0.0.mrv2 is now deprecated. It is not recommended for usage. It is better to use MapR 5.1.0.mrv2 instead. This version of plugin will be removed in the Ocata release.
Bug Fixes¶
Fixed issues with hardcoded password during creation MySQL database for Oozie, bug 1541122.
Fixed issues with hardcoded password during starting Hive process, bug 1498035.
5.0.0.0b2¶
Prelude¶
Documentation about distributed periodics are extended with steps about installation additional libs required for correct work of coordination backend. Please refer Advanced Configuration Guide for details.
New Features¶
Add impala 2.2 to MapR plugin
Separate transport URL can be used for notifications purposes now, to enable this feature ‘transport_url’ should be provided in ‘oslo_messaging_notifications’ section.
Upgrade Notes¶
Migration from novaclient.v2.images to glanceclient
Deprecation Notes¶
Convert to cluster template feature is no longer supported by all plugins.
Support of HDP 2.0.6 plugin was removed. Use Ambari plugin instead.
Bug Fixes¶
Fixed launching Hive jobs in Ambari plugin.
Starting Ambari clusters on Centos 7 is fixed.
CA certificate handling in Keystone, Nova, Neutron and Cinder clients are fixed (#330635)
After decommissioning hosts all associated configs groups will be removed in Ambari plugin.
Fix uncompleted event logs for Oozie and Drill in MapR plugin.
5.0.0.0b1¶
Upgrade Notes¶
migration from keystoneclient to keystoneauth is done for using auth features of Keystone.
Bug Fixes¶
Fixed incorrect down scaling of Ambari cluster
This patch adds ability to work with swift by using Keystone API v3
Fixed issue with handling endpoint_type during creation neutron client, closed bug 1564805
