Data Processing API v2¶
Cluster templates¶
A cluster template configures a cluster. A cluster template lists node groups with the number of instances in each group. You can also define cluster-scoped configurations in a cluster template.
Shows details for a cluster template.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_template_id |
path |
string |
The unique identifier of the cluster template. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description (Optional) |
body |
string |
Description of the cluster template |
use_autoconfig |
body |
boolean |
If set to |
cluster_configs |
body |
object |
A set of key and value pairs that contain the cluster configuration. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
default_image_id |
body |
string |
The default ID of the image. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
plugin_name |
body |
string |
The name of the plugin. |
is_default |
body |
boolean |
If set to |
is_protected |
body |
boolean |
If set to |
shares |
body |
string |
The sharing of resources in the cluster. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
project_id |
body |
string |
The UUID of the project. |
node_groups |
body |
object |
The detail properties of the node in key-value pairs. |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
id |
body |
string |
The UUID of the cluster template. |
name |
body |
string |
The name of the cluster template. |
Response Example¶
{
"cluster_templates": [
{
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
},
{
"is_public": true,
"anti_affinity": [],
"name": "asd",
"created_at": "2015-08-18T08:39:39",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"general": {}
},
"shares": null,
"id": "5a9c787c-2078-4f7d-9a66-27759be9051b",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T08:41:15",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "2",
"id": "a65864dd-3f99-4d29-a011-f7711cc23fa0",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-18T08:39:39",
"node_group_template_id": "42ce49de-1b8f-41d5-8f4a-244ec0826d92",
"updated_at": null,
"volumes_per_node": 1,
"is_proxy_gateway": false,
"name": "asd",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"jobtracker"
],
"volumes_size": 10,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": "",
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "2c76e0d3-56cd-4d28-bb4f-4808e538c7b9"
}
}
Updates a cluster template.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_template_id |
body |
string |
The UUID of the cluster template. |
Request Example¶
{
"description": "Updated template",
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"name": "vanilla-updated",
"cluster_configs": {
"HDFS": {
"dfs.replication": 2
}
}
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description (Optional) |
body |
string |
Description of the cluster template |
use_autoconfig |
body |
boolean |
If set to |
cluster_configs |
body |
object |
A set of key and value pairs that contain the cluster configuration. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
default_image_id |
body |
string |
The default ID of the image. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
plugin_name |
body |
string |
The name of the plugin. |
is_default |
body |
boolean |
If set to |
is_protected |
body |
boolean |
If set to |
shares |
body |
string |
The sharing of resources in the cluster. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
project_id |
body |
string |
The UUID of the project. |
node_groups |
body |
object |
The detail properties of the node in key-value pairs. |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
id |
body |
string |
The UUID of the cluster template. |
name |
body |
string |
The name of the cluster template. |
Deletes a cluster template.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_template_id |
body |
string |
The UUID of the cluster template. |
Lists available cluster templates.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting cluster templates.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
description (Optional) |
body |
string |
Description of the cluster template |
use_autoconfig |
body |
boolean |
If set to |
cluster_configs |
body |
object |
A set of key and value pairs that contain the cluster configuration. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
default_image_id |
body |
string |
The default ID of the image. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
plugin_name |
body |
string |
The name of the plugin. |
is_default |
body |
boolean |
If set to |
is_protected |
body |
boolean |
If set to |
shares |
body |
string |
The sharing of resources in the cluster. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
project_id |
body |
string |
The UUID of the project. |
node_groups |
body |
object |
The detail properties of the node in key-value pairs. |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
id |
body |
string |
The UUID of the cluster template. |
name |
body |
string |
The name of the cluster template. |
{
"cluster_templates": [
{
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
},
{
"is_public": true,
"anti_affinity": [],
"name": "asd",
"created_at": "2015-08-18T08:39:39",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"general": {}
},
"shares": null,
"id": "5a9c787c-2078-4f7d-9a66-27759be9051b",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T08:41:15",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "2",
"id": "a65864dd-3f99-4d29-a011-f7711cc23fa0",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-18T08:39:39",
"node_group_template_id": "42ce49de-1b8f-41d5-8f4a-244ec0826d92",
"updated_at": null,
"volumes_per_node": 1,
"is_proxy_gateway": false,
"name": "asd",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"jobtracker"
],
"volumes_size": 10,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": "",
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "2c76e0d3-56cd-4d28-bb4f-4808e538c7b9"
}
}
Creates a cluster template.
Normal response codes:202
Request Example¶
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_groups": [
{
"name": "worker",
"count": 3,
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251"
},
{
"name": "master",
"count": 1,
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae"
}
],
"name": "cluster-template"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description (Optional) |
body |
string |
Description of the cluster template |
use_autoconfig |
body |
boolean |
If set to |
cluster_configs |
body |
object |
A set of key and value pairs that contain the cluster configuration. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
default_image_id |
body |
string |
The default ID of the image. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
plugin_name |
body |
string |
The name of the plugin. |
is_default |
body |
boolean |
If set to |
is_protected |
body |
boolean |
If set to |
shares |
body |
string |
The sharing of resources in the cluster. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
project_id |
body |
string |
The UUID of the project. |
node_groups |
body |
object |
The detail properties of the node in key-value pairs. |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
id |
body |
string |
The UUID of the cluster template. |
name |
body |
string |
The name of the cluster template. |
Clusters¶
A cluster is a group of nodes with the same configuration.
Lists available clusters.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting clusters.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
count |
body |
integer |
The number of nodes in the cluster. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
cluster_template_id |
body |
string |
The UUID of the cluster template. |
is_transient |
body |
boolean |
If set to |
provision_progress |
body |
array |
A list of the cluster progresses. |
status |
body |
string |
The status of the cluster. |
neutron_management_network |
body |
string |
The UUID of the neutron management network. |
clusters |
body |
array |
The list of clusters. |
management_public_key |
body |
string |
The SSH key for the management network. |
status_description |
body |
string |
The description of the cluster status. |
trust_id |
body |
integer |
The id of the trust. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
{
"clusters": [
{
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {
"YARN": {
"Web UI": "http://172.18.168.115:8088",
"ResourceManager": "http://172.18.168.115:8032"
},
"HDFS": {
"Web UI": "http://172.18.168.115:50070",
"NameNode": "hdfs://vanilla-cluster-master-0:9000"
},
"JobFlow": {
"Oozie": "http://172.18.168.115:11000"
},
"MapReduce JobHistory Server": {
"Web UI": "http://172.18.168.115:19888"
}
},
"user_keypair_id": "apavlov",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:36",
"id": "4867d92e-cc7b-4cde-9a1a-149e91caa491",
"management_ip": "172.18.168.115",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "b9f16a07-88fc-423e-83a3-489598fe6737",
"internal_ip": "10.50.0.60",
"instance_name": "vanilla-cluster-master-0"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:36",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:37",
"id": "f3633b30-c1e4-4144-930b-ab5b780b87be",
"management_ip": "172.18.168.118",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "0cf1ee81-aa72-48da-be2c-65bc2fa51f8f",
"internal_ip": "10.50.0.63",
"instance_name": "vanilla-cluster-worker-0"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0d66fd93-f277-4a94-b46a-f5866aa0c38f",
"management_ip": "172.18.168.117",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "4a937391-b594-4ad0-9a53-00a99a691383",
"internal_ip": "10.50.0.62",
"instance_name": "vanilla-cluster-worker-1"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0982cefd-5c58-436e-8f1e-c1d0830f18a7",
"management_ip": "172.18.168.116",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "839b1d56-6d0d-4aa4-9d05-30e029c276f8",
"internal_ip": "10.50.0.61",
"instance_name": "vanilla-cluster-worker-2"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:37",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [
{
"created_at": "2015-09-14T10:57:18",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "0a6d95f9-30f4-4434-823a-a38a7999a5af",
"step_type": "Engine: create cluster",
"step_name": "Create Heat stack",
"updated_at": "2015-09-14T10:57:38",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:16",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "29f2b587-c34c-4871-9ed9-9235b411cd9a",
"step_type": "Engine: create cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:58:22",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:27",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "36f1efde-90f9-41c1-b409-aa1cf9623e3e",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): Oozie",
"updated_at": "2015-09-14T11:01:15",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:22",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "602bcc27-3a2d-42c8-8aca-ebc475319c72",
"step_type": "Plugin: configure cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:59:21",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "7e291df1-2d32-410d-ae89-33ab6f83cf17",
"step_type": "Plugin: configure cluster",
"step_name": "Configure topology data",
"updated_at": "2015-09-14T10:59:37",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:01",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "8ab7933c-ad61-4a4f-88db-23ce78ee10f6",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): DataNodes, NodeManagers",
"updated_at": "2015-09-14T11:00:11",
"successful": true,
"total": 3,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:11",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "9c8dc016-8c5b-4e80-9857-80c41f6bd971",
"step_type": "Plugin: start cluster",
"step_name": "Await DataNodes start up",
"updated_at": "2015-09-14T11:00:21",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "c6327532-222b-416c-858f-73dbb32b8e97",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): HistoryServer",
"updated_at": "2015-09-14T11:00:27",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:41",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "d3eca726-8b44-473a-ac29-fba45a893725",
"step_type": "Engine: create cluster",
"step_name": "Wait for instance accessibility",
"updated_at": "2015-09-14T10:58:14",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:14",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "d7a875ff-64bf-41aa-882d-b5061c8ee152",
"step_type": "Engine: create cluster",
"step_name": "Mount volumes to instances",
"updated_at": "2015-09-14T10:58:15",
"successful": true,
"total": 0,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:55",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "ded7d227-10b8-4cb0-ab6c-25da1462bb7a",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): ResourceManager",
"updated_at": "2015-09-14T11:00:00",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:38",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "e1701ff5-930a-4212-945a-43515dfe24d1",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): NameNode",
"updated_at": "2015-09-14T10:59:54",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:38",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "eaf0ab1b-bf8f-48f0-8f2c-fa4f82f539b9",
"step_type": "Engine: create cluster",
"step_name": "Assign IPs",
"updated_at": "2015-09-14T10:57:41",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
}
],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T11:01:15",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Active"
}
]
}
Creates a cluster.
Normal response codes: 202
Request Example¶
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"user_keypair_id": "test",
"name": "vanilla-cluster",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
count |
body |
integer |
The number of nodes in the cluster. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
cluster_template_id |
body |
string |
The UUID of the cluster template. |
is_transient |
body |
boolean |
If set to |
provision_progress |
body |
array |
A list of the cluster progresses. |
status |
body |
string |
The status of the cluster. |
neutron_management_network |
body |
string |
The UUID of the neutron management network. |
management_public_key |
body |
string |
The SSH key for the management network. |
status_description |
body |
string |
The description of the cluster status. |
trust_id |
body |
integer |
The id of the trust. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Shows details for a cluster, by ID.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
path |
string |
The ID of the cluster |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
count |
body |
integer |
The number of nodes in the cluster. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
cluster_template_id |
body |
string |
The UUID of the cluster template. |
is_transient |
body |
boolean |
If set to |
provision_progress |
body |
array |
A list of the cluster progresses. |
status |
body |
string |
The status of the cluster. |
neutron_management_network |
body |
string |
The UUID of the neutron management network. |
management_public_key |
body |
string |
The SSH key for the management network. |
status_description |
body |
string |
The description of the cluster status. |
trust_id |
body |
integer |
The id of the trust. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Response Example¶
{
"cluster": {
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {},
"user_keypair_id": "test",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T10:57:12",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Validating"
}
}
Deletes a cluster.
Normal response codes: 204 or 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
path |
string |
The ID of the cluster |
force (Optional) |
body |
boolean |
If set to |
Scales a cluster.
Normal response codes: 202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
body |
string |
The UUID of the cluster. |
Request Example¶
{
"add_node_groups": [
{
"count": 1,
"name": "b-worker",
"node_group_template_id": "bc270ffe-a086-4eeb-9baa-2f5a73504622"
}
],
"resize_node_groups": [
{
"count": 4,
"name": "worker"
}
]
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
count |
body |
integer |
The number of nodes in the cluster. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
cluster_template_id |
body |
string |
The UUID of the cluster template. |
is_transient |
body |
boolean |
If set to |
provision_progress |
body |
array |
A list of the cluster progresses. |
status |
body |
string |
The status of the cluster. |
neutron_management_network |
body |
string |
The UUID of the neutron management network. |
management_public_key |
body |
string |
The SSH key for the management network. |
status_description |
body |
string |
The description of the cluster status. |
trust_id |
body |
integer |
The id of the trust. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Updates a cluster.
Normal response codes: 202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
path |
string |
The ID of the cluster |
Request Example¶
{
"name": "public-vanilla-cluster",
"is_public": true
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
count |
body |
integer |
The number of nodes in the cluster. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
cluster_template_id |
body |
string |
The UUID of the cluster template. |
is_transient |
body |
boolean |
If set to |
provision_progress |
body |
array |
A list of the cluster progresses. |
status |
body |
string |
The status of the cluster. |
neutron_management_network |
body |
string |
The UUID of the neutron management network. |
management_public_key |
body |
string |
The SSH key for the management network. |
status_description |
body |
string |
The description of the cluster status. |
trust_id |
body |
integer |
The id of the trust. |
domain_name (Optional) |
body |
string |
Domain name for internal and external hostname resolution. Required if DNS service is enabled. |
Shows provisioning progress for a cluster.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
path |
string |
The ID of the cluster |
Response Example¶
{
"status": "Error",
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076",
"is_transient": false,
"description": "",
"user_keypair_id": "vgridnev",
"updated_at": "2015-03-31 14:10:59",
"plugin_name": "spark",
"provision_progress": [
{
"successful": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:20",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:35",
"events": [
{
"instance_name": "sample-worker-spark-004",
"successful": false,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": "Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"instance_id": "b5ba5ba8-e9c1-47f7-9355-3ce0ec0e449d",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "34afcfc7-bdb0-43cb-b142-283d560dc6ad"
},
{
"instance_name": "sample-worker-spark-001",
"successful": true,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": null,
"instance_id": "c532ab71-38da-475a-95f8-f8eb93b8f1c2",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "4ba50414-5216-4161-bc7a-12716122b99d"
}
],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Wait for instances to become active",
"total": 5,
"id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6"
},
{
"successful": true,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:12",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:19",
"events": [],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Run instances",
"total": 5,
"id": "407ba50a-c799-46af-9dfb-6aa5f6ade426"
}
],
"anti_affinity": [],
"node_groups": [],
"management_public_key": "Sahara",
"status_description": "Creating cluster failed for the following reason(s): Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"plugin_version": "1.0.0",
"id": "c26ec982-ba6b-4d75-1f8c-a50240164af0",
"trust_id": null,
"info": {},
"cluster_template_id": "5a9a09a3-9349-43bd-9058-16c401fad2d5",
"name": "sample",
"cluster_configs": {},
"created_at": "2015-03-31 14:10:07",
"default_image_id": "e6a6c5da-67be-4017-a7d2-81f466efe67e",
"project_id": "9cd1314a0a31493282b6712b76a8fcda"
}
Data sources¶
A data source object defines the location of input or output for MapReduce jobs and might reference different types of storage.
The Data Processing service does not validate data source locations.
Shows details for a data source.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
data_source_id |
path |
string |
The UUID of the data source. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the data source object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the data source. |
name |
body |
string |
The name of the data source. |
Response Example¶
{
"data_source": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10.691493",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"updated_at": null,
"name": "swift_input",
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
}
}
Deletes a data source.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
data_source_id |
path |
string |
The UUID of the data source. |
Updates a data source.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
data_source_id |
path |
string |
The UUID of the data source. |
Request Example¶
{
"description": "This is public input",
"is_protected": true
}
Lists all data sources.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting data sources.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
description |
body |
string |
The description of the data source object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the data source. |
name |
body |
string |
The name of the data source. |
{
"data_sources": [
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"name": "swift_input",
"updated_at": null,
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
},
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:09:36",
"id": "d7fffe9c-3b42-46a9-8be8-e98f586fa7a9",
"name": "hdfs_input",
"updated_at": null,
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs"
}
]
}
Creates a data source.
Normal response codes:202
Request Example¶
{
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs",
"name": "hdfs_input"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the data source object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the data source. |
name |
body |
string |
The name of the data source. |
Event log¶
The event log feature provides information about cluster provisioning. In the event of errors, the event log shows the reason for the failure.
Shows provisioning progress of cluster.
Normal response codes: 200 Error response codes:
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
cluster_id |
body |
string |
The UUID of the cluster. |
Response Example¶
{
"status": "Error",
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076",
"is_transient": false,
"description": "",
"user_keypair_id": "vgridnev",
"updated_at": "2015-03-31 14:10:59",
"plugin_name": "spark",
"provision_progress": [
{
"successful": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:20",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:35",
"events": [
{
"instance_name": "sample-worker-spark-004",
"successful": false,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": "Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"instance_id": "b5ba5ba8-e9c1-47f7-9355-3ce0ec0e449d",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "34afcfc7-bdb0-43cb-b142-283d560dc6ad"
},
{
"instance_name": "sample-worker-spark-001",
"successful": true,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": null,
"instance_id": "c532ab71-38da-475a-95f8-f8eb93b8f1c2",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "4ba50414-5216-4161-bc7a-12716122b99d"
}
],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Wait for instances to become active",
"total": 5,
"id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6"
},
{
"successful": true,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:12",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:19",
"events": [],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Run instances",
"total": 5,
"id": "407ba50a-c799-46af-9dfb-6aa5f6ade426"
}
],
"anti_affinity": [],
"node_groups": [],
"management_public_key": "Sahara",
"status_description": "Creating cluster failed for the following reason(s): Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"plugin_version": "1.0.0",
"id": "c26ec982-ba6b-4d75-1f8c-a50240164af0",
"trust_id": null,
"info": {},
"cluster_template_id": "5a9a09a3-9349-43bd-9058-16c401fad2d5",
"name": "sample",
"cluster_configs": {},
"created_at": "2015-03-31 14:10:07",
"default_image_id": "e6a6c5da-67be-4017-a7d2-81f466efe67e",
"project_id": "9cd1314a0a31493282b6712b76a8fcda"
}
Image registry¶
Use the image registry tool to manage images, add tags to and remove tags from images, and define the user name for an instance operating system. Each plugin lists required tags for an image. To run remote operations, the Data Processing service requires a user name with which to log in to the operating system for an instance.
Adds tags to an image.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
tags |
body |
array |
List of tags to add. |
image_id |
path |
string |
The UUID of the image. |
Request Example¶
{
"tags": [
"vanilla",
"2.7.1",
"some_other_tag"
]
}
Shows details for an image.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
image_id |
path |
string |
The UUID of the image. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
status |
body |
string |
The status of the cluster. |
username |
body |
string |
The name of the user for the image. |
updated |
body |
string |
The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
description |
body |
string |
The description of the image. |
created |
body |
string |
The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
image |
body |
object |
A set of key and value pairs that contain image properties. |
tags |
body |
array |
List of tags to add. |
minDisk |
body |
integer |
The minimum disk space, in GB. |
name |
body |
string |
The name of the operating system image. |
progress |
body |
integer |
A progress indicator, as a percentage value, for the amount of image content that has been processed. |
minRam |
body |
integer |
The minimum amount of random access memory (RAM) for the image, in GB. |
id |
body |
string |
The UUID of the image. |
metadata |
body |
object |
A set of key and value pairs that contain image metadata. |
Response Example¶
{
"image": {
"updated": "2015-02-03T10:29:32Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true,
"_sahara_tag_2.6.0": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"2.6.0"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.6.0-ubuntu-14.04",
"description": null,
"OS-EXT-IMG-SIZE:size": 1101856768
}
}
Registers an image in the registry.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
username |
body |
string |
The name of the user for the image. |
description |
body |
string |
The description of the image. |
image_id |
path |
string |
The UUID of the image. |
Request Example¶
{
"username": "ubuntu",
"description": "Ubuntu image for Hadoop 2.7.1"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
status |
body |
string |
The status of the cluster. |
username |
body |
string |
The name of the user for the image. |
updated |
body |
string |
The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
description |
body |
string |
The description of the image. |
created |
body |
string |
The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
image |
body |
object |
A set of key and value pairs that contain image properties. |
tags |
body |
array |
List of tags to add. |
minDisk |
body |
integer |
The minimum disk space, in GB. |
name |
body |
string |
The name of the operating system image. |
progress |
body |
integer |
A progress indicator, as a percentage value, for the amount of image content that has been processed. |
minRam |
body |
integer |
The minimum amount of random access memory (RAM) for the image, in GB. |
id |
body |
string |
The UUID of the image. |
metadata |
body |
object |
A set of key and value pairs that contain image metadata. |
Removes an image from the registry.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
image_id |
path |
string |
The UUID of the image. |
Removes tags from an image.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
tags |
body |
array |
List of tags to add. |
image_id |
path |
string |
The UUID of the image. |
Request Example¶
{
"tags": [
"some_other_tag"
]
}
Lists all images registered in the registry.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
tags |
body |
array |
List of tags to add. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
status |
body |
string |
The status of the cluster. |
username |
body |
string |
The name of the user for the image. |
updated |
body |
string |
The date and time when the image was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
description |
body |
string |
The description of the image. |
created |
body |
string |
The date and time when the image was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
For example, The |
image |
body |
object |
A set of key and value pairs that contain image properties. |
tags |
body |
array |
List of tags to add. |
minDisk |
body |
integer |
The minimum disk space, in GB. |
name |
body |
string |
The name of the operating system image. |
images |
body |
array |
The list of images and their properties. |
progress |
body |
integer |
A progress indicator, as a percentage value, for the amount of image content that has been processed. |
minRam |
body |
integer |
The minimum amount of random access memory (RAM) for the image, in GB. |
id |
body |
string |
The UUID of the image. |
metadata |
body |
object |
A set of key and value pairs that contain image metadata. |
Response Example¶
{
"images": [
{
"name": "ubuntu-vanilla-2.7.1",
"id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"created": "2015-08-06T08:17:14Z",
"metadata": {
"_sahara_tag_2.7.1": true,
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 998716928,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"vanilla",
"2.7.1"
],
"updated": "2015-09-04T09:35:09Z",
"minRam": 0,
"description": null
},
{
"name": "cdh-latest",
"id": "ff74035b-9da7-4edf-981d-57f270ed337d",
"created": "2015-09-04T11:56:44Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_5.4.0": true,
"_sahara_tag_cdh": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 3281453056,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"5.4.0",
"cdh"
],
"updated": "2015-09-04T12:46:42Z",
"minRam": 0,
"description": null
}
]
}
Job binaries¶
Job binary objects represent data processing applications and libraries that are stored in Object Storage service(S3 or Swift) or in Manila Shares.
Lists the available job binaries.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting job binaries.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
description |
body |
string |
The description of the job binary object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
binaries |
body |
array |
The list of job binary internal objects. |
id |
path |
string |
The UUID of the job binary. |
name |
body |
string |
The name of the object. |
{
"binaries": [
{
"is_public": false,
"description": "",
"url": "internal-db://d2498cbf-4589-484a-a814-81436c18beb3",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:36:59.375060",
"updated_at": null,
"id": "84248975-3c82-4206-a58d-6e7fb3a563fd",
"name": "example.pig",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "internal-db://22f1d87a-23c8-483e-a0dd-cb4a16dde5f9",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:43:52.265899",
"updated_at": null,
"id": "508fc62d-1d58-4412-b603-bdab307bb926",
"name": "udf.jar",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "swift://container/jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
]
}
Creates a job binary.
Normal response codes:202
Request Example¶
{
"url": "swift://container/jar-example.jar",
"name": "jar-example.jar",
"description": "This is a job binary",
"extra": {
"password": "swordfish",
"user": "admin"
}
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the job binary object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
id |
path |
string |
The UUID of the job binary. |
name |
body |
string |
The name of the object. |
Shows details for a job binary.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_binary_id |
path |
string |
The UUID of the job binary. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the job binary object. |
url |
body |
string |
The url of the data source object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
is_protected |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
id |
path |
string |
The UUID of the job binary. |
name |
body |
string |
The name of the object. |
Response Example¶
{
"job_binary": {
"is_public": false,
"description": "an example jar file",
"url": "swift://container/jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
}
Deletes a job binary.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_binary_id |
path |
string |
The UUID of the job binary. |
Updates a job binary.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_binary_id |
path |
string |
The UUID of the job binary. |
Request Example¶
{
"url": "swift://container/new-jar-example.jar",
"name": "new-jar-example.jar",
"description": "This is a new job binary"
}
Shows data for a job binary.
The response body shows the job binary raw data and the response headers show the data length.
Example response:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 161
Content-Type: text/html; charset=utf-8
Date: Sat, 28 Mar 2016 02:42:48 GMT
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_binary_id |
path |
string |
The UUID of the job binary. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
Content-Length |
header |
string |
The length of the data, in bytes. |
Response Example¶
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Job templates¶
A job templates object lists the binaries that a job needs to run. To run a job, you must specify data sources and job parameters.
You can run a job on an existing or new transient cluster.
Lists all job templates.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting jobs.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_templates |
body |
array |
The list of the job templates. |
description |
body |
string |
The description of the job object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
mains |
body |
array |
The list of the job object and their properties. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
libs |
body |
array |
The list of the job object properties. |
is_protected |
body |
boolean |
If set to |
interface |
body |
array |
The interfaces of the job object. |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the job template object. |
name |
body |
string |
The name of the job template object. |
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
Response Example¶
..rest_method:: GET /v2/job-templates?limit=2
{
"job_templates": [
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job-3d60854e",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job-339c2d1a.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job-339c2d1a.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
},
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "4d1f3759-3497-4927-8352-910bacf24e62",
"name": "Edp-test-job-6b6953c8",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e0d47800-4ac1-4d63-a2e1-c92d669a44e2",
"name": "binary-job-6f21a2f8.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-6f21a2f8.jar"
}
],
"type": "Pig",
"mains": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e073e896-f123-4b76-995f-901d786262df",
"name": "binary-job-d4f8bd75.pig",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-d4f8bd75.pig"
}
],
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "c53832da-6e7b-449e-a166-9f9ce1718d03"
}
}
Creates a job object.
Normal response codes:202
Request Example¶
{
"description": "This is pig job example",
"mains": [
"90d9d5ec-11aa-48bd-bc8c-34936ce0db6e"
],
"libs": [
"320a2ca7-25fd-4b48-9bc3-4fb1b6c4ff27"
],
"type": "Pig",
"name": "pig-job-example"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the job object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
mains |
body |
array |
The list of the job object and their properties. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
libs |
body |
array |
The list of the job object properties. |
is_protected |
body |
boolean |
If set to |
interface |
body |
array |
The interfaces of the job object. |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the job template object. |
name |
body |
string |
The name of the job template object. |
Shows details for a job template.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_template_id |
path |
string |
The UUID of the template job. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the job object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
mains |
body |
array |
The list of the job object and their properties. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
libs |
body |
array |
The list of the job object properties. |
is_protected |
body |
boolean |
If set to |
interface |
body |
array |
The interfaces of the job object. |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the job template object. |
name |
body |
string |
The name of the job template object. |
Response Example¶
{
"job_template": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
}
}
Removes a job.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_template_id |
path |
string |
The UUID of the template job. |
Updates a job template object.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_template_id |
path |
string |
The UUID of the template job. |
Request Example¶
{
"description": "This is public pig job example",
"name": "public-pig-job-example"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
description |
body |
string |
The description of the job object. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
mains |
body |
array |
The list of the job object and their properties. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
libs |
body |
array |
The list of the job object properties. |
is_protected |
body |
boolean |
If set to |
interface |
body |
array |
The interfaces of the job object. |
is_public |
body |
boolean |
If set to |
type |
body |
string |
The type of the data source object. |
id |
body |
string |
The UUID of the job template object. |
name |
body |
string |
The name of the job template object. |
Get job template config hints
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_type |
path |
string |
The job type. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_config |
body |
string |
The job configuration. |
args |
body |
array |
The list of arguments. |
configs |
body |
object |
The mappings of the job tasks. |
Job types¶
Each plugin that supports EDP also supports specific job types. Different versions of a plugin might actually support different job types. Configuration options vary by plugin, version, and job type.
The job types provide information about which plugins support which job types and how to configure the job types.
Lists all job types.
You can use query parameters to filter the response.
Normal response codes: 200 Error response codes:
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
plugin (Optional) |
path |
string |
Filters the response by a plugin name. |
version |
path |
string |
Filters the response by a plugin version. |
type |
body |
string |
The type of the data source object. |
hints (Optional) |
path |
boolean |
Includes configuration hints in the response. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
versions |
body |
array |
The list of plugin versions. |
title |
body |
string |
The title of the plugin. |
description |
body |
string |
The full description of the plugin. |
job_types |
body |
array |
The list of plugins and their job types. |
name |
body |
string |
The name of the plugin. |
Response Example¶
{
"job_types": [
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Hive"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Java"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce.Streaming"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Pig"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Shell"
},
{
"plugins": [
{
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": {
"1.0.0": {}
},
"title": "Apache Spark",
"name": "spark"
}
],
"name": "Spark"
}
]
}
Jobs¶
A job object represents a job that runs on a cluster. A job polls the status of a running job and reports it to the user.
Executes a job.
Normal response codes: 200
{
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"job_template_id": "548ea8d4-a5sd-33a4-bt22-asf4n87a8e2dh",
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"job_configs": {
"configs": {
"mapred.map.tasks": "1",
"mapred.reduce.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
}
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
info |
body |
object |
A set of key and value pairs that contain cluster information. |
output_id |
body |
string |
The UUID of the output of job execution object. |
start_time |
body |
string |
The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
job_template_id |
body |
string |
The UUID of the job template object. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
args |
body |
array |
The list of arguments. |
data_source_urls |
body |
object |
The data source URLs. |
return_code |
body |
string |
The code returned after job has executed. |
oozie_job_id |
body |
string |
The UUID of the |
is_protected |
body |
boolean |
If set to |
cluster_id |
body |
string |
The UUID of the cluster. |
end_time |
body |
string |
The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
params |
body |
object |
The mappings of values to the parameters. |
is_public |
body |
boolean |
If set to |
input_id |
body |
string |
The UUID of the input. |
configs |
body |
object |
The mappings of the job tasks. |
job |
body |
object |
A set of key and value pairs that contain the job object. |
id |
body |
string |
The UUID of the job object. |
Response Example¶
{
"job": {
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"is_protected": false,
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"created_at": "2015-09-15T09:49:24",
"is_public": false,
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"project_id": "808d5032ea0446889097723bfc8e919d",
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"info": {
"status": "PENDING"
}
}
}
Lists available jobs.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting job executions.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
info |
body |
object |
A set of key and value pairs that contain cluster information. |
output_id |
body |
string |
The UUID of the output of job execution object. |
start_time |
body |
string |
The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
job_template_id |
body |
string |
The UUID of the job template object. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
args |
body |
array |
The list of arguments. |
data_source_urls |
body |
object |
The data source URLs. |
return_code |
body |
string |
The code returned after job has executed. |
oozie_job_id |
body |
string |
The UUID of the |
is_protected |
body |
boolean |
If set to |
cluster_id |
body |
string |
The UUID of the cluster. |
end_time |
body |
string |
The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
params |
body |
object |
The mappings of values to the parameters. |
is_public |
body |
boolean |
If set to |
input_id |
body |
string |
The UUID of the input. |
configs |
body |
object |
The mappings of the job tasks. |
job |
body |
object |
A set of key and value pairs that contain the job object. |
id |
body |
string |
The UUID of the job object. |
jobs |
body |
array |
The list of job objects. |
{
"jobs": [
{
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
]
}
Shows details for a job, by ID.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_id |
path |
string |
The UUID of the job. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
info |
body |
object |
A set of key and value pairs that contain cluster information. |
output_id |
body |
string |
The UUID of the output of job execution object. |
start_time |
body |
string |
The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
job_template_id |
body |
string |
The UUID of the job template object. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
args |
body |
array |
The list of arguments. |
data_source_urls |
body |
object |
The data source URLs. |
return_code |
body |
string |
The code returned after job has executed. |
oozie_job_id |
body |
string |
The UUID of the |
is_protected |
body |
boolean |
If set to |
cluster_id |
body |
string |
The UUID of the cluster. |
end_time |
body |
string |
The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
params |
body |
object |
The mappings of values to the parameters. |
is_public |
body |
boolean |
If set to |
input_id |
body |
string |
The UUID of the input. |
configs |
body |
object |
The mappings of the job tasks. |
job |
body |
object |
A set of key and value pairs that contain the job object. |
id |
body |
string |
The UUID of the job object. |
Response Example¶
{
"job": {
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"is_protected": false,
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"created_at": "2015-09-15T09:49:24",
"is_public": false,
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"project_id": "808d5032ea0446889097723bfc8e919d",
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"info": {
"status": "PENDING"
}
}
}
Deletes a job.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_id |
path |
string |
The UUID of the job. |
Updates a job.
Normal response codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
job_id |
path |
string |
The UUID of the job. |
Request Example¶
{
"is_public": true
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
info |
body |
object |
A set of key and value pairs that contain cluster information. |
output_id |
body |
string |
The UUID of the output of job execution object. |
start_time |
body |
string |
The date and time when the job started. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
job_template_id |
body |
string |
The UUID of the job template object. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
args |
body |
array |
The list of arguments. |
data_source_urls |
body |
object |
The data source URLs. |
return_code |
body |
string |
The code returned after job has executed. |
oozie_job_id |
body |
string |
The UUID of the |
is_protected |
body |
boolean |
If set to |
cluster_id |
body |
string |
The UUID of the cluster. |
end_time |
body |
string |
The end date and time of the job execution. The date and time when the job completed execution. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
params |
body |
object |
The mappings of values to the parameters. |
is_public |
body |
boolean |
If set to |
input_id |
body |
string |
The UUID of the input. |
configs |
body |
object |
The mappings of the job tasks. |
job |
body |
object |
A set of key and value pairs that contain the job object. |
id |
body |
string |
The UUID of the job object. |
Node group templates¶
A cluster is a group of nodes with the same configuration. A node group template configures a node in the cluster.
A template configures Hadoop processes and VM characteristics, such as the number of reduced slots for task tracker, the number of CPUs, and the amount of RAM. The template specifies the VM characteristics through an OpenStack flavor.
Lists available node group templates.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
limit (Optional) |
path |
integer |
Maximum number of objects in response data. |
marker (Optional) |
path |
string |
ID of the last element on the list which won’t be in response. |
sort_by (Optional) |
path |
string |
The field for sorting node group templates.
this parameter accepts the following values:
|
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
markers (Optional) |
body |
object |
The markers of previous and following pages of data.
This field exists only if |
prev (Optional) |
body |
string |
The marker of previous page. May be |
next (Optional) |
body |
string |
The marker of next page of list data. |
volume_local_to_instance |
body |
boolean |
If set to |
availability_zone |
body |
string |
The availability of the node in the cluster. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
use_autoconfig |
body |
boolean |
If set to |
volumes_per_node |
body |
integer |
The number of volumes for the node. |
id |
body |
string |
The UUID of the node group template. |
security_groups |
body |
string |
The security groups of the node. |
shares |
body |
string |
The sharing of resources in the cluster. |
node_configs |
body |
object |
A set of key and value pairs that contain the node configuration in the cluster. |
auto_security_group |
body |
boolean |
If set to |
volumes_availability_zone |
body |
string |
The availability zone of the volumes. |
description (Optional) |
body |
string |
Description of the node group template |
volume_mount_prefix |
body |
string |
The mount point of the node. |
plugin_name |
body |
string |
The name of the plugin. |
floating_ip_pool |
body |
string |
The UUID of the pool in the template. |
is_default |
body |
boolean |
If set to |
image_id |
body |
string |
The UUID of the image. |
volumes_size |
body |
integer |
The size of the volumes in a node. |
is_proxy_gateway |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
name |
body |
string |
The name of the node group template. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
volume_type |
body |
string |
The type of volume in a node. |
is_protected |
body |
boolean |
If set to |
node_processes |
body |
array |
The list of the processes performed by the node. |
flavor_id |
body |
string |
The ID of the flavor. |
{
"node_group_templates": [
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
},
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"description": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:27:00",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
],
"markers": {
"prev":"39dfc852-8588-4b61-8d2b-eb08a67ab240",
"next":"eaa0bd97-ab54-43df-83ab-77a9774d7358"
}
}
Creates a node group template.
Normal response codes: 202
Request Example¶
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"name": "master",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
volume_local_to_instance |
body |
boolean |
If set to |
availability_zone |
body |
string |
The availability of the node in the cluster. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
use_autoconfig |
body |
boolean |
If set to |
volumes_per_node |
body |
integer |
The number of volumes for the node. |
id |
body |
string |
The UUID of the node group template. |
security_groups |
body |
string |
The security groups of the node. |
shares |
body |
string |
The sharing of resources in the cluster. |
node_configs |
body |
object |
A set of key and value pairs that contain the node configuration in the cluster. |
auto_security_group |
body |
boolean |
If set to |
volumes_availability_zone |
body |
string |
The availability zone of the volumes. |
description (Optional) |
body |
string |
Description of the node group template |
volume_mount_prefix |
body |
string |
The mount point of the node. |
plugin_name |
body |
string |
The name of the plugin. |
floating_ip_pool |
body |
string |
The UUID of the pool in the template. |
is_default |
body |
boolean |
If set to |
image_id |
body |
string |
The UUID of the image. |
volumes_size |
body |
integer |
The size of the volumes in a node. |
is_proxy_gateway |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
name |
body |
string |
The name of the node group template. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
volume_type |
body |
string |
The type of volume in a node. |
is_protected |
body |
boolean |
If set to |
node_processes |
body |
array |
The list of the processes performed by the node. |
flavor_id |
body |
string |
The ID of the flavor. |
Shows a node group template, by ID.
Normal response codes: 200
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
node_group_template_id |
path |
string |
The UUID of the node group template. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
volume_local_to_instance |
body |
boolean |
If set to |
availability_zone |
body |
string |
The availability of the node in the cluster. |
updated_at |
body |
string |
The date and time when the cluster was updated. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
use_autoconfig |
body |
boolean |
If set to |
volumes_per_node |
body |
integer |
The number of volumes for the node. |
id |
body |
string |
The UUID of the node group template. |
security_groups |
body |
string |
The security groups of the node. |
shares |
body |
string |
The sharing of resources in the cluster. |
node_configs |
body |
object |
A set of key and value pairs that contain the node configuration in the cluster. |
auto_security_group |
body |
boolean |
If set to |
volumes_availability_zone |
body |
string |
The availability zone of the volumes. |
description (Optional) |
body |
string |
Description of the node group template |
volume_mount_prefix |
body |
string |
The mount point of the node. |
plugin_name |
body |
string |
The name of the plugin. |
floating_ip_pool |
body |
string |
The UUID of the pool in the template. |
is_default |
body |
boolean |
If set to |
image_id |
body |
string |
The UUID of the image. |
volumes_size |
body |
integer |
The size of the volumes in a node. |
is_proxy_gateway |
body |
boolean |
If set to |
is_public |
body |
boolean |
If set to |
plugin_version |
body |
string |
The version of the Plugin used in the cluster. |
name |
body |
string |
The name of the node group template. |
project_id |
body |
string |
The UUID of the project. |
created_at |
body |
string |
The date and time when the cluster was created. The date and time stamp format is ISO 8601: CCYY-MM-DDThh:mm:ss±hh:mm
The For example, |
volume_type |
body |
string |
The type of volume in a node. |
is_protected |
body |
boolean |
If set to |
node_processes |
body |
array |
The list of the processes performed by the node. |
flavor_id |
body |
string |
The ID of the flavor. |
Response Example¶
{
"node_group_template": {
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
}
Deletes a node group template.
Normal response codes:204
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
node_group_template_id |
path |
string |
The UUID of the node group template. |
Updates a node group template.
Normal respose codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
node_group_template_id |
path |
string |
The UUID of the node group template. |
Request Example¶
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_processes": [
"datanode"
],
"name": "new",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}
Exports a node group template.
Normal respose codes:202
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
node_group_template_id |
path |
string |
The UUID of the node group template. |
Request Example¶
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_processes": [
"datanode"
],
"name": "new",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}
Plugins¶
A plugin object defines the Hadoop or Spark version that it can install and which configurations can be set for the cluster.
Shows details for a plugin.
Normal response codes: 200 Error response codes: 400, 500
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
plugin_name |
path |
string |
Name of the plugin. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
versions |
body |
array |
The list of plugin versions. |
title |
body |
string |
The title of the plugin. |
description |
body |
string |
The full description of the plugin. |
name |
body |
string |
The name of the plugin. |
Response Example¶
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component."
}
}
Lists all registered plugins.
Normal response codes: 200 Error response codes: 400, 500
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
title |
body |
string |
The title of the plugin. |
versions |
body |
array |
The list of plugin versions. |
plugins |
body |
array |
The list of plugins. |
description |
body |
string |
The full description of the plugin. |
name |
body |
string |
The name of the plugin. |
Response Example¶
{
"plugins": [
{
"name": "vanilla",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop"
},
{
"name": "hdp",
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": [
"1.3.2",
"2.0.6"
],
"title": "Hortonworks Data Platform"
},
{
"name": "spark",
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": [
"1.0.0",
"0.9.1"
],
"title": "Apache Spark"
},
{
"name": "cdh",
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": [
"5",
"5.3.0"
],
"title": "Cloudera Plugin"
}
]
}
Shows details for a plugin version.
Normal response codes: 200 Error response codes: 400, 500
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
plugin_name |
path |
string |
Name of the plugin. |
version |
path |
string |
Filters the response by a plugin version. |
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
versions |
body |
array |
The list of plugin versions. |
title |
body |
string |
The title of the plugin. |
description |
body |
string |
The full description of the plugin. |
name |
body |
string |
The name of the plugin. |
Response Example¶
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"required_image_tags": [
"vanilla",
"2.6.0"
],
"node_processes": {
"JobFlow": [
"oozie"
],
"HDFS": [
"namenode",
"datanode",
"secondarynamenode"
],
"YARN": [
"resourcemanager",
"nodemanager"
],
"MapReduce": [
"historyserver"
],
"Hadoop": [],
"Hive": [
"hiveserver"
]
},
"configs": [
{
"default_value": "/tmp/hadoop-${user.name}",
"name": "hadoop.tmp.dir",
"priority": 2,
"config_type": "string",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "A base for other temporary directories."
},
{
"default_value": true,
"name": "hadoop.native.lib",
"priority": 2,
"config_type": "bool",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "Should native hadoop libraries, if present, be used."
},
{
"default_value": 1024,
"name": "NodeManager Heap Size",
"config_values": null,
"priority": 1,
"config_type": "int",
"applicable_target": "YARN",
"is_optional": false,
"scope": "node",
"description": null
},
{
"default_value": true,
"name": "Enable Swift",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": false,
"scope": "cluster",
"description": null
},
{
"default_value": true,
"name": "Enable MySQL",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": true,
"scope": "cluster",
"description": null
}
],
"title": "Vanilla Apache Hadoop"
}
}
Updates details for a plugin.
Normal response codes: 202 Error response codes: 400, 500
Request¶
Name |
In |
Type |
Description |
|---|---|---|---|
plugin_name |
path |
string |
Name of the plugin. |
Request Example¶
{
"plugin_labels": {
"enabled": {
"status": false
}
}
}
Response Parameters¶
Name |
In |
Type |
Description |
|---|---|---|---|
title |
body |
string |
The title of the plugin. |
versions |
body |
array |
The list of plugin versions. |
description |
body |
string |
The full description of the plugin. |
name |
body |
string |
The name of the plugin. |
Response Example¶
{
"plugin": {
"plugin_labels": {
"hidden": {
"status": true,
"mutable": true,
"description": "Existence of plugin or its version is hidden, but still can be used for cluster creation by CLI and directly by client."
},
"enabled": {
"status": false,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
},
"description": "It's a fake plugin that aimed to work on the CirrOS images. It doesn't install Hadoop. It's needed to be able to test provisioning part of Sahara codebase itself.",
"versions": [
"0.1"
],
"tenant_id": "993f53c1f51845e48e013aeb632358d8",
"title": "Fake Plugin",
"version_labels": {
"0.1": {
"enabled": {
"status": true,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
}
},
"name": "fake"
}
}
