Installation Guide¶

Introduction¶

In the single pod installation guide, we discuss how to deploy the Tricircle in one single pod with DevStack. Besides the Tricircle API and the central Neutron server, only one pod(one pod means one OpenStack instance) is running. Network is created with the default network type: local. Local type network will be only presented in one pod. If a local type network is already hosting virtual machines in one pod, you can not use it to boot virtual machine in another pod. That is to say, local type network doesn’t support cross-Neutron l2 networking.

With multi-pod installation of the Tricircle, you can try out cross-Neutron l2 networking and cross-Neutron l3 networking features.

To support cross-Neutron l2 networking, we have added both VLAN and VxLAN network type to the Tricircle. When a VLAN type network created via the central Neutron server is used to boot virtual machines in different pods, local Neutron server in each pod will create a VLAN type network with the same VLAN ID and physical network as the central network, so each pod should be configured with the same VLAN allocation pool and physical network. Then virtual machines in different pods can communicate with each other in the same physical network with the same VLAN tag. Similarly, for VxLAN network type, each pod should be configured with the same VxLAN allocation pool, so local Neutron server in each pod can create a VxLAN type network with the same VxLAN ID as is allocated by the central Neutron server.

Cross-Neutron l3 networking is supported in two ways in the Tricircle. If two networks connected to the router are of local type, we utilize a shared VLAN or VxLAN network to achieve cross-Neutron l3 networking. When a subnet is attached to a router via the central Neutron server, the Tricircle not only creates corresponding subnet and router in the pod, but also creates a “bridge” network. Both tenant network and “bridge” network are attached to the router. Each tenant will have one allocated VLAN or VxLAN ID, which is shared by the tenant’s “bridge” networks across Neutron servers. The CIDRs of “bridge” networks for one tenant are also the same, so the router interfaces in “bridge” networks across different Neutron servers can communicate with each other. By adding an extra route as following:

destination: CIDR of tenant network in another pod
nexthop: "bridge" network interface ip in another pod

When a virtual machine sends a packet whose receiver is in another network and in another pod, the packet first goes to router, then is forwarded to the router in another pod according to the extra route, at last the packet is sent to the target virtual machine. This route configuration job is triggered when user attaches a subnet to a router via the central Neutron server and the job is finished asynchronously.

If one of the network connected to the router is not local type, meaning that cross-Neutron l2 networking is supported in this network(like VLAN type), and the l2 network can be stretched into current pod, packets sent to the virtual machine in this network will not pass through the “bridge” network. Instead, packets first go to router, then are directly forwarded to the target virtual machine via the l2 network. A l2 network’s presence scope is determined by the network’s availability zone hint. If the l2 network is not able to be stretched into the current pod, the packets will still pass through the “bridge network”. For example, let’s say we have two pods, pod1 and pod2, and two availability zones, az1 and az2. Pod1 belongs to az1 and pod2 belongs to az2. If the availability zone hint of one VLAN type network is set to az1, this network can not be stretched to pod2. So packets sent from pod2 to virtual machines in this network still need to pass through the “bridge network”.

Prerequisite¶

In this guide we take two nodes deployment as an example. One node to run the Tricircle API, the central Neutron server and one pod, the other one node to run another pod. For VLAN network, both nodes should have two network interfaces, which are connected to the management network and provider VLAN network. The physical network infrastructure should support VLAN tagging. For VxLAN network, you can combine the management plane and data plane, in this case, only one network interface is needed. If you would like to try north-south networking, too, you should prepare one more network interface in the second node for the external network. In this guide, the external network is also VLAN type, so the local.conf sample is based on VLAN type external network setup. For the resource requirements to setup each node, please refer to All-In-One Single Machine for installing DevStack in bare metal server and All-In-One Single VM for installing DevStack in virtual machine.

If you want to experience cross Neutron VxLAN network, please make sure compute nodes are routable to each other on data plane, and enable L2 population mechanism driver in OpenStack RegionOne and OpenStack RegionTwo.

Setup¶

In pod1 in node1 for Tricircle service, central Neutron and OpenStack RegionOne,

• 1 Install DevStack. Please refer to DevStack document on how to install DevStack into single VM or bare metal server.

• 2 In DevStack folder, create a file local.conf, and copy the content of local.conf node1 sample to local.conf, change password in the file if needed.

• 3 Change the following options according to your environment

  • change HOST_IP to your management interface ip:

    HOST_IP=10.250.201.24
    

  • the format of Q_ML2_PLUGIN_VLAN_TYPE_OPTIONS is (network_vlan_ranges=<physical network name>:<min vlan>:<max vlan>), you can change physical network name, but remember to adapt your change to the commands showed in this guide; also, change min VLAN and max vlan to adapt the VLAN range your physical network supports. You need to additionally specify the physical network “extern” to ensure the central neutron can create “extern” physical network which located in other pods:

    Q_ML2_PLUGIN_VLAN_TYPE_OPTIONS=(network_vlan_ranges=bridge:2001:3000,extern:3001:4000)
    

  • if you would like to also configure vxlan network, you can set Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS. the format of it is (vni_ranges=<min vxlan>:<max vxlan>):

    Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS=(vni_ranges=1001:2000)
    

  • the format of OVS_BRIDGE_MAPPINGS is <physical network name>:<ovs bridge name>, you can change these names, but remember to adapt your change to the commands showed in this guide. You do not need specify the bridge mapping for “extern”, because this physical network is located in other pods:

    OVS_BRIDGE_MAPPINGS=bridge:br-vlan
    

    this option can be omitted if only VxLAN networks are needed

  • if you would like to also configure flat network, you can set Q_ML2_PLUGIN_FLAT_TYPE_OPTIONS, the format of it is (flat_networks=phy_net1,phy_net2,…). Besides specifying a list of physical network names, you can also use ‘*’ to allow flat networks with arbitrary physical network names; or use an empty list to disable flat networks. For simplicity, we use the same physical networks and bridge mappings for vlan and flat network configuration. Similar to vlan network, You need to additionally specify the physical network “extern” to ensure the central neutron can create “extern” physical network which located in other pods:

    Q_ML2_PLUGIN_FLAT_TYPE_OPTIONS=(flat_networks=bridge,extern)
    

  • set TRICIRCLE_START_SERVICES to True to install the Tricircle service and central Neutron in node1:

    TRICIRCLE_START_SERVICES=True
    

• 4 Create OVS bridge and attach the VLAN network interface to it

  sudo ovs-vsctl add-br br-vlan
  sudo ovs-vsctl add-port br-vlan eth1
  

  br-vlan is the OVS bridge name you configure on OVS_PHYSICAL_BRIDGE, eth1 is the device name of your VLAN network interface, this step can be omitted if only VxLAN networks are provided to tenants.

• 5 Run DevStack. In DevStack folder, run

  ./stack.sh
  

• 6 After DevStack successfully starts, begin to setup node2.

In pod2 in node2 for OpenStack RegionTwo,

• 1 Install DevStack. Please refer to DevStack document on how to install DevStack into single VM or bare metal server.

• 2 In DevStack folder, create a file local.conf, and copy the content of local.conf node2 sample to local.conf, change password in the file if needed.

• 3 Change the following options according to your environment

  • change HOST_IP to your management interface ip:

    HOST_IP=10.250.201.25
    

  • change KEYSTONE_SERVICE_HOST to management interface ip of node1:

    KEYSTONE_SERVICE_HOST=10.250.201.24
    

  • change KEYSTONE_AUTH_HOST to management interface ip of node1:

    KEYSTONE_AUTH_HOST=10.250.201.24
    

  • the format of Q_ML2_PLUGIN_VLAN_TYPE_OPTIONS is (network_vlan_ranges=<physical network name>:<min vlan>:<max vlan>), you can change physical network name, but remember to adapt your change to the commands showed in this guide; also, change min vlan and max vlan to adapt the vlan range your physical network supports:

    Q_ML2_PLUGIN_VLAN_TYPE_OPTIONS=(network_vlan_ranges=bridge:2001:3000,extern:3001:4000)
    

  • if you would like to also configure vxlan network, you can set Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS. the format of it is (vni_ranges=<min vxlan>:<max vxlan>):

    Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS=(vni_ranges=1001:2000)
    

  • the format of OVS_BRIDGE_MAPPINGS is <physical network name>:<ovs bridge name>, you can change these names, but remember to adapt your change to the commands showed in this guide:

    OVS_BRIDGE_MAPPINGS=bridge:br-vlan,extern:br-ext
    

    if you only use vlan network for external network, it can be configured like:

    OVS_BRIDGE_MAPPINGS=extern:br-ext
    

  • if you would like to also configure flat network, you can set Q_ML2_PLUGIN_FLAT_TYPE_OPTIONS, the format of it is (flat_networks=phy_net1,phy_net2,…). Besides specifying a list of physical network names, you can also use ‘*’ to allow flat networks with arbitrary physical network names; or use an empty list to disable flat networks. For simplicity, we use the same physical networks and bridge mappings for vlan and flat network configuration:

    Q_ML2_PLUGIN_FLAT_TYPE_OPTIONS=(flat_networks=bridge,extern)
    

  • set TRICIRCLE_START_SERVICES to False(it’s True by default) so Tricircle services and central Neutron will not be started in node2:

    TRICIRCLE_START_SERVICES=False
    

  In this guide, we define two physical networks in node2, one is “bridge” for bridge network, the other one is “extern” for external network. If you do not want to try l3 north-south networking, you can simply remove the “extern” part. The external network type we use in the guide is VLAN, if you want to use other network type like flat, please refer to DevStack document.

• 4 Create OVS bridge and attach the VLAN network interface to it

  sudo ovs-vsctl add-br br-vlan
  sudo ovs-vsctl add-port br-vlan eth1
  sudo ovs-vsctl add-br br-ext
  sudo ovs-vsctl add-port br-ext eth2
  

  br-vlan and br-ext are the OVS bridge names you configure on OVS_PHYSICAL_BRIDGE, eth1 and eth2 are the device names of your VLAN network interfaces, for the “bridge” network and the external network. Omit br-vlan if you only use vxlan network as tenant network.

• 5 Run DevStack. In DevStack folder, run

  ./stack.sh
  

• 6 After DevStack successfully starts, the setup is finished.

<Directory /usr/local/bin>
    Require all granted
</Directory>

How to play¶

• 1 After DevStack successfully starts, we need to create environment variables for the user (admin user as example in this guide). In DevStack folder

  source openrc admin demo
  

• 2 Unset the region name environment variable, so that the command can be issued to specified region in following commands as needed

  unset OS_REGION_NAME
  

• 3 Check if services have been correctly registered. Run

  openstack --os-region-name=RegionOne endpoint list
  

  you should get output looks like as following

  +----------------------------------+---------------+--------------+----------------+
  | ID                               | Region        | Service Name | Service Type   |
  +----------------------------------+---------------+--------------+----------------+
  | 4adaab1426d94959be46314b4bd277c2 | RegionOne     | glance       | image          |
  | 5314a11d168042ed85a1f32d40030b31 | RegionTwo     | nova_legacy  | compute_legacy |
  | ea43c53a8ab7493dacc4db079525c9b1 | RegionOne     | keystone     | identity       |
  | a1f263473edf4749853150178be1328d | RegionOne     | neutron      | network        |
  | ebea16ec07d94ed2b5356fb0a2a3223d | RegionTwo     | neutron      | network        |
  | 8d374672c09845f297755117ec868e11 | CentralRegion | tricircle    | Tricircle      |
  | e62e543bb9cf45f593641b2d00d72700 | RegionOne     | nova_legacy  | compute_legacy |
  | 540bdedfc449403b9befef3c2bfe3510 | RegionOne     | nova         | compute        |
  | d533429712954b29b9f37debb4f07605 | RegionTwo     | glance       | image          |
  | c8bdae9506cd443995ee3c89e811fb45 | CentralRegion | neutron      | network        |
  | 991d304dfcc14ccf8de4f00271fbfa22 | RegionTwo     | nova         | compute        |
  +----------------------------------+---------------+--------------+----------------+
  

  “CentralRegion” is the region you set in local.conf via CENTRAL_REGION_NAME, whose default value is “CentralRegion”, we use it as the region for the Tricircle API and central Neutron server. “RegionOne” and “RegionTwo” are the normal OpenStack regions which includes Nova, Neutron and Glance. Shared Keystone service is registered in “RegionOne”.

• 4 Create pod instances for the Tricircle to manage the mapping between availability zones and OpenStack instances

  openstack multiregion networking pod create --region-name CentralRegion
  
  openstack multiregion networking pod create --region-name RegionOne --availability-zone az1
  
  openstack multiregion networking pod create --region-name RegionTwo --availability-zone az2
  

  Pay attention to “region_name” parameter we specify when creating pod. Pod name should exactly match the region name registered in Keystone. In the above commands, we create pods named “CentralRegion”, “RegionOne” and “RegionTwo”.

• 5 Create necessary resources in central Neutron server

  neutron --os-region-name=CentralRegion net-create --availability-zone-hint RegionOne net1
  neutron --os-region-name=CentralRegion subnet-create net1 10.0.1.0/24
  neutron --os-region-name=CentralRegion net-create --availability-zone-hint RegionTwo net2
  neutron --os-region-name=CentralRegion subnet-create net2 10.0.2.0/24
  

  Please note that the net1 and net2 ID will be used in later step to boot VM.

• 6 Get image ID and flavor ID which will be used in VM booting

  glance --os-region-name=RegionOne image-list
  nova --os-region-name=RegionOne flavor-list
  glance --os-region-name=RegionTwo image-list
  nova --os-region-name=RegionTwo flavor-list
  

• 7 Boot virtual machines

  nova --os-region-name=RegionOne boot --flavor 1 --image $image1_id --nic net-id=$net1_id vm1
  nova --os-region-name=RegionTwo boot --flavor 1 --image $image2_id --nic net-id=$net2_id vm2
  

• 8 Verify the VMs are connected to the networks

  neutron --os-region-name=CentralRegion port-list
  neutron --os-region-name=RegionOne port-list
  nova --os-region-name=RegionOne list
  neutron --os-region-name=RegionTwo port-list
  nova --os-region-name=RegionTwo list
  

  The ip address of each VM could be found in local Neutron server and central Neutron server. The port has same uuid in local Neutron server and central Neutron Server.

• 9 Create external network and subnet

  neutron --os-region-name=CentralRegion net-create --router:external --provider:network_type vlan --provider:physical_network extern --availability-zone-hint RegionTwo ext-net
  neutron --os-region-name=CentralRegion subnet-create --name ext-subnet --disable-dhcp ext-net 163.3.124.0/24
  

  Pay attention that when creating external network, we need to pass “availability_zone_hints” parameter, which is the name of the pod that will host external network.

  Currently external network needs to be created before attaching subnet to the router, because plugin needs to utilize external network information to setup bridge network when handling interface adding operation. This limitation will be removed later.

• 10 Create router and attach subnets in central Neutron server

  neutron --os-region-name=CentralRegion router-create router
  neutron --os-region-name=CentralRegion router-interface-add router $subnet1_id
  neutron --os-region-name=CentralRegion router-interface-add router $subnet2_id
  

• 11 Set router external gateway in central Neutron server

  neutron --os-region-name=CentralRegion router-gateway-set router ext-net
  

  Now virtual machine in the subnet attached to the router should be able to ping machines in the external network. In our test, we use hypervisor tool to directly start a virtual machine in the external network to check the network connectivity.

• 12 Launch VNC console and test connection

  nova --os-region-name=RegionOne get-vnc-console vm1 novnc
  nova --os-region-name=RegionTwo get-vnc-console vm2 novnc
  

  You should be able to ping vm1 from vm2 and vice versa.

• 13 Create floating ip in central Neutron server

  neutron --os-region-name=CentralRegion floatingip-create ext-net
  

• 14 Associate floating ip

  neutron --os-region-name=CentralRegion floatingip-list
  neutron --os-region-name=CentralRegion port-list
  neutron --os-region-name=CentralRegion floatingip-associate $floatingip_id $port_id
  

  Now you should be able to access virtual machine with floating ip bound from the external network.

Installation with Central Neutron Server¶

• 1 Install the Tricircle package:

  git clone https://github.com/openstack/tricircle.git
  cd tricircle
  pip install -e .
  

• 2 Register the Tricircle administrator API to Keystone:

  openstack user create tricircle --password password
  openstack role add --project service --user tricircle service
  openstack service create tricircle --name tricircle --description "Cross Neutron Networking Automation Service"
  service_id=$(openstack service show tricircle -f value -c id)
  service_host=162.3.124.201
  service_port=19999
  service_region=CentralRegion
  service_url=http://$service_host:$service_port/v1.0
  openstack endpoint create $service_id --publicurl $service_url --adminurl $service_url --internalurl $service_url --region $service_region
  

  change password, service_host, service_port and service_region in the above commands to adapt your deployment. OpenStack CLI tool will automatically find the endpoints to send to registration requests. If you would like to specify the region for endpoints, use:

  openstack --os-region-name <region_name> <command>
  

• 3 Generate the Tricircle configuration sample:

  cd tricircle
  oslo-config-generator --config-file=etc/api-cfg-gen.conf
  oslo-config-generator --config-file=etc/xjob-cfg-gen.conf
  

  The generated sample files are located in tricircle/etc

• 4 Configure the Tricircle administrator API:

  cd tricircle/etc
  cp api.conf.sample api.conf
  

  Edit etc/api.conf, for detail configuration information, please refer to the configuration guide. Below only options necessary to be changed are listed.

• 5 Create the Tricircle database(take mysql as an example):

  mysql -uroot -p -e "create database tricircle character set utf8;"
  cd tricircle
  tricircle-db-manage --config-file etc/api.conf db_sync
  

• 6 Start the Tricircle administrator API:

  sudo mkdir /var/cache/tricircle
  sudo chown $(whoami) /var/cache/tricircle/
  cd tricircle
  tricircle-api --config-file etc/api.conf
  

• 7 Configure the Tricircle Xjob daemon:

  cd tricircle/etc
  cp xjob.conf.sample xjob.conf
  

  Edit etc/xjob.conf, for detail configuration information, please refer to the configuration guide. Below only options necessary to be changed are listed.

• 8 Start the Tricircle Xjob daemon:

  cd tricircle
  tricircle-xjob --config-file etc/xjob.conf
  

• 9 Setup central Neutron server

  In this guide we assume readers are familiar with how to install Neutron server, so we just briefly discuss the steps and extra configuration needed by central Neutron server. For detail information about the configuration options in “client” and “tricircle” groups, please refer to the configuration guide. Neutron server can be installed alone, or you can install a full OpenStack instance then remove or stop other services.

  • install Neutron package

  • configure central Neutron server

    edit neutron.conf

  Option	Description	Example
  [database] connection	database connection string for central Neutron server	mysql+pymysql://root:password@ 127.0.0.1/neutron?charset=utf8
  [DEFAULT] bind_port	Port central Neutron server binds to	change to a different value rather than 9696 if you run central and local Neutron server in the same host
  [DEFAULT] core_plugin	core plugin central Neutron server uses	tricircle.network.central_plugin. TricirclePlugin
  [DEFAULT] service_plugins	service plugin central Neutron server uses	(leave empty)
  [DEFAULT] tricircle_db_connection	database connection string for tricircle	mysql+pymysql://root:password@ 127.0.0.1/tricircle?charset=utf8
  [client] auth_url	keystone authorization url	http://$keystone_service_host/identity
  [client] identity_url	keystone service url	http://$keystone_service_host/identity/v3
  [client] auto_refresh_endpoint	if set to True, endpoint will be automatically refreshed if timeout accessing	True
  [client] top_region_name	name of central region which client needs to access	CentralRegion
  [client] admin_username	username of admin account	admin
  [client] admin_password	password of admin account	password
  [client] admin_tenant	project name of admin account	demo
  [client] admin_user_domain_name	user domain name of admin account	Default
  [client] admin_tenant_domain_name	project name of admin account	Default
  [tricircle] type_drivers	list of network type driver entry points to be loaded	vxlan,vlan,flat,local
  [tricircle] tenant_network_types	ordered list of network_types to allocate as tenant networks	vxlan,vlan,flat,local
  [tricircle] network_vlan_ranges	physical network names and VLAN tags range usable of VLAN provider	bridge:2001:3000
  [tricircle] vni_ranges	VxLAN VNI range	1001:2000
  [tricircle] flat_networks	physical network names with which flat networks can be created	bridge
  [tricircle] bridge_network_type	l3 bridge network type which is enabled in tenant_network_types and is not local type	vxlan
  [tricircle] default_region_for_external_network	Default Region where the external network belongs to	RegionOne
  [tricircle] enable_api_gateway	whether the API gateway is enabled	False

  Note

  Change keystone_service_host to the address of Keystone service.

  • create database for central Neutron server
  • register central Neutron server endpoint in Keystone, central Neutron should be registered in the same region with the Tricircle
  • start central Neutron server

Installation with Local Neutron Server¶

• 1 Install the Tricircle package:

  git clone https://github.com/openstack/tricircle.git
  cd tricircle
  pip install -e .
  

• 2 Setup local Neutron server

  In this guide we assume readers have already installed a complete OpenStack instance running services like Nova, Cinder, Neutron, etc, so we just discuss how to configure Neutron server to work with the Tricircle. For detail information about the configuration options in “client” and “tricircle” groups, please refer to the configuration guide. After the change, you just restart the Neutron server.

  edit neutron.conf.

  Note

  Pay attention to the service_plugins configuration item, make sure the plugin which is configured can support the association of floating IP to a port whose network is not directly attached to the router. To support it, TricircleL3Plugin is inherited from Neutron original L3RouterPlugin and overrides the original “get_router_for_floatingip” implementation. In order to configure local Neutron to use original L3RouterPlugin, you will need to patch the function “get_router_for_floatingip” in the same way that has been done for TricircleL3Plugin.

  It’s not necessary to configure the service plugins if cross Neutron L2 networking is the only need in the deployment.

  Option	Description	Example
  [DEFAULT] core_plugin	core plugin local Neutron server uses	tricircle.network.local_plugin. TricirclePlugin
  [DEFAULT] service_plugins	service plugins local Neutron server uses	tricircle.network.local_l3_plugin. TricircleL3Plugin
  [client] auth_url	keystone authorization url	http://$keystone_service_host/identity
  [client] identity_url	keystone service url	http://$keystone_service_host/identity/v3
  [client] auto_refresh_endpoint	if set to True, endpoint will be automatically refreshed if timeout accessing	True
  [client] top_region_name	name of central region which client needs to access	CentralRegion
  [client] admin_username	username of admin account	admin
  [client] admin_password	password of admin account	password
  [client] admin_tenant	project name of admin account	demo
  [client] admin_user_domain_name	user domain name of admin account	Default
  [client] admin_tenant_domain_name	project name of admin account	Default
  [tricircle] real_core_plugin	the core plugin the Tricircle local plugin invokes	neutron.plugins.ml2.plugin. Ml2Plugin
  [tricircle] central_neutron_url	central Neutron server url	http://$neutron_service_host :9696

  Note

  Change keystone_service_host to the address of Keystone service, and neutron_service_host to the address of central Neutron service.

  edit ml2_conf.ini

  Option	Description	Example
  [ml2] mechanism_drivers	add l2population if vxlan network is used	openvswitch,l2population
  [agent] l2_population	set to True if vxlan network is used	True
  [agent] tunnel_types	set to vxlan if vxlan network is used	vxlan
  [ml2_type_vlan] network_vlan_ranges	for a specific physical network, the vlan range should be the same with tricircle.network_vlan_ranges option for central Neutron, configure this option if vlan network is used	bridge:2001:3000
  [ml2_type_vxlan] vni_ranges	should be the same with tricircle.vni_ranges option for central Neutron, configure this option if vxlan network is used	1001:2000
  [ml2_type_flat] flat_networks	should be part of the tricircle.network_vlan_ranges option for central Neutron, configure this option if flat network is used	bridge
  [ovs] bridge_mappings	map the physical network to an ovs bridge	bridge:br-bridge

  Note

  In tricircle.network_vlan_ranges option for central Neutron, all the available physical networks in all pods and their vlan ranges should be configured without duplication. It’s possible that one local Neutron doesn’t contain some of the physical networks configured in tricircle.network_vlan_ranges, in this case, users need to specify availability zone hints when creating network or booting instances in the correct pod, to ensure that the required physical network is available in the target pod.

Setup¶

• 1 Follow “Multi-pod Installation with DevStack” document to prepare your local.conf for both nodes, and set TRICIRCLE_DEPLOY_WITH_CELL to True for both nodes. Start DevStack in node1, then node2.

• 2 Enter the screen section in node2, stop n-api and n-sch.

• 3 In node2, run the following command:

  mysql -u$user -p$password -Dnova -e 'select host, mapped from compute_nodes
  

  you can see that this command returns you one row showing the host of node2 is already mapped:

  +-----------+--------+
  | host      | mapped |
  +-----------+--------+
  | zhiyuan-2 |      1 |
  +-----------+--------+
  

  This host is registered to Nova API in node2, which is already stopped by us, We need to update this row to set “mapped” to 0:

  mysql -u$user -p$password -Dnova -e 'update compute_nodes set mapped = 0 where host = "zhiyuan-2"'
  

  then we can register this host again in step4.

• 4 In node1, run the following commands to register the new cell:

  nova-manage cell_v2 create_cell --name cell2 \
    --transport-url rabbit://$rabbit_user:$rabbit_passwd@$node2_ip:5672/ \
    --database_connection mysql+pymysql://$db_user:$db_passwd@$node2_ip/nova?charset=utf8
  
  nova-manage cell_v2 discover_hosts
  

  then you can see the new cell and host are added in the database:

  mysql -u$user -p$password -Dnova -e 'select cell_id, host from host_mappings'
  
  +---------+-----------+
  | cell_id | host      |
  +---------+-----------+
  |       2 | zhiyuan-1 |
  |       3 | zhiyuan-2 |
  +---------+-----------+
  
  mysql -u$user -p$password -Dnova -e 'select id, name from cell_mappings'
  
  +----+-------+
  | id | name  |
  +----+-------+
  |  1 | cell0 |
  |  2 | cell1 |
  |  3 | cell2 |
  +----+-------+
  

• 5 In node1, check if compute services in both hosts are registered:

  openstack --os-region-name CentralRegion compute service list
  
  +----+------------------+-----------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host      | Zone     | Status  | State | Updated At                 |
  +----+------------------+-----------+----------+---------+-------+----------------------------+
  |  5 | nova-conductor   | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:37.000000 |
  |  7 | nova-scheduler   | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:36.000000 |
  |  8 | nova-consoleauth | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:39.000000 |
  |  9 | nova-compute     | zhiyuan-1 | nova     | enabled | up    | 2017-05-27T06:04:42.000000 |
  |  4 | nova-conductor   | zhiyuan-2 | internal | enabled | up    | 2017-05-27T06:04:40.000000 |
  |  6 | nova-scheduler   | zhiyuan-2 | internal | enabled | down  | 2017-05-27T02:56:50.000000 |
  |  7 | nova-consoleauth | zhiyuan-2 | internal | enabled | up    | 2017-05-27T06:04:36.000000 |
  |  8 | nova-compute     | zhiyuan-2 | nova     | enabled | up    | 2017-05-27T06:04:38.000000 |
  +----+------------------+-----------+----------+---------+-------+----------------------------+
  

  Nova scheduler in host zhiyuan-2 is down because we stop it in step2.

• 6 Create two aggregates and put the two hosts in each aggregate:

  nova --os-region-name CentralRegion aggregate-create ag1 az1
  nova --os-region-name CentralRegion aggregate-create ag2 az2
  nova --os-region-name CentralRegion aggregate-add-host ag1 zhiyuan-1
  nova --os-region-name CentralRegion aggregate-add-host ag2 zhiyuan-2
  

• 7 Create pods, tricircle client is used:

  openstack --os-region-name CentralRegion multiregion networking pod create --region-name CentralRegion
  openstack --os-region-name CentralRegion multiregion networking pod create --region-name RegionOne --availability-zone az1
  openstack --os-region-name CentralRegion multiregion networking pod create --region-name RegionTwo --availability-zone az2
  

• 8 Create network and boot virtual machines:

  net_id=$(openstack --os-region-name CentralRegion network create --provider-network-type vxlan net1 -c id -f value)
  openstack --os-region-name CentralRegion subnet create --subnet-range 10.0.1.0/24 --network net1 subnet1
  image_id=$(openstack --os-region-name CentralRegion image list -c ID -f value)
  
  openstack --os-region-name CentralRegion server create --flavor 1 --image $image_id --nic net-id=$net_id --availability-zone az1 vm1
  openstack --os-region-name CentralRegion server create --flavor 1 --image $image_id --nic net-id=$net_id --availability-zone az2 vm2
  

Trouble Shooting¶

• 1 After you run “compute service list” in step5, you only see services in node1, like:

  +----+------------------+-----------+----------+---------+-------+----------------------------+
  | ID | Binary           | Host      | Zone     | Status  | State | Updated At                 |
  +----+------------------+-----------+----------+---------+-------+----------------------------+
  |  5 | nova-conductor   | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:37.000000 |
  |  7 | nova-scheduler   | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:36.000000 |
  |  8 | nova-consoleauth | zhiyuan-1 | internal | enabled | up    | 2017-05-27T06:04:39.000000 |
  |  9 | nova-compute     | zhiyuan-1 | nova     | enabled | up    | 2017-05-27T06:04:42.000000 |
  +----+------------------+-----------+----------+---------+-------+----------------------------+
  

  Though new cell has been registered in the database, the running n-api process in node1 may not recognize it. We find that restarting n-api can solve this problem.

• 2 After you create a server, the server turns into Error status with error “No valid host was found”

  We check the log of n-sch process in node1 and find it says: “Found 2 cells”, this is incorrect since we have three cells including cell0. After restarting n-sch, the log says “Found 3 cells”. Try creating server again after that.

  Another reason for this problem we have found is that n-cpu doesn’t sync available resource with n-sch, restarting n-cpu can slove this problem.