[ English | Indonesia | Deutsch | 日本語 ]

Customizing the OpenStack Compute (nova) Scheduler

Many OpenStack projects allow for customization of specific features using a driver architecture. You can write a driver that conforms to a particular interface and plug it in through configuration. For example, you can easily plug in a new scheduler for Compute. The existing schedulers for Compute are feature full and well documented at Scheduling. However, depending on your user’s use cases, the existing schedulers might not meet your requirements. You might need to create a new scheduler.

To create a scheduler, you must inherit from the class nova.scheduler.driver.Scheduler. Of the five methods that you can override, you must override the two methods marked with an asterisk (*) below:

• update_service_capabilities

• hosts_up

• group_hosts

• * schedule_run_instance

• * select_destinations

To demonstrate customizing OpenStack, we’ll create an example of a Compute scheduler that randomly places an instance on a subset of hosts, depending on the originating IP address of the request and the prefix of the hostname. Such an example could be useful when you have a group of users on a subnet and you want all of their instances to start within some subset of your hosts.

Warning

This example is for illustrative purposes only. It should not be used as a scheduler for Compute without further development and testing.

When you join the screen session that stack.sh starts with screen -r stack, you are greeted with many screen windows:

0$ shell*  1$ key  2$ horizon  ...  9$ n-api  ...  14$ n-sch ...

shell

A shell where you can get some work done

key

The keystone service

horizon

The horizon dashboard web application

n-{name}

The nova services

n-sch

The nova scheduler service

To create the scheduler and plug it in through configuration

1. The code for OpenStack lives in /opt/stack, so go to the nova directory and edit your scheduler module. Change to the directory where nova is installed:

   $ cd /opt/stack/nova
   

2. Create the ip_scheduler.py Python source code file:

   $ vim nova/scheduler/ip_scheduler.py
   

3. The code shown below is a driver that will schedule servers to hosts based on IP address as explained at the beginning of the section. Copy the code into ip_scheduler.py. When you are done, save and close the file.

   # vim: tabstop=4 shiftwidth=4 softtabstop=4
   # Copyright (c) 2014 OpenStack Foundation
   # All Rights Reserved.
   #
   #    Licensed under the Apache License, Version 2.0 (the "License"); you may
   #    not use this file except in compliance with the License. You may obtain
   #    a copy of the License at
   #
   #         http://www.apache.org/licenses/LICENSE-2.0
   #
   #    Unless required by applicable law or agreed to in writing, software
   #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
   #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
   #    License for the specific language governing permissions and limitations
   #    under the License.
   
   """
   IP Scheduler implementation
   """
   
   import random
   
   from oslo_config import cfg
   
   from nova.compute import rpcapi as compute_rpcapi
   from nova import exception
   from nova.openstack.common import log as logging
   from nova.openstack.common.gettextutils import _
   from nova.scheduler import driver
   
   CONF = cfg.CONF
   CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
   LOG = logging.getLogger(__name__)
   
   class IPScheduler(driver.Scheduler):
       """
       Implements Scheduler as a random node selector based on
       IP address and hostname prefix.
       """
   
       def __init__(self, *args, **kwargs):
           super(IPScheduler, self).__init__(*args, **kwargs)
           self.compute_rpcapi = compute_rpcapi.ComputeAPI()
   
       def _filter_hosts(self, request_spec, hosts, filter_properties,
           hostname_prefix):
           """Filter a list of hosts based on hostname prefix."""
   
           hosts = [host for host in hosts if host.startswith(hostname_prefix)]
           return hosts
   
       def _schedule(self, context, topic, request_spec, filter_properties):
           """Picks a host that is up at random."""
   
           elevated = context.elevated()
           hosts = self.hosts_up(elevated, topic)
           if not hosts:
               msg = _("Is the appropriate service running?")
               raise exception.NoValidHost(reason=msg)
   
           remote_ip = context.remote_address
   
           if remote_ip.startswith('10.1'):
               hostname_prefix = 'doc'
           elif remote_ip.startswith('10.2'):
               hostname_prefix = 'ops'
           else:
               hostname_prefix = 'dev'
   
           hosts = self._filter_hosts(request_spec, hosts, filter_properties,
               hostname_prefix)
           if not hosts:
               msg = _("Could not find another compute")
               raise exception.NoValidHost(reason=msg)
   
           host = random.choice(hosts)
           LOG.debug("Request from %(remote_ip)s scheduled to %(host)s" % locals())
   
           return host
   
       def select_destinations(self, context, request_spec, filter_properties):
           """Selects random destinations."""
           num_instances = request_spec['num_instances']
           # NOTE(timello): Returns a list of dicts with 'host', 'nodename' and
           # 'limits' as keys for compatibility with filter_scheduler.
           dests = []
           for i in range(num_instances):
               host = self._schedule(context, CONF.compute_topic,
                       request_spec, filter_properties)
               host_state = dict(host=host, nodename=None, limits=None)
               dests.append(host_state)
   
           if len(dests) < num_instances:
               raise exception.NoValidHost(reason='')
           return dests
   
       def schedule_run_instance(self, context, request_spec,
                                 admin_password, injected_files,
                                 requested_networks, is_first_time,
                                 filter_properties, legacy_bdm_in_spec):
           """Create and run an instance or instances."""
           instance_uuids = request_spec.get('instance_uuids')
           for num, instance_uuid in enumerate(instance_uuids):
               request_spec['instance_properties']['launch_index'] = num
               try:
                   host = self._schedule(context, CONF.compute_topic,
                                         request_spec, filter_properties)
                   updated_instance = driver.instance_update_db(context,
                           instance_uuid)
                   self.compute_rpcapi.run_instance(context,
                           instance=updated_instance, host=host,
                           requested_networks=requested_networks,
                           injected_files=injected_files,
                           admin_password=admin_password,
                           is_first_time=is_first_time,
                           request_spec=request_spec,
                           filter_properties=filter_properties,
                           legacy_bdm_in_spec=legacy_bdm_in_spec)
               except Exception as ex:
                   # NOTE(vish): we don't reraise the exception here to make sure
                   #             that all instances in the request get set to
                   #             error properly
                   driver.handle_schedule_error(context, ex, instance_uuid,
                                                request_spec)
   

   There is a lot of useful information in context, request_spec, and filter_properties that you can use to decide where to schedule the instance. To find out more about what properties are available, you can insert the following log statements into the schedule_run_instance method of the scheduler above:

   LOG.debug("context = %(context)s" % {'context': context.__dict__})
   LOG.debug("request_spec = %(request_spec)s" % locals())
   LOG.debug("filter_properties = %(filter_properties)s" % locals())
   

4. To plug this scheduler into nova, edit one configuration file, /etc/nova/nova.conf:

   $ vim /etc/nova/nova.conf
   

5. Find the scheduler_driver config and change it like so:

   scheduler_driver=nova.scheduler.ip_scheduler.IPScheduler
   

6. Restart the nova scheduler service to make nova use your scheduler. Start by switching to the n-sch screen:

   1. Press Ctrl+A followed by 9.

   2. Press Ctrl+A followed by N until you reach the n-sch screen.

   3. Press Ctrl+C to kill the service.

   4. Press Up Arrow to bring up the last command.

   5. Press Enter to run it.

7. Test your scheduler with the nova CLI. Start by switching to the shell screen and finish by switching back to the n-sch screen to check the log output:

   1. Press  Ctrl+A followed by 0.

   2. Make sure you are in the devstack directory:

      $ cd /root/devstack
      

   3. Source openrc to set up your environment variables for the CLI:

      $ . openrc
      

   4. Put the image ID for the only installed image into an environment variable:

      $ IMAGE_ID=`openstack image list | egrep cirros | egrep -v "kernel|ramdisk" | awk '{print $2}'`
      

   5. Boot a test server:

      $ openstack server create --flavor 1 --image $IMAGE_ID scheduler-test
      

8. Switch back to the n-sch screen. Among the log statements, you’ll see the line:

   2014-01-23 19:57:47.262 DEBUG nova.scheduler.ip_scheduler
   [req-... demo demo] Request from xx.xx.xx.xx scheduled to devstack-havana
   _schedule /opt/stack/nova/nova/scheduler/ip_scheduler.py:76
   

Warning

Functional testing like this is not a replacement for proper unit and integration testing, but it serves to get you started.

A similar pattern can be followed in other projects that use the driver architecture. Simply create a module and class that conform to the driver interface and plug it in through configuration. Your code runs when that feature is used and can call out to other services as necessary. No project core code is touched. Look for a “driver” value in the project’s .conf configuration files in /etc/<project> to identify projects that use a driver architecture.

When your scheduler is done, we encourage you to open source it and let the community know on the OpenStack mailing list. Perhaps others need the same functionality. They can use your code, provide feedback, and possibly contribute. If enough support exists for it, perhaps you can propose that it be added to the official Compute schedulers.