1. Results of measuring performance of Cobbler

Abstract:This document includes performance test results of Cobbler service as a provisioning system. All tests have been performed regarding Measuring performance of provisioning systems

1.1. Environment description

1.1.1. Hardware configuration of each server

Description of servers hardware

server	name	728983-comp-disk-242	nodes-{1..195}
	role	Cobbler	nodes to be provisioned
	vendor,model	HP,DL380 Gen9	HP,DL380 Gen9
	operating_system	3.13.0-76-generic Ubuntu-trusty x86_64	3.13.0-83-generic Ubuntu-trusty x86_64
CPU	vendor,model	Intel,E5-2680 v3	Intel,E5-2680 v3
	processor_count	2	2
	core_count	12	12
	frequency_MHz	2500	2500
RAM	vendor,model	HP,752369-081	HP,752369-081
	amount_MB	262144	262144
NETWORK	interface_name	p1p1	p1p1
	vendor,model	Intel,X710 Dual Port	Intel,X710 Dual Port
	bandwidth	10G	10G
STORAGE	dev_name	/dev/sda	/dev/sda
	vendor,model	raid10 - HP P840 12 disks EH0600JEDHE	raid10 - HP P840 12 disks EH0600JEDHE
	SSD/HDD	HDD	HDD
	size	3,6TB	3,6TB

1.1.2. Network scheme and part of configuration of hardware network switches

Network scheme of the environment:

Here is the part of switch configuration for each switch port which connected to p1p1 interface of a server:

switchport mode trunk
switchport trunk native vlan 600
switchport trunk allowed vlan 600-602,630-649
spanning-tree port type edge trunk
spanning-tree bpduguard enable
no snmp trap link-status

1.1.3. Software configuration of Cobbler service

1.1.3.1. Installation of Cobbler:

To install and configure Cobbler section Installation script and config files has been used.

The nodes has Intel X710 NIC therefore we had to provide i40e driver to Debian installer. The following steps was performed to do that:

mkdir /var/www/html/ubuntu_custom_packages
cd /var/www/html/ubuntu_custom_packages
wget <address_to_builded_deb_package>
dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Cobbler has the following issue https://github.com/cobbler/cobbler/issues/1530. Due the bug we can’t add one more repository for provisioning step. Also we need to allow root access via ssh after provisioning. Therefore:

sed -i s/"\$SNIPPET('late_apt_repo_config')"/\
         "\$SNIPPET('late_apt_repo_config')\\n\
          echo \"deb http:\/\/\$server\/ubuntu_custom_packages\/ .\/\" \>\> \/etc\/apt\/sources.list\\n\
          apt\-get update \&\& apt\-get \-y \-\-force\-yes install i40e\-dkms\\n\
          sed \-i \'s\/PermitRootLogin\.\*\/PermitRootLogin yes\/g\' \/etc\/ssh\/sshd_config"/ \
        /var/lib/cobbler/scripts/preseed_late_default

Versions of some software

Software	Version
Ubuntu	Ubuntu 14.04.3 LTS
Cobbler	2.6.11
Apache2	2.4.7
Bind9	1:9.9.5
tftpd-hpa	5.2
isc-dhcp-server	4.2.4
syslinux	6.03

1.1.3.2. Test tool:

Script to start provisioning and collect metrics was used to collect performance metrics during the tests.

1.1.3.3. Operating system configuration:

You can find outputs of some commands and /etc folder in the following archive: server_description_of_728983-comp-disk-242.tar.gz

1.1.4. Software configuration of provisioned nodes

Versions of some software

Software	Version
Ubuntu	Ubuntu 14.04.3 LTS

1.1.4.1. Operating system:

You can find outputs of some commands and /etc folder in the following archive: server_description_of_728997-comp-disk-228.tar.gz

1.2. Testing process

1.2.1. Preparation

1. Cobbler was installed on top of 728983-comp-disk-242 server as described in Installation of Cobbler: section.
2. systems.list file (with matching of server name, server ILO address and mac address of interface connected to Cobbler) was created:

root@728983-comp-disk-242:~# tail -3 systems.list
729691-comp-disk-201,10.15.242.170,68:05:ca:38:64:d4
729692-comp-disk-200,10.15.242.171,3c:fd:fe:9c:62:30
729693-comp-disk-199,10.15.242.172,3c:fd:fe:9c:68:3c

3. Cobbler systems was added using Script to add cobbler systems.

1.2.2. Testing. Case when pxlinux downloads kernel via HTTP

During Installation of Cobbler: we copied /etc/cobbler/pxe/pxesystem.template (pxesystem_http). It means that on the step when pxelinux downloads kernel and initrd files via HTTP protocol.

1. Performed Script to start provisioning and collect metrics with option “1” to provision 1 server.

bash -ex measure.sh 1

2. Save /var/log/results.csv file

mv /var/log/results.csv /var/log/results-http-1.csv

3. The steps 1 and 2 was repeated with the following numbers of nodes: 1,10,20,40

As a result of this part we got the following CSV files:

METRICS(NUMBER_OF_NODES=1) METRICS(NUMBER_OF_NODES=10) METRICS(NUMBER_OF_NODES=20) METRICS(NUMBER_OF_NODES=40)

During the las test when we provisioned 40 nodes 2 nodes wasn’t provisioned due the issue Linux kernel can’t be downloaded by pxelinux via HTTP

1.2.3. Testing. Case when pxlinux downloads kernel via TFTP

During the following steps we change pxe configurations for all systems to download kernel and initrd files via TFTP protocol.

1. Change pxe template

sed -i s/"http:\/\/\$server:\$http_port\/cblr\/"// \
    /etc/cobbler/pxe/pxesystem.template

2. Performed Script to start provisioning and collect metrics with option “1” to provision 1 server.

bash -ex measure.sh 1

3. Save /var/log/results.csv file

mv /var/log/results.csv /var/log/results-http-1.csv

4. The steps 2 and 3 was repeated with the following numbers of nodes: 1,10,20,40,80,160,195

As a result of this part we got the following CSV files:

METRICS(NUMBER_OF_NODES=1) METRICS(NUMBER_OF_NODES=10) METRICS(NUMBER_OF_NODES=20) METRICS(NUMBER_OF_NODES=40) METRICS(NUMBER_OF_NODES=80) METRICS(NUMBER_OF_NODES=160) METRICS(NUMBER_OF_NODES=195)

1.3. Results

1.3.1. Case when pxlinux downloads kernel via HTTP

After simple processing results the following plots for performance metrics collected during provisioning of the nodes in depend on time created (click to expand an image):

1.3.1.1. CPU(TIME), RAM(TIME)

Number of nodes	Plot CPU(TIME)	Plot RAM(TIME)
1
10
20
40

1.3.1.2. NET(TIME), DISK(TIME)

Number of nodes	Plot NET(TIME)	Plot DISK(TIME)
1
10
20
40

The following table and graphs show how performance metrics and provisioning time parameters depend on numbers of nodes.

numbers of nodes	1	10	20	40
provisioning time	633.341015606	642.138856745	667.484074798	1322.96068455
cpu_usage_max	1.228	5.316	9.954	19.386
cpu_usage_min	0	0	0	0
cpu_usage_average	0.04806940063	0.09161897356	0.1349446108	0.1451919879
cpu_usage_percentile 90%	0.083	0.146	0.25	0.271
ram_usage_max	5200.3	5305.52	5426.92	5672.05
ram_usage_min	5179.25	5165.86	5175.59	5183.7
ram_usage_average	5183.245536	5174.417621	5191.933563	5205.502789
ram_usage_percentile 90%	5185.981	5177.15	5199.619	5220.494
net_recv_max	4.922	10.5239	19.0217	17.0064
net_recv_min	0	0	0	0
net_recv_average	0.0235053613	0.188453268	0.3578001869	0.35373526
net_recv_percentile 90%	0.0126724	0.428462	0.8075162	0.9857374
net_send_max	491.92	1300.81	2184.08	1857.75
net_send_min	0	0	0	0
net_send_average	3.658875403	36.04228255	69.38244342	68.31148501
net_send_percentile 90%	0.01753312	126.2174	233.9985	255.4942
dsk_io_read_max	0.074	0.074	0.074	0.073
dsk_io_read_min	0	0	0	0
dsk_io_read_average	0.0001167192429	0.0001150855365	0.0001107784431	0.00005517762661
dsk_io_read_percentile 90%	0	0	0	0
dsk_io_writ_max	60	104	223	265
dsk_io_writ_min	0	0	0	0
dsk_io_writ_average	1.433591483	1.735454121	2.142062874	1.789037793
dsk_io_writ_percentile 90%	5	4	6	5

1.3.1.3. PROVISIONING_TIME(NODES)

1.3.1.4. CPU(NODES), RAM(NODES)

Plot CPU(NODES)	Plot RAM(NODES)

1.3.1.5. NET(NODES), DISK(NODES)

Plot NET(NODES)	Plot DISK(NODES)

1.3.2. Case when pxlinux downloads kernel via TFTP

After simple processing results the following plots for performance metrics collected during provisioning of the nodes in depend on time created (click to expand an image):

1.3.2.1. CPU(TIME), RAM(TIME)

Number of nodes	Plot CPU(TIME)	Plot RAM(TIME)
1
10
20
40
80
160
195

1.3.2.2. NET(TIME), DISK(TIME)

Number of nodes	Plot NET(TIME)	Plot DISK(TIME)
1
10
20
40
80
160
195

The following table and graphs show how performance metrics and provisioning time parameters depend on numbers of nodes.

numbers of nodes	1	10	20	40	80	160	195
provisioning time	655.309167699	685.366625243	697.005017299	716.179839426	768.240187372	795.676431454	798.103271441
cpu_usage_max	1.271	4.88	6.857	19.866	38.46	75.475	85.182
cpu_usage_min	0	0	0	0	0	0	0
cpu_usage_average	0.04915091463	0.09638921283	0.1376284075	0.2399679219	0.3951248375	0.7407713568	0.9038723404
cpu_usage_percentile 90%	0.083	0.167	0.25	0.488	0.9172	2.0905	2.2302
ram_usage_max	5205.15	5323.44	5448.61	5684.64	6172.27	7134.59	7582.7
ram_usage_min	5181.74	5185.72	5203.85	5191.63	5201.07	5206.32	5226.64
ram_usage_average	5185.197835	5194.588353	5222.031549	5218.299512	5240.278635	5276.185992	5301.640676
ram_usage_percentile 90%	5186.965	5197.37	5228.904	5228.812	5249.982	5280.4	5280.912
net_recv_max	4.03802	16.0916	23.9195	31.5682	45.9824	60.1388	93.5101
net_recv_min	0	0	0	0	0	0	0
net_recv_average	0.02883827959	0.2355115738	0.4605028878	0.8857573161	1.643228416	3.174715098	3.858486644
net_recv_percentile 90%	0.01241305	0.4111175	0.9639786	2.218254	3.943012	8.279855	9.897108
net_send_max	370.962	1535.99	1815.63	1987.61	3184.7	5157.32	7434.88
net_send_min	0	0	0	0	0	0	0
net_send_average	3.53548533	33.7695734	66.46850655	129.2245215	240.9720221	465.5923141	565.352486
net_send_percentile 90%	0.01602555	96.58555	206.7048	489.8658	862.1248	1643.095	1964.3
dsk_io_read_max	0.072	0.072	0.072	0.072	1	0.071	0.07
dsk_io_read_min	0	0	0	0	0	0	0
dsk_io_read_average	0.0001097560976	0.0001049562682	0.0001032998565	0.00010041841	0.001394018205	0.0000891959799	0.00008760951189
dsk_io_read_percentile 90%	0	0	0	0	0	0	0
dsk_io_writ_max	61	463	427	463	55	463	428
dsk_io_writ_min	0	0	0	0	0	0	0
dsk_io_writ_average	1.321489329	1.940080175	2.288228121	2.612129707	1.923149545	3.551388191	3.694494368
dsk_io_writ_percentile 90%	5	3	3	6	4	9	11

1.3.2.3. PROVISIONING_TIME(NODES)

1.3.2.4. CPU(NODES), RAM(NODES)

Plot CPU(NODES)	Plot RAM(NODES)

1.3.2.5. NET(NODES), DISK(NODES)

Plot NET(NODES)	Plot DISK(NODES)

1.4. Issues which have been found during the tests

1.4.1. Linux kernel can’t be downloaded by pxelinux via HTTP

During testing the case when pxelinux downloads kernel via HTTP, we have found that during provisioning numbers of nodes more then 20 some servers (2 for case when we tries to provision 40 nodes and up to 7 when tries to provision 195 nodes) can be stacked on the downloading of ubuntu installer kernel. Here is example of screen:

In Apache log files we can see the following lines

10.50.11.2 - - [29/Mar/2016:11:57:34 +0000] 300100342 "GET /cblr//images/ubuntu14-x86_64/linux HTTP/1.0" 200 102200 "-" "Syslinux/6.03"

It means that transferred data (linux file) is too small (102200 bytes). Actually the size of the linux kernel was 5778968 bytes. Apache return 200 code after “Timeout” parameter which specified in its configuration file.

1.5. Applications

1.5.1. Installation script and config files

#!/bin/bash -ex

PXE_INTERFACE=${PXE_INTERFACE:-p1p1}
PXE_ADDRESS=${PXE_ADDRESS:-"10.50.0.10"}
MANAGEMENT_INTERFACE=${MANAGEMENT_INTERFACE:-p1p1.602}
COBBLER_NET=${COBBLER_NET:-"10.50.0.0"}
COBBLER_NETMASK=${COBBLER_NETMASK:-255.255.0.0}
COBBLER_DYN_RANGE=${COBBLER_DYN_RANGE:-"10.50.1.1 10.50.10.254"}
COBBLER_DNS_DOMAIN=${COBBLER_DNS_DOMAIN:-"cobbler-test.local"}
UBUNTU_REPOSITORY="http://archive.ubuntu.com/ubuntu/"

preparation ()
{
  apt-get -y install curl software-properties-common make prips fence-agents ipmitool sshpass
}
install_tftp ()
{
  apt-get -y install tftpd-hpa
  start tftpd-hpa
}
install_dhcp ()
{
  apt-get -y install isc-dhcp-server
  sed -i s/"INTERFACES=\"\""/"INTERFACES=\"${PXE_INTERFACE}\""/ /etc/default/isc-dhcp-server
}
install_dns ()
{
  apt-get -y install bind9
}
install_cobbler ()
{
  # Install Cobbler from package from OpenSuse repository
  curl -s 'http://download.opensuse.org/repositories/home:/libertas-ict:/cobbler26/xUbuntu_14.04/Release.key' | apt-key add -
  add-apt-repository "deb http://download.opensuse.org/repositories/home:/libertas-ict:/cobbler26/xUbuntu_14.04/ ./"
  apt-get update
  apt-get -y install cobbler python-urlgrabber libapache2-mod-wsgi python-django
  # Apply few workarounds for
  SECRET_KEY=$(python -c 'import re;from random import choice; import sys; sys.stdout.write(re.escape("".join([choice("abcdefghijklmnopqrstuvwxyz0123456789^&*(-_=+)") for i in range(100)])))')
  sed --in-place "s/^SECRET_KEY = .*/SECRET_KEY = '${SECRET_KEY}'/" /usr/share/cobbler/web/settings.py
  rm -f /etc/apache2/conf-enabled/cobbler.conf
  rm -f /etc/apache2/conf-enabled/cobbler_web.conf
  cp /etc/cobbler/cobbler.conf /etc/apache2/conf-available/
  cp /etc/cobbler/cobbler_web.conf /etc/apache2/conf-available/
  ln -s /etc/apache2/conf-available/cobbler.conf /etc/apache2/conf-enabled/
  ln -s /etc/apache2/conf-available/cobbler_web.conf /etc/apache2/conf-enabled/
  a2enconf cobbler cobbler_web
  a2enmod proxy proxy_http
  chown -R www-data /var/lib/cobbler/webui_sessions
  # Change configs regarding variables values
  for INSTALL_VAR in PXE_ADDRESS COBBLER_NET COBBLER_NETMASK COBBLER_DYN_RANGE COBBLER_DNS_DOMAIN
  do
    find ./ -type f -exec sed -i s/"${INSTALL_VAR}"/"${!INSTALL_VAR}"/g {} \;
  done
  # Need to copy configs
  cp -rf configs/etc/cobbler/* /etc/cobbler/
  service cobblerd restart
  update-rc.d cobblerd defaults
  service apache2 restart
  # download pxe loaders
  cobbler get-loaders
  wget 'https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.xz'
  tar xJf syslinux-6.03.tar.xz
  if [ ! -d "/var/lib/tftpboot/" ]
  then
    mkdir /var/lib/tftpboot/
  fi
  ln -s /var/lib/tftpboot /srv/www/cobbler/
  cp syslinux-6.03/bios/core/lpxelinux.0 /var/lib/tftpboot/
  cp syslinux-6.03/bios/core/lpxelinux.0 /var/lib/cobbler/loaders/
  cp syslinux-6.03/bios/com32/lib/libcom32.c32 /var/lib/tftpboot/
  cp syslinux-6.03/bios/com32/libutil/libutil.c32 /var/lib/tftpboot/
  cp syslinux-6.03/bios/com32/elflink/ldlinux/ldlinux.c32 /var/lib/tftpboot/
  cp syslinux-6.03/bios/com32/chain/chain.c32 /var/lib/tftpboot
  # Configure Ubuntu repository
  if [ ! -d "/var/lib/cobbler/.gnupg/" ]
  then
    mkdir -p /var/lib/cobbler/.gnupg/
  fi
  rm -rf /srv/www/cobbler/repo_mirror
  ln -s /var/www/cobbler/repo_mirror /srv/www/cobbler/
  ln -s /etc/apt/trusted.gpg /var/lib/cobbler/.gnupg/trustedkeys.gpg
  cobbler repo add --name=ubuntu14-x86_64
                   --apt-components='main restricted universe multiverse main/debian-installer restricted/debian-installer'
                   --apt-dists='trusty trusty-updates trusty-security'
                   --breed=apt
                   --keep-updated=yes
                   --mirror=${UBUNTU_REPOSITORY}
                   --mirror-locally=yes
  # Copy kernel and initrd which will be used as a installers.
  rsync -av rsync://archive.ubuntu.com:/ubuntu/dists/trusty/main/installer-amd64  /var/www/cobbler/repo_mirror/ubuntu14-x86_64/dists/trusty/main/
  # Create Cobbler distro and Cobbler profile
  cobbler distro add --name=ubuntu14-x86_64
                     --kernel=/var/www/cobbler/repo_mirror/ubuntu14-x86_64/dists/trusty/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
                     --initrd=/var/www/cobbler/repo_mirror/ubuntu14-x86_64/dists/trusty/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
                     --ksmeta=tree=http://@@http_server@@/cblr/repo_mirror/ubuntu14-x86_64
                     --arch=x86_64
                     --breed=ubuntu
                     --os-version=trusty
  cobbler profile add --name=ubuntu14-x86_64
                      --distro=ubuntu14-x86_64
                      --kickstart=/var/lib/cobbler/kickstarts/sample.seed
                      --kopts='ksdevice=bootif lang= locale=en_US text priority=critical'
  cobbler sync
  # Mirror configured repositories
  cobbler reposync
  echo "Ubuntu repository is mirroring now. It'll keep up to 100 GB disk space"
  echo "and take few hours (depend on your network connection)"
}
main ()
{
  preparation
  install_dhcp
  install_dns
  install_cobbler
}

main

Here you can find the configs for the script:

configs/etc/cobbler/settings:

---
# cobbler settings file
# restart cobblerd and run "cobbler sync" after making changes
# This config file is in YAML 1.0 format
# see http://yaml.org
# ==========================================================
# if 1, cobbler will allow insertions of system records that duplicate
# the --dns-name information of other system records.  In general,
# this is undesirable and should be left 0.
allow_duplicate_hostnames: 0

# if 1, cobbler will allow insertions of system records that duplicate
# the ip address information of other system records.  In general,
# this is undesirable and should be left 0.
allow_duplicate_ips: 0

# if 1, cobbler will allow insertions of system records that duplicate
# the mac address information of other system records.  In general,
# this is undesirable.
allow_duplicate_macs: 0

# if 1, cobbler will allow settings to be changed dynamically without
# a restart of the cobblerd daemon. You can only change this variable
# by manually editing the settings file, and you MUST restart cobblerd
# after changing it.
allow_dynamic_settings: 0

# by default, installs are *not* set to send installation logs to the cobbler
# # # server.  With 'anamon_enabled', kickstart templates may use the pre_anamon
# # # snippet to allow remote live monitoring of their installations from the
# # # cobbler server.  Installation logs will be stored under
# # # /var/log/cobbler/anamon/.  NOTE: This does allow an xmlrpc call to send logs
# # # to this directory, without authentication, so enable only if you are
# # # ok with this limitation.
anamon_enabled: 0

# If using authn_pam in the modules.conf, this can be configured
# to change the PAM service authentication will be tested against.
# The default value is "login".
authn_pam_service: "login"

# How long the authentication token is valid for, in seconds
auth_token_expiration: 3600

# Email out a report when cobbler finishes installing a system.
# enabled: set to 1 to turn this feature on
# sender: optional
# email: which addresses to email
# smtp_server: used to specify another server for an MTA
# subject: use the default subject unless overridden
build_reporting_enabled: 0
build_reporting_sender: ""
build_reporting_email: [ 'root@localhost' ]
build_reporting_smtp_server: "localhost"
build_reporting_subject: ""
build_reporting_ignorelist: [ "" ]

# Cheetah-language kickstart templates can import Python modules.
# while this is a useful feature, it is not safe to allow them to
# import anything they want. This whitelists which modules can be
# imported through Cheetah.  Users can expand this as needed but
# should never allow modules such as subprocess or those that
# allow access to the filesystem as Cheetah templates are evaluated
# by cobblerd as code.
cheetah_import_whitelist:
 - "random"
 - "re"
 - "time"

# Default createrepo_flags to use for new repositories. If you have
# createrepo >= 0.4.10, consider "-c cache --update -C", which can
# dramatically improve your "cobbler reposync" time.  "-s sha"
# enables working with Fedora repos from F11/F12 from EL-4 or
# EL-5 without python-hashlib installed (which is not available
# on EL-4)
createrepo_flags: "-c cache -s sha"

# if no kickstart is specified to profile add, use this template
default_kickstart: /var/lib/cobbler/kickstarts/default.ks

# configure all installed systems to use these nameservers by default
# unless defined differently in the profile.  For DHCP configurations
# you probably do /not/ want to supply this.
default_name_servers: []

# if using the authz_ownership module (see the Wiki), objects
# created without specifying an owner are assigned to this
# owner and/or group.  Can be a comma seperated list.
default_ownership:
 - "admin"

# cobbler has various sample kickstart templates stored
# in /var/lib/cobbler/kickstarts/.  This controls
# what install (root) password is set up for those
# systems that reference this variable.  The factory
# default is "cobbler" and cobbler check will warn if
# this is not changed.
# The simplest way to change the password is to run
# openssl passwd -1
# and put the output between the "" below.
default_password_crypted: "$1$ti7vIgk2$d1ZurbDd9nV61u7LWSLi90"

# the default template type to use in the absence of any
# other detected template. If you do not specify the template
# with '#template=<template_type>' on the first line of your
# templates/snippets, cobbler will assume try to use the
# following template engine to parse the templates.
#
# Current valid values are: cheetah, jinja2
default_template_type: "cheetah"

# for libvirt based installs in koan, if no virt bridge
# is specified, which bridge do we try?  For EL 4/5 hosts
# this should be xenbr0, for all versions of Fedora, try
# "virbr0".  This can be overriden on a per-profile
# basis or at the koan command line though this saves
# typing to just set it here to the most common option.
default_virt_bridge: virbr0

# use this as the default disk size for virt guests (GB)
default_virt_file_size: 5

# use this as the default memory size for virt guests (MB)
default_virt_ram: 512

# if koan is invoked without --virt-type and no virt-type
# is set on the profile/system, what virtualization type
# should be assumed?  Values: xenpv, xenfv, qemu, vmware
# (NOTE: this does not change what virt_type is chosen by import)
default_virt_type: qemu

# enable gPXE booting? Enabling this option will cause cobbler
# to copy the undionly.kpxe file to the tftp root directory,
# and if a profile/system is configured to boot via gpxe it will
# chain load off pxelinux.0.
# Default: 0
enable_gpxe: 0

# controls whether cobbler will add each new profile entry to the default
# PXE boot menu.  This can be over-ridden on a per-profile
# basis when adding/editing profiles with --enable-menu=0/1.  Users
# should ordinarily leave this setting enabled unless they are concerned
# with accidental reinstalls from users who select an entry at the PXE
# boot menu.  Adding a password to the boot menus templates
# may also be a good solution to prevent unwanted reinstallations
enable_menu: 1

# enable Func-integration?  This makes sure each installed machine is set up
# to use func out of the box, which is a powerful way to script and control
# remote machines.
# Func lives at http://fedorahosted.org/func
# read more at https://github.com/cobbler/cobbler/wiki/Func-integration
# you will need to mirror Fedora/EPEL packages for this feature, so see
# https://github.com/cobbler/cobbler/wiki/Manage-yum-repos if you want cobbler
# to help you with this
func_auto_setup: 0
func_master: overlord.example.org

# change this port if Apache is not running plaintext on port
# 80.  Most people can leave this alone.
http_port: 80

# kernel options that should be present in every cobbler installation.
# kernel options can also be applied at the distro/profile/system
# level.
kernel_options:
 ksdevice: bootif
 lang: ' '
 text: ~

# s390 systems require additional kernel options in addition to the
# above defaults
kernel_options_s390x:
 RUNKS: 1
 ramdisk_size: 40000
 root: /dev/ram0
 ro: ~
 ip: off
 vnc: ~

# configuration options if using the authn_ldap module. See the
# the Wiki for details.  This can be ignored if you are not using
# LDAP for WebUI/XMLRPC authentication.
ldap_server: "ldap.example.com"
ldap_base_dn: "DC=example,DC=com"
ldap_port: 389
ldap_tls: 1
ldap_anonymous_bind: 1
ldap_search_bind_dn: ''
ldap_search_passwd: ''
ldap_search_prefix: 'uid='
ldap_tls_cacertfile: ''
ldap_tls_keyfile: ''
ldap_tls_certfile: ''

# cobbler has a feature that allows for integration with config management
# systems such as Puppet.  The following parameters work in conjunction with
# --mgmt-classes  and are described in furhter detail at:
# https://github.com/cobbler/cobbler/wiki/Using-cobbler-with-a-configuration-management-system
mgmt_classes: []
mgmt_parameters:
 from_cobbler: 1

# if enabled, this setting ensures that puppet is installed during
# machine provision, a client certificate is generated and a
# certificate signing request is made with the puppet master server
puppet_auto_setup: 0

# when puppet starts on a system after installation it needs to have
# its certificate signed by the puppet master server. Enabling the
# following feature will ensure that the puppet server signs the
# certificate after installation if the puppet master server is
# running on the same machine as cobbler. This requires
# puppet_auto_setup above to be enabled
sign_puppet_certs_automatically: 0

# location of the puppet executable, used for revoking certificates
puppetca_path: "/usr/bin/puppet"

# when a puppet managed machine is reinstalled it is necessary to
# remove the puppet certificate from the puppet master server before a
# new certificate is signed (see above). Enabling the following
# feature will ensure that the certificate for the machine to be
# installed is removed from the puppet master server if the puppet
# master server is running on the same machine as cobbler. This
# requires puppet_auto_setup above to be enabled
remove_old_puppet_certs_automatically: 0

# choose a --server argument when running puppetd/puppet agent during kickstart
#puppet_server: 'puppet'

# let cobbler know that you're using a newer version of puppet
# choose version 3 to use: 'puppet agent'; version 2 uses status quo: 'puppetd'
#puppet_version: 2

# choose whether to enable puppet parameterized classes or not.
# puppet versions prior to 2.6.5 do not support parameters
#puppet_parameterized_classes: 1

# set to 1 to enable Cobbler's DHCP management features.
# the choice of DHCP management engine is in /etc/cobbler/modules.conf
manage_dhcp: 1

# set to 1 to enable Cobbler's DNS management features.
# the choice of DNS mangement engine is in /etc/cobbler/modules.conf
manage_dns: 1

# set to path of bind chroot to create bind-chroot compatible bind
# configuration files.  This should be automatically detected.
bind_chroot_path: ""

# set to the ip address of the master bind DNS server for creating secondary
# bind configuration files
bind_master: 127.0.0.1

# set to 1 to enable Cobbler's TFTP management features.
# the choice of TFTP mangement engine is in /etc/cobbler/modules.conf
manage_tftpd: 1

# set to 1 to enable Cobbler's RSYNC management features.
manage_rsync: 0

# if using BIND (named) for DNS management in /etc/cobbler/modules.conf
# and manage_dns is enabled (above), this lists which zones are managed
# See the Wiki (https://github.com/cobbler/cobbler/wiki/Dns-management) for more info
manage_forward_zones: ['COBBLER_DNS_DOMAIN']
manage_reverse_zones: ['10.50']

# if using cobbler with manage_dhcp, put the IP address
# of the cobbler server here so that PXE booting guests can find it
# if you do not set this correctly, this will be manifested in TFTP open timeouts.
next_server: PXE_ADDRESS

# settings for power management features.  optional.
# see https://github.com/cobbler/cobbler/wiki/Power-management to learn more
# choices (refer to codes.py):
#    apc_snmp bladecenter bullpap drac ether_wake ilo integrity
#    ipmilan ipmitool lpar rsa virsh wti
power_management_default_type: 'ipmitool'

# the commands used by the power management module are sourced
# from what directory?
power_template_dir: "/etc/cobbler/power"

# if this setting is set to 1, cobbler systems that pxe boot
# will request at the end of their installation to toggle the
# --netboot-enabled record in the cobbler system record.  This eliminates
# the potential for a PXE boot loop if the system is set to PXE
# first in it's BIOS order.  Enable this if PXE is first in your BIOS
# boot order, otherwise leave this disabled.   See the manpage
# for --netboot-enabled.
pxe_just_once: 1

# the templates used for PXE config generation are sourced
# from what directory?
pxe_template_dir: "/etc/cobbler/pxe"

# Path to where system consoles are
consoles: "/var/consoles"

# Are you using a Red Hat management platform in addition to Cobbler?
# Cobbler can help you register to it.  Choose one of the following:
#   "off"    : I'm not using Red Hat Network, Satellite, or Spacewalk
#   "hosted" : I'm using Red Hat Network
#   "site"   : I'm using Red Hat Satellite Server or Spacewalk
# You will also want to read: https://github.com/cobbler/cobbler/wiki/Tips-for-RHN
redhat_management_type: "off"

# if redhat_management_type is enabled, choose your server
#   "management.example.org" : For Satellite or Spacewalk
#   "xmlrpc.rhn.redhat.com"  : For Red Hat Network
# This setting is also used by the code that supports using Spacewalk/Satellite users/passwords
# within Cobbler Web and Cobbler XMLRPC.  Using RHN Hosted for this is not supported.
# This feature can be used even if redhat_management_type is off, you just have
# to have authn_spacewalk selected in modules.conf
redhat_management_server: "xmlrpc.rhn.redhat.com"

# specify the default Red Hat authorization key to use to register
# system.  If left blank, no registration will be attempted.  Similarly
# you can set the --redhat-management-key to blank on any system to
# keep it from trying to register.
redhat_management_key: ""

# if using authn_spacewalk in modules.conf to let cobbler authenticate
# against Satellite/Spacewalk's auth system, by default it will not allow per user
# access into Cobbler Web and Cobbler XMLRPC.
# in order to permit this, the following setting must be enabled HOWEVER
# doing so will permit all Spacewalk/Satellite users of certain types to edit all
# of cobbler's configuration.
# these roles are:  config_admin and org_admin
# users should turn this on only if they want this behavior and
# do not have a cross-multi-org seperation concern.  If you have
# a single org in your satellite, it's probably safe to turn this
# on and then you can use CobblerWeb alongside a Satellite install.
redhat_management_permissive: 0

# if set to 1, allows /usr/bin/cobbler-register (part of the koan package)
# to be used to remotely add new cobbler system records to cobbler.
# this effectively allows for registration of new hardware from system
# records.
register_new_installs: 0

# Flags to use for yum's reposync.  If your version of yum reposync
# does not support -l, you may need to remove that option.
reposync_flags: "-l -n -d"

# when DHCP and DNS management are enabled, cobbler sync can automatically
# restart those services to apply changes.  The exception for this is
# if using ISC for DHCP, then omapi eliminates the need for a restart.
# omapi, however, is experimental and not recommended for most configurations.
# If DHCP and DNS are going to be managed, but hosted on a box that
# is not on this server, disable restarts here and write some other
# script to ensure that the config files get copied/rsynced to the destination
# box.  This can be done by modifying the restart services trigger.
# Note that if manage_dhcp and manage_dns are disabled, the respective
# parameter will have no effect.  Most users should not need to change
# this.
restart_dns: 1
restart_dhcp: 1

# install triggers are scripts in /var/lib/cobbler/triggers/install
# that are triggered in kickstart pre and post sections.  Any
# executable script in those directories is run.  They can be used
# to send email or perform other actions.  They are currently
# run as root so if you do not need this functionality you can
# disable it, though this will also disable "cobbler status" which
# uses a logging trigger to audit install progress.
run_install_triggers: 1

# enables a trigger which version controls all changes to /var/lib/cobbler
# when add, edit, or sync events are performed.  This can be used
# to revert to previous database versions, generate RSS feeds, or for
# other auditing or backup purposes. "git" and "hg" are currently suported,
# but git is the recommend SCM for use with this feature.
scm_track_enabled: 0
scm_track_mode: "git"

# this is the address of the cobbler server -- as it is used
# by systems during the install process, it must be the address
# or hostname of the system as those systems can see the server.
# if you have a server that appears differently to different subnets
# (dual homed, etc), you need to read the --server-override section
# of the manpage for how that works.
server: PXE_ADDRESS

# If set to 1, all commands will be forced to use the localhost address
# instead of using the above value which can force commands like
# cobbler sync to open a connection to a remote address if one is in the
# configuration and would traceback.
client_use_localhost: 0

# If set to 1, all commands to the API (not directly to the XMLRPC
# server) will go over HTTPS instead of plaintext. Be sure to change
# the http_port setting to the correct value for the web server
client_use_https: 0

# this is a directory of files that cobbler uses to make
# templating easier.  See the Wiki for more information.  Changing
# this directory should not be required.
snippetsdir: /var/lib/cobbler/snippets

# Normally if a kickstart is specified at a remote location, this
# URL will be passed directly to the kickstarting system, thus bypassing
# the usual snippet templating Cobbler does for local kickstart files. If
# this option is enabled, Cobbler will fetch the file contents internally
# and serve a templated version of the file to the client.
template_remote_kickstarts: 0

# should new profiles for virtual machines default to auto booting with the physical host when the physical host reboots?
# this can be overridden on each profile or system object.
virt_auto_boot: 1

# cobbler's web directory.  Don't change this setting -- see the
# Wiki on "relocating your cobbler install" if your /var partition
# is not large enough.
webdir: /srv/www/cobbler

# cobbler's public XMLRPC listens on this port.  Change this only
# if absolutely needed, as you'll have to start supplying a new
# port option to koan if it is not the default.
xmlrpc_port: 25151

# "cobbler repo add" commands set cobbler up with repository
# information that can be used during kickstart and is automatically
# set up in the cobbler kickstart templates.  By default, these
# are only available at install time.  To make these repositories
# usable on installed systems (since cobbler makes a very convient)
# mirror, set this to 1.  Most users can safely set this to 1.  Users
# who have a dual homed cobbler server, or are installing laptops that
# will not always have access to the cobbler server may wish to leave
# this as 0.  In that case, the cobbler mirrored yum repos are still
# accessable at http://cobbler.example.org/cblr/repo_mirror and yum
# configuration can still be done manually.  This is just a shortcut.
yum_post_install_mirror: 1

# the default yum priority for all the distros.  This is only used
# if yum-priorities plugin is used.  1=maximum.  Tweak with caution.
yum_distro_priority: 1

# Flags to use for yumdownloader.  Not all versions may support
# --resolve.
yumdownloader_flags: "--resolve"

# sort and indent JSON output to make it more human-readable
serializer_pretty_json: 0

# replication rsync options for distros, kickstarts, snippets set to override default value of "-avzH"
replicate_rsync_options: "-avzH"

# replication rsync options for repos set to override default value of "-avzH"
replicate_repo_rsync_options: "-avzH"

# always write DHCP entries, regardless if netboot is enabled
always_write_dhcp_entries: 0

# external proxy - used by: get-loaders, reposync, signature update
# eg: proxy_url_ext: "http://192.168.1.1:8080"
proxy_url_ext: ""

# internal proxy - used by systems to reach cobbler for kickstarts
# eg: proxy_url_int: "http://10.0.0.1:8080"
proxy_url_int: ""

configs/etc/cobbler/dhcp.template:

# ******************************************************************
# Cobbler managed dhcpd.conf file
#
# generated from cobbler dhcp.conf template ($date)
# Do NOT make changes to /etc/dhcpd.conf. Instead, make your changes
# in /etc/cobbler/dhcp.template, as /etc/dhcpd.conf will be
# overwritten.
#
# ******************************************************************

ddns-update-style interim;

allow booting;
allow bootp;

ignore client-updates;
set vendorclass = option vendor-class-identifier;

option pxe-system-type code 93 = unsigned integer 16;
option space pxelinux;
option pxelinux.magic      code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;

subnet COBBLER_NET netmask COBBLER_NETMASK {
     interface                  p1p1;
     option routers             $server;
     option domain-name-servers $server;
     option domain-search       "COBBLER_DNS_DOMAIN";
     option subnet-mask         COBBLER_NETMASK;
     range dynamic-bootp        COBBLER_DYN_RANGE;
     default-lease-time         21600;
     max-lease-time             43200;
     next-server                $next_server;
}

#for dhcp_tag in $dhcp_tags.keys():
    ## group could be subnet if your dhcp tags line up with your subnets
    ## or really any valid dhcpd.conf construct ... if you only use the
    ## default dhcp tag in cobbler, the group block can be deleted for a
    ## flat configuration
    #if $dhcp_tag == "default":
 #group for Cobbler DHCP tag: $dhcp_tag
group {
        #for mac in $dhcp_tags[$dhcp_tag].keys():
            #set iface = $dhcp_tags[$dhcp_tag][$mac]
    host $iface.name {
        hardware ethernet $mac;
        #if $iface.ip_address:
        fixed-address $iface.ip_address;
        #end if
        #if $iface.hostname:
        option host-name "$iface.hostname";
        #end if
        #if $iface.netmask:
        option subnet-mask $iface.netmask;
        #end if
        #if $iface.gateway:
        option routers $iface.gateway;
        #end if
        filename "lpxelinux.0";
        ## Cobbler defaults to $next_server, but some users
        ## may like to use $iface.system.server for proxied setups
        next-server $iface.next_server;
    }
        #end for
}
    #else
 #group for Cobbler DHCP tag: $dhcp_tag
group {
        #for mac in $dhcp_tags[$dhcp_tag].keys():
            #set iface = $dhcp_tags[$dhcp_tag][$mac]
    host $iface.name {
        hardware ethernet $mac;
        #if $iface.ip_address:
        fixed-address $iface.ip_address;
        #end if
        #if $iface.hostname:
        option host-name "$iface.hostname";
        #end if
        #if $iface.netmask:
        option subnet-mask $iface.netmask;
        #end if
        #if $iface.gateway:
        option routers $iface.gateway;
        #end if
        filename "lpxelinux.0";
        ## Cobbler defaults to $next_server, but some users
        ## may like to use $iface.system.server for proxied setups
        next-server $iface.next_server;
    }
        #end for
}
#group for Cobbler DHCP tag: $dhcp_tag but for native vlan
group {
       #for mac in $dhcp_tags[$dhcp_tag].keys():
           #set iface = $dhcp_tags[$dhcp_tag][$mac]
   host $iface.name-native-vlan {
       hardware ethernet $mac;
       if exists user-class and option user-class = "iPXE" {
           filename "http://$next_server/cblr/ks_mirror/config/vlan-$dhcp_tag";
       } else {
           filename "ipxe.kpxe";
       }
       ## Cobbler defaults to $next_server, but some users
       ## may like to use $iface.system.server for proxied setups
       next-server $next_server;
   }
       #end for
}
   #end if
#end for

configs/etc/cobbler/pxe/pxesystem.template:

default linux
prompt 0
timeout 1
label linux
        kernel http://$server:$http_port/cblr/$kernel_path
        append initrd=http://$server:$http_port/cblr/$initrd_path $append_line

configs/etc/cobbler/pxe/pxelocal.template:

default local
prompt 0
timeout 0
totaltimeout 0
ontimeout 0
label local
        kernel http://$server:$http_port/cblr/tftpboot/chain.c32
        append hd

configs/var/lib/cobbler/kickstarts/sample.seed:

# Mostly based on the Ubuntu installation guide
# https://help.ubuntu.com/12.04/installation-guide/

# Preseeding only locale sets language, country and locale.
d-i debian-installer/locale string en_US

# Keyboard selection.
# Disable automatic (interactive) keymap detection.
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layoutcode string us
d-i keyboard-configuration/variantcode string

# netcfg will choose an interface that has link if possible. This makes it
# skip displaying a list if there is more than one interface.
#set $myhostname = $getVar('hostname',$getVar('name','cobbler')).replace("_","-")
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string $myhostname

# If non-free firmware is needed for the network or other hardware, you can
# configure the installer to always try to load it, without prompting. Or
# change to false to disable asking.
# d-i hw-detect/load_firmware boolean true

# NTP/Time Setup
d-i time/zone string US/Eastern
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server  string $server

# Setup the installation source
d-i mirror/country string manual
d-i mirror/http/hostname string $http_server
d-i mirror/http/directory string $install_source_directory
d-i mirror/http/proxy string

#set $os_v = $getVar('os_version','')
#if $os_v and $os_v.lower()[0] > 'p'
# Required at least for 12.10+
# d-i live-installer/net-image string http://$http_server/cobbler/links/$distro_name/install/filesystem.squashfs
#end if

# Suite to install.
# d-i mirror/suite string precise
# d-i mirror/udeb/suite string precise

# Components to use for loading installer components (optional).
#d-i mirror/udeb/components multiselect main, restricted

# Disk Partitioning
# Use LVM, and wipe out anything that already exists
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-partitioning/confirm_write_new_label boolean true

d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string regular
partman-auto partman-auto/init_automatically_partition select Guided - use entire disk
partman-auto partman-auto/automatically_partition select
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# You can choose one of the three predefined partitioning recipes:
# - atomic: all files in one partition
# - home:   separate /home partition
# - multi:  separate /home, /usr, /var, and /tmp partitions
#d-i partman-auto/choose_recipe select atomic

# If you just want to change the default filesystem from ext3 to something
# else, you can do that without providing a full recipe.
# d-i partman/default_filesystem string ext4

# root account and password
d-i passwd/root-login boolean true
d-i passwd/root-password-crypted password $default_password_crypted

# skip creation of a normal user account.
d-i passwd/make-user boolean false

# You can choose to install restricted and universe software, or to install
# software from the backports repository.
# d-i apt-setup/restricted boolean true
# d-i apt-setup/universe boolean true
# d-i apt-setup/backports boolean true

# Uncomment this if you don't want to use a network mirror.
# d-i apt-setup/use_mirror boolean false

# Select which update services to use; define the mirrors to be used.
# Values shown below are the normal defaults.
d-i apt-setup/services-select multiselect security
d-i apt-setup/security_host string $server
d-i apt-setup/security_path string /cblr/repo_mirror/ubuntu14-x86_64
# d-i apt-setup/local0/repository string http://$http_server/ubuntu_custom_packages ./

$SNIPPET('preseed_apt_repo_config')

# Enable deb-src lines
# d-i apt-setup/local0/source boolean true

# URL to the public key of the local repository; you must provide a key or
# apt will complain about the unauthenticated repository and so the
# sources.list line will be left commented out
# d-i apt-setup/local0/key string http://local.server/key

# By default the installer requires that repositories be authenticated
# using a known gpg key. This setting can be used to disable that
# authentication. Warning: Insecure, not recommended.
d-i debian-installer/allow_unauthenticated boolean true

# Individual additional packages to install
# wget is REQUIRED otherwise quite a few things won't work
# later in the build (like late-command scripts)
d-i pkgsel/include string ntp ssh wget

# Use the following option to add additional boot parameters for the
# installed system (if supported by the bootloader installer).
# Note: options passed to the installer will be added automatically.
d-i debian-installer/add-kernel-opts string $kernel_options_post

# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note

## Figure out if we're kickstarting a system or a profile
#if $getVar('system_name','') != ''
#set $what = "system"
#else
#set $what = "profile"
#end if

# This first command is run as early as possible, just after preseeding is read.
# d-i preseed/early_command string [command]
d-i preseed/early_command string wget -O- \
   http://$http_server/cblr/svc/op/script/$what/$name/?script=preseed_early_default | \
   /bin/sh -s

# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
# d-i partman/early_command \
#       string debconf-set partman-auto/disk "\$(list-devices disk | head -n1)"

# This command is run just before the install finishes, but when there is
# still a usable /target directory. You can chroot to /target and use it
# directly, or use the apt-install and in-target commands to easily install
# packages and run commands in the target system.
# d-i preseed/late_command string [command]
d-i preseed/late_command string wget -O- \
   http://$http_server/cblr/svc/op/script/$what/$name/?script=preseed_late_default | \
   chroot /target /bin/sh -s

1.5.2. Script to start provisioning and collect metrics

#!/bin/bash -ex

REQUESTED_NODES=$1
ENV_NAME=cobbler-test
INTERFACE=p1p1
DSTAT_OUTPUT_FILE=/var/log/dstat.csv
RESULTS_FILE=/var/log/results.csv

# Need to install the required packages on provisioning system servers:
if (("`dpkg -l | grep dstat | grep ^ii > /dev/null; echo $?` == 1"))
then
  apt-get -y install dstat bc
fi

# Release all nodes from the environment and disable net booting
for SYSTEM in `cobbler system find --comment=${ENV_NAME}`
do
  cobbler system edit --name ${SYSTEM} --comment= --netboot-enabled=no
done

# Check if we have anought nodes
if [ `cobbler system find --comment= | wc -l` < ${REQUESTED_NODES} ]
then
  echo "You have less nodes then requested" | tee -a ${RESULTS_FILE}
  exit 1
fi

# Add requested number of nodes to the env and enable net booting
for SYSTEM in `cobbler system find --comment= | head -${REQUESTED_NODES}`
do
  cobbler system edit --name=${SYSTEM} --comment=${ENV_NAME} --netboot-enabled=yes
done
cobbler sync

# Need to prepare the following script on provisioning system server to collect
# values of CPU,RAM,NET and IO loads per second. You need to change "INTERFACE"
# variable regarding the interface which connected to nodes to communicare with
# them during provisioning process. As a result of this command we'll get
# running in backgroud dstat programm which collecting needed parametes in CSV
# format into /var/log/dstat.log file.:
rm -f ${DSTAT_OUTPUT_FILE}
dstat --nocolor --time --cpu --mem --net -N ${INTERFACE} --io --output ${DSTAT_OUTPUT_FILE} > /dev/null &

# Need to prepare script which starts provisioning process and gets the time when
# provisioning started and when provisioning ended ( when all nodes reachable via
# ssh). We'll analyze results collected during this time window. For getting
# start time we can add "date" command before API call or CLI command and forward
# the output of the command to some log file. Here is example for cobbler:
start_time=`date +%s.%N`
echo "Provisioning started at "`date` > ${RESULTS_FILE}
for SYSTEM in `cobbler system find --comment=${ENV_NAME}`
do
  cobbler system reboot --name=${SYSTEM} &
done

# For getting end-time we can use the script below. This script tries to reach
# nodes via ssh and write "Provisioning finished at <date/time>" into
# /var/log/provisioning.log file. You'll need to provide ip addresses of the
# nodes (from file nodes_ips.list, where IPs listed one per line) and
# creadentials (SSH_PASSWORD and SSH_USER variables):
SSH_OPTIONS="StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
SSH_PASSWORD="r00tme"
SSH_USER="root"
unset NODE_IPS[@]
NODE_IPS=()
for SYSTEM in `cobbler system find --comment=cobbler-test`
do
  NODE_IPS+=(`cobbler system dumpvars --name=${SYSTEM} | grep -w ip_address_${INTERFACE} | awk -F": " '{print $2}'`)
done
TIMER=0
TIMEOUT=50
while (("${TIMER}" < "${TIMEOUT}"))
do
     sleep 10
     for ARRAY_ELEMENT_NUM in $(seq 0 ${#NODE_IPS[@]})
     do
             SSH_CMD="sshpass -p ${SSH_PASSWORD} ssh -o ${SSH_OPTIONS} ${SSH_USER}@${NODE_IPS[${ARRAY_ELEMENT_NUM}]}"
             ${SSH_CMD} "hostname" && UNHAPPY_SSH=0 || UNHAPPY_SSH=1
             if (("${UNHAPPY_SSH}" == "0"))
             then
                     echo "Node with ip "${NODE_IPS[${ARRAY_ELEMENT_NUM}]}" is reachable via ssh"
                     unset NODE_IPS[${ARRAY_ELEMENT_NUM}] && NODE_IPS=(${NODE_IPS[@]})
             else
                     echo "Node with ip "${NODE_IPS[${ARRAY_ELEMENT_NUM}]}" is still unreachable via ssh"
             fi
      done
      TIMER=$((${TIMER} + 1))
      if (("${TIMER}" == "${TIMEOUT}"))
      then
              echo "The following "${#NODE_IPS[@]}" are unreachable" | tee -a ${RESULTS_FILE}
              echo ${NODE_IPS[@]} | tee -a ${RESULTS_FILE}
              break 
      fi
      if ((${#NODE_IPS[@]} == 0 ))
      then
              break
      fi
      # Check that nodes are reachable once per 1 seconds
      sleep 1
done
echo "Provisioning finished at "`date` >> ${RESULTS_FILE}

end_time=`date +%s.%N`
elapsed_time=$(echo "$end_time - $start_time" | bc -l)
echo "Total elapsed time for provisioning: $elapsed_time seconds" >> ${RESULTS_FILE}

# Stop dstat command
kill `ps aux | grep dstat | grep python | awk '{print $2}'`

# Delete excess values and convert to needed metrics. So, we'll get the
# following csv format:
# time,cpu_usage,ram_usage,net_recv,net_send,net_all,dsk_io_read,dsk_io_writ,dsk_all
awk -F "," 'BEGIN {getline;getline;getline;getline;getline;getline;getline;
                   print "time,cpu_usage,ram_usage,net_recv,net_send,net_all,dsk_io_read,dsk_io_writ,dsk_all"}
            {print $1","100-$4","$8/1048576","$12/131072","$13/131072","($12+$13)/131072","$14","$15","$14+$15}' \
$DSTAT_OUTPUT_FILE >> ${RESULTS_FILE}

1.5.3. Script to add cobbler systems

#!/bin/bash

DNS_DOMAIN="cobbler-test.local"
PROFILE="ubuntu14-x86_64"
ENV_NAME="cobbler-test"
INTERFACE_1="p1p1"
IP_RANGE_1="10.50.11.1 10.50.20.254"
NETMASK_1="255.255.0.0"
GATEWAY_1="10.50.0.10"
DNS1="10.50.0.10"

SYSTEM_IPMI_USER="root"
SYSTEM_IPMI_PASS="calvincalvin"

SYSTEMS_LIST_FILE="systems.list"
SYSTEMS_COUNT=`wc -l ${SYSTEMS_LIST_FILE} | awk '{print $1}'`

for EXISTED_SYSTEM in `cobbler system list`
do
  EXISTED_IP_ADDRESSES=(`cobbler system dumpvars --name=${EXISTED_SYSTEM} | grep ^ip_address | awk -F": " '{print $2}'`)
done

IP_ADDRESSES=(`prips ${IP_RANGE_1}`)

for IP in ${EXISTED_IP_ADDRESSES[@]}
do
  for ARRAY_ELEMENT_NUM in $(seq 0 ${#IP_ADDRESSES[@]})
  do
    if [ "${IP_ADDRESSES[${ARRAY_ELEMENT_NUM}]}" == "${IP}" ]
    then
       unset IP_ADDRESSES[${ARRAY_ELEMENT_NUM}] && IP_ADDRESSES=(${IP_ADDRESSES[@]})
    fi
  done
done


for SYSTEM_NUM in $(seq 1 ${SYSTEMS_COUNT})
do
  SYSTEM_NAME=`awk -F"," '{print $1}' ${SYSTEMS_LIST_FILE} | head -${SYSTEM_NUM} | tail -1`
  SYSTEM_MAC=`awk -F"," '{print $3}' ${SYSTEMS_LIST_FILE} | head -${SYSTEM_NUM} | tail -1`
  SYSTEM_IPMI_ADDRESS=`awk -F"," '{print $2}' ${SYSTEMS_LIST_FILE} | head -${SYSTEM_NUM} | tail -1`
  NODE_IP_1=${IP_ADDRESSES[${SYSTEM_NUM}]}
  cobbler system add --name=${SYSTEM_NAME} \
                     --hostname=${SYSTEM_NAME} \
                     --dns-name=${SYSTEM_NAME}.${DNS_DOMAIN} \
                     --name-servers-search=${DNS_DOMAIN} \
                     --profile=${PROFILE} \
                     --comment=${ENV_NAME}\
                     --netboot-enabled=yes \
                     --interface=${INTERFACE_1} \
                     --mac-address=${SYSTEM_MAC} \
                     --ip-address=${NODE_IP_1} \
                     --netmask=${NETMASK_1} \
                     --virt-bridge=pxe \
                     --virt-file-size=100 \
                     --power-type=ipmilan \
                     --power-address=${SYSTEM_IPMI_ADDRESS} \
                     --power-user=${SYSTEM_IPMI_USER} \
                     --power-pass=${SYSTEM_IPMI_PASS} \
                     --power-id=lanplus \
                     --kickstart=/var/lib/cobbler/kickstarts/sample.seed
done