Requirements¶

Before running a deployment, there are some requirements to understand the way Spinal Stack works do be deployed.

The minimal requirements to install Spinal Stack are:

Build eDeploy roles¶

To build eDeploy roles by yourself on your eDeploy server, you can run:

$ export ROLES="openstack-full install-server"
$ export VIRTUALIZED=/srv/edeploy/tools/virt.conf
$ /srv/edeploy/tools/jenkins-build.sh <src dir> <build dir> <archive dir> [<make params>]

When making continuous integration, this step can be done by Jenkins (upstream) which will distribute eDeploy roles on the installation servers (downstream).

Deploy the install-server role¶

This role contains all these components:

• eDeploy
• PXEmngr
• PuppetDB: running on port 8080 and a vhost acting as a proxy to terminate the SSL listening on this URL: http://install-server:8081
• Puppet Dashboard: monitor Puppet nodes using this URL: http://install-server:82
• Kibana3: View logs in real-time through UI using this URL: http://install-server:8300
• Jenkins: manage lifecyle of our infrastructure (configuration, sanity and upgrade) using this URL: http://install-server:8282

The first step, is to bootstrap the install-server by using eDeploy itself. There is several ways to do it and this manual explains them in details.

In the case you don’t have an eDeploy server to bootstrap the install-server node, you can use the local installation with an USB key method.

You have to follow eDeploy manual to finish the installation of eDeploy service on the install-server node. Once we have eDeploy working, we can continue to prepare the configuration.

Deploy the openstack-full roles¶

Once the install-server node is ready, we can deploy all our infrastructure by using eDeploy over the network. To start the deployment, boot the targeted server by using the proper boot device regarding the kind of deployment you choose (PXE versus USB).

puppet-openstack-cloud¶

puppet-openstack-cloud module is the Spinal Stack composition layer. All parameters and details are documented on this page.

Before deploying Spinal Stack, you should learn about the composition layer capabilities, so you will have the desired environment in your deployment scenario.

Infrastructure YAML files¶

Currently, there are two examples of infrastructure YAML file which work for 2 scenarios: The infrastructure YAML repository is flexible enough to support multiple scenarios.

In “scenario” directory, you will find differents scenarios that contain a “infra.yaml” file which will fit (or not) with your deployment. If none of these scenarios fits with your deployment, you have to create your own, respecting this YAML architecture:

• name: Name of the profile (ex: install-server)
• arity: Define a rule for the number of servers expected in the deployment (ex: 1+n means at least one server with this profile)
• edeploy: eDeploy role to use for the deployment. Usually, we use install-server role for installation server node and openstack-full role for all other nodes.
• steps/roles: contains the different steps of the deployments by defining for each step the Puppet classes expected to be applied on the nodes. It’s optionnal to define the steps for each profile.
• serverspec_config: parameters to make Spinal Stack serverspec tests working on the profiles.
• ordered_profiles: describe the order to run Puppet for each step. For example, if we want to run puppet on compute nodes before the controller.
• serverspec: Spinal Stack Puppet classes / Spinal Stack serverspec tests names assotiation

This directory is common to all scenarios:

data

data directory contains Hiera files. Spinal Stack is structured in 3 Hiera levels:

• FQDN (Full qualified domain name): parameters and classes for some hosts only. (ex: set VRRP priority on a load-balancer node in particular)
• type: defines some classes and parameters for each profile that we created in the infra YAML file
• common: common classes & parameters that will be applied on all the nodes of the deployment.

data/fqdn

Later, we will describe another YAML file which is the environment. In this file, we will see how Hiera will use at this level. FQDN hiera level is useful when we want to set specific parameters for a host (like the storage devices for Swift and Ceph, etc).

data/type

This file contains all the parameters that we pass to the different Puppet classes. config-tools will generate the different profiles with the corresponding Puppet classes for each profile. If we want to override the paramater of a class in particular, we can do this at the FQDN Hiera level. If there are too many modifications to do at this level, it may be because the deployment does not fit with this scenario and you may have to create a new infrastructure environment.

data/common

YAML file which will contain all parameters consummed by data/type to configure the Puppet classes. The actual data could be pulled from the environment file that we will write later (ex: config.mysql_root_password) or from common itself (ex: galera_ip: “%{hiera(‘vip_public_ip’)}”) by using Hiera syntax.

heat

OpenStack Heat (Orchestration) template that could be used to run the infrastructure in OpenStack.

Environment YAML file¶

The environment that we just talked in infrastructure is something specific for each deployment and has to be generated each time you deploy Spinal Stack.

There is an example of environment file here.

Let’s create this file (in the example it will be named joe.yml) and describe its structure: * infra: infrastructure name (ex: 3nodes)

For each host:

• hostname: hostname of the host
• profile: profile name used in infra.yaml in the infrastructure YAML file
• ip: IP used to reach the servers from internal network.

• config: contains FQDN Hiera data level parameters.

After hosts description:

• config: contains confidentials & deployment specific informations, like IP addresses, subnets, passwords, SSH keys, etc.

This file should be stored on a private versionned git repository since it contains confidential informations.

This is an example of a 3 nodes deployment: From the installation server, the configuration is generated by first creating this YAML file:

**env-joe.yaml**

module:
  git@github.com:stackforge/puppet-openstack-cloud
ansible:
  git@github.com:enovance/edeploy-roles
serverspec:
  git@github.com:enovance/openstack-serverspec.git
edeploy_repo:
  git@github.com:enovance/edeploy.git
environment:
  joe
infrastructure:
  git@github.com:enovance/openstack-yaml-infra.git
scenario:
  ref-arch
jenkins:
  git@github.com:enovance/jjb-openstack.git

We are now ready to generate the Spinal Stack configuration.

Generate the configuration¶

From the installation server, run:

$ git clone git@github.com:enovance/config-tools.git
$ cd config-tools
$ git checkout J.1.0.0
$ ./provision.sh -i J.1.0.0 git@my-env-git-repo:spinalstack-env/env-joe.yaml version=RH7.0-J.1.0.0

Run the configuration¶

From the installation server, connect to the Jenkins server by using this URL: http://install-server:8282 and run the job puppet.

Depending of the infrastructure size, the configuration can take 30 minutes or more.

During the configuration¶

1. Puppet master is being prepared by config-tools provision.sh script. It will take care of Puppet configuration, Hiera data dynamic management (according to the steps) and the install-server will be puppetized itself.
2. Then the deployment of OpenStack nodes is starting:
   • Step 1: Memcached, MySQL, MongoDB, RabbitMQ,i logging agent, Ceph monitor and Ceph Key management
   • Step 2: HAproxy, Ceph OSD, Horizon, Swift storage nodes
   • Step 3: Keystone users, tenants, services and endpoints; Create Ceph storage pools
   • Step 4: OpenStack API, schedulers and compute services
   • Step 5: OpenStack Networking services

For each step, Puppet is run on the nodes where the step is needed. Puppet is run until serverspec tests pass with a limit of 5 times. If after 5 times of Puppet run, the serverspec tests still fail, Jenkins job will fail and provide an output to let the deployer know which tests fail. The concept of steps make easy the debugging when something is wrong during the deployment process.

Javelin¶

Javelin is a tool that allows the creation, check and destruction of some OpenStack resources. We use it to validate that existing resources are able to survive a version upgrades. Here’s how it works:

1. Before running, Javelin will compare the current version of Spinal Stack with the latest version tested. The current version is provided by the tool edeploy and the latest version tested is fetched from disk. This will determine if Javelin is running before or after an upgrade process.
2. If Javelin is running before an upgrade (or if it’s a first run after the initial deployment), it will simply create a bunch of resources.
3. If Javelin is running after an upgrade (in other words, if the current version of Spinal Stack is superior to the latest version tested), Javelin will check for the existence of the OpenStack resources. If it cannot find them, Javelin considers that the resources didn’t survive the upgrade and will report an error.
4. After every run, Javelin will update the latest tested version on disk with the current version of Spinal Stack.

The latest version tested by Javelin can be found on the install server, in the file: /opt/tempest-scripts/javelin-latest-run

The resources manipulated by Javelin are described in a Yaml file managed by the Sanity suite. As of the J.1.1.0 release, here are the resources:

Tempest¶

Tempest is the suite of functional tests that is used in the gate to validate every API call against new developments. We use it to validate that the deployment went well. When Sanity is launched, it will take care of:

• Generating a tempest.conf relevant to the Spinal Stack deployment
• Create the users, tenants, roles and images that will be used in the tests
• Launch over 1600 functional tests, testing API and CLI
• Report the tests result in the Jenkins interface

A couple of things to note about Tempest:

• Tempest will create a bunch of resources during its tests. These resources will be cleaned up after Sanity, even in the case of tests failure.
• Sanity enables a feature of Tempest called “tenant isolation”, which means that every resource created by a test will be only visible inside its own tenant. This allows parallel execution of the tests while preventing potential race conditions.

Glance¶

glance image-create --name <name> --disk-format raw --container-format bare --file <path_to_img> --is-public True --is-protected True --progress

glance image-create --name Microsoft_Windows_Server_2012_R2 --disk-format raw --container-format bare --file <path_to_img> --is-public True --is-protected True --property hw_video_model=qxl --property hw_video_ram=64 --progress

glance image-update <image-id> --property hw_video_model=qxl --property hw_video_ram=6

nova flavor-key m1.microsoft.windows set hw_video:ram_max_mb=128

Kernel panic - not syncing: IO-APIC + timer doesn't work!

glance image-create --name GNU_Linux_CentOS_5.1 --disk-format raw --container-format bare --file <path_to_img> --is-public True --is-protected True --progress --property hw_disk_bus=ide --property hw_vif_model=rtl8139

Nova¶

nova flavor-delete 1
nova flavor-create m1.tiny 1 512 6 1

scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateInstanceExtraSpecsFilter

nova aggregate-create microsoft-windows

nova aggregate-add-host <aggregate> <server>

nova aggregate-set-metadata <microsoft-windows aggregate ID> windowshypervisors=true

nova flavor-create m1.microsoft.windows 6 4096 40 2

nova flavor-key m1.microsoft.windows set windowshypervisors=true

Neutron¶

neutron net-create public --router:external=True
neutron subnet-create public --name ext-subnet --allocation-pool start=$ALLOCATION_PUBLIC_POOL_START,end=ALLOCATION_PUBLIC_POOL_END --disable-dhcp --gateway $ALLOCATION_PUBLIC_POOL_GW 193.191.68

neutron net-create public --provider:network_type vlan --provider:physical_network public --provider:segmentation_id 100 --shared --router:external=True

neutron meter-label-create public-in
neutron meter-label-rule-create public-in 0.0.0.0/0 --direction ingress

neutron meter-label-create public-out
neutron meter-label-rule-create public-out 0.0.0.0/0 --direction egress

Cinder¶

cinder qos-create high-iops consumer="front-end" read_iops_sec=2000 write_iops_sec=1000
cinder type-create high-iops
cinder qos-associate c38d72f8 9c746ca5

Overview¶

Put an image here, with rows showing which part is https ans which part is not

Configuration¶

mkdir self-signed-cert && cd self-signed-cert
# change theses values to match your requirements
# country, 2 letters
country="my_country"
location="my_town"
organisation="my_organisation"
domain="my_domain.com"

# generate the CA
openssl req -days 3650 -out ca.pem -new -x509 -subj /C=$country/L=$location/O=$organisation

# generate a private key for your servers
openssl genrsa -out star_$domain.key 2048

# create csr
openssl req -key star_$domain.key -new -out star_$domain.req -subj /C=$country/L=$location/O=$organisation/CN=*.$domain
echo '00' > file.srl

# generate the crt file
openssl x509 -req -days 3650 -in star_$domain.req -CA ca.pem -CAkey privkey.pem -CAserial file.srl -out star_$domain.crt

# generate the pem file for haproxy
cat star_$domain.key star_$domain.crt > star_$domain.pem

# you can rename star_my_domain.com.* star_my_domain_com.* if needed

cp ca.pem /etc/pki/ca-trust/source/anchors/
update-ca-trust

---
cloud::loadbalancer::nova_bind_options:
  - ssl
  - crt
  - /patch/to/pem

---
cloud::identity::ks_nova_public_proto: https

---
cloud::identity::ks_nova_internal_proto: https

---
cloud::identity::ks_nova_admin_proto: https

Overview¶

Security-Enhanced Linux (SELinux) is a Linux kernel security module that provides a mechanism for supporting access control security policies, including United States Department of Defense–style mandatory access controls (MAC). Spinal Stack is able to enable SElinux in ‘enforced’ mode in a flexible way where you can configure your own booleans and modules.

Configuration¶

selinux_mode: enforcing
selinux_directory: /usr/share/selinux/custom/

---
cloud::selinux_booleans:
  - global_ssp
  - haproxy_connect_any
  - rsync_full_access
  - swift_can_network
  - nis_enabled
  - domain_kernel_load_modules
  - virt_use_execmem
  - httpd_can_network_connect
cloud::selinux_modules:
  - systemctl
  - nova-cert
  - nova-consoleauth
  - nova-scheduler
  - keepalived_haproxy
  - keepalived_init
  - mysql_rsync
  - swiftobject-sps

Overview¶

The firewalling implementation is very flexible and disabled by default. If you enable it without specific paramters, all the rules will be created depending of the services that are actually running. Example, if you have a Nova API node, this node will automatically have Nova API firewall rules (8774 tcp by default).

Configuration¶

manage_firewall: true

---
cloud::firewall_rules:
  '300 allow custom application 1':
    port: 999
    proto: udp
    action: accept
  '301 allow custom application 2':
    port: 8081
    proto: tcp
    action: accept

---
cloud::identity::firewall_settings:
  '301 limit keystone API requests':
      rate_limiting: '50/sec'

Provisioning¶

The first step of an upgrade is to re-provision the install-server. The procedure is already documented on Upgrade guide page. During this process, the install-server will update: * Puppet modules * Serverspec tests and configuration * eDeploy roles * Ansible playbooks (see next section for details)

Ansible playbooks¶

Ansible playbooks manage the upgrade orchestration, so we don’t have to worry about doing the process manually.

The playbooks are generated by config-tools during the provisioning step. Each Puppet class is associated with a playbook snippet.

For example, to manage Ceilometer Collector upgrade, we run:

- name: restart ceilometer-collector
  service: name={{ ceilometer_collector }} state=restarted
  tags: 7

You can find more information on snippets in the openstack-yaml-infra page.

For each profile (ie. controller/compute/load-balancer/network/storage) defined in your infra.yaml file, config-tools will build Ansible playbooks according to the Puppet classes you are running on each node. In other words, Ansible playbooks will be generated for every deployment so you don’t ever have to worry about doing an upgrade manually.

As of version J.1.1.0, the association between Puppet classes and Ansible playbooks is still done manually. We are working on a tool inside eDeploy to associate Puppet resources with eDeploy roles. That way, we will know what is actually updated and what we need to execute during the upgrade.

Example: if in the previous eDeploy role we detect an upgrade in Ceilometer collector, we will associate the ceilometer-collector snippet with the node where cloud::telemetry::collector is run.

The goal here is to upgrade Spinal Stack in a continuous way with automatic orchestration thanks to config-tools, eDeploy, Ansible and Puppet.

eDeploy upgrade¶

eDeploy updates each node by copying the roles from the install-server using rsync. Currently, eDeploy repository contains metadatas to upgrade roles release after release. The most important file is the ‘exclude’ file. It contains all the files/directories that we want to exclude from the upgrade. Basically, we exclude all the data so we don’t loose it during the upgrade. Once the rsync process is done, we execute a script called ‘post’. This script is automatically generated during the build of the eDeploy roles. It contains all actions executed by packaging during the build (create users, directories, etc). Since this script is executed after an upgrade, you don’t have to worry about new packages in the role, all packaging scripts are automatically executed and your system will be ready to use the service.

Puppet¶

After a run of the Ansible playbooks, we run step 0 and step 5 of Puppet. This is the usual process, so Puppet will be run in a maximum of 5 times with serverspec testing that everything is configured correctly.

Sanity¶

After Upgrade job, you should run Sanity job to ensure OpenStack is up and running. During Sanity process, Tempest will check both API & CLI feature are still working. Javelin will check that resources created in a previous release are still alive (servers, networks, volumes, containers, etc).

Principles¶

The following is the declaration of our messaging class

class cloud::messaging(
  $cluster_node_type = 'disc',
  $rabbit_names      = $::hostname,
  $rabbit_password   = 'rabbitpassword'
){

As one can see it has default value for each parameters. With this current state they are three ways to specify a value to the cloud::messaging class. Hiera being the way Spinal Stack works.

1. Let the default

   One can let the default value and simply include the cloud::messaging class to use it

2. Specify specific valude

   One can specify specific values in another manifest, the call would look like the following

class {'cloud::messaging' :
  cluster_node_type = 'foo',
  rabbit_names      = 'rabbit.example.com',
  rabbit_password   = 'secret',
}

3. Use Hiera

   One can use hiera to specify those values in a configured backend (yaml file, json file, mysql). Spinal Stack relies on a yaml backend at the moment. (More in Configuration file)

   The yaml hiera file would look like the following

---
cloud::messaging::cluster_node_type: foo
cloud::messaging::rabbit_names: rabbit.example.com
cloud::messaging::rabbit_password: secret

Configuration file¶

The configuration files is located at /etc/puppte/hiera.yaml and the default provided by Spinal Stack is the following

---
:backends:
  - yaml
:yaml:
  :datadir: /etc/puppet/data
:hierarchy:
  - "%{::type}/%{::fqdn}"
  - "%{::type}/common"
  - common

Feel free to change it to meet your needs, here the explanation of the default hiera.yaml.

• backend: Specify the backend one wants to use (can be json, mysql, etc...)
• yaml: Specify backend specifics, here for the yaml backend. Simply specify the directory where to find the files
• hierarchy: Indicates, in prefered order, where to find the datas. The actual file name on the filesystem should end with .yaml.

Admiting that the fact type is ‘controller’ and the fact fqdn is ‘controller01.example.com’, this will be the order in which data will be looked for :

• /etc/puppet/data/controller/controller01.example.com.yaml
• /etc/puppet/data/controller/common.yaml
• /etc/puppet/data/common.yaml

Debug¶

From time to time one might find hiera, not popullating the variable as one would have expect, then it’s time to get one’s hands dirty.

Log onto the puppet master server, and start looking what hiera actually returns to puppet, using the following commands.

hiera MYVARIABLE -c /etc/puppet/hiera.yaml

hiera MYVARIABLE -c /etc/puppet/hiera.yaml --debug

hiera MYVARIABLE -c /etc/puppet/hiera.yaml ::type=controller ::fqdn=lb001.example.com

How Spinal Stack prevents it ?¶

To prevent OSSA-2014-041 on your deployed platforms, a user needs to set specific policies for Glance API via the glance_api_policies configuration setting in the environment file.

The needed policies are the following :

‘delete_image_location’: ‘role:admin’ ‘get_image_location’: ‘role:admin’ ‘set_image_location’: ‘role:admin’

By default, if the user does not specify any glance_api_policies in his configuration file, those parameters will be added automatically else the user will need to add them manually in addition of the policies she’s adding.

Find below the snippet to use

glance_api_policies:
  delete_image_location:
    key: delete_image_location
    value: 'role:admin'
  get_image_location:
    key: get_image_location
    value: 'role:admin'
  set_image_location:
    key: set_image_location
    value: 'role:admin'

To completely disable Glance API policies, set the following in your environment file

glance_api_policies: false

Am I affected ?¶

If you have untrusted cloud user or hosting internet facing application then you are at risk.

How to get protected from VENOM ?¶

In order to get an early fix and avoid a full upgrade of the plateform, here is a simple procedure to apply manually on each compute node:

1/ Activate package manager::

# edeploy activate-pkgmngr

2/ Register to Red Hat CDN. (Please read this documentation)

3/ Update the Qemu package::

# yum update qemu-kvm-rhev

4/ Deactivate package manager::

# edeploy deactivate-pkgmngr

5/ Following the update, the guests (virtual machines) need to be powered off

and started up again for the update to take effect. It is also possible to migrate guests away from the affected host, update the host, and then migrate the guests back. Please note that it is not enough to restart the guests because a restarted guest would continue running using the same (old, not updated) QEMU binary:

# nova evacuate

Will I get protected after a full upgrade ?¶

Unless the new version is J-1.4.0, the qemu package is not fixed and the above procedure needs to be re-applied.