Verifying the Cloud with Rally and Tempest

Rally is a test framework, and Tempest is the OpenStack API test suite. In this guide, Rally is used to run Tempest tests.

Requirements

OpenStack tests are run from a host with access to the OpenStack APIs and external network for instances.

The following software environment is needed:

• OpenStack admin credentials, eg public-openrc.sh from kayobe-config/etc/kolla

• A virtualenv setup with python-openstackclient installed

source venv/bin/activate
pip install python-openstackclient
source ~/kayobe-env/src/kayobe-config/etc/kolla/public-openrc.sh

Setup Rally for a new user

A good directory hierarchy would be ~/rally/shakespeare/tempest-recipes. This is assumed in this guide.

Install Rally into the virtualenv:

pip install rally-openstack \
     --constraint https://releases.openstack.org/constraints/upper/master

Create the Rally test database and configuration file. For this you will need the virtualenv and public-openrc as described above:

mkdir -p ~/.rally ~/rally/data
echo "[database]" | tee ~/.rally/rally.conf
echo "connection=sqlite:///${HOME}/rally/data/rally.db" | tee -a ~/.rally/rally.conf
rally db recreate
rally verify create-verifier --name default --type tempest
rally deployment create --fromenv --name production

Check:

rally deployment show

Install Shakespeare

Shakespeare is used for writing Tempest test configuration.

cd ~/rally
git clone https://github.com/stackhpc/shakespeare.git
cd shakespeare
pip install -r requirements.txt

Install Tempest Recipe

A custom set of Tempest recipes should be maintained for Acme, defining key parameters needed and fine-grained control on the test cases to run (and which to skip).

In your shakespeare directory, install the Tempest recipes in your Tempest configuration.

git clone https://github.com/acme-openstack/tempest-recipes.git
ansible-playbook template.yml -e @tempest-recipes/production.yml
mkdir -p  ../config/production
rally verify configure-verifier --reconfigure --extend ../config/production/production.conf

Rally invocation

Invoke the tests (this will take several hours to complete):

rally --debug verify start --concurrency 1 --skip-list tempest-recipes/production-skiplist.yml

Report generation

Generate an HTML report of the results:

rally verify report --type html --to ~/rally/report-$(date -d "today" +"%Y%m%d%H%M").html