Tempest 101

OpenStack's Testing Engine

Sean Dague - IBM Linux Technology Center / @sdague

What is Tempest?

  • Test Suite for OpenStack
  • Aims to provide validation that OpenStack is working
  • Runs on ever core servers commit to OpenStack
  • Can be used on real clouds (with configuration)
  • Written in Python
  • Does not use python-clients (*)

Getting started with Tempest

The Devstack way

git clone git://github.com/openstack-dev/devstack.git
cd devstack
./stack.sh
cd /opt/stack/tempest
./run_tests.sh [tempest/tests/...]
  • Step back, get a cup of coffee (or two) and watch things run.
  • Will spin up images, servers, volumes, do things to them, bring them down.
  • Takes about 45 minutes on my dev machines.

Tempest outside of devstack

creating etc/tempest.conf manually


[identity]
# Ignore SSL certificate validation failures? Use when in testing
# environments that have self-signed SSL certs.
disable_ssl_certificate_validation = False
# URL for where to find the OpenStack Identity API endpoint (Keystone)
uri = http://127.0.0.1:5000/v2.0/

[compute]
# Reference data for tests. The ref and ref_alt should be
# distinct images/flavors.
image_ref = {$IMAGE_ID}
image_ref_alt = {$IMAGE_ID_ALT}
flavor_ref = 1
flavor_ref_alt = 2

Tempest Test Case - positive

tempest/tests/compute/servers/test_servers.py

    def test_create_with_existing_server_name(self):
        # Creating a server with a name that already exists is allowed
        server_name = rand_name('server')
        resp, server = self.create_server(name=server_name,
                                          wait_until='ACTIVE')
        id1 = server['id']
        resp, server = self.create_server(name=server_name,
                                          wait_until='ACTIVE')
        id2 = server['id']
        self.assertNotEqual(id1, id2, "Did not create a new server")
        resp, server = self.client.get_server(id1)
        name1 = server['name']
        resp, server = self.client.get_server(id2)
        name2 = server['name']
        self.assertEqual(name1, name2)

Tempest Test Case - negative

tempest/tests/compute/servers/test_servers_negative.py

    @attr(type='negative')
    def test_rebuild_deleted_server(self):
        # Rebuild a deleted server

        resp, server = self.create_server()
        self.server_id = server['id']
        self.client.delete_server(self.server_id)
        self.client.wait_for_server_termination(self.server_id)

        self.assertRaises(exceptions.NotFound,
                          self.client.rebuild,
                          self.server_id, self.image_ref_alt)

    @attr(type='negative')
    def test_create_numeric_server_name(self):
        # Create a server with a numeric name
        if self.__class__._interface == "xml":
            raise self.skipException("Not testable in XML")

        server_name = 12345
        self.assertRaises(exceptions.BadRequest,
                          self.create_server,
                          name=server_name)

Tempest in the Gate

We run Tempest 3 ways on every commit to gate

  • Tempest Full on MySQL with nova-network
  • Tempest Full on PostgreSQL with nova-network
  • Tempest Smoke on MySQL with quantum

Tempest Full

  • Spins up blank Ubuntu 12.04 node
  • Installs with devstack
  • Runs 700+ tests
  • Takes about 40 minutes in gate
  • Minimum of 4GB RAM needed to run

When things fail...

Start by following the log links

When things fail...

Some Real Issues Tempest Found

  • Broken Nova XML API in Folsom
  • Broken Postgresql Support
  • Resize testing in Conductor
  • Stack traces being thrown even on successful API calls
Source
                                  http://www.flickr.com/photos/pforret/187520198/sizes/m/in/photostream/

Tempest-Two-Step

Issue: Tempest enforces an API, a commit in a core project changes that API, Tempest blocks the commit.

  • Disable test in Tempest
  • Land core project change
  • Fix and re-enable test in Tempest

Tempest's Evolution

  • Started as a small number of integration tests
  • New Focus on API testing
  • Boto tests added in Grizzly (testing EC2)
  • CLI test directory added late in Grizzly
  • Configured by default in devstack in Grizzly

Possibilities in Havana

  • Replace Devstack exercises with subset of Tempest tests
  • Tempest parallel - increase our time budget
  • Restructuring tempest to specific test approaches
    • API testing (can't use clients)
    • Integration with clients
    • CLI
    • Stress
  • API validation with Tempest
  • Tooling to make getting to the bottom of gate fails easier

Where we need help

  • New tests to cover rest of API
  • Tempest is strong on Nova, weaker on other projects
  • Lots of extensions untested
  • Evaluation of tests for duplication
  • Maybe move some content back to core projects

Getting Involved in Tempest

  • Come participate in QA Track (Tues & Wed)
  • Hang out with use in #openstack-qa
  • Weekly QA meetings Thursday 17:00 UTC
  • Get on openstack-qa mailing list
  • Always looking for help filling test gaps