Articles

Introduction

Testing is a very important part of any software development process. Tests act as verifier for a program or a component. They tell us if the system works the way we expect it to. If tests don't work, then the implementation is broken and we need to fix it. Later, when the implementation is changed, refined or expanded, the same tests can be used to determine that the changes did not break anything. OakUT is a simple, flexible and powerful unit test framework, providing many features as explained in this article.

Test Execution Model

The OakUT framework specifies the execution model of the tests. It helps users approach tests in a uniform way and provides a neat structure for the whole test suite of the system.

The test environment is divided into two distinct parts: testexecutor and the user written test code. Tests do not depend upon the executor at the compile time or link time. The aim is that the tests should be separated from the details and complexeties of the execution environment. The executor provides a host of services for the tests like logging, timeout etc. The Executor is the interface that the users interact with to run the tests.

Secondly, tests are independent of each other, at compile and link time. This enables us to avoid one big monolithic test system. It also cuts down the compilation time of the tests. The physical unit of a test is a DLL or a shared object. In principle, it could also support an executable file, but I have not provided that option.

A group of related tests form a test group and can be assembled together in the execution unit. To run the test, user specifies the test DLL to the test executor:

This causes the testexecutor to run all the tests in the testgroup and report the output and results at the end of the tests run. The executor loads the DLL, looks for a testgroup in it and runs all the tests in the group. It then unloads the DLL. As you will see in the next section, users do not have to worry about these details. They need to concentrate on writing the test logic, while the availability of the tests to the test group and to the executor is handled by the framework behind the scenes.

The executor also provides an option to run the tests in a separate address space (different from executor's address space). I will describe this and various options in a later section.

Writing Tests

Suppose we were to write an implementation of a C standard library function strstr() without using any standard library function.

char* strstr(const char* s1, const char* s2);

The function locates the first occurence of string s2 in s1 (excluding the terminating null character) and returns a pointer to it, or a null pointer if the string is not found. The implementation is in StringUtils namespace. I have deliberately written a buggy implementation. Later, we will write a test for it.

//stringutils.h

namespace StringUtils
{
    namespace impl
    {
        inline bool startsWith(const char* s1, const char* s2);
    }

    //does not work for all the cases
    inline const char* strstr(const char* s1, const char* s2)
    {
        char ch = s2[0];

        if(0 == ch)
            return s1;

        while(*s1)
        {
            if(*s1 == ch && impl::startsWith(s1, s2))
                return s1;
            ++s1;
        }
        return 0;
    }

    namespace impl
    {
        inline bool startsWith(const char* s1, const char* s2)
        {
            while(*s1 && *s2)
            {
                if(*s1++ != *s2++)
                    return false;
            }

            //if(*s2)
            //    return false;

            return true;
        }
    }
}

To write a test for this function, we need to perform two steps: Write a new test class, inherit it from oak::test::Test class and override its run() member function. Second step is to add the test class to the TestGroup. See the test program below. The constructor of the test class calls the base class constructor passing it the name of the test. User can also optionally pass test description as a second parameter. Note that the run() member function accpets a parameter of type oak::util::Properties. We can ignore it for now. I will talk about it later.

//stringutils_test.cpp

#include <oak/test/tmain.cpp>
//or #include <oak/test/test.h> (see notes below)
#include "stringutils.h"

//Step 1: Write the test class
class StrStrTest : public oak::test::Test
{
public:
    StrStrTest(): oak::test::Test("StringUtilsTest::StrStrTest")
    {
    }

    void run(const oak::util::Properties& props)
    {
        const char* s = "StrStr Test";

        const char* str1 = "Test";
        const char* str2 = "Str";
        const char* str3 = "Testing";
        const char* str4 = "";
        const char* str5 = "xyz";

        const char* p1 = StringUtils::strstr(s, str1);
        const char* p2 = StringUtils::strstr(s, str2);
        const char* p3 = StringUtils::strstr(s, str3);
        const char* p4 = StringUtils::strstr(s, str4);
        const char* p5 = StringUtils::strstr(s, str5);

        OAK_TEST_ASSERT_NOT_EQUAL(p1, (const char*)0);
        OAK_TEST_ASSERT_NOT_EQUAL(p2, (const char*)0);
        OAK_TEST_ASSERT_EQUAL(p3, (const char*)0);
        OAK_TEST_ASSERT_EQUAL(p4, s);
        OAK_TEST_ASSERT_EQUAL(p5, (const char*)0);
    }
};

//Step 2: Add to the TestGroup
OAK_TEST_ADD(StrStrTest, g_t1);

To run this test, build this file as a DLL and pass it to the executor:

On running the test, it fails with the following message:

This tells that there was an assertion failure in file stringutils_test.cpp at line number 32. If we inspect the code, we will find that the following call failed:

StringUtils::strstr("StrStr Test", "Testing");

To fix it, uncomment the comments in startsWith() function in the program above and rerun the test. Now, all the assertions will pass.

Note that the first argument to the oak::test::Test base class is the test name. Test name is very important and is used at many places in the executor, as you will see in later sections.

The framework calls test class's run member function. The test duration is the start and end of run member function. If the return is normal (return statement), that is considered a success. The failure is reported by throwing oak::test::TestException. If any other exception escapes the run member function, that is considered an error. Error is also a test failure but the intention here is that if the test code calls some interface that throws an exception, it should be aware of it and deal with it.

The OAK_TEST_ASSERT macros used above are defined in oak/test/util.h. This file defines a number of macros that throw oak::test::TestException if the condition does not evaluate to true. Suppose, you want to test the result of the function that returns an int, and further suppose that the result returned should be equal to 5. We could call the OAK_TEST_ASSERT_EQUAL macro that will throw oak::test::TestException if the two values are not equal:

These macros expand to calls to function templates. So the arguments to these macros could be of any type as long as long as operator== is defined for these types or is meaningful for them. For instance, operator== is not sufficient to test the equality of two strings (char pointers). We can define a functor for such a case and pass it to OAK_TEST_ASSERT_EQUAL_IF macro:

class StrEqual
{
public:
    bool operator()(const char* p1, const char* p2) const
    {
       return 0 == strcmp(p1, p2);
    }
};

OAK_TEST_ASSERT_EQUAL_IF(s1, s2, StrEqual());

//partial oak/test/util.h file

#define OAK_TEST_ASSERT_EQUAL(t1, t2) ::oak::test::assertEqual(\
                            t1, t2, oak::test::getComparator(t1), \
                            OAK_TEST_ASSERT_EQUAL_MSG)

#define OAK_TEST_ASSERT_EQUAL2(t1, t2, Msg) ::oak::test::assertEqual(\
                            t1, t2, oak::test::getComparator(t1), Msg)

#define OAK_TEST_ASSERT_EQUAL_IF(t1, t2, Cmp) \
                        ::oak::test::assertEqual(\
                        t1, t2, Cmp, OAK_TEST_ASSERT_EQUAL_MSG)

#define OAK_TEST_ASSERT_EQUAL2_IF(t1, t2, Cmp, Msg) \
                        ::oak::test::assertEqual(t1, t2, Cmp, Msg)

#define OAK_TEST_ASSERT_NOT_EQUAL(t1, t2) ::oak::test::assertNotEqual(\
                        t1, t2, oak::test::getComparator(t1), \
                        OAK_TEST_ASSERT_NOT_EQUAL_MSG)

//...

namespace oak
{
    namespace test
    {
        template<class T>
        class Equal
        {
        public:
            bool operator()(const T& t1, const T& t2) const
            {
                return t1 == t2;
            }
        };

        template<class T>
        Equal<T> getComparator(const T&)
        {
            return Equal<T>();
        }

        template <class T, class Cmp>
        bool assertEqual(const T& t1, const T& t2, Cmp cmp,
                         const char* msg = "assertEqual")
        {
            if(!cmp(t1, t2))
                throw TestException(msg);
            return true;
        }

        template<class T, class Cmp>
        bool assertNotEqual(const T& t1, const T& t2, Cmp cmp,
                            const char* msg = "assertNotEqual")
        {
            if(cmp(t1, t2))
                throw TestException(msg);
            return true;
        }
        
        //...
    }
}

To add the test class to the testgroup, test writer should provide the name of the test class to OAK_TEST_ADD macro:

OAK_TEST_ADD(TestClassName, id);

Properties

Users can also specify properties for the test in a property file. The properties file is a xml file. Each property file contains the modulename and optional properties for the tests. We specify the properties file directly to the testexecutor using the -p option:

oakte -p mytest_props.xml

The executor reads the modulename from the xml file and loads the dll containing the testgroup. It then executes each test in the group, passing it the properties read from the xml file, if present. The run member function of the oak::test::Test class has a parameter of type oak::util::Properties:

void run(const oak::util::Properties&);</b>

Property is a name-value pair of type std::string and a oak::util::Properties object holds a list of properties for the test. User can iterate through the properties or lookup for a property using getProperty member function:

//class oak::util::Properties

iterator getproperty(const std::string& name);
const_iterator getProperty(const std::string& name) const;

iterator begin();
iterator end();

const_iterator begin() const;
const_iterator end() const;

The function returns end iterator (Properties::end()) if the property is not found:

Properties::const_iterator iter = props.getProperty("test1");
if(iter != props.end())
{
    const std::string& value = iter->second;
    //...
}

To show the usage of property files, let's add two more functions to the StringUtils namespace: toupper and tolower. These functions operate on ASCII encoding:

void toupper(const char* s1, char* s2);</b>

The toupper function converts the characters in s1 to uppercase, if possible, and puts the resulting characters in the second parameter. To convert lowercase ASCII alphabetic character to uppercase, we subtract 32 from it:

namespace StringUtils
{
    inline void toupper(const char* s1, char* s2)
    {
        int ch = *s1;
        while(ch != 0)
        {
            if(ch >= 'a' && ch <= 'z')
                *s2 = ch - 32;
            else
                *s2 = ch;
            ++s1;
            ++s2;

            ch = *s1;
        }
        *s2 = 0;
    }

    inline void tolower(const char* s1, char* s2)
    {
        int ch = *s1;

        while(ch != 0)
        {
            if(ch >= 'A' && ch <= 'Z')
                *s2 = ch + 32;
            else
                *s2 = ch;
            ++s1;
            ++s2;

            ch = *s1;
        }
        *s2 = 0;
    }
}

For this case, we can specify the set of input and the expected value as properties. For instance:

<property>
  <name>toupper</name>
  <value>TOUPPER</value>
</property>

Here, the content of &ltname> element is used as a test value for StringUtils::topupper function and the return value of this function should be equal to the contents of the <value> element above. The test iterates through the properties, passing the name contents to the StringUtils::topupper function and comparing the result to the contents of the value element.

First, let's look at the test properties file:

//stringutils_test.xml

<testgroup>
    
    <!-- module name is required -->
    <module>stringutils_test</module>

    <test name="StringUtilsTest::ToLower">
        <property>
            <name>ToLower</name>
            <value>tolower</value>
        </property>
        
        <property>
            <!-- use entity references like &lt; instead of '<' -->
            <name>&lt;#sTr1+</name>
            <value>&lt;#str1+</value>
        </property>
    </test>
    
    <test name="StringUtilsTest::ToUpper">
        <!-- timeout (in milliseconds) is optional --> 
        <timeout>1000</timeout>

        <!-- properties are also optional -->
        <property>
            <name>n1&</name>
            <value>N1&</value>
        </property>
    </test>
   

</testgroup>

The root element is <testgroup>. The <modulename> element is the required element and specifies the name of the DLL or shared object. The DLL must be present in the same directory as the properties file. Secondly, as you will notice, the module name does not contain path information or the DLL extension. The testexecutor appends the DLL extension to the module name depending on the operating system. For instance, for Windows, it will be .dll, while it will be .so for linux or solaris. The test element is optional and may be specified to provide properties for the test. The test element contains required name attribute and an optional timeout and property element.

The test code is shown below. Notice, that the test name below passed to the base class constructor (oak::test::Test) matches the test name in the xml properties file above.

#include <oak/test/tmain.cpp>
#include "stringutils.h"

namespace StringUtilsTest
{
    class ToLower : public oak::test::Test
    {
    public:
        ToLower() : oak::test::Test("StringUtilsTest::ToLower") 
        {
        }

        void run(const oak::util::Properties& props)
        {
            oak::util::Properties::const_iterator iter = props.begin();
            oak::util::Properties::const_iterator end = props.end();

            while(iter != end)
            {
                const std::string& name = iter->first;
                const std::string& value = iter->second;
                
                //define a big enough array, assuming for convenience here
                //that the size will not exceed the array size
                char lstr[1024];

                StringUtils::tolower(name.c_str(), lstr);
                OAK_TEST_ASSERT_EQUAL(value, std::string(lstr));

                ++iter;
            }
        }
    };

    class ToUpper : public oak::test::Test
    {
    public:
        ToUpper() : oak::test::Test("StringUtilsTest::ToUpper") 
        {
        }

        void run(const oak::util::Properties& props)
        {
            oak::util::Properties::const_iterator iter = props.begin();
            oak::util::Properties::const_iterator end = props.end();

            while(iter != end)
            {
                const std::string& name = iter->first;
                const std::string& value = iter->second;
                
                //define a big enough array, assuming for convenience here
                //that the size will not exceed the array size
                char ustr[1024];

                StringUtils::toupper(name.c_str(), ustr);
                OAK_TEST_ASSERT_EQUAL(value, std::string(ustr));

                ++iter;
            }
        }
    };
}

//Add the tests to the test group
OAK_TEST_ADD(StringUtilsTest::ToLower, g_t2);
OAK_TEST_ADD(StringUtilsTest::ToUpper, g_t3);

To run the tests, use the -p option:

oakte -p stringutils_test.xml

The test framework contains a handcrafted xml parser to parse the property file. Consequently, It is strict about what it parses and what it does not and recognizes a subset of the XML grammar. The property file is a well-formed XML file. The validation of the document is built within the parser and it does not use or recognize DTD or XML schema for validation. The encoding of the file is ISO8859-1. Comments are allowed in the document. Entity references are replaced by the parser. For instance, for string '&lt;', the parser returns '<'.

Usually, the spaces and lines around the text are discarded from the element content. To preserve space and provide unparsed data, use CDATA element. The CDATA element is allowed for <modulename>, <name> and <value> elements:

<value>  
<![CDATA[
      void foo()
      {
          std::cout << "hello, world!" << std::endl;
      }
]]>
</value>

One-Time Initialization

Users may wish to do one-time initialization when the test module loads. Users can provide a class that derives from oak::test::Init class to perform this task:

class MyInit: public oak::test::Init
{
public:
    bool init(const oak::util::Properties& props)
    {
        //...
    }
    
    bool finish()
    {
        //...
    }
};

We need to set the object of this type as init object: OAK_TEST_SET_INIT_OBJ(MyInit, id); The property file may specify properties for the Init object by using a special reserved name for the test - "all":

<test name="all">
  <property>
    <name>...</name>
    <value>...</value> 
  </property>
  .....
</test>

Test Executor Options

Users can provide -h option to the testexecutor to find help about various options. Multiple -t or -p options can be provided and they can be mixed together:

oakte -t test1.dll -t test2.dll -p test3.xml -p test4.xml

Alternatively, users can assemble the module names in a file, with each name present on a new line, and provide the name of the file to the executor. The following tells the executor to read all the test module names from the tests1 file and execute the testgroups in these modules. Similarly, it tells the executor to read all the property file names from the tests2 file and execute the testsgroup specified by each property file:

oakte -tf tests1 -pf tests2

The executor loads the test modules in its address space. If the test is unstable, it may cause the executor to crash with it, thus affecting the run of other tests. Users may wish to run the tests in a separate address space. They can do so by specifying the -a option to the executor:

oakte -t /tmp/test1.so -a

This tells the executor to run each test in a new address space and communicate the results to the executor over the pipe. A user may also wish to specify the timeout value for the tests. The following tells the executor to wait for a maximum of 1000 milliseconds for each test to run. If the test does not complete within the specified time, the executor kills the test and reports a timed out message.

oakte -t c:\tmp\test1.dll -a -timeout 1000

Timeout value is used only for tests that are executing in an external process (with -a option).

Users may also exclude some test from running by specifying the name of the test to -e option. The following tells the executor to exclude StringUtilsTest::ToUpper test while running all other tests of the testgroup:

oakte -e StringUtilsTest::ToUpper -p StringUtilsTest.xml

There is also a corresponding -ef option that tells the executor to read test exclude names from a file (with each name present on a new line). Users can specify to run a specific test in the testgroup instead of all the tests by embedding the name of the test to run with the module name or property file name, separating the module name and test name by '@' character. Users may also specify the timeout value specific for the test or testgroup embedded in the module or property file name (separated by '#' character). For instance, the following tells the executor to execute StringUtils::ToLower test from the testgroup with a timeout of 500 milliseconds:

oakte -p StringUtilsTest.xml@StringUtilsTest::ToLower#500 -a

The timeout value embedded in the module or property file name overrides the timeout value for that module or test specified with -timeout option or specified in the property file.

Test Output

The oak::test::Test base class defines two member functions to control the test output:

The loggers present C++ style output interface. In fact, by default, they refer to std::cout and std::cerr. Tests should use these loggers to write the output:

The benefit of using these interfaces is that the executor can change the ouput stream without affecting the test code. For example, it could send the output to a GUI window or to files.

The testexecutor provides the facility to send the test output to files, using -o and -r option. The -r option specifies the result file to which it writes the summary of all the tests run. For example, it will show the list of all the tests that passed, list of all the tests that failed etc. The -o option specifies the directory to which the output of the test should go. Each test output goes to its own file. The executor uses the name of the test to create the output file name. If, for instance, the name of the test is net::T1, the executor creates the output file net/T1.log under the directory specified by -o option.

Note For Multithreaded Tests

In a multi-threaded test, the run member function executing in the main thread will spawn off other threads. The test writer must make sure that the run member function does not return to the caller while other threads spawned by it are running. Once the run function returns, that is considered the end of the test run and the framework unloads the test dll. This will be disastrous since the other threads will be trying to execute the code of the DLL that has been unloaded. It will most likely lead to a crash. Hence, the main thread (in run member function) must wait for all the threads, that it spawned, to finish before returning the result to the framework.

If that is not an option, users may wish to run the test in an external address space (-a option to the testexecutor). Here, the main thread in the run member function will still wait, but some other thread may communicate the result to the test framework, followed by process termination. User may do so by calling getTestResultReporter member function on the oak::test::Test base class and calling one of its member functions to report the result. Note that the reporting of the result marks the end of the test, so these functions also terminate the process (which should be an external process, as noted above. No check is made).

class TestResultReporter
{
private:
    //Not implemented
    TestResultReporter& operator==(const TestResultReporter&);
    TestResultReporter(const TestResultReporter&);

public:
    void passed();
    void failed(const char* reason);
    void error(const char* reason); //error is also a failure, see comments
                                    //in the article
};

Building Tests

To build individual tests, one needs to specify the the root of the oaklib directory in the compiler include path settings (/I or -I or specifying it in MS VC++ project settings tab).

Users also need to build oak/test/tmain.cpp and link it with their tests. However, this step can be easily avoided if tests #include <oak/test/tmain.cpp> file in one source file. Note that only one test source file should include tmain.cpp.

For MS VC++ or Borland compiler, user must use (link) the dynamic multithread version of the C/C++ library (debug or non-debug).

Please see here for build instructions. It gives instructions on how to build testexecutor and example.

Copyright © Vishal Kochhar, 2004

Vishal Kochhar is a software engineer with Sybase Inc. in Waterloo , canada . He can be contacted at [email protected]