Functional REST Testing

Owned by bcarlsen (Unlicensed)
Last updated: Aug 25, 2015
5 min read

Overview

Documents the approach to functional/integration testing for the application REST services layer.

Coverage

These service are currently covered by REST integration testing.

  • Security Service
  • Metadata Service
  • Content Service
  • History Service
  • Project Service

Complete coverage would address each mode of use of each method call of each service, and combinations of calls across them, as well as any ad hoc test cases for identified errors not covered by existing test cases.

Coverage Definition

The attached spreadsheet formally defines REST service layer coverage from an integration testing perspective. 

Assumptions

  • A dev database exists that is loaded with exactly the sample "full" data from the config/ project.
  • Any changes made are reverted to initial conditions so all tests can assume identical initial conditions
  • A server built using the latest code is deployed to the URL specified in the base.url property of the config file used to run the integration tests.
  • The server is configured to use "default" authentication in which "guest" and "admin" users can be authenticated with any password (and the auth token matches the usernames)

Coverage is defined in a multi-dimensional matrix that takes into account the services, the methods of the services, and the modes of use of the services (including normal use, degenerate use, edge case use, and role check). NOTE: in many cases there are many "normal" modes of use which may extend the number of columns in the spreadsheet.  For example, methods that take optional parameters (e.g. Pfs).

The following sample coverage spreadsheet is the current coverage definition.

Organization

Integration tests are organized under the integration-testing project in the "wci.com.server.umls.test.rest" package.

Generally each column of each tab of the coverage spreadsheet is represented by a single testing class, where each row within that column corresponds to a particular method of that testing class.  The coverage spreadsheet defines both the class names used to implement tests as well as the method names (so they can be easily cross referenced).

The test names themselves are "test cases" that are organized into "test suites" according to the structure of the spreadsheet.

For each test suite and test case, there should also exist a documentation page in this wiki.  A test suite is a collection of test cases and a test case is a "script" detailing the actions involved and the expected outcomes with any other needed information.

Integration Test Suites

Test suites are organized by spreadsheet tab.  It is time consuming to create documentation to reflect the procedures and activities of the various integration tests here and we do not have the resources to do so at this time.  What this means is that the spreadsheet defines coverage, and the classes themselves contain details of the tests.  When we are able, tests will be comprehensively documented here.

 

Degenerate Use Test Automation

For repetitive testing of degenerate use of methods with null and INVALID values, a class DegenerateUseMethodTestHelper is provided, which provides a mechanism for testing a method

  • A call with supplied valid parameters is made to verify that the method invocation can be successfully made
  • For each parameter, the value is alternately set to null and an invalid value, the method is invoked, and behavior evaluated as described below
    • NOTE: For primitive argument types (e.g. int/long), the null value is not tested.

The four methods available for degenerate use testing are:

  • DegenerateUseMethodTestHelper(Object o, Method m, Object[] parameters)
  • DegenerateUseMethodTestHelper(Object o, Method m, Object[] parameters, Object[] invalidParameters)
  • DegenerateUseMethodTestHelper(Object o, Method m, Object[] parameters, ExpectedFailures[] failureTypes)
  • DegenerateUseMethodTestHelper(Object o, Method m, Object[] parameters,  Object[] invalidParameters), ExpectedFailures[] failureTypes)

Where:

  • Object:  The service client undergoing testing
  • Method:  The method of the service client to be tested
  • Parameters:  An object array of values that that produce successful invocation of the method
  • InvalidParameters (optional):  An object array of values which will be used to test invalid parameters for the method invocation
    • If not provided, the default values for class types are:
      • String: "" (empty string)
      • Long/long: -5L
      • Integer/int : -5
    • For all other classes, the invalid value is set to null, and is not tested.  If additional default behavior is desired, the Test Helper can be modified to create invalid objects.
  • ExpectedFailures (optional):  An array of Enum constants ExpectedFailures
    • If not provided, the default behavior expected is EXCEPTION for both null and invalid values
    • The general behaviors that can be specified are:
      • EXCEPTION: Null and invalid values both throw exceptions.
      • SKIP: Behavior of this parameter will not be evaluated
      • NONE: Null and invalid values should not throw exceptions
      • NO_RESULTS: Null and invalid values will both return no results
    • Value-specific behavior.   

      NOTE:  This will be refactored to only require 9 enums spanning all combinations of (SUCCESS, EXCEPTION, NO RESULTS).  Invalid values will be parameterizable. 

      Here, differing behavior is expected from invalid and null values.  The following enums are named in the format TYPE_PARAMETER1_RESULT1_PARAMETER2_RESULT2.  For each of these cases, the parameter passed in is checked for type safety, and behavior evaluated according to the result for each parameter.
      • Class:  Long
        • Available enums are:
          • LONG_INVALID_NO_RESULTS_NULL_EXCEPTION
            • For an invalid Long value, no results should be returned
            • For a null Long value, an exception should be thrown
      • Class: String
        • Available enums are:
          • STRING_INVALID_EXCEPTION_NULL_EXCEPTION
            • For both INVALID and null String values, an exception should be thrown
            • This is identical to EXCEPTION above, but restricted to String parameters
          • STRING_INVALID_EXCEPTION_NULL_NO_RESULTS
            • An INVALID string will cause an exception
            • A null string will return no results
          • STRING_INVALID_EXCEPTION_NULL_SUCCESS
            • An INVALID string will cause an exception
            • A null value will cause a successful invocation
          • STRING_INVALID_NO_RESULTS_NULL_EXCEPTION
            • An INVALID string will return no results
            • A null value will throw an exception
          • STRING_INVALID_NO_RESULTS_NULL_NO_RESULTS
            • Both INVALID strings and null values return no results.
          • STRING_INVALID_NO_RESULTS_NULL_SUCCESS
            • An INVALID string will return no results
            • A null value will cause a successful invocation
          • STRING_INVALID_SUCCESS_NULL_EXCEPTION
            • An INVALID string will cause a successful invocation
            • A null string will throw an exception
          • STRING_INVALID_SUCCESS_NULL_NO_RESULTS
            • An INVALID string will cause a successful invocation
            • A null string will return no results
          • STRING_INVALID_SUCCESS_NULL_SUCCESS
            • Both null and INVALID strings will cause a successful invocation
            • This is identical to NONE above, but restricted to String parameters

Paging/Filtering/Sorting objects (PfsParameter) are explicitly handled with the following tests:

  • Test invalid sort field (specified by string "-")
  • Test invalid start index (specified by integer -5);
  • Test invalid query restriction (specified by string "BAD_SYNTAX*:*!~bad")

 

Example: 

 

// the class for which methods are to be tested, in this case SecurityClientRest
SecurityClientRest service = new SecurityClientRest(ConfigUtility.getConfigProperties());
 
// use reflection to get the desired method
Method method =
  service.getClass().getMethod(
    "authenticate",                               // method name and parameters
    new Class<?>[] {String.class, String.class}); // i.e. SecurityClientRest.authenticate(String username, String password)
 
// specify the valid parameters
Object[] parameters = new Object[] {
  new String("name"), new String("password")      // valid parameters that when passed to method produce successful outcome (e.g. authentication)
    };
 
// if all invalid and null parameters should throw exceptions, use this call
DegenerateUseMethodTestHelper.testDegenerateArguments(service, method,parameters);
 
// if different parameters cause different behavior, use this type of call
DegenerateUseMethodTestHelper.testDegenerateArguments(service, method,
  parameters, new ExpectedFailure[] {
  ExpectedFailure.STRING_INVALID_NO_RESULTS_NULL_EXCEPTION, // an INVALID string will result in no result, a NULL string will result in exception
  ExpectedFailure.EXCEPTION                                 // both INVALID strings and NULL strings will result in exception
});
 
// if specific invalid values are desired, use this type of call
DegenerateUseMethodTestHelper.testDegenerateArguments(service, method,
  parameters, new Object[] {                     
  new String("Invalid value"),    // the first argument should test "Invalid value", and the method should throw an exception
  null                  // no invalid value will be tested
});