# Adding New Tests

This testing tool is intended to be straightforward to extend. If you encounter an implementation which is operating outside of the specification and a current test suite does not identify this behaviour, please consider adding a test as follows:

1.  First, raise an Issue against this repository. Even if you do not have the time to write additional tests, a good explanation of the issue identified could allow someone else to do so on your behalf.
2.  Once an issue has been raised, feel free to assign it to yourself. We would welcome any Pull Requests which add to the set of tests available. Once a Pull Request is raised, one of the specification maintainers will review it before including it in the test suite.

## Test Suite Structure

All test suite classes inherit from `GenericTest` which implements some basic schema checks on GET/HEAD/OPTIONS methods from the specification. It also provides access to a 'Specification' object which contains a parsed version of the API RAML, and provides access to schemas for the development of additional tests.

Each manually defined test case is expected to be defined as a method starting with `test_`, taking an object of class `Test`. This will allow it to be automatically discovered and run as part of the test suite.
The return type for each test case must be the result of calling one of the methods on the `Test` object shown below.

*   The first argument, `details`, is used to specify the reason for the test result.
  It is required for `FAIL`, `OPTIONAL` (Not Implemented), or `NA` (Not Applicable), and is recommended for all cases other than a straightforward `PASS`.

*   The second argument, `link`, is optional. It may be used to specify a link to more information, such as to a sub-heading on one of the NMOS Wiki [Specifications](https://github.com/AMWA-TV/nmos/wiki/Specifications) pages.
  It is recommended especially to provide further explanation of the effect of an `OPTIONAL` feature being unimplemented.

Examples of each result are included below:

```python
from .TestResult import Test

def test_my_stuff(self, test):
    """My test description"""

    # Test code
    if test_passed:
        return test.PASS()
    elif test_failed:
        return test.FAIL("Reason for failure")
    elif test_warning:
        return test.WARNING("Reason the API configuration or response is not recommended")
    elif test_disabled:
        return test.DISABLED("Explanation of why the test is disabled and e.g. how to change the test suite "
                             "config to allow it to be run")
    elif test_could_not_test:
        return test.UNCLEAR("Explanation of what prior responses prevented this test being run")
    elif test_not_implemented:
        return test.OPTIONAL("Explanation of what wasn't implemented, and why you might require it",
                             "https://github.com/AMWA-TV/nmos/wiki/Specifications#what-is-required-vs-optional")
    elif test_manual:
        return test.MANUAL("Explanation of why the test is not (yet) tested automatically, and e.g. how to "
                           "run it manually")
    elif test_not_applicable:
        return test.NA("Explanation of why the test is not applicable, e.g. due to the version of the "
                       "specification being tested")
```

The following methods may be of use within a given test definition.

**Requesting from an API**
```python
# All keyword parameters are optional
# Where 'json' is the body of the request in json and 'data' is the body as url encoded form data
self.do_request(method, url, json=json, data=data, headers=headers, auth=auth)
```
Returns a tuple of the request status (True/False) and a Requests library Response object.

**Testing an API's response**
```python
self.check_response(schema, method, response)
```
Return a tuple of the test status (True/False) and a string indicating the error in the case this is False.

**Accessing response schemas**
```python
self.get_schema(api_name, method, path, status_code)
```
Returns a JSON schema, or None if it is unavailable.

**Validating a JSON schema**
```python
self.validate_schema(payload, schema)
```
Raises an exception upon validation failure.

## Adding New Controller Tests

Controller test suite classes inherit from `ControllerTest` which in turn inherits from `GenericTest`.

`ControllerTest` provides helper functions for setting up mock resources, and sending and receiving quesions/answers to a Testing Façade.

**Mock resources**

The registry is populated as part of `set_up_tests()` which should be overridden by the test suite.

However it is important that the `ControllerTest` base implementation of `set_up_tests()` is explicitly called at the end of the overridden function as this triggers the population of the mock Registry and mock Node.

```python
ControllerTest.set_up_tests(self)
```

The Senders and Receivers registered are specified by `self.senders` and `self.receivers` which should be initialized in `set_up_tests()`.

**Specifying Senders**

Each Sender should have a `label`, `description` and a `registered` flag.
Only Senders that have a `registered` flag set to `True` will be registered with the mock Registry.
```python
self.senders = [{'label': 's1/gilmour', 'description': 'Sender 1', 'registered': True},
                {'label': 's2/waters', 'description': 'Sender 2', 'registered': False},
                {'label': 's3/wright', 'description': 'Sender 3', 'registered': False},
                {'label': 's4/mason', 'description': 'Sender 4', 'registered': True},
                {'label': 's5/barrett', 'description': 'Sender 5', 'registered': False}]
```

**Specifying Receivers**

Each Receiver should have a `label`, `description`, a `connectable` flag, and a `registered` flag.
Only Receivers that have a `connectable` flag set to `True` will have an IS-05 connection API.
Only Receivers that have a `registered` flag set to `True` will be registered with the mock Registry.

```python
self.receivers = [{'label': 'r1/palin', 'description': 'Receiver 1', 'connectable': True, 'registered': False},
                  {'label': 'r2/cleese', 'description': 'Receiver 2', 'connectable': True, 'registered': False},
                  {'label': 'r3/jones', 'description': 'Receiver 3', 'connectable': True, 'registered': False},
                  {'label': 'r4/chapman', 'description': 'Receiver 4', 'connectable': True, 'registered': False},
                  {'label': 'r5/idle', 'description': 'Receiver 5', 'connectable': True, 'registered': False},
                  {'label': 'r6/gilliam', 'description': 'Receiver 6', 'connectable': True, 'registered': False}]
```

**Sending Questions to the Testing Façade**

Questions are sent to the Testing Façade using the `_invoke_testing_facade` helper function.
The question to be displayed to the User on the Testing Façade is specified as plain text.
The list of possible answers are specified according to [question-schema.json](../testingfacade/APIs/schemas/question-schema.json).
Questions can be of the following type:
* `action`: User responds by pressing the 'Next' button.
* `single_choice`: User responds by selecting a single answer.
* `multi_choice`: User responds by selecting multiple answers.

The format of the answer returned by `_invoke_testing_facade` is specified according to [answer-schema.json](../testingfacade/APIs/schemas/answer-schema.json).

Examples of the JSON posted to the TestingFaçade, and the TestingFaçade's JSON responses posted back to the Testing tool can be found in [examples](../testingfacade/examples/).
