# Usage

The following guide indicates how to run the tool against an implementation of one of the NMOS APIs.

Description | Screenshot
--- | ---
Open a browser and go to the address of the server you [deployed](1.0.%20Installation.md). | ![Testing Tool Launcher](images/initial-launch.png "Testing Tool Launcher")
Select a test suite from the dropdown. | ![Test Suite Dropdown](images/test-dropdown.png "Test Suite Dropdown")
Provide the IP or hostname and port of the relevant API under test, along with the preferred API version. | ![Test Configuration](images/test-setup.png "Test Configuration")
Optionally, select a subset of the tests to run. This should default to 'all'. | ![Test Selection](images/test-selection.png "Test Selection")
Once all of the field have been completed, press 'Run'. The result of the tests will be shown after a few seconds. |![Example Results Window](images/test-results.png "Example Results Window")

## Understanding the Results

The result of each test case will be one of the following:

| Test Result | Reason |
| - | - |
| ![Pass](https://place-hold.it/128x32/28a745.png?text=Pass&fontsize=12&bold) | Successful test case. |
| ![Fail](https://place-hold.it/128x32/dc3545.png?text=Fail&fontsize=12&bold) | Required feature of the specification has been found to be implemented incorrectly. |
| ![Warning](https://place-hold.it/128x32/ffc107.png?text=Warning&fontsize=12&bold) | Not a failure, but the API being tested is responding or configured in a way which is not recommended in most cases. |
| ![Test Disabled](https://place-hold.it/128x32/ffc107.png?text=Test%20Disabled&fontsize=12&bold) | Test is disabled due to test suite configuration; change the config or test manually. |
| ![Could Not Test](https://place-hold.it/128x32/ffc107.png?text=Could%20Not%20Test&fontsize=12&bold) | Test was not run due to prior responses from the API, which may be OK, or indicate a fault. |
| ![Not Implemented](https://place-hold.it/128x32/ffc107.png?text=Not%20Implemented&fontsize=12&bold) | Recommended/optional feature of the specifications has been found to be not implemented. |
| ![Manual](https://place-hold.it/128x32/007bff.png?text=Manual&fontsize=12&bold) | Test suite does not currently test this feature, so it must be tested manually. |
| ![Not Applicable](https://place-hold.it/128x32/6c757d.png?text=Not%20Applicable&fontsize=12&bold) | Test is not applicable, e.g. due to the version of the specification being tested. |

## Configuration

The testing tool has a number of general configuration settings that affect its operation, defined in `nmostesting/Config.py`.
 
To alter these, copy `nmostesting/UserConfig.example.py` to `nmostesting/UserConfig.py` and make your changes there.

`MAX_TEST_ITERATIONS` can be set to restrict the maximum number of resources or test points that time-consuming tests run against.
The default of `0` means unlimited (all available resources or test points) for a really thorough test!

## Test-specific Documentation

The following pages provide information on additional instructions for testing specific features:

- [Unicast Discovery](2.1.%20Usage%20-%20Testing%20Unicast%20Discovery.md)
- [BCP-003-01](2.2.%20Usage%20-%20Testing%20BCP-003-01%20TLS.md)
- [Authorization](2.3.%20Usage%20-%20Testing%20IS-10%20Authorization.md)
- [SDP Files](2.4.%20Usage%20-%20Testing%20of%20SDP%20Files.md)
- [IS-07 MQTT](2.7.%20Usage%20-%20Testing%20IS-07%20MQTT.md)
- [Controllers](2.8.%20Usage%20-%20Testing%20Controllers.md)
- [Invasive Device Model Testing](2.10%20Usage%20-%20Invasive%20Device%20Model%20Testing.md)

## Non-interactive Testing

The testing tool supports two modes of operation that support usage in larger scale or continuous integration tests.

- [Non-Interactive Mode](2.5.%20Usage%20-%20Non-Interactive%20Mode.md)
- [Using the API](2.6.%20Usage%20-%20Using%20the%20API.md)
- [Controllers](2.9.%20Usage%20-%20Fully%20Automated%20Testing%20of%20Controllers.md)

## Advanced Testing

Some of the tests contained within this tool perform a number of steps which may not be obvious without viewing the source code.
To help with debugging, descriptions of such behaviour is covered by the [advanced testing](6.0.%20Advanced%20Testing.md) pages.
