Skip to content

Run tests

A component in OpenProblems will typically come with at least two unit tests out of the box:

  • The first test checks whether the config file has all required fields.
  • The second test checks whether the component runs correctly and whether the output has the expected dimensions and fields.

Use viash test to run all unit tests for a component.

Example

Section titled “Example”
viash test src/methods/logistic_regression/config.vsh.yaml

Test multiple components

Section titled “Test multiple components”

Use viash ns test to unit test all components of a given task at once.

viash ns test --parallel
Example output
          namespace                 name               runner               engine            test_name exit_code duration               result
    control_methods          true_labels           executable               docker                start
    control_methods          true_labels           executable               docker     build_executable         0        1              SUCCESS
    control_methods          true_labels           executable               docker run_and_check_output.py         0        3              SUCCESS
    control_methods          true_labels           executable               docker      check_config.py         0        3              SUCCESS
    data_processors      process_dataset           executable               docker                start
    data_processors      process_dataset           executable               docker     build_executable         0        1              SUCCESS
    data_processors      process_dataset           executable               docker run_and_check_output.py         0        3              SUCCESS
            methods  logistic_regression           executable               docker                start
            methods  logistic_regression           executable               docker     build_executable         0        1              SUCCESS
            methods  logistic_regression           executable               docker run_and_check_output.py         0        3              SUCCESS
            methods  logistic_regression           executable               docker      check_config.py         0        3              SUCCESS
            metrics             accuracy           executable               docker                start
            metrics             accuracy           executable               docker     build_executable         0        1              SUCCESS
            metrics             accuracy           executable               docker run_and_check_output.py         0        3              SUCCESS
            metrics             accuracy           executable               docker      check_config.py         0        3              SUCCESS
All 11 configs built and tested successfully

Common errors

Section titled “Common errors”

Below is a listing of common errors and how to solve them. If you encounter other problems, check the troubleshooting page or open a GitHub issue.

Assertion error

Section titled “Assertion error”

An assertion error typically occurs when the data format of an input or output parameter is incorrect.

Script errors — the output file is missing or has wrong fields:

+/tmp/viash_test_knn12471306149427017048/test_generic_test/test_executable
>> Running script as test
>> Checking whether output file exists
>> Reading h5ad files
>> Checking whether predictions were added
Traceback (most recent call last):
  File "/viash_automount/.../viash-run-knn-QTKmUM.py", line 57, in <module>
    assert "label_predi" in output.obs
AssertionError

Config errors — a required metadata field is missing or misspelled. Add an empty string "" to include it even when the value is not yet known:

+/tmp/viash_test_knn12945373156205296243/test_check_method_config/test_executable
Load config data
check general fields
Check info fields
Traceback (most recent call last):
  File "/viash_automount/.../viash-run-knn-Xn2Vd7.py", line 42, in <module>
    assert "summary" in info is not None, "summary not an info field or is empty"
AssertionError: summary not an info field or is empty

Python / R dependency does not exist

Section titled “Python / R dependency does not exist”

When a dependency is missing from the Docker setup, you will see a ModuleNotFoundError. Add the dependency to the setup section of the config.

ModuleNotFoundError: No module named 'yaml'

Docker image not found

Section titled “Docker image not found”

Check for spelling mistakes in the image name.

#3 ERROR: docker.io/library/python:3.1: not found
...
FROM python:3.1
...
ERROR: failed to solve: python:3.1: docker.io/library/python:3.1: not found
[error] Error occurred while building container
ERROR! Setup failed!

Script error

Section titled “Script error”

When the executed script fails, the stack trace is printed. Look for the root cause there.

+/tmp/viash_test_knn14797416935521308344/test_generic_test/test_executable
>> Running script as test
Load input data
Traceback (most recent call last):
  File "/tmp/viash-run-knn-p4pvkA.py", line 31, in <module>
    input_test = ad.read_h5ad(par['input_test'])
  ...
TypeError: expected str, bytes or os.PathLike object, not NoneType
Method script with returncode ...