Testing¶
In this section, we will talk about how to test different modules and algorithms in garage.
In garage, we use pytest to carry out tests.
Test Modules¶
All test files are stored under tests/
in the
garage respository.
Test modules are organized in the same way as the
garage main repository. Ideally, all main modules
and files under src/garage/
should be covered
with test cases.
Test modules of garage are structured in the following ways:
tests/
contains all tests and supporting codetests/garage
contains unit teststests/integration_tests
contains integration testsmodules in
tests/
andtests/fixtures
contain helper codes and fixtures which make writing good tests easier.
tests
├── fixtures
├── garage
├── integration_tests
├── helpers.py
└── (helper_files.py)
Prerequisites¶
To begin testing, we suggest using the same
Python manager environment i.e. virtualenv
, conda
as we develop on garage repository so that all
packages and dependencies required for testing
are installed in the virutal environment.
Let’s begin by activating the virtual environment:
# Conda
conda activate myenv
# Virtualenv
source myenv/bin/activate
Next, we need to install pytest. Generally, pytest
should have already been installed upon garage’s
dev
installation. To install the garage environment
for testing, you will need to install the
garage[all,dev]
for dependencies such as pytest,
mujoco and so on.
Also, you may want to check out our installation guide first before diving into tests. It is also recommened to check out official pytest documentation.
cd path/to/garage/
pip install -e '.[all,dev]'
To get started, run pytest as follows:
pytest
Basic Pytest Usage¶
Congrats! Now you’re ready for testing garage!
We will start with a simple test. To be consistent
with pytest requirement and garage modules,
we name our test .py
file with pre-fix test_
.
Let’s write a simple test case for the vpg
algorithm src/garage/tf/algos/vpg.py
.
We begin by creating a file called test_vpg.py
and put it under the tree tests/garage/tf/algos/
.
We want to test VPG in the cart pole environment.
Hence, we create a unitest named test_vpg_cartpole
.
Inside the function, we define the environemnt,
the policy and the baselien and feed them to the
VPG algorithm and run the experiment. Finally,
we make sure the return value is identical to our
expectation by using assert
.
In new xunit-style tests, multiple tests are modeled
into a class for modular and scalable structure.
There is no need to subclass or anything,
but make sure the prefix of the class starts
with Test
, otherwise the class will be skipped.
Note that it’s not encouraged to use this style,
especially when a test doesn’t require setup_method
and teardown_method
.
# test_vpg.py
class TestVPG(...):
def test_vpg_cartpole(self):
...
env = GymEnv('CartPole-v1')
policy = CategoricalMLPPolicy(name='policy',
env_spec=env.spec,
...)
baseline = LinearFeatureBaseline(env_spec=env.spec)
algo = VPG(env_spec=env.spec,
policy=policy,
baseline=baseline,
discount=0.99,
optimizer_args=dict(learning_rate=0.01, ))
trainer.setup(algo, env)
last_avg_ret = trainer.train(n_epochs=10, batch_size=10000)
assert last_avg_ret > 90
env.close()
Run Tests¶
In general, we can start running tests simply by:
pytest
However, in most of the cases, we simply don’t have the time to test everything, plus Travis CI will take care of the majority of tests upon the deployment workflow. We can use the following ways to carry out specific tests to make life easier.
Specifying Tests / Selecting Tests¶
Pytest supports several ways to run and select tests.
Run tests in a directory¶
Run a test on particular module(s) by specifying a directory path.
pytest tests/garage/tf/algo/
Run tests in a module¶
Run a test on a particular module by specifying a file path.
pytest tests/garage/tf/algo/test_ppo.py
Run tests by keyword expressions¶
Run tests by keyword expressions. This is useful for running particular test function(s).
pytest -k test_ppo_pendulum_continuous_baseline
Useful Pytest Methods¶
Below are the pytest methods and functions that we found helpful for testing garage.
Setup and teardown methods¶
setup_method
is called before every tests to
set up the test environment. It setups any state
tied to the execution of the given method in
a class. setup_method
is invoked for every
test method of a class.
teardown_method
is called after every tests to
teardown any state that was previously setup
with a setup_method
.
For details on setup_method
and teardown_method
, check this.
class TestSampleClass:
def setup_method(self):
"""Setup method which is called before every test."""
...
def teardown_method(self):
"""Teardown method which is called after every test."""
...
Parametrized test functions¶
Parametrized test function is a delightful solution to save us from tedious testing in same scenarios with different parameters. We can simply specify the name of the arguments that will be pass to the test function and a list of arguments corresponding to the names.
import pytest
@pytest.mark.parametrize('filters, in_channels, strides', [
(((32, (1, 1)),), (3, ), (1, )),
(((32, (3, 3)),), (3, ), (1, )),
(((32, (3, 3)),), (3, ), (2, )),
(((32, (1, 1)), (64, (1, 1))), (3, 32), (1, 1)),
(((32, (3, 3)), (64, (3, 3))), (3, 32), (1, 1)),
(((32, (3, 3)), (64, (3, 3))), (3, 32), (2, 2)),
])
def test_output_value(self, filters, in_channels, strides):
model = CNNModel(filters=filters,
strides=strides,
name='cnn_model',
padding='VALID',
hidden_w_init=tf. constant_initialize(1,
hidden_nonlinearity=None)
...
This page was authored by Iris Liu (@irisliucy).