aodncore.testlib package

Subpackages

Submodules

aodncore.testlib.basetest module

class aodncore.testlib.basetest.BaseTestCase(methodName='runTest')

Bases: TestCase

assertNoException(callableObj=None, *args, **kwargs)

Fail if any exception is raised

Returns

None

property config

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

property temp_dir

property temp_nc_file

aodncore.testlib.dummyhandler module

class aodncore.testlib.dummyhandler.DummyHandler(*args, **kwargs)

Bases: HandlerBase

This is an example handler, used for testing and also to demonstrate some of the kinds of things that each step might perform.

Bear in mind that this inherits everything from the HandlerBase, but only some of the things are suitable to be overridden in this child class.

Inherited instance members that will nearly always be relevant:

• self.file_collection - the main collection of files, which is a PipelineFileCollection object whose elements

  are all PipelineFile instances (see handlerlib.files to see what these are)

• self.logger - log things!

Inherited Things that you would typically implement here:

• dest_path static method to

• preprocess

• process

• postprocess methods

static dest_path(filename)

This method is optional, but if it is defined, it will be used to determine the destination path for a given input path. This could be based purely on the filename, or you could import the netcdf library and open the file and work it out based on the contents, or of course any other way you want to determine it :param filename: individual input file :return: destination path for the input file

postprocess()

Here you can run code that needs to run after the publishing step but before the notify step

Returns

None

preprocess()

Here you can run code that needs to run before the compliance checker step. This might be where you specify which files in the “eligible_files” list are “UPLOAD_ONLY”, or not published at all

Returns

None

process()

Here you can run code that needs to run after the compliance checker step but before the publishing step.

Returns

None

aodncore.testlib.handlertest module

class aodncore.testlib.handlertest.HandlerTestCase(methodName='runTest')

Bases: BaseTestCase

run_handler(*args, **kwargs)

Instantiate and run the handler class defined in self.handler_class.

Raise an AssertionError if the handler completes with any error

Parameters

• args – args passed directly to handler instance

• kwargs – kwargs passed directly to handler instance

Returns

the handler instance

run_handler_with_exception(expected_error, *args, **kwargs)

Instantiate and run the handler class defined in self.handler_class

Raise an AssertionError if the handler completes without the error attribute being the expected Exception class

Parameters

• expected_error – expected Exception class

• args – args passed directly to handler instance

• kwargs – kwargs passed directly to handler instance

Returns

the handler instance

setUp()

Hook method for setting up the test fixture before exercising it.

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

test_base()

aodncore.testlib.testutil module

class aodncore.testlib.testutil.NullStorageBroker(prefix, fail=False)

Bases: BaseStorageBroker

assert_delete_call_count(count)

assert_delete_called()

assert_delete_not_called()

assert_download_call_count(count)

assert_download_called()

assert_query_call_count(count)

assert_query_called()

assert_query_not_called()

assert_upload_call_count(count)

assert_upload_called()

assert_upload_not_called()

delete(pipeline_files, is_stored_attr='is_stored', dest_path_attr='dest_path')

Delete the given PipelineFileCollection or PipelineFile from the storage backend

Parameters

• pipeline_files – collection to delete

• is_stored_attr – PipelineFile attribute which will be set to True if delete is successful

• dest_path_attr – PipelineFile attribute containing the destination path

Returns

None

download(remote_pipeline_files, local_path, dest_path_attr='dest_path')

Download the given RemotePipelineFileCollection or RemotePipelineFile from the storage backend

Parameters

• remote_pipeline_files – collection to download

• local_path – local path to download the files into

Returns

None

query(query='')

Query the storage for existing files

A trailing slash will result in a directory listing type of query, recursively listing all files underneath the given directory.

Omitting the trailing slash will cause a prefix style query, where the results will be any path that matches the query including partial file names.

Parameters

query – S3 prefix style string (if omitted, will search with a blank prefix)

Returns

RemotePipelineFileCollection of files matching the prefix

upload(pipeline_files, is_stored_attr='is_stored', dest_path_attr='dest_path')

Upload the given PipelineFileCollection or PipelineFile to the storage backend

Parameters

• pipeline_files – collection to upload

• is_stored_attr – PipelineFile attribute which will be set to True if upload is successful

• dest_path_attr – PipelineFile attribute containing the destination path

Returns

None

aodncore.testlib.testutil.dest_path_testing(filename)

Example/test function for destination path resolution

Parameters

filename –

Returns

relative path prepended with DUMMY

aodncore.testlib.testutil.get_nonexistent_path(relative=False)

Return a path that is guaranteed not to exist

Returns

string containing guaranteed non-existent path

aodncore.testlib.testutil.get_test_config(temp_dir)

aodncore.testlib.testutil.load_runtime_patched_pipeline_config_file(config_file, rel_path, temp_dir, additional_patch=None)

Load and update pipeline config file with dynamic runtime values (e.g. temp directories)

Parameters

• config_file – Pipeline config file to load, patch and return

• rel_path – Path for wip_dir

• temp_dir – temporary directory used to update values

• additional_patch – any additional pipeline config items to patch

Returns

ConfigParser object patched with runtime test values

aodncore.testlib.testutil.make_test_file(filename, attributes=None, **variables)

Create a netcdf file with the given global and variable attributes. Variables are created as dimensionless doubles.

For example this:

make_test_file(testfile,
               {'title': 'test file', 'site_code': 'NRSMAI'},
               TEMP = {'standard_name': 'sea_water_temperature'},
               PSAL = {'standard_name': 'sea_water_salinity'}
)

will create (in cdl):

netcdf testfile {
variables:
    double PSAL ;
            PSAL:standard_name = "sea_water_salinity" ;
    double TEMP ;
            TEMP:standard_name = "sea_water_temperature" ;

// global attributes:
            :site_code = "NRSMAI" ;
            :title = "test file" ;
}

aodncore.testlib.testutil.make_zip(temp_dir, file_list)

Create a zip file in tmp_dir containing the files listed in file_list. Return the full path to the zip file.

Module contents

This package contains test “helper” code, which may also be used by other packages (e.g. aodndata).

class aodncore.testlib.BaseTestCase(methodName='runTest')

Bases: TestCase

assertNoException(callableObj=None, *args, **kwargs)

Fail if any exception is raised

Returns

None

property config

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

property temp_dir

property temp_nc_file

class aodncore.testlib.DummyHandler(*args, **kwargs)

Bases: HandlerBase

This is an example handler, used for testing and also to demonstrate some of the kinds of things that each step might perform.

Bear in mind that this inherits everything from the HandlerBase, but only some of the things are suitable to be overridden in this child class.

Inherited instance members that will nearly always be relevant:

• self.file_collection - the main collection of files, which is a PipelineFileCollection object whose elements

  are all PipelineFile instances (see handlerlib.files to see what these are)

• self.logger - log things!

Inherited Things that you would typically implement here:

• dest_path static method to

• preprocess

• process

• postprocess methods

static dest_path(filename)

This method is optional, but if it is defined, it will be used to determine the destination path for a given input path. This could be based purely on the filename, or you could import the netcdf library and open the file and work it out based on the contents, or of course any other way you want to determine it :param filename: individual input file :return: destination path for the input file

postprocess()

Here you can run code that needs to run after the publishing step but before the notify step

Returns

None

preprocess()

Here you can run code that needs to run before the compliance checker step. This might be where you specify which files in the “eligible_files” list are “UPLOAD_ONLY”, or not published at all

Returns

None

process()

Here you can run code that needs to run after the compliance checker step but before the publishing step.

Returns

None

class aodncore.testlib.HandlerTestCase(methodName='runTest')

Bases: BaseTestCase

run_handler(*args, **kwargs)

Instantiate and run the handler class defined in self.handler_class.

Raise an AssertionError if the handler completes with any error

Parameters

• args – args passed directly to handler instance

• kwargs – kwargs passed directly to handler instance

Returns

the handler instance

run_handler_with_exception(expected_error, *args, **kwargs)

Instantiate and run the handler class defined in self.handler_class

Raise an AssertionError if the handler completes without the error attribute being the expected Exception class

Parameters

• expected_error – expected Exception class

• args – args passed directly to handler instance

• kwargs – kwargs passed directly to handler instance

Returns

the handler instance

setUp()

Hook method for setting up the test fixture before exercising it.

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

test_base()

class aodncore.testlib.NullStorageBroker(prefix, fail=False)

Bases: BaseStorageBroker

assert_delete_call_count(count)

assert_delete_called()

assert_delete_not_called()

assert_download_call_count(count)

assert_download_called()

assert_query_call_count(count)

assert_query_called()

assert_query_not_called()

assert_upload_call_count(count)

assert_upload_called()

assert_upload_not_called()

delete(pipeline_files, is_stored_attr='is_stored', dest_path_attr='dest_path')

Delete the given PipelineFileCollection or PipelineFile from the storage backend

Parameters

• pipeline_files – collection to delete

• is_stored_attr – PipelineFile attribute which will be set to True if delete is successful

• dest_path_attr – PipelineFile attribute containing the destination path

Returns

None

download(remote_pipeline_files, local_path, dest_path_attr='dest_path')

Download the given RemotePipelineFileCollection or RemotePipelineFile from the storage backend

Parameters

• remote_pipeline_files – collection to download

• local_path – local path to download the files into

Returns

None

query(query='')

Query the storage for existing files

A trailing slash will result in a directory listing type of query, recursively listing all files underneath the given directory.

Omitting the trailing slash will cause a prefix style query, where the results will be any path that matches the query including partial file names.

Parameters

query – S3 prefix style string (if omitted, will search with a blank prefix)

Returns

RemotePipelineFileCollection of files matching the prefix

upload(pipeline_files, is_stored_attr='is_stored', dest_path_attr='dest_path')

Upload the given PipelineFileCollection or PipelineFile to the storage backend

Parameters

• pipeline_files – collection to upload

• is_stored_attr – PipelineFile attribute which will be set to True if upload is successful

• dest_path_attr – PipelineFile attribute containing the destination path

Returns

None

aodncore.testlib.dest_path_testing(filename)

Example/test function for destination path resolution

Parameters

filename –

Returns

relative path prepended with DUMMY

aodncore.testlib.get_nonexistent_path(relative=False)

Return a path that is guaranteed not to exist

Returns

string containing guaranteed non-existent path

aodncore.testlib.get_test_config(temp_dir)

aodncore.testlib.make_test_file(filename, attributes=None, **variables)

Create a netcdf file with the given global and variable attributes. Variables are created as dimensionless doubles.

For example this:

make_test_file(testfile,
               {'title': 'test file', 'site_code': 'NRSMAI'},
               TEMP = {'standard_name': 'sea_water_temperature'},
               PSAL = {'standard_name': 'sea_water_salinity'}
)

will create (in cdl):

netcdf testfile {
variables:
    double PSAL ;
            PSAL:standard_name = "sea_water_salinity" ;
    double TEMP ;
            TEMP:standard_name = "sea_water_temperature" ;

// global attributes:
            :site_code = "NRSMAI" ;
            :title = "test file" ;
}

aodncore.testlib.make_zip(temp_dir, file_list)

Create a zip file in tmp_dir containing the files listed in file_list. Return the full path to the zip file.