Developing Clowder Extractors
Developing the Computing Pipeline with Clowder Extractors
The TERRA REF computing pipeline and data management is managed by Clowder. The pipeline consists of 'extractors' that take a file or other piece of information and generate new files or information. In this way, each extractor is a step in the pipeline.
An extractor 'wraps' an algorithm in code that watches for files that it can convert into new data products and phenotypes. These extractors wait silently alongside the Clowder interface and databases. Extractors can be configured to wait for specific file types and automatically execute operations on those files to process them and extract metadata.
If you want to add an algorithm to the TERRAREF pipeline, or use the Clowder software to manage your own pipeline, extractors provide a way of automating and scaling the algorithms that you have. The NCSA Extractor Development wiki provides instructions, including:
Setting up a pipeline development environment on your own computer.
Using the web development interface) (currently in beta testing)
Using the Clowder API
TERRA-REF Extractor Resources
terrautils
To make working with the TERRA-REF pipeline as easy as possible, the terrautils Python library was written. By importing this library in an extractor script, developers can ensure that code duplication is minimized and standard practices are used for common tasks such as GeoTIFF creation and georeferencing. It also provides modules for managing metadata, downloading and uploading, and BETYdb/geostreams API wrapping.
Modules include:
betydb BETYdb API wrapper
extractors General extractor tools e.g. for creating metadata JSON objects and generating folder hierarchies
formats Standard methods for creating output files e.g. images from numpy arrays
gdal GDAL general image tools
geostreams Geostreams API wrapper
influx InfluxDB logging API wrapper
lemnatec LemnaTec-specific data management methods
metadata Getting and cleaning metadata
products Get file lists
sensors Standard sensor information resources
spatial Geospatial metadata management
Science packages
To keep code and algorithms broadly applicable, TERRA-REF is developing a series of science-driven packages to collect methods and algorithms that are generic to an input and output from the pipeline. That is, these packages should not refer to Clowder or extraction pipelines, but instead can be used in applications to manipulate data products. They are organized by sensor.
These packages will also include test suites to verify that any changes are consistent with previous outputs. The test directories can also act as examples on how to instantiate and use the science packages in actual code.
stereo_rgb stereo RGB camera (stereoTop in rawdata, rgb prefix elsewhere)
flir_ir FLIR infrared camera (flirIrCamera in rawData, ir prefix elsewhere)
scanner_3d laser 3D scanner (scanner3DTop in rawData, laser3d elsewhere)
Extractor repositories
Extractors can be considered wrapper scripts that call methods in the science packages to do work, but include the necessary components to communicate with TERRA's RabbitMQ message bus to process incoming data as it arrives and upload outputs to Clowder. There should be no science-oriented code in the extractor repos - this code should be implemented in science packages instead so it is easier for future developers to leverage.
Each repository includes extractors in the workflow chain corresponding to the named sensor.
Contact:
Extractor development and deployment: Max Burnette
Development environments: Craig Willis
On our Slack Channel
On GitHub
Last updated