About this book

This book describes the TERRA-REF data collection, computing, and analysis pipelines. The following links provide quick access to

• Available Data

About TERRA-REF

The ARPA-E-funded Transportation Energy Resources from Renewable Agriculture Phenotyping Reference Platform (TERRA-REF) program aims to transform plant breeding by using remote sensing to quantify plant traits such as plant architecture, carbon uptake, tissue chemistry, water use, and other features to predict the yield potential and stress resistance of 300+ diverse Sorghum lines.

The data storage and computing system provides researchers with access to the reference phenotyping data and analytics resources using a high performance computing environment. The reference phenotyping data includes direct measurements and sensor observations, derived plant phenotypes, and genetic and genomic data.

Our objectives are to ensure that the software and data in the reference data and computing pipeline are interoperable, reusable, extensible, and understandable. Providing clear definitions of common formats will make it easier to analyze and exchange data and results.

Versions

• The first edition (alpha release) was published November 2016.

• The second edition (beta release) will be published November 2018

• The third edition (version 1.0) will be published November 2019

TERRA-REF uses a suite of databases and software components that are described below.

Clowder (sensor data and computation management with web user interface)

Clowder is the primary system used to organize, annotate, and process raw data generated by the phenotyping platforms as well as information about sensors. Use Clowder to explore the raw TERRA-REF data, perform exploratory analysis, and develop custom extractors. For more information, see Using Clowder.

Globus Connect (large data transfer)

Raw data is transferred to the primary TERRA-REF compute pipeline using Globus Online. Globus also provides access to TERRA REF files, but this is not a primary portal and metadata in Clowder may be required to locate and interpret these files. Use Globus Online when you want to transfer data from the TERRA-REF system for local analysis by accessing the . For more information, see .

BETYdb (phenotype data)

BETYdb is a database and web interface to the trait / phenotype data and agronomic metadata. This is where you can find plant and plot level trait data as well as plot locations and other information associated with agronomic experimental design. Use BETYdb to access derived trait and agronomic data. For more information, see .

Algorithms (a.k.a. 'extractors')

Plant CV

is an imaging processing package specific for plants that is built upon open-source software platforms , , and . Plant CV is used for trait identification, the output is stored in both Clowder and BETYdb.

Other Algorithms

Each step in the pipeline is performed by an algorithm. These are maintained in the TERRA REF GitHub organization in repositories with names that begin in extractors-* such as .

Analysis Tools

The NDS Workbench enables users to access the large filesystem and databases with familiar development environments. We provide a variety of environments for developing new algorithms and integrating them into the TERRA REF pipeline. These include and configured for specific use cases such as sensor data processing, trait analysis, database queries, and piepline development.

CoGe

CoGe contains genomic information and sequence data. For more information, see .

Field phenotyping research sites

Maricopa Agricultural Center (MAC), Arizona

The Maricopa field site is located at the the University of Arizona Maricopa Agricultural Center and USDA Arid Land Research Station in Maricopa, Arizona. At this site, we have deployed the following phenotyping platforms.

• The is the largest field crop analytics robot in the world. This high-throughput phenotyping field-scanning robot has a 30-ton steel gantry that autonomously moves along two 200-meter steel rails while continuously imaging the crops growing below it with a diverse array of .

• The PhenoTractor is fitted with a sensor frame that supports a real time kinematic (RTK) satellite navigation antenna, a sonar transducer, an infrared temperature (IRT) scanner, and three .

Kansas State University

• Tractor - coming 2017

• UAV - coming 2017

Controlled-environment phenotyping

Donald Danforth Plant Science Center, Missouri

The is a climate controlled 70 m2 growth house with a conveyor belt system for moving plants to and from fluorescence, color, and near infrared imaging cabinets. This automated, high-throughput platform allows repeated non-destructive time-series image capture and multi-parametric analysis of 1,140 plants in a single experiment.

Genomics

Genomic data includes whole-genome resequencing data from the HudsonAlpha Institute for Biotechnology, Alabama for 384 samples for accessions from the sorghum Bioenergy Association Panel (BAP) and genotyping-by-sequencing (GBS) data from Kansas State University for 768 samples from a population of sorghum recombinant inbred lines (RIL).

see https://gist.github.com/dlebauer/6b7b0e181cc5ae5034b992f725712ba4#file-sorghum-lines-genomics-md

CoGe contains genomic data.

About CoGe

CoGe is a platform for performing Comparative Genomics research. It provides an open-ended network of interconnected tools to manage, analyze, and visualize next-gen data.

Requesting Access

Coming soon

Overview

This user manual is divided into the following sections:

• Data Products: A summary of the available data products and the processes used to create them

• : Instructions for how to access the data products using Clowder, Globus, BETYdb, and CoGe

• Description of the

• : Information about data use and attribution

• : In-depth examples of how to access and use the TERRA-REF data

What data is available?

• Raw output from sensors deployed on Lemnatec field and greenhouse systems, UAVs and tractors

• Manually-collected fieldbooks and associated protocols

• Derived data, including phenomics data, from computational approaches

Audience

The TERRA-REF reference dataset may be of interest to a variety of research communities including:

• Computer vision\/remote sensing\/image analysis (raw sensor data and metadata)

• Physiologists (plants and how they are growing)

• Robotics (gantry\/location\/orientation)

About the Analysis Workbench

The Analysis Workbench allows you to launch private Jupyter Notebook and RStudio instances to explore and analyze TERRA-REF data products.

Requesting Access

To create an account, sign up at the TERRA-REF site and wait for your account to be approved. Once access is granted, you can launch analysis environments.

Scratch Space

Each user has a "home" directory mounted into the analysis tools under /home/userid. This is read-write scratch space.

Data Access

Data access is provided via a read-only NFS mount to the TERRA-REF dataset on ROGER. The data is mounted to each container under /data/terraref and linked to the analysis environment working directory. For example, in Jupyter this is /home/jovyan/work/data.

Phenotyping

The TERRA-REF project is phenotyping the same genotypes of sorghum at multiple locations

• Automated Lemnatec Scanalyzer Field System at (MAC)

Abstract

Automated VIS and NIR imaging in a controlled growth environment

Materials

Real-time sensor data transfer by file number and size can be viewed .

See for more information about individual data products and for instructions to access the data products.

Overview

TERRA’s data standards facilitate the exchange of genomic and phenomic data across teams and external researchers. Applying common standards makes it easier to exchange analytical methods and data across domains and to leverage existing tools.

When practical, existing conventions and standards have been used to create data standards. Spatial data adopts Federal Geographic Data Committee (FGDC) and Open Geospatial Consortium (OGC) data and meta-data standards. CF variable naming convention was adopted for meteorological data and biophysical data. Data formats and variable naming conventions were adapted from NEON and NASA.

Field phenotyping

The data processing pipeline transmits data from origination sites to a controlled directory structure on the CyberGIS supercomputer.

The data is generally structured as follows:

...where raw outputs from sensors per site are stored in a raw_data subdirectory and corresponding outputs from different extractor algorithms are stored in Level_1 (and eventually Level_2, etc) subdirectories.

When possible, sensor directories will be divided into days and then into individual datasets.

The following protocols have been contributed by TERRA-REF team members:

• Field Scanner - Coming 2017

• Genomics - Coming 2017

Maricopa Agricultural Center, Arizona

Automated controlled-environment phenotyping, Missouri

Current Practice

In the TERRA-REF release, sensor metadata is generally stored and exchanged using formats defined by LemnaTec. Sensor metadata is stored in metadata.json files for each dataset. This information is ingested into Clowder and available via the "Metadata" tab .

Manufacturer information about devices and sensors are available via Clowder in the collection. This collection includes datasets representing each sensor or calibration target containing specifications\/datasheets, calibration certificates, and associated reference data.

About CyVerse

is a National Science Foundation funded cyberinfrastructure that aims to democratize access to supercomputing capabilities.

Logging

Automated checks

visualizations

testing and continuous integration framework

checking that scans align with plots

• Blue Waters Nearline: NCSA 300PB+ Tape Archive (2PB Allocation)

• ROGER: CyberGIS R&D server for GIS applications, 5PB storage + variety of nodes, including large memory. roger.ncsa.illinois.edu (1PB Allocation)

Abstract

Materials

TERRA members may submit data to Clowder, BETYdb, and CoGe.

• Clowder contains data related to the field scanner operations and sensor box, including bounding box of each image / dataset as well as location of the sensor, data types and processing level, scanner missions.

• BETYdb contains plot locations and other geolocations of interest (e.g. fields, rows, plants) that are associated with agronomic experimental design / meta-data (what was planted where, field boundaries, treatments, etc).

As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.

We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.

Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.

This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.

We are developing a set of tutorials described here

Note that the tutorials assume that you are using terraref.ndslabs.org which provides all of the software dependencies along with data access.

Experiment LT1A (TM015)

ATLAS LEOTI PI_144134 PI_145619 PI_145626 PI_145632 PI_145633 PI_146890 PI_147224 PI_152591 PI_152651 PI_152694 PI_152727 PI_152728 PI_152730 PI_152733 PI_152751 PI_152771 PI_152816 PI_152828 PI_152860 PI_152862 PI_152923 PI_152961 PI_152963 PI_152965 PI_152966 PI_152967 PI_152971 PI_153877 PI_154750 PI_154844 PI_154846 PI_154944 PI_154987 PI_154988 PI_155149 PI_155516 PI_155760 PI_155885 PI_156178 PI_156203 PI_156217 PI_156268 PI_156326 PI_156330 PI_156393 PI_156463 PI_156487 PI_156871 PI_156890 PI_157030 PI_157033 PI_157035 PI_157804 PI_167093 PI_170787 PI_175919 PI_176766 PI_179749 PI_180348 PI_181080 PI_181083 PI_195754 PI_196049 PI_196583 PI_196586 PI_196598 PI_197542 PI_19770 PI_213900 PI_217691 PI_218112 PI_221548 PI_221651 PI_226096 PI_22913 PI_229841 PI_251672 PI_253986 PI_255239 PI_255744 PI_257599 PI_257600 PI_266927 PI_267573 PI_273465 PI_273969 PI_276837 PI_297130 PI_297155 PI_297171 PI_302252 PI_303658 PI_329256 PI_329286 PI_329299 PI_329300 PI_329301 PI_329310 PI_329319 PI_329326 PI_329333 PI_329338 PI_329351 PI_329394 PI_329403 PI_329435 PI_329440 PI_329465 PI_329466 PI_329471 PI_329473 PI_329478 PI_329480 PI_329501 PI_329506 PI_329510 PI_329511 PI_329517 PI_329518 PI_329519 PI_329541 PI_329545 PI_329546 PI_329550 PI_329569 PI_329570 PI_329584 PI_329585 PI_329605 PI_329614 PI_329615 PI_329618 PI_329632 PI_329644 PI_329645 PI_329646 PI_329665 PI_329673 PI_329699 PI_329702 PI_329710 PI_329711 PI_329841 PI_329843 PI_329864 PI_329865 PI_330168 PI_330169 PI_330181 PI_330182 PI_330184 PI_330185 PI_330195 PI_330196 PI_330199 PI_330796 PI_330803 PI_330807 PI_330833 PI_330858 PI_337680 PI_337689 PI_35038 PI_365512 PI_452542 PI_452619 PI_452692 PI_453696 PI_455217 PI_455221 PI_455280 PI_455301 PI_455307 PI_505717 PI_505722 PI_505735 PI_506030 PI_506069 PI_506114 PI_506122 PI_508366 PI_510757 PI_511355 PI_513898 PI_514456 PI_521019 PI_521152 PI_521280

Experiment LT1B (TM016)

PI_521290 PI_524475 PI_525049 PI_52606 PI_526905 PI_527045 PI_533792 PI_533902 PI_533998 PI_534120 PI_534165 PI_535783 PI_535785 PI_535792 PI_535793 PI_535794 PI_535795 PI_535796 PI_540518 PI_542718 PI_550604 PI_561840 PI_562730 PI_562732 PI_562781 PI_562897 PI_562970 PI_562971 PI_562981 PI_562982 PI_562985 PI_562990 PI_562991 PI_562994 PI_562997 PI_562998 PI_563002 PI_563006 PI_563009 PI_563020 PI_563021 PI_563022 PI_563032 PI_563196 PI_563222 PI_563295 PI_563329 PI_563330 PI_563331 PI_563332 PI_563338 PI_563348 PI_563350 PI_563355 PI_564163 PI_566819 PI_568717 PI_569090 PI_569097 PI_569148 PI_569244 PI_569264 PI_569416 PI_569418 PI_569419 PI_569420 PI_569421 PI_569422 PI_569423 PI_569425 PI_569427 PI_569433 PI_569435 PI_569443 PI_569444 PI_569445 PI_569447 PI_569452 PI_569453 PI_569454 PI_569455 PI_569457 PI_569458 PI_569459 PI_569460 PI_569462 PI_569465 PI_569886 PI_570031 PI_570038 PI_570042 PI_570047 PI_570053 PI_570071 PI_570073 PI_570074 PI_570075 PI_570076 PI_570085 PI_570087 PI_570090 PI_570091 PI_570096 PI_570106 PI_570109 PI_570110 PI_570114 PI_570145 PI_570254 PI_570371 PI_570373 PI_570388 PI_570393 PI_570400 PI_570431 PI_573193 PI_576399 PI_576401 PI_583832 PI_585406 PI_585448 PI_585452 PI_585454 PI_585461 PI_585467 PI_585577 PI_585608 PI_585954 PI_585961 PI_585966 PI_586435 PI_586443 PI_586541 PI_593916 PI_619807 PI_619838 PI_620072 PI_620157 PI_63715 PI_641807 PI_641810 PI_641815 PI_641817 PI_641821 PI_641824 PI_641829 PI_641830 PI_641835 PI_641836 PI_641850 PI_641860 PI_641862 PI_641892 PI_641909 PI_642998 PI_643008 PI_643016 PI_646242 PI_646251 PI_646266 PI_651491 PI_651493 PI_651495 PI_651496 PI_651497 PI_653616 PI_653617 PI_655972 PI_655978 PI_655981 PI_655983 PI_656015 PI_656026 PI_656035 PI_656065 PI_92270 PI_329471 PI_329506 PI_329569 PI_337680 PI_452692 PI_455217 PI_152730 PI_329311 NTJ2 M81e CK60B B_Az9504 ICSV700 China 17

About Clowder

Clowder is an active data repository designed to enable collaboration around a set of shared datasets. TERRAREF uses Clowder to organize, annotate, and process data generated by phenotyping platforms. Datafiles are available via the Clowder web interface or API.

See the Clowder documentation for more information about the software and its applications.

Requesting Access

To create an account, sign up at the and wait for your account to be approved. Once access is granted, you can explore collections and datasets.

Data organization

Data is organized into spaces, collections, and datasets, collections.

• Spaces contain collections and datasets. TERRA-REF uses one space for each of the phenotyping platforms.

• Collections consist of one or more datasets. TERRA-REF collections are organized by acquisition date and sensor. Users can also create their own collections.

• Datasets consist of one or more files with associated metadata collected by one sensor at one time point. Users can annotate, download, and use these sensor datasets.

Searching the database

Clowder allows users to search metadata and filter datasets and files with particular attributes. Simply enter your search terms in the search box.

Analyzing data in Clowder

Clowder includes support for launching integrated analysis environments from your browser, including RStudio and Jupyter Notebooks.

After selecting a dataset, under the "Analysis Environment Instances", select the "Launch new instance with dataset" drop-down, select the desired tool, then the "Launch" button. Select the "Environment manager" link to view the list of active instances. Find your instance and select the title link. This will display the tool with the selected dataset mounted. If you have a running instance, you can also "Upload dataset to existing instance".

Clowder Extractors

Through it's extractor architecture, Clowder supports automated computational workflows. For more information about developing Clowder extractors, see the documentation

We will release the data in stages or tiers.

First Tier

The first tier will be an internal release to the TERRA-REF team and the standards committee. This first tier release will be to initially quality check and calibrate the data and will take place as data sets are produced and compiled.

• By November 2016, it is an objective of the TERRA-REF team to establish a data release pipeline, wherein the release of data to this first tier will be within 21 days from the date of collection.

• Access to the data will be arranged for by the resource producer (i.e. limiting access to selected users).

Second Tier

The second tier will enable the release of the data generated solely by the TERRA-REF team to other TERRA teams as well as non-TERRA entities.

• By November 2017, it is an objective of the TERRA-REF team to establish a data release pipeline, wherein the release of data to this second tier will be within 10 days from the data of collection.

• It is noted that release of the data to the second tier may occur prior to publication and that access is granted with the understanding that the contributions and interests of the TERRA-REF team should be recognized and respected by the users of the data. The TERRA-REF team reserves the right to analyze and published its own data, provided that this is done in a timely fashion. Resource users should appropriately cite the source of the data and acknowledge the resource produces. The publication the data, as suggested in the TERRA-REF Authorship Guidelines, should specify the collaborative nature of the project, and authorship is expected to include all those contributing significantly to the work.

Abstract

Materials

Platforms

• SenseFly eBee fixed-wing drone

• Hexacopter

Cameras

UAV data are collected using one of three cameras:

• 5-band

• 4-band + RGB

• SenseFly thermal

Cameras are carried singly or in tandem on the SenseFly eBee fixed-wing drone (Sequoia and thermoMap, individually only), or a hexacopter (RedEdge or Sequoia, individually or in tandem).

Procedure

Flight

Standard flight altitude is 44m with 75% image overlap (both sequentially and laterally), and missions are programmed and managed by either or senseFly .

Calibration

No radiometric calibration was conducted as of Nov 5, 2016.

Analysis

Pix4D software was used to generate gray-scale orthomosaic geotiff files containing NDVI data after georegistration to the WGS84/UTM 12 N coordinate reference system using three to five 2D geo-located ground control points. These are manually matched to 5-40 images each. Ground control points for the Lemnatec Field Scanner are on the concrete pylons and were geolocated using an RTK base station maintained by the USDA-ARS at Maricopa (see section on ).

QGIS software was used to confirm geospatial alignment of NDVI geotiffs with shape files containing geolocated positions of the rail foundations. A shape file containing polygons aligning with the middle two rows of each of the 350 experimental units (for sorghum crop Aug-Nov 2016) was kindly generated by Dr. A French of USDA-ARS. Zonal Statistics in QGIS was used to calculate NDVI means for each plot polygon.

References

• MicaSense:

• SenseFly

• QGIS

Overview

Genomic data have reached a high level of standardization in the scientific community. Today, all high-impact journals typically ask the author to deposit their genomic data in either or both of these databases before publication.

Below are the most widely accepted formats that are relevant to the data and analyses generated in TERRA-REF.

Raw reads + quality scores

Raw reads + quality scores are stored in . FASTQ files can be manipulated for QC with

Reference genome assembly

Reference genome assembly (for alignment of reads or BLAST) is in . FASTA files generally need indexing and formatting that can be done by aligners, BLAST, or other applications that provide built-in commands for this purpose.

Sequence alignment

Sequence alignments are in BAM format – in addition to the nucleotide sequence, the BAM format contains fields to describe mapping and read quality. BAM files are binary files but can be visualized with . If needed, BAM can be converted in SAM (text file) with

BAM is the preferred format for sra database (sequence read archive).

SNP and genotype variants

SNP and genotype variants are in . VCF contains all information about read mapping and SNP and genotype calling quality. VCF files are typically manipulated with

VCF format is also the format required by dbSNP, the largest public repository all SNPs.

Genomic coordinates

Genomic coordinates are given in a BED format – gives the start and end positions of a feature in the genome (for single nucleotides, start = end). can be edited with .

See Also

Maricopa Agricultural Center, Arizona

Environmental Sensors Log of files transfered from Arizona to NCSA

Transferring ima

Data is sent to the gantry-cache server located inside the main UA-MAC building's telecom room via FTP over a private 10GbE interface. Path to each file being transferred is logged to /var/log/xferlog. Docker container running on the gantry-cache reads through this log file, tracking the last line it has read and scans the file regularly looking for more lines. File paths are scraped from the log and are bundled into groups of 500 to be transferred to the Spectrum Scale file systems that backs the ROGER cluster at NCSA via the Globus Python API. The log file is rolled daily and compressed to keep size in check. Sensor directories on the gantry-cache are white listed for being monitored to prevent accidental or junk data from being ingested into the Clowder pipeline.

A Docker container in the terra-clowder VM running in ROGER's Openstack environment gets pinged about incoming transfers and watches for when they complete, once completed the same files are queued to be ingested into Clowder.

Once files have been successfully received by the ROGER Globus endpoint, the files are then removed from the gantry-cache server by the Docker container running on the gantry-cache server. A clean up script walks the gantry-cache daily looking for files older than two days that have not been transferred and queues any if found.

Automated controlled-environment phenotyping, Missouri

Transferring images

Processes at Danforth monitor the database repository where images captured from the Scanalyzer are stored. After initial processing, files are transferred to NCSA servers for additional metadata extraction, indexing and storage.

At the start of the transfer process, metadata collected and derived during Danforth's initial processing will be pushed.

The current "beta" Python script can be viewed . During transfer tests of data from Danforth's sorghum pilot experiment, 2,725 snapshots containing 10 images each were uploaded in 775 minutes (3.5 snapshots\/minute).

Transfer volumes

The Danforth Center transfers approximately X GB of data to NCSA per week.

Kansas State University

HudsonAlpha - Genomics

About BETYdb

BETYdb is used to manage and distribute agricultural and ecological data. It contains phenotype and agronomic data including plot locations and other geolocations of interest (e.g. fields, rows, plants).

Requesting access

To request access to BETYdb, register on the. You will be notified once you have been granted access.

Data organization

The primary BETYdb is largely relevant here, noting the following usages:

• Genotypes are stored in the cultivars table

• Plots are stored in the sites table. Plots are nested hierarchically based on geolocation.

Using the Advanced Search box

Most tables in BETYdb have search boxes. We describe below how to use the Advanced Search box to query data from these tables and download the results as a CSV file.

The Advanced Search box is the easiest way to download summary datasets designed to have enough information (location, time, species, citations) to be useful for a wide range of use cases.

(For more information about querying data from specific tables, see the BETYdb .)

Using the Search Box

On the Welcome page of BETYdb there is a search option for trait and yield data (Figure 1). This tool allows users to search the entire collection of trait and yield data for specific sites, citations, species, and traits.

The results page provides a map interface and the option to download a file containing search results. The downloaded file is in CSV format. This file provides meta-data and provenance information, including the SQL query used to extract the data, the date and time the query was made, the citation source of each result row, and a citation for BETYdb itself.

Instructions

Using the search box to search trait and yield data is very simple: Type the site (city or site name), species (scientific or common name), cultivar, citation (author and/or year), or trait (variable name or description) into the search box and the results will show contents of BETYdb that match the search. The number of records per page can be changed to accord with the viewer's preference and the search results can be downloaded in the Excel-compatible CSV format.

The search map may be used in conjunction with search terms to restrict search results to a particular geographical area—or even a specific site—by clicking on a map. Clicking on a particular site will restrict results to that site. Clicking in the vicinity of a group of sites but not on a particular site will restrict the search to the region around the point clicked. Alternatively, if a search using search terms is done without clicking on the map, all sites associated with the returned results are highlighted on the map. Then, to zero in on results for a particular geographic area, click on or near highlighted locations on the map.

See also

Produced with Gitbook version

Location: The Automated controlled-environment phenotyping at the Donald Danforth Plant Science Center Bellwether Foundation Phenotyping Facility

The Scanalyzer 3D platform consists of multiple digital imaging chambers connected to the Conviron growth house by a conveyor belt system, resulting in a continuous imaging loop. Plants are imaged from the top and/or multiple sides, followed by digital construction of images for analysis.

• RGB imaging allows visualization and quantification of plant color and structural morphology, such as leaf area, stem diameter and plant height.

• NIR imaging enables visualization of water distribution in plants in the near infrared spectrum of 900–1700 nm.

• Fluorescent imaging uses red light excitation to visualize chlorophyll fluorescence between 680 – 900 nm. The system is equipped with a dark adaptation tunnel preceding the fluorescent imaging chamber, allowing the analysis of photosystem II efficiency.

The LemnaTec software suite is used to program and control the Scanalyzer platform, analyze the digital images and mine resulting data. Data and images are saved and stored on a secure server for further review or reanalysis.

Experiments LT1A (TM015) and LT1B (TM016)

Duration: 10 days on LemnaTec platform

Experimental Design:

• 3 replicates of were grown in a randomized complete block design

• Watering regimes = 30% FC and 100% FC

• Drought conditions were imposed 10 days after planting

The TERRA hyperspectral data pipeline processes imagery from hyperspectral camera, and ancillary metadata. The pipeline converts the "raw" ENVI-format imagery into netCDF4/HDF5 format with (currently) lossless compression that reduces their size by ~20%. The pipeline also adds suitable ancillary metadata to make the netCDF image files truly self-describing. At the end of the pipeline, the files are typically [ready for xxx]/[uploaded to yyy]/[zzz].

Installation

Software dependencies

The pipeline currently depends on three pre-requisites: . .

Pipeline source code

Once the pre-requisite libraries above have been installed, the pipeline itself may be installed by checking-out the TERRAREF computing-pipeline repository. The relevant scripts for hyperspectral imagery are:

• Main script * JSON metadata->netCDF4 script

Setup

The pipeline works with input from any location (directories, files, or stdin). Supply the raw image filename(s) (e.g., meat_raw), and the pipeline derives the ancillary filename(s) from this (e.g., meat_raw.hdr, meat_metadata.json). When specifying a directory without a specifice filename, the pipeline processes all files with the suffix "_raw".

shmkdir ~/terrarefcd ~/terrarefgit clone [email protected]:terraref/computing-pipeline.gitgit clone [email protected]:terraref/documentation.git

Run the Hyperspectral Pipeline

shterraref.sh -i ${DATA}/terraref/foo_raw -O ${DATA}/terrarefterraref.sh -I /projects/arpae/terraref/raw_data/lemnatec_field -O /projects/arpae/terraref/outputs/lemnatec_field

The software that makes up the terraref system runs on different VM's. Some of the services leveraged by the systems runs in a replicated mode so that the overall system will not stop working if any of the underlying VM's goes down.

Following is the overview of the system as it is running now:

• terraref is the frontend for everything, runs nginx

• terra-clowder runs the data management system clowder, connected to terra-mongo-[123], terra-es-[123], terra-postgres and the (using NFS mount)

• terra-geodashboard runs the geodashboard software, connected to terra-clowder

• terra-thredds runs the thredds server (experimental), connected to roger filesystem (using NFS moutn)

• terra-es-[123] run elasticsearch 2.4 and for a cluster

• terra-mongo-[123] run mongo 3.6 in a replicated cluster, terra-mongo-3 is an arbiter and does not hold any data

• terra-postgres runs postgres 9.5

About Globus Connect

The service provides high-performance, secure, file transfer and synchronization between endpoints. It also allows you to securely share your data with other Globus users.

Summary

Infrared heat imaging data is collected collected using the FLIR SC615 thermal sensor. These data are provided as geotiff image raster files as well as plot level means.

• Algorithms are in the repository; see the readme for details.

Data Product Levels

Data products are processed at various levels ranging from Level 0 to Level 4. Level 0 products are raw data at full instrument resolution. At higher levels, the data are converted into more useful parameters and formats. These are derived from NASA and NEON

Raw data

Running nightly on ROGER.

Script is hosted at: /gpfs/smallblockFS/home/malone12/terra_backup

Script uses the Spectrum Scale policy engine to find all files that were modified the day prior, and passes that list to a job in the batch system. The job bundles the files into a .tar file, then uses pigz to compress it in parallel across 18 threads. Since this script is run as a job in the batch system, with variables passed with the date, if the batch system is busy, the backups won't need to preclude each other. The .tgz files are then sent over to NCSA Nearline using Globus, then purged from file system.

Genomic data includes whole-genome resequencing data from the HudsonAlpha Institute for Biotechnology, Alabama for 384 samples for accessions from the sorghum (BAP) and genotyping-by-sequencing (GBS) data from Kansas State University for 768 samples from a population of sorghum recombinant inbred lines (RIL).

These data are available to Beta Users and require permission to access. The form to sign up for our beta user program is at . Once you have signed up for our beta user program you can access genomics data in one of the following locations:

• Download via .

Summary

Fluorescence intensity data is collected using the PSII camera.

Raw data access

Genomic data includes whole-genome resequencing data from the HudsonAlpha Institute for Biotechnology, Alabama for 384 samples for accessions from the sorghum Bioenergy Association Panel (BAP) and genotyping-by-sequencing (GBS) data from Kansas State University for 768 samples from a population of sorghum recombinant inbred lines (RIL).

Whole-genome resequencing

Experimental Design:

supports the genomics pipeline required for the TERRA program for Sorghum sequence alignment and analysis. It has a web interface and REST API. CoGe is developed by Eric Lyons and hosted at the University of Arizona, where it is made available for researchers to use. CoGe can be hosted on any server, VM, or Docker container.

Submitting Sequences to the CoGe Pipeline

Overview

TERRA-REF data is available through four different approaches: Globus Connect, Clowder, BETYdb, and CoGe. Raw data is transfered to the primary compute pipeline using Globus Online. Data is ingested into Clowder to support exploratory analysis. The Clowder extractor system is used to transform the data and create derived data products, which are either available via Clowder or published to specialized services, such as BETYdb.

For more information, see the .

This section includes the following:

Release with Attribution

We plan to make data from the Transportation Energy Resources from Renewable Agriculture Phenotyping Reference Platform (TERRA-REF) project available for use with attribution. Each type of data will include or point to the appropriate attribution policy.

Timing and Control of Release

We plan to release the data in stages or tiers. For pre-release access please complete the .

1. The first tier will be an internal release to the TERRA-REF team and the standards committee. This first tier release will be to initially quality check and calibrate the data and will take place as data sets are produced and compiled.

   1. By November 2016, it is an objective of the TERRA-REF team to establish a data release pipeline, wherein the release of data to this first tier will be within 21 days from the date of collection.

Genomic Data

Restrictions on dataset usage

Genomic data for the Sorghum bicolor Bioenergy Association Panel (BAP) from the TERRA-REF project is available pre-publication to maximize the community benefit of these resources. Use of the raw and processed data that is available should follow the principles of the and the .

By accessing these data, you agree not to publish any articles containing analyses of genes or genomic data on a whole genome or chromosome scale prior to publication by TERRA-REF and/or its collaborators of a comprehensive genome analysis ("Reserved Analyses"). "Reserved analyses" include the identification of complete (whole genome) sets of genomic features such as genes, gene families, regulatory elements, repeat structures, GC content, or any other genome feature, and whole-genome- or chromosome-scale comparisons with other species. The embargo on publication of Reserved Analyses by researchers outside of the TERRA-REF project is expected to extend until the publication of the results of the sequencing project is accepted. Scientific users are free to publish papers dealing with specific genes or small sets of genes using the sequence data. If these data are used for publication, the following acknowledgment should be included: 'These sequence data were produced by the US Department of Energy Transportation Energy Resources from Renewable Agriculture Phenotyping Reference Platform (TERRA-REF) Project'. These data may be freely downloaded and used by all who respect the restrictions in the previous paragraphs. The assembly and sequence data should not be redistributed or repackaged without permission from TERRA-REF. Any redistribution of the data during the embargo period should carry this notice: "The TERRA-REF project provides these data in good faith, but makes no warranty, expressed or implied, nor assumes any legal liability or responsibility for any purpose for which the data are used. Once the sequence is moved to unreserved status, the data will be freely available for any subsequent use."

We prefer that potential users of these sequence data contact the individuals listed under Contacts with their plans to ensure that proposed usage of sequence data are not considered Reserved Analyses.

Software and Algorithms

For algorithms, we intend to release via or MIT compatible license (e.g. BSD, UIUC/NCSA, Apache v2).

Images, Phenotypes, and Other Raw Data

For other raw data, such as phenotypic data and associated metadata, we intend to release via .

Contacts

Todd Mockler, Project/Genomics Lead (email: tmockler AT danforthcenter DOT org)

David LeBauer, Computing Pipeline Lead (email: dlebauer AT illinois DOT edu)

Erica Fishel, Technology Transfer Lead (email: efischel AT danforthcenter DOT org)

Nadia Shakoor, Associate Project Director (email: nshakoor AT danforthcenter DOT org)

For use by the TERRA Reference Phenotyping Standards Committee.

Overview

All of the web-based software below provides the ability to organize projects hierarchically, facilitate sharing, and support collaboration. Much of this is publicly viewable.

Core Communication Tools

• project management, website content and hosting, collaborative software development

• Google Drive collaborative editing of documents that we create (notes, manuscripts, etc)

• ()

GitHub

TERRA Ref sites on Github:

• Data products repository

  • issues and milestones:

• Computational Pipeline Repository

Using Github:

• Features

  • Interface to 'git', a specialized command-line tool for version control.

  • :

Current Practice

In TERRA-REF v0 release, agronomic and phenotype data is stored and exchanged using the BETYdb API. Agronomic data is stored in the sites, managements, and treatments tables. Phenotype data is stored in the traits, variables, and methods tables. Data is ingested and accessed via the BETYdb API formats.

Standardization Efforts

In cooperation with participants from , the , and groups, the TERRA-REF team is pursuing the development of a format to facilitate the exchange of data across systems based on the ICASA Vocabulary and AgMIP JSON Data Objects. An initial draft of this format is available for comment on

In addition, we plan to enable the TERRA-REF databases to import and export data via the .

Summary

3D point cloud data is collected using the Fraunhofer 3D laserscanner. .

Data access

Data is available via Clowder and Globus.

• Clowder:

• Globus path: /sites/ua_mac/raw_data/scanner3DTop

• Sensor information:

For details about using this data via Clowder or Globus, please see section.

Computational pipeline

Raw sensor output (PLY) is converted to LAS format using the ply2las extractor

• Description: PLY data is converted to LAS using the 3D point cloud extractor

• Output:

  • Clowder: LAS file is added to the dataset

See also

Summary

Hyperspectral imaging data is collected using the Headwall VNIR and SWIR sensors. In the Nov 2017 Beta Release only VNIR data is provided because we do not have the measurements of downwelling spectral radiation required by the pipeline.

Please see the for more information about how the data are generated and known issues.

Authors: Matthew Maimaitiyiming, Wasit Wulamu, and David LeBauer

Center for Sustainability, Saint Louis University, St. Louis, MO 63108

Abstract

This document provides a brief summary of methods, procedures, and workflows to process the tractor data.

Maricopa Agricultural Center, Arizona

• The Lemnatec Scanalyzer Field Gantry System

This section describes sensor calibration processes and how to access additional information about specific calibration protocols, calibration targets, and associated reference data.

LemnaTec Field Scanalyzer

Calibration protocols

Calibration protocols have been defined by LemnaTec in cooperation with vendors and the TERRA-REF Sensor Steering Committee. Draft calibration protocols are currently in and have been incorporated into the .

A detailed calibration process is also provided for the , with further information below.

Calibration targets

The following calibration targets are available:

• Aluminum 3D test object

Sensor Calibration

Environmental sensor calibration

The environmental sensor has been calibrated by LemnaTec. The output of the spectrometer is raw counts, users will need to use the calibration files to convert to units of µW m-2 s-1, taking into account the bandwidth of the chip (0.4nm) if converting to µmol m-2 s-1.

Calibration reference data is available via Globus /sites/ua-mac/EnvironmentLogger/CalibrationDataor in Github

Hyperspectral calibration

Sources:

For the SWIR and VNIR sensors, factory calibration is repeated each year using the calibration lamp provided by Headwall. To convert the hyperspectral exposure image to reflectance requires the wavelength-dependent, factory calibrated reflectance of the spectralon at all VNIR and SWIR wavelengths and a good image of a spectralon panel from each camera. This includes periodic measurements of a white spectralon reflectance panel run with 20ms exposure to match panel calibration.

Dark reference measurement:

• VNIR

  • Dark measurement for VNIR camera is taken at exposure times 20, 25, 30, 35, 40, 45, 50, 55ms.

  • Data is in the same hypercube format with 180-200 lines, 955 bands, and 1600 pixel samples.

White reference measurement:

• VNIR

  • White measurement for VNIR camera is taken at exposure times 20, 25, 30, 25, 40, 25, 50, 55 ms.

  • The name of the folder is the exposure time. Data are 1600 sample, 955 bands and 268-298 lines. White reference is located in the lines between 60 to 100 and in the samples between 600 to 1000.

Stereo 3D height scanner

LemnaTec applied calibration matrix to the 3D scanners.

UAV calibration

Source:

• There are calibrated reference panels and blackbody images taken with UAV sensors before and\/or after the each flight mission.

• There are also 4 white,grey and black panels laid on the ground during the flight. Knowing the proprieties of these targets would helps us radiometrically correct the UAV images.

• What are the reflectance properties of calibrated reference panels for multispectral camera?

Halogen spectrum

No per-wavelength analysis of light produced by the halogen lights is available from the vendor for Showtec 240V\/75W. Measurements are available for a similar halogen bulb Philips Twistline Halogen 230V 50W 18072 in Github:

Spectral response data

Relative spectral response data is available for the following sensors:

• NDVI

• PRI

• PAR

Calibration data

Where available, per device calibration certificates are included in the collections.

Meteorological data will use Climate Forecasting 'standard names' and 'canonical units' conventions. CF is widely used in climate, meteorology, and earth sciences.

Here are some examples (note that we can change from canonical units to match the appropriate scale, e.g. "C" instead of "K"; time can use any base time and time step (e.g. hours since 2015-01-01 00:00:00 UTC, etc. But the time zone has to be UTC, where 12:00:00 is approx (+/- 15 min). solar noon at Greenwich.

• standard_name is CF-convention standard names (except irrigation)

• units can be converted by udunits, so these can vary (e.g. the time denominator may change with time frequency of inputs)

Running The Pipeline

Before the Running

The pipepline is developed in Python, so a Python Interpreter is a must. Other than the basic Python standard librarys, the following third-party libraries are required:

• netCDF4 for Python

• numpy

Other than official CPython interpreter, Pypy is also welcomed, but please make sure that these third-party modules are correctly installed for the target interpreter. The pipeline can only works in Python 2.X versions (2.7 recommended) since numpy does not support Python 3.X versions.

Cloning from the Git:

The extractor for this pipeline is developed and maintained by Max in branch "EnvironmentalLogger-extractor" under the same repository.

Get the Environmental Logger Pipeline to Work

To trigger the pipeline, use the following command:

python ${environmental_logger_source_path}/environmental_logger_json2netcdf.py ${input_JSON_file} ${output_netCDF_file}

Where:

• ${environmental_logger_source_path} is where the three environmental_logger files are located

• ${input_JSON_file} is where the input JSON files are located

• ${output_netCDF_file}

Please note that the parameter for the output file can be a path to either a directory or a file, and it is not necessarily to be existed. If the output is a path to a folder, the final product will be in this folder as a netCDF file that has the same name as the imported JSON file but with a different filename extension (.nc for standard netCDF file); if this path does not exist, environmental_logger pipeline will automatically make one.

Calculation

The calculation in the Environmental Logger is mainly finished by the module under the support of numpy.

The following table lists available TERRA-REF data products. The table will be updated as new datasets are released. Links are provided to pages with detailed information about each data product including sensor descriptions, algorithm (extractor) information, protocols, and data access instructions.

See also

• ?

Web Interface Data Uploads

1. Log in with your account

2. Click 'Datasets' > 'Create'

3. Provide a name and description

4. Click 'Select Files' to choose which files to add

5. Click 'Upload' to save selected files to dataset

6. Click 'View Dataset' to confirm. You can add more content with 'Add Files'.

7. Add metadata, terms of use, etc.

Some metadata may automatically be generated depending on the types of files uploaded. Metadata can be manually added to files or datasets at any time.

API Data Uploads

Clowder also includes a RESTful API that allows programmatic interactions such as creating new datasets and downloading files. For example, one can request a list of datasets using: GET _clowder home URL_/api/datasets. The current API schema for a Clowder instance can be accessed by selecting API from the ? Help menu in the upper-right corner of the application.

For typical workflows, the following steps are sufficient to push data into Clowder in an organized fashion:

1. Create a collection to hold relevant datasets (optional) POST /api/collections provide a name; returns collection ID

2. Create a dataset to hold relevant files and add it to the collection POST /api/datasets/createempty provide a name; returns dataset ID POST /api/collections/<collection id>/datasets/<dataset id>

An extensive API reference can be found .

Uploading Data Using Globus

Some files, e.g. those transferred via Globus, will be moved to the server without triggering Clowder's normal upload paths. These must be transmitted in a certain way to ensure proper handling.

1. Log into and click 'Transfer Files'.

2. Select your source endpoint, and Terraref as the destination. You need to contact NCSA to ensure you have the necessary credentials and folder space to utilize Globus - unrecognized Globus accounts will not be trusted.

3. Transfer your files. You will receive a Task ID when the transfer starts.

In this way Clowder indexes a pointer to the file on disk rather than making a new copy of the file; thus the file will still be accessible via Globus, FTP, or other methods directed at the filesystem.

Several extractors push data to the Clowder Geostreams API, which allows registration of data streams that accumulate datapoints over time. These streams can then be queried, visualized and downloaded to get time series of various measurements across plots and sensors.

TERRA-REF organizes data into three levels:

• Location (e.g. plot, or a stationary sensor)

  • Information stream (a particular instrument's data, or a subset of one instrument's data)

    • Datapoint (a single observation from the information stream at a particular point in time)

Sensor destinations

Here, the various streams that are used in the pipeline and their contents are listed.

• Location group

  • Stream name

    • Datapoint property [units / sample value]

Season 1 sorghum (April - July 2016) Season 2 sorghum (August - November 2016) Durum wheat (January 2017 -

Season 1 soghum (April - July 2016)

.

Developing the Computing Pipeline with Clowder Extractors

The TERRA REF computing pipeline and data management is managed by Clowder. The pipeline consists of 'extractors' that take a file or other piece of information and generate new files or information. In this way, each extractor is a step in the pipeline.

An extractor 'wraps' an algorithm in code that watches for files that it can convert into new data products and phenotypes. These extractors wait silently alongside the Clowder interface and databases. Extractors can be configured to wait for specific file types and automatically execute operations on those files to process them and extract metadata.