TERRA-REF Documentation
WebsiteGitHubTutorials
Primary version
Primary version
  • Introduction
  • Scientific Objectives
  • Experimental Design
    • The Maricopa Agricultural Center (MAC)
    • Controlled Environment Phenotyping
    • Genomics
  • Data
    • How to Access Data
    • Data Products
      • Environmental conditions
      • Phenotype Data
      • Genomics data
      • Fluorescence intensity imaging
      • Geospatial information
      • Hyperspectral imaging data
      • Infrared heat imaging data
      • Meteorological data
      • Point Cloud Data
      • Controlled Environment phenotype data
    • Data Use Policy
    • Manuscripts and Authorship Guidelines
  • Protocols
    • Field Scanner
    • Sensor Calibration
    • Hyperspectral Data
    • Controlled Environment Protocols
    • Manual Field Data Protocols
    • Phenotractor Protocols
    • UAV Protocols
    • Genomic Protocols
  • Technical Documentation
    • Software
    • Data Standards
      • Existing Data Standards
      • Agronomic and Phenotype Data Standards
      • Genomic Data Standards
      • Sensor Data Standards
      • Data Standards Committee
    • Data Product Levels
    • Directory Structure
    • Data Transfer
    • Data Processing Pipeline
    • Time Series Data in Geostreams
    • Data Backup
    • Systems Configuration
  • Code of Conduct
  • Appendix
    • Glossary
    • Accessing BETYdb with GIS Software
  • References
  • Archived Documentation
    • Developer Manual
      • Submitting data to Clowder
      • Submitting data to BETYdb
      • Submitting Data to CoGe
      • Developing Clowder Extractors
Powered by GitBook
On this page
  • Overview
  • Tutorials (Recommended!)
  • Globus: Browse and Transfer Files
  • BETYdb: Trait Data and Agronomic Metadata
  • Accessing data in R
  • Using SQL and PostGIS with Docker (Advanced Users)
  • Clowder: Sensor Data and Metadata Browser
  • CyVerse: Genomics Data
  • CoGe: Genomics Data
Export as PDF
  1. Data

How to Access Data

Overview

TERRA-REF data can be accessed through many different interfaces: Globus, Clowder, BETYdb, CyVerse, and CoGe. Raw data is transfered to the primary compute pipeline using Globus Online. Data is ingested into Clowder to support exploratory analysis. The Clowder extractor system is used to transform the data and create derived data products, which are either available via Clowder or published to specialized services, such as BETYdb.

Resource

Use

Web User Interface

API*

clients

Sensor Data

Globus

Browse directories; transfer large sensor files

globus.org #TERRAREF endpoint

R, Python

Clowder

Browse and Download small Sensor Data

terraref.org/clowder

Python

Trait Data

BETYdb

Trait and Agronomic Metadata

terraref.org/bety

R traits package, Python: terrautils; SQL: Postgres in Docker

traitvis

View available trait data

terraref.org/traitvis

NA

NA

Genomics Data

CyVerse

Download Genomics data

terraref.org/cyverse-genomics

yes

CoGe

Download, process, visualize Genomics data

terraref.org/coge

Other

Tutorials

R and Python scripts for accessing data

terraref.org/tutorials

NA

Advanced Search

Search across sensor and trait data

search.terraref.org (under development)

yes

Tutorials (Recommended!)

We have developed tutorials to provide users with both 'quick start' vignettes and more detailed introductions to TERRA REF datasets. Tutorials for accessing trait data, sensor data, and genomics data are organized by directory ("traits", "sensors", and "genomics").

The tutorials assume familiarity with or willingness to learn Python and / or R, and provide the greatest flexibility and access to available data.

Globus: Browse and Transfer Files

Raw data is transferred to the primary TERRA-REF file system at the National Center for Computing Applications at the University of Illinois.

Use Globus Online when you want to transfer data from the TERRA-REF system for local analysis.

Transferring data using Globus Connect:

To access data via Globus, you must first have a Globus account and endpoint.

  1. Select source

    • Endpoint: #Terraref

    • Path: Navigate to the subdirectory that you want.

    • Select (click) a folder

    • Select (highlight) files that you want to download at destination

    • Select the endpoint that you set up above of your local computer or server

    • Select the destination folder (e.g. /~/Downloads/)

  2. Click 'go'

  3. Files will be transfered to your computer

Requesting Access to unpublished data in TERRA-REF BETYdb:

To request access to unpublished data, send your Globus id to David LeBauer (dlebauer@email.arizona.edu) with 'TERRAREF Globus Access Request' in the subject.

  1. fill out the terraref.org/beta user form

  2. email dlebauer@email.arizona.edu with your globusid to request access.

BETYdb: Trait Data and Agronomic Metadata

BETYdb contains the derived trait data with plot locations and other information associated with agronomic experimental design.

Accessing data in R

Requesting Access to unpublished data in TERRA-REF BETYdb:

  1. email dlebauer@email.arizona.edu for your account to be approved.

Using SQL and PostGIS with Docker (Advanced Users)

The fastest and most comprehensive way to access the database using SQL and other database interfaces (such as the R package dplyr interface described below, or GIS programs described in . You can run an instance of the database using docker, as described below

This is how you can access the TERRA REF trait database. It requires that you install the Docker software on your computer.

#git clone https://github.com/terraref/data-paper 
cd data-paper/code/betydb_docker 
docker-compose up -d postgres
docker-compose run --rm bety initialize
docker-compose run --rm bety sync

psql

psql -d bety -U bety -W bety

R

library(dplyr)
bety_src <- src_postgres(dbname = "bety", 
                         password = 'bety',
                         host = 'localhost', 
                         user = 'bety',
                         port = 5433)

GIS software

Clowder: Sensor Data and Metadata Browser

Data organization in Clowder

Data is organized into spaces, collections, and datasets, collections.

  • Spaces contain collections and datasets. TERRA-REF uses one space for each of the phenotyping platforms.

  • Collections consist of one or more datasets. TERRA-REF collections are organized by acquisition date and sensor. Users can also create their own collections.

  • Datasets consist of one or more files with associated metadata collected by one sensor at one time point. Users can annotate, download, and use these sensor datasets.

Requesting Access to unpublished data in Clowder:

  1. email dlebauer@email.arizona.edu for your account to be approved.

CyVerse: Genomics Data

TERRA-REF genomics data is accessible on the CyVerse Data Store and Discovery Environment. Accessing data through the CyVerse Discovery Environment requires signing up for a free CyVerse account. The Discovery Environment gives users access to software and computing resources, so this method has the advantage that TERRA-REF data can be utilized directly without the need to copy the data elsewhere.

You can also find these in the CyVerse discovery environment in the TERRA-REF Community Data folder: /iplant/home/shared/terraref.

CoGe: Genomics Data

PreviousDataNextData Products

Last updated 4 years ago

and

These can be found at .

Public domain data is available for Globus transfer via the . Non-public (but available with permission) data are at the endpoint

See also Globus

The service provides high-performance, secure, file transfer and synchronization between endpoints. It also allows you to securely share your data with other Globus users.

Sign up for Globus at

Download and install Globus Connect or

Log into Globus

Add an endpoint for the destination (e.g. your local computer)

Go to the 'transfer files' page:

is used to manage and distribute agricultural and ecological data. It contains phenotype and agronomic data including plot locations and other geolocations of interest (e.g. fields, rows, plants).

The easiest way to access data is to use the . This is documented in the .

fill out the user form

create an account at the TERRA-REF BETYdb: (not betydb.org)

The easiest way to get the entire database, including metadata. Assuming you are familiar with the Postgres and / or the R dbplyr library documentation. See the terraref.org/tutorials, the for additional examples.

Interested researchers can access BETYdb directly from GIS software such as ESRI ArcMap and QGIS. In some cases direct access can simplify the use of spatial data in BETYdb. See the Appendix for more information.

Clowder is an active data repository designed to enable collaboration around a set of shared datasets. TERRAREF uses Clowder to organize, annotate, and process data generated by phenotyping platforms. Datafiles are available via the Clowder or .

Clowder is the used to organize, annotate, and process raw data generated by the field scanner and other phenotyping platforms. It also stores information about sensors. Learn more about Clowder software from

fill out the user form

create an account at the

is a National Science Foundation funded cyberinfrastructure that aims to democratize access to supercomputing capabilities.

Genomics data can be browsed and downloaded from the CyVerse data store at

is a platform for performing Comparative Genomics research. It provides an open-ended network of interconnected tools to manage, analyze, and visualize next-gen data.

CoGe contains genomic information and sequence data. You can find the TERRA REF Genomics data on CoGe in this notebook:

terraref.org/tutorials
ncsa#terra-public
#Terraref
Getting Started
Globus Connect
globus.org
Personal
Server.
https://www.globus.org
https://www.globus.org/app/endpoints/create-gcp
https://www.globus.org/app/transfer
BETYdb
R traits package
tutorials
terraref.org/beta
terraref.org/bety
TERRA REF Tutorials
BETYdb Data Access guide
Accessing BETYdb with GIS Software
web interface
API
https://clowderframework.org
terraref.org/beta
TERRA-REF Clowder site
CyVerse
http://datacommons.cyverse.org/browse/iplant/home/shared/terraref
CoGe
https://genomevolution.org/coge/NotebookView.pl?nid=2137
docs.globus.org/api/
terraref.org/clowder/swaggerUI
terraref.org/bety/api/v1
terraref.org/brapi/v1/ui
genomevolution.org/apidocs/