Revision as of 23:59, 3 March 2017

LiPD: Linked Paleo Data

The LiPD vision

Paleoclimate investigators have made a major effort over the past decade to make their data available to the broader community, largely through online archiving systems like the World Data Center for Paleoclimatology and Pangaea. However, there is no agreed-upon data standard for how to store and exchange such data. As the number of records in these archives has grown, making connections manually has thus become more and more challenging, hampering integrative efforts at the very time they should be flourishing. Paleoclimatologists thus need a common tongue to describe their datasets to each other and to machines.

LiPD (Linked Paleo Data) proposes such a common tongue. It is a universally-readable data container that organizes data and metadata in a uniform way, such that they may be web-searchable, and such that a variety of code functionalities may be built and apply instantly to any dataset that observes that standard.

For more details, see this article.

LiPD in LinkedEarth

LiPD is a convenient way to store and exchange paleoclimate format, and is an important building block of the LinkedEarth edifice. However, it is important to note that the LinkedEarth database grown via the wiki is not reliant on LiPD. LiPD is merely a convenient way to exchange paleoclimate data, and is convenient way to import data to the wiki, or extract a dataset from the wiki for analysis. LiPD is closely aligned with the LinkedEarth ontology, changes in one are often reflected in the other (with a small lag).

Structuring data : the LiPD Way

LiPD components

structure of a lipd file

There are 6 possible components to a LiPD dataset. To be considered valid, a LiPD file must observe the structure to the right, and contain basic metadata for at least some of these categories.

Root Metadata

This describes metadata that applies to the whole dataset. Common examples are:

• Dataset Name
• Investigators
• Link to online dataset
• LiPD version

Geographic Metadata

Here the the site location is described following the geoJSON standard. This includes:

• Coordinates
• Sitename
• Descriptive location (e.g.,)
  • Country
  • State
  • Province
  • Ocean

Publication Metadata

Describes publications, either traditional or data publications are described following the bibJSON standard. This includes:

• Authors
• Title
• Journal
• DOI
• Year
• URL

Funding Metadata

Includes information about the research that produced the data were funded, including:

• Funding agency
• Funding grant

PaleoData Table

This includes all of the measured and inferred paleoenvironmental data - essentially all of the measurements and data from the study that were not used to infer the age. This includes:

Measurement Tables

• Column names, units and descriptions
• Interpretation metadata
• Calibration metadata

Models

• Methodology used to produce the model results
• Summary table of model output
• Ensemble tables of data produced by the model
• Distribution tables of data produced by the model

ChronData Table

This mirrors PaleoData but only includes the data from the study that were used to infer the age of the sequence. This includes:

Measurement Tables

• Column names, units and descriptions

Models

• Methodology used to produce the model results
• Summary table of model output
• Ensemble tables of data produced by the model
• Distribution tables of data produced by the model

LiPD implementation

LiPD is centered on JSON-LD, a JSON-based format compliant with the Linked Data paradigm. JavaScript Object Notation (JSON) is an extremely lightweight and flexible way to encode information, and has become the leading format for data exchange on the Web. Linked Data are datasets that observe common rules to be able to be automatically linked through the World Wide Web. A LiPD file (.lpd) is in fact a zipped folder that follows the BagIt standard:

This includes:

• The BagIt "payload":
  • One json-ld file that describes all of the metadata
  • Csv tables that include data from all of the tables
• BagIt complementary information, including:
  • a “bag-info.txt” file which details metadata for the bag, using colon-separated key/value pairs
  • a tag manifest file which lists tag files and their associated MD5 checksums

BagIt is needed to ensure that all the bits on your computer are the same as the bits on the server.

Working with LiPD data

Because LiPD structures such a complex set of data and metadata, navigating through the native json-ld and csv files is complicated, and is not the best way to harness the power of LiPD files. A growing number of utilities and software packages can read and write LiPD files, and enable users to readily take advantage of its rich structure:

List of utilities and software that read and write LiPD

• The Linked Earth Wiki
• LiPD Utilities in:
  • Matlab
  • Python
  • R
• GeoChronR
• CSciBox
• PyleoClim