Data cataloguing for Xarray

Goals: At the end of this tutorial, you’ll have an overview of what data cataloging is, why it is done, what tools are available. You’ll also know how to open and browse some data catalogues.

A data catalog is a detailed inventory of data assets within an organization. It helps users easily discover, understand, manage, curate and access data. -IBM

Within an organisation, you may work with a lot of different datasets. As someone managing this data, you might want the following things:

• An organized, easily browsable collection: By easing discovery and loading of your datasets you can reduce the friction for those analysing the data. Users can eaily get an overview of available datasets, discover datasets of interest, and load them for analysis.

• Access control and logging: You can limit who is able to access these datasets, and track when datasets are being accessed - keeping the data private, and providing metrics that can inform data management (e.g., by removing little-used datasets).

• Combine individual data assets: Combine individual data assets that are similar into a larger object (e.g., combining individual time snapshots into a larger data cube)

There are a bunch of solutions available in Xarray that meet some of these needs - we’ll broadly call these “data catalogs”. These solutions differ in functionality, and the correct data cataloguing solution depends on your needs as an institute.

If you feel there are data cataloguing tools missing from this page, please submit a PR.

intake v2

Intake is a lightweight Python package which allows one to specify data catalogues (or a hierarchy of catalogues) via YAML files. These YAML files describe:

• how to load the dataset. This is done via “readers”, which support different file formats, and support lazily defining additional transformations to the dataset using Dask

• additional metadata for the dataset

Uploading the catalogue file to a Git forge like GitHub allows for versioning of this file.

This catalogue can be browsed in Python, or via third-party tools which parse the datalog, allowing the user to select and open a dataset. This tool has been adopted in the geosciences community for working with Xarray datasets, but is also flexible to work with other dataset types.

There is no separate authentication layer in Intake, hence access control and logging depends entirely on the way the data is accessed. If the dataset is loaded from a file location on a mounted drive, then you need to have access to that mounted drive. If the dataset is located in an S3 bucket, you need to have the appropriate access credentials for the resource.

Intake was originally developed within Anaconda by the team behind Dask.

Resources

Project Pythia has a cookbook on Intake.

Example code

Here we work based off an Intake tutorial on easy.gems! from the DKRZ (German Climate Supercomputing Center). Note that the tutorial is based on v1 of Intake (which also uses intake-xarray).

# Browse an existing catalogue

import intake

cat = intake.open_catalog("https://data.nextgems-h2020.eu/online.yaml")
# TIP: Its just a YAML file - open it in your browser and you'll see!

# view the available subcatalogues or datasets
list(cat)

['ICON', 'ERA5', 'JRA3Q', 'MERRA2', 'IFS', 'IMERG', 'GSMaP', 'tutorial']

# Access catalogue elements using `["attr"]` or `.attr` notation
# NOTE: The `.attr` notation only works if the entry is a valid Python attribute name - which the entry might not be

# Let's see what the `tutorial` item is...
cat["tutorial"]

tutorial:
  args:
    path: https://data.nextgems-h2020.eu/tutorial.yaml
  description: Example datasets for use in tutorials
  driver: intake.catalog.local.YAMLFileCatalog
  metadata:
    catalog_dir: https://data.nextgems-h2020.eu

# it's a subcatalogue!

print(list(cat["tutorial"]))

# Let's look at an entry within...
cat["tutorial"]["ICON.native.2d_PT6H_inst"]

['ICON.native.2d_PT6H_inst', 'ICON.native.2d_P1D_mean', 'ICON.native.3d_P1D_mean']

ICON.native.2d_PT6H_inst:
  args:
    chunks: null
    consolidated: true
    urlpath: https://swift.dkrz.de/v1/dkrz_948e7d4bbfbb445fbff5315fc433e36a/easygems_tutorial/ICON.native.2d_PT6H_inst.zarr
  description: ''
  driver: intake_xarray.xzarr.ZarrSource
  metadata:
    catalog_dir: https://data.nextgems-h2020.eu

# Let's open it as an xarray dataset

ds = cat["tutorial"]["ICON.native.2d_PT6H_inst"].to_dask()
ds

/home/runner/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 3GB
Dimensions:             (time: 124, ncells: 1310720)
Coordinates:
  * time                (time) datetime64[ns] 992B 2020-01-01T05:59:59 ... 20...
    cell_sea_land_mask  (ncells) float64 10MB ...
    lat                 (ncells) float64 10MB ...
    lon                 (ncells) float64 10MB ...
Dimensions without coordinates: ncells
Data variables:
    hus2m               (time, ncells) float32 650MB ...
    psl                 (time, ncells) float32 650MB ...
    ts                  (time, ncells) float32 650MB ...
    uas                 (time, ncells) float32 650MB ...
    vas                 (time, ncells) float32 650MB ...

xarray.Dataset

• Dimensions:
  • time: 124
  • ncells: 1310720
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2020-01-01T05:59:59 ... 2020-01-...

    array(['2020-01-01T05:59:59.000000000', '2020-01-01T11:59:59.000000000',
           '2020-01-01T17:59:59.000000000', '2020-01-01T23:59:59.000000000',
           '2020-01-02T05:59:59.000000000', '2020-01-02T11:59:59.000000000',
           '2020-01-02T17:59:59.000000000', '2020-01-02T23:59:59.000000000',
           '2020-01-03T05:59:59.000000000', '2020-01-03T11:59:59.000000000',
           '2020-01-03T17:59:59.000000000', '2020-01-03T23:59:59.000000000',
           '2020-01-04T05:59:59.000000000', '2020-01-04T11:59:59.000000000',
           '2020-01-04T17:59:59.000000000', '2020-01-04T23:59:59.000000000',
           '2020-01-05T05:59:59.000000000', '2020-01-05T11:59:59.000000000',
           '2020-01-05T17:59:59.000000000', '2020-01-05T23:59:59.000000000',
           '2020-01-06T05:59:59.000000000', '2020-01-06T11:59:59.000000000',
           '2020-01-06T17:59:59.000000000', '2020-01-06T23:59:59.000000000',
           '2020-01-07T05:59:59.000000000', '2020-01-07T11:59:59.000000000',
           '2020-01-07T17:59:59.000000000', '2020-01-07T23:59:59.000000000',
           '2020-01-08T05:59:59.000000000', '2020-01-08T11:59:59.000000000',
           '2020-01-08T17:59:59.000000000', '2020-01-08T23:59:59.000000000',
           '2020-01-09T05:59:59.000000000', '2020-01-09T11:59:59.000000000',
           '2020-01-09T17:59:59.000000000', '2020-01-09T23:59:59.000000000',
           '2020-01-10T05:59:59.000000000', '2020-01-10T11:59:59.000000000',
           '2020-01-10T17:59:59.000000000', '2020-01-10T23:59:59.000000000',
           '2020-01-11T05:59:59.000000000', '2020-01-11T11:59:59.000000000',
           '2020-01-11T17:59:59.000000000', '2020-01-11T23:59:59.000000000',
           '2020-01-12T05:59:59.000000000', '2020-01-12T11:59:59.000000000',
           '2020-01-12T17:59:59.000000000', '2020-01-12T23:59:59.000000000',
           '2020-01-13T05:59:59.000000000', '2020-01-13T11:59:59.000000000',
           '2020-01-13T17:59:59.000000000', '2020-01-13T23:59:59.000000000',
           '2020-01-14T05:59:59.000000000', '2020-01-14T11:59:59.000000000',
           '2020-01-14T17:59:59.000000000', '2020-01-14T23:59:59.000000000',
           '2020-01-15T05:59:59.000000000', '2020-01-15T11:59:59.000000000',
           '2020-01-15T17:59:59.000000000', '2020-01-15T23:59:59.000000000',
           '2020-01-16T05:59:59.000000000', '2020-01-16T11:59:59.000000000',
           '2020-01-16T17:59:59.000000000', '2020-01-16T23:59:59.000000000',
           '2020-01-17T05:59:59.000000000', '2020-01-17T11:59:59.000000000',
           '2020-01-17T17:59:59.000000000', '2020-01-17T23:59:59.000000000',
           '2020-01-18T05:59:59.000000000', '2020-01-18T11:59:59.000000000',
           '2020-01-18T17:59:59.000000000', '2020-01-18T23:59:59.000000000',
           '2020-01-19T05:59:59.000000000', '2020-01-19T11:59:59.000000000',
           '2020-01-19T17:59:59.000000000', '2020-01-19T23:59:59.000000000',
           '2020-01-20T05:59:59.000000000', '2020-01-20T11:59:59.000000000',
           '2020-01-20T17:59:59.000000000', '2020-01-20T23:59:59.000000000',
           '2020-01-21T05:59:59.000000000', '2020-01-21T11:59:59.000000000',
           '2020-01-21T17:59:59.000000000', '2020-01-21T23:59:59.000000000',
           '2020-01-22T05:59:59.000000000', '2020-01-22T11:59:59.000000000',
           '2020-01-22T17:59:59.000000000', '2020-01-22T23:59:59.000000000',
           '2020-01-23T05:59:59.000000000', '2020-01-23T11:59:59.000000000',
           '2020-01-23T17:59:59.000000000', '2020-01-23T23:59:59.000000000',
           '2020-01-24T05:59:59.000000000', '2020-01-24T11:59:59.000000000',
           '2020-01-24T17:59:59.000000000', '2020-01-24T23:59:59.000000000',
           '2020-01-25T05:59:59.000000000', '2020-01-25T11:59:59.000000000',
           '2020-01-25T17:59:59.000000000', '2020-01-25T23:59:59.000000000',
           '2020-01-26T05:59:59.000000000', '2020-01-26T11:59:59.000000000',
           '2020-01-26T17:59:59.000000000', '2020-01-26T23:59:59.000000000',
           '2020-01-27T05:59:59.000000000', '2020-01-27T11:59:59.000000000',
           '2020-01-27T17:59:59.000000000', '2020-01-27T23:59:59.000000000',
           '2020-01-28T05:59:59.000000000', '2020-01-28T11:59:59.000000000',
           '2020-01-28T17:59:59.000000000', '2020-01-28T23:59:59.000000000',
           '2020-01-29T05:59:59.000000000', '2020-01-29T11:59:59.000000000',
           '2020-01-29T17:59:59.000000000', '2020-01-29T23:59:59.000000000',
           '2020-01-30T05:59:59.000000000', '2020-01-30T11:59:59.000000000',
           '2020-01-30T17:59:59.000000000', '2020-01-30T23:59:59.000000000',
           '2020-01-31T05:59:59.000000000', '2020-01-31T11:59:59.000000000',
           '2020-01-31T17:59:59.000000000', '2020-01-31T23:59:59.000000000'],
          dtype='datetime64[ns]')

  • cell_sea_land_mask
    (ncells)
    float64
    ...

    long_name :

    sea (-2 inner, -1 boundary) land (2 inner, 1 boundary) mask for the cell

    units :

    2,1,-1,-

    grid_type :

    unstructured

    number_of_grid_in_reference :

    1

    dtype :

    int32

    zlib :

    False

    szip :

    False

    zstd :

    False

    bzip2 :

    False

    blosc :

    False

    shuffle :

    False

    complevel :

    0

    fletcher32 :

    False

    contiguous :

    True

    chunksizes :

    None

    source :

    /work/bm1344/DKRZ/kerchunks_batched/erc2002/icon_grid_0033_R02B08_G.nc

    original_shape :

    [5242880]

    [1310720 values with dtype=float64]

  • lat
    (ncells)
    float64
    ...

    [1310720 values with dtype=float64]

  • lon
    (ncells)
    float64
    ...

    [1310720 values with dtype=float64]

• Data variables: (5)
  • hus2m
    (time, ncells)
    float32
    ...

    CDI_grid_type :

    unstructured

    long_name :

    specific humidity in 2m

    number_of_grid_in_reference :

    1

    param :

    6.0.0

    standard_name :

    qv2m

    units :

    kg kg-1

    [162529280 values with dtype=float32]

  • psl
    (time, ncells)
    float32
    ...

    CDI_grid_type :

    unstructured

    long_name :

    mean sea level pressure

    number_of_grid_in_reference :

    1

    param :

    1.3.0

    standard_name :

    mean sea level pressure

    units :

    Pa

    [162529280 values with dtype=float32]

  • ts
    (time, ncells)
    float32
    ...

    CDI_grid_type :

    unstructured

    long_name :

    surface temperature

    number_of_grid_in_reference :

    1

    param :

    0.0.0

    standard_name :

    surface_temperature

    units :

    K

    [162529280 values with dtype=float32]

  • uas
    (time, ncells)
    float32
    ...

    CDI_grid_type :

    unstructured

    long_name :

    zonal wind in 10m

    number_of_grid_in_reference :

    1

    param :

    2.2.0

    standard_name :

    uas

    units :

    m s-1

    [162529280 values with dtype=float32]

  • vas
    (time, ncells)
    float32
    ...

    CDI_grid_type :

    unstructured

    long_name :

    meridional wind in 10m

    number_of_grid_in_reference :

    1

    param :

    3.2.0

    standard_name :

    vas

    units :

    m s-1

    [162529280 values with dtype=float32]

That’s it for the example! Feel free to poke around further to see what other datasets are available in this catalogue, or to see which other catalogues are available online.

STAC

SpatioTemporal Asset Catalogs The STAC specification is a common language to describe geospatial information, so it can more easily be worked with, indexed, and discovered. -stacspec.org

Let’s set the scene a bit: During satellite data processing, many image files (predominantly TIFF) get generated. Each of these files can have different spatial and temporal extents, and over time these individual files can number in the millions, containing terrabytes of data in total. When doing analyses on these files, the first thing one needs to know is whether the file is even relevant to the analysis at hand (i.e., at least “what is the spatial and temporal extent of the file?”), however downloading the file just to check this metadata within is very wasteful. This is the context from which the STAC project was born.

The goal of the STAC project is to make geospatial data more accessible by providing a specification to describe assets and their metadata. Adding this metadata alongside the asset (or lifting it out above the asset itself) allows:

• The efficient querying of assets based on properties of interest without downloading the entire asset.

• The combining of these assets into a datacube object (i.e., Xarray dataset) by reprojecting or interpolating the data to a common grid.

Within the STAC project is:

• The STAC Specification: A set of specifications describing (from most atomic to least) STAC Items, STAC Catalogs, and STAC collections. There is also a spec for the STAC API.

• Tooling for working with STAC: More on this just now

• The community

There are various tools in the STAC ecosystem. There are online STAC browsers such as the Radiant Earth STAC browser. There are Python packages for the creating of STAC items and searching of STAC catalogues (e.g., rustac-py and pystac). There are also Python packages for combining STAC objects into larger datacubes (e.g., odc-stac, stacstac and xpystac).

There are many packages with some overlapping functionality. An example workflow you might follow would be using an online STAC browser to get a better understanding of the STAC assets you’re working with, and what properties are available to query on, using rustac-py to generate and run the query, and then (once you’re happy with the result) using stackstac or odc-stac to open the result as an Xarray Dataset.

Resources

• https://stacspec.org/en

• https://stacindex.org/

• Example using odc-stac: See example notebook from odc-stacdocs “Access Sentinel 2 Data from AWS”.

Commercial offering - Arraylake

There are also commercial offerings available that have data cataloguing functionality.

Arraylake is Earthmover’s managed cloud platform built on top of the open-source Icechunk data format (where Icechunk is a project offering versioning of Zarr-like datasets).

See here for a full comparison between Icechunk and Arraylake.

A notable feature is that Arraylake integrates with Earthmover’s data marketplace, powering data discovery and allowing for the dissemination of datasets (or subsets of datasets) to other interested users.

More resources

https://guide.cloudnativegeo.org/cookbooks/zarr-stac-report/data-consumers/