Xarray in 45 minutes

Xarray in 45 minutes#

In this lesson, we discuss cover the basics of Xarray data structures. By the end of the lesson, we will be able to:

Understand the basic data structures in Xarray
Inspect DataArray and Dataset objects.
Read and write netCDF files using Xarray.
Understand that there are many packages that build on top of xarray

We’ll start by reviewing the various components of the Xarray data model, represented here visually:

https://docs.xarray.dev/en/stable/_images/dataset-diagram.png

import matplotlib.pyplot as plt
import numpy as np
import xarray as xr

xr.set_options(keep_attrs=True, display_expand_data=False)
np.set_printoptions(threshold=10, edgeitems=2)

%xmode minimal
%matplotlib inline
%config InlineBackend.figure_format='retina'

Exception reporting mode: Minimal

Xarray has a few small real-world tutorial datasets hosted in the xarray-data GitHub repository.

xarray.tutorial.load_dataset is a convenience function to download and open DataSets by name (listed at that link).

Here we’ll use air temperature from the National Center for Environmental Prediction. Xarray objects have convenient HTML representations to give an overview of what we’re working with:

Note that behind the scenes the tutorial.open_dataset downloads a file. It then uses xarray.open_dataset function to open that file (which for this datasets is a netCDF file).

A few things are done automatically upon opening, but controlled by keyword arguments. For example, try passing the keyword argument mask_and_scale=False… what happens?

Why Xarray?#

Metadata provides context and provides code that is more legible. This reduces the likelihood of errors from typos and makes analysis more intuitive and fun!

Analysis without xarray: `X(`#

# plot the first timestep
lat = ds.air.lat.data  # numpy array
lon = ds.air.lon.data  # numpy array
temp = ds.air.data  # numpy array

plt.figure()
plt.pcolormesh(lon, lat, temp[0, :, :]);

../_images/9c909f839c75f6cb6b3a1ba9256a1ce20d61af5b6c4662f13feac326ea58a301.png

temp.mean(axis=1)  ## what did I just do? I can't tell by looking at this line.

array([[279.39798, 279.6664 , ..., 280.31522, 280.6624 ],
       [279.05722, 279.538  , ..., 280.27002, 280.79764],
       ...,
       [279.398  , 279.66602, ..., 280.34198, 280.834  ],
       [279.27   , 279.354  , ..., 279.96997, 280.48196]], dtype=float32)

Analysis with xarray `=)`#

How readable is this code?

ds.air.isel(time=0).plot(x="lon");

../_images/52aee6f13c831276ea4623b208480dd928089ffc61b4d8f5c1604b38f398b54a.png

Use dimension names instead of axis numbers

ds.air.mean(dim="time").plot(x="lon")

<matplotlib.collections.QuadMesh at 0x7ff9f88bfa90>

../_images/43d50fc4f51b9a26b18dceb626227cc2749c00e3eb7a4f522073a349514ee9a2.png

Visualization#

(.plot)

We have seen very simple plots earlier. Xarray also lets you easily visualize 3D and 4D datasets by presenting multiple facets (or panels or subplots) showing variations across rows and/or columns.

# facet the seasonal_mean
seasonal_mean.air.plot(col="season", col_wrap=2);

../_images/8bc1e84381af7b591838c97cead308c2ce4512887b9d60e016638e3cbca77a7c.png

# contours
seasonal_mean.air.plot.contour(col="season", levels=20, add_colorbar=True);

../_images/84671bd0e7e48e7675d30bbb275f81dfbd107c1ec0da239fed5e3a330caeaeb8.png

# line plots too? wut
seasonal_mean.air.mean("lon").plot.line(hue="season", y="lat");

../_images/65bce763612ab84823b51b8d00e2976f5276d507b283acf8be66c3f3227717a3.png

For more see the user guide, the gallery, and the tutorial material.

The scientific python ecosystem#

Xarray ties in to the larger scientific python ecosystem and in turn many packages build on top of xarray. A long list of such packages is here: https://docs.xarray.dev/en/stable/related-projects.html.

Now we will demonstrate some cool features.

Pandas: tabular data structures#

You can easily convert between xarray and pandas structures. This allows you to conveniently use the extensive pandas ecosystem of packages (like seaborn) for your work.

# convert to pandas dataframe
df = ds.isel(time=slice(10)).to_dataframe()
df

			air
lat	time	lon
75.0	2013-01-01 00:00:00	200.0	241.199997
		202.5	242.500000
		205.0	243.500000
		207.5	244.000000
		210.0	244.099991
...	...	...	...
15.0	2013-01-03 06:00:00	320.0	297.000000
		322.5	297.290009
		325.0	296.899994
		327.5	296.790009
		330.0	297.100006

13250 rows × 1 columns

Alternative array types#

This notebook has focused on Numpy arrays. Xarray can wrap other array types! For example:

distributed parallel arrays & Xarray user guide on Dask

pydata/sparse : sparse arrays

GPU arrays & cupy-xarray

pint : unit-aware arrays & pint-xarray

Dask#

Dask cuts up NumPy arrays into blocks and parallelizes your analysis code across these blocks

# demonstrate dask dataset
dasky = xr.tutorial.open_dataset(
    "air_temperature",
    chunks={"time": 10},  # 10 time steps in each block
)

dasky.air

<xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)>
dask.array<chunksize=(10, 25, 53), meta=np.ndarray>
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
    long_name:     4xDaily Air temperature at sigma level 995
    units:         degK
    precision:     2
    GRIB_id:       11
    GRIB_name:     TMP
    var_desc:      Air temperature
    dataset:       NMC Reanalysis
    level_desc:    Surface
    statistic:     Individual Obs
    parent_stat:   Other
    actual_range:  [185.16 322.1 ]

All computations with dask-backed xarray objects are lazy, allowing you to build up a complicated chain of analysis steps quickly

# demonstrate lazy mean
dasky.air.mean("lat")

<xarray.DataArray 'air' (time: 2920, lon: 53)>
dask.array<chunksize=(10, 53), meta=np.ndarray>
Coordinates:
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
    long_name:     4xDaily Air temperature at sigma level 995
    units:         degK
    precision:     2
    GRIB_id:       11
    GRIB_name:     TMP
    var_desc:      Air temperature
    dataset:       NMC Reanalysis
    level_desc:    Surface
    statistic:     Individual Obs
    parent_stat:   Other
    actual_range:  [185.16 322.1 ]

To get concrete values, call .compute or .load

HoloViz#

Quickly generate interactive plots from your data!

The hvplot package attaches itself to all xarray objects under the .hvplot namespace. So instead of using .plot use .hvplot

import hvplot.xarray

ds.air.hvplot(groupby="time", clim=(270, 300), widget_location='bottom')

Note

The time slider will only work if you’re executing the notebook, rather than viewing the website

Other cool packages#

xgcm : grid-aware operations with xarray objects
xrft : fourier transforms with xarray
xclim : calculating climate indices with xarray objects
intake-xarray : forget about file paths
rioxarray : raster files and xarray
xesmf : regrid using ESMF
MetPy : tools for working with weather data

Check the Xarray Ecosystem page and this tutorial for even more packages and demonstrations.

Next#

Read the tutorial material and user guide
See the description of common terms used in the xarray documentation:
Answers to common questions on “how to do X” with Xarray are here
Ryan Abernathey has a book on data analysis with a chapter on Xarray
Project Pythia has foundational and more advanced material on Xarray. Pythia also aggregates other Python learning resources.
The Xarray Github Discussions and Pangeo Discourse are good places to ask questions.
Tell your friends! Tweet!

Welcome!#

Xarray is an open-source project and gladly welcomes all kinds of contributions. This could include reporting bugs, discussing new enhancements, contributing code, helping answer user questions, contributing documentation (even small edits like fixing spelling mistakes or rewording to make the text clearer). Welcome!

Xarray in 45 minutes

Contents

Xarray in 45 minutes#

What’s in a Dataset?#

What’s in a DataArray?#

Name (optional)#

Named dimensions#

Coordinate variables#

Arbitrary attributes#

Underlying data#

Review#

Why Xarray?#

Analysis without xarray: `X(`#

Analysis with xarray `=)`#

Extracting data or “indexing”#

Label-based indexing#

Position-based indexing#

Concepts for computation#

Broadcasting: expanding data#

Alignment: putting data on the same grid#

High level computation#

groupby#

resample#

weighted#

Visualization#

Reading and writing files#

The scientific python ecosystem#

Pandas: tabular data structures#

Alternative array types#

Dask#

HoloViz#

cf_xarray#

Other cool packages#

Next#

Welcome!#

Xarray in 45 minutes

Contents

Xarray in 45 minutes#

What’s in a Dataset?#

What’s in a DataArray?#

Name (optional)#

Named dimensions#

Coordinate variables#

Arbitrary attributes#

Underlying data#

Review#

Why Xarray?#

Analysis without xarray: X(#

Analysis with xarray =)#

Extracting data or “indexing”#

Label-based indexing#

Position-based indexing#

Concepts for computation#

Broadcasting: expanding data#

Alignment: putting data on the same grid#

High level computation#

groupby#

resample#

weighted#

Visualization#

Reading and writing files#

The scientific python ecosystem#

Pandas: tabular data structures#

Alternative array types#

Dask#

HoloViz#

cf_xarray#

Other cool packages#

Next#

Welcome!#

Analysis without xarray: `X(`#

Analysis with xarray `=)`#