Introduction to Xarray Backends

Introduction to Xarray Backends#

In this lesson, we will learn about and how to use xarray’s backends

Learning Goals

Learn about xarray’s internal and external backends
Learn how xarray selects backend engines
Learn how to specify backend engines

What are Xarray Backends#

Xarray supports direct serialization and IO to several file formats, through the use of internal and external backends. Xarray’s internal backends cover many common data formats such as “netCDF4”, “h5netcdf”, and “zarr”. These backends provide a set of instructions that tells xarray how read data and store it into a Dataset, DataTree or DataTree model and are stored in the underlying “backend”.

Xarray also supports external backends such as “rioxarray” and “cfgrib” and even creating your own backend.

Internal Backend Selection#

When opening a file or URL without explicitly specifying the engine parameter, xarray automatically selects an appropriate backend based on the file path or URL. The backends are tried in order: netcdf4 → h5netcdf → scipy → pydap → zarr.

Check for available backends#

We can get a list of our available engines with list_engines

import xarray as xr

xr.backends.list_engines()

{'netcdf4': <NetCDF4BackendEntrypoint>
   Open netCDF (.nc, .nc4 and .cdf) and most HDF5 files using netCDF4 in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.NetCDF4BackendEntrypoint.html,
 'h5netcdf': <H5netcdfBackendEntrypoint>
   Open netCDF (.nc, .nc4 and .cdf) and most HDF5 files using h5netcdf in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.H5netcdfBackendEntrypoint.html,
 'scipy': <ScipyBackendEntrypoint>
   Open netCDF files (.nc, .cdf and .nc.gz) using scipy in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.ScipyBackendEntrypoint.html,
 'pydap': <PydapBackendEntrypoint>
   Open remote datasets via OPeNDAP using pydap in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.PydapBackendEntrypoint.html,
 'rasterio': <RasterioBackend>,
 'store': <StoreBackendEntrypoint>
   Open AbstractDataStore instances in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.StoreBackendEntrypoint.html,
 'zarr': <ZarrBackendEntrypoint>
   Open zarr files (.zarr) using zarr in Xarray
   Learn more at https://docs.xarray.dev/en/stable/generated/xarray.backends.ZarrBackendEntrypoint.html}

Setting engine order#

You can customize the order in which netcdf4 backends are resolved using xr.set_options(netcdf_engine_order=)

Let’s update our engine order to have “h5netcdf” and the “netCDF4” and reopening the dataset

xr.set_options(netcdf_engine_order=["h5netcdf", "netcdf4", "scipy"])

<xarray.core.options.set_options at 0x7f515c0c86e0>

Specifying the wrong backend#

Lets trying specifying a bad engine for opening our dataset

Warning

This will fail because we cannot use the “pydap” engine to read this dataset

xr.tutorial.open_dataset("air_temperature", engine="pydap")

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 xr.tutorial.open_dataset("air_temperature", engine="pydap")

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/xarray/tutorial.py:170, in open_dataset(name, cache, cache_dir, engine, **kws)
# retrieve the file
filepath = pooch.retrieve(
   url=url, known_hash=None, path=cache_dir, downloader=downloader
)
--> 170 ds = _open_dataset(filepath, engine=engine, **kws)
if not cache:
   ds = ds.load()

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/api.py:613, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, create_default_indexes, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
decoders = _resolve_decoders_kwargs(
   decode_cf,
   open_backend_dataset_parameters=backend.open_dataset_parameters,
   (...)    609     decode_coords=decode_coords,
)
overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 613 backend_ds = backend.open_dataset(
   filename_or_obj,
   drop_variables=drop_variables,
   **decoders,
   **kwargs,
)
ds = _dataset_from_backend_dataset(
   backend_ds,
   filename_or_obj,
   (...)    632     **kwargs,
)
return ds

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/pydap_.py:279, in PydapBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, application, session, output_grid, timeout, verify, user_charset, checksums)
def open_dataset(
   self,
   filename_or_obj: (
   (...)    277     checksums=True,
) -> Dataset:
--> 279     store = PydapDataStore.open(
       url=filename_or_obj,
       group=group,
       application=application,
       session=session,
       output_grid=output_grid,
       timeout=timeout,
       verify=verify,
       user_charset=user_charset,
       checksums=checksums,
   )
   store_entrypoint = StoreBackendEntrypoint()
   with close_on_error(store):

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/pydap_.py:141, in PydapDataStore.open(cls, url, group, application, session, output_grid, timeout, verify, user_charset, checksums)
kwargs = {
   "url": url,
   "application": application,
   (...)    137     "user_charset": user_charset,
}
if isinstance(url, str):
   # check uit begins with an acceptable scheme
--> 141     dataset = open_url(**kwargs)
elif hasattr(url, "ds"):
   # pydap dataset
   dataset = url.ds

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/pydap/client.py:179, in open_url(url, application, session, output_grid, flat, timeout, verify, checksums, user_charset, protocol, batch, use_cache, session_kwargs, cache_kwargs, get_kwargs)
if not session:
   session = create_session(
       use_cache=use_cache,
       session_kwargs=session_kwargs,
       cache_kwargs=cache_kwargs,
   )
--> 179 handler = DAPHandler(
   url,
   application,
   session,
   output_grid=output_grid,
   flat=flat,
   timeout=timeout,
   verify=verify,
   checksums=checksums,
   user_charset=user_charset,
   protocol=protocol,
   get_kwargs=get_kwargs,
)
dataset = handler.dataset
dataset._session = session

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/pydap/handlers/dap.py:131, in DAPHandler.__init__(self, url, application, session, output_grid, flat, timeout, verify, checksums, user_charset, protocol, get_kwargs)
       self.scheme = "https"
else:
--> 131     self.protocol = self.determine_protocol()
self.projection, self.selection = parse_ce(self.query, self.protocol)
arg = (
   self.scheme,
   self.netloc,
   (...)    140     self.fragment,
)

File ~/work/xarray-tutorial/xarray-tutorial/.pixi/envs/default/lib/python3.14/site-packages/pydap/handlers/dap.py:157, in DAPHandler.determine_protocol(self)
       return protocol
   else:
--> 157         raise TypeError(
           "Invalid URL scheme - acceptable options are"
           "`dap2`, `dap4`. `https` and `http`",
       )
if self.query[:4] == "dap4":
   return "dap4"

TypeError: Invalid URL scheme - acceptable options are`dap2`, `dap4`. `https` and `http`

Introduction to Xarray Backends

Contents

Introduction to Xarray Backends#

What are Xarray Backends#

Reading a dataset with the netCDF4 engine#

Internal Backend Selection#

Check for available backends#

Reading without specifying engine#

Setting engine order#

Specifying the wrong backend#