Overview¶

nwparray opens NWP weather forecast archives (such as those from NOAA or ECMWF) as xarray datasets, so that the data can be organized for model training or forecast evaluation. It's designed to be highly scalable via Dask, to quickly download and process terabytes of archive data.

The package relies on templates from Herbie, and uses many of the same function arguments.

Example¶

import pandas as pd
from nwparray import NwpCollection

# describe the HRRR data to get
searches = [
    ':TMP:2 m above ground:',
    ':DPT:2 m above ground:'
]
runs = pd.date_range(start='2021-04-01 12:00', periods=4, freq='D')
fxx = range(13) # forecast hours 0-12

hrrr = NwpCollection(runs, fxx, 'hrrr', searches=searches)
hrrr.collection_size() # estimate the complete download size

# create a list of xarray datasets
dataset_list = hrrr.open_datasets()
print(dataset_list[0])

<xarray.Dataset> Size: 823MB
 Dimensions:            (time: 4, step: 13, y: 1059, x: 1799)
 Coordinates:
   * time               (time) datetime64[us] 32B 2021-04-01T12:00:00 ... 2021...
   * step               (step) int64 104B 0 1 2 3 4 5 6 7 8 9 10 11 12
   * y                  (y) int64 8kB 0 1 2 3 4 5 ... 1054 1055 1056 1057 1058
   * x                  (x) int64 14kB 0 1 2 3 4 5 ... 1794 1795 1796 1797 1798
     latitude           (y, x) float64 15MB 21.14 21.15 21.15 ... 47.85 47.84
     longitude          (y, x) float64 15MB 237.3 237.3 237.3 ... 299.0 299.1
     heightAboveGround  float64 8B 2.0
 Data variables:
     t2m                (time, step, y, x) float32 396MB dask.array<chunksize=(1, 13, 1059, 1799), meta=np.ndarray>
     d2m                (time, step, y, x) float32 396MB dask.array<chunksize=(1, 13, 1059, 1799), meta=np.ndarray>

nwparray vs. Herbie¶

nwparray is modeled on Herbie and even includes modified Herbie source code. There are two major differences that allow nwparray to handle larger datasets:

1. Herbie writes the original data files to the local system, and nwparray does not. For large datasets, reading the files becomes slow and impractical. To store datasets locally with nwparray, the data can be written directly from the xarray dataset to netcdf or zarr format, which is more practical for large datasets.
2. While both Herbie and nwparray use Dask for parallel processing once the data is in xarray, nwparray adjusts the chunk size used by Dask, which allows Dask to process much larger datasets.