cuNumeric.jl

Appearance

HDF5 ​

h5read and h5write transfer datasets between HDF5 files and runtime-managed NDArrays without gathering them into Julia Arrays.

Example ​

julia
using cuNumeric

field = cuNumeric.fill(3.5f0, 128, 64)
cuNumeric.h5write("checkpoint.h5", "field", field)
cuNumeric.Legate.runtime_sync()

restored = cuNumeric.h5read("checkpoint.h5", "field"; layout=:row)
@assert size(restored) == (128, 64)
@assert eltype(restored) == Float32
@assert Array(restored) == fill(3.5f0, 128, 64)

h5write submits work to Legate and can return before the file write has completed. Synchronize before accessing the file outside the runtime, moving or deleting it, or exiting immediately after the write.

h5write removes a leftover empty or truncated .h5 before launching the write, which is what otherwise aborts HDF5CombineVDS. A valid HDF5 file is left for Legate to overwrite. The *_legate_vds sidecar is left alone: after a Legate write that directory is the data, and the .h5 is only an index. h5read rejects a missing path, a directory, or a file that does not start with the HDF5 signature, instead of opening it in a Legate task.

Dataset layout ​

julia
row_major = cuNumeric.h5read("python.h5", "field")
column_major = cuNumeric.h5read("julia.h5", "field"; layout=:col)

layout=:row is the default for NumPy/h5py, cuPyNumeric, and cuNumeric.h5write. Use layout=:col for multidimensional datasets written by HDF5.jl. One-dimensional datasets are unaffected. Other keywords are forwarded to Legate.h5read.

Tests cover Float32, Float64, Int32, and Int64 arrays with one to three dimensions. Other types depend on the Legate HDF5 backend.

API reference ​

h5read ​

cuNumeric.h5read Function
julia
h5read(path::String, dataset::String; layout::Symbol=:row) -> NDArray

Read a dataset from an HDF5 file into an NDArray.

Arguments

  • path: Path to the HDF5 file.

  • dataset: Name of the dataset to read.

Keywords

  • layout: On-disk memory order, either :row (default) or :col.
source

h5write ​

cuNumeric.h5write Function
julia
h5write(path::String, dataset::String, arr::NDArray)

Write an NDArray directly to an HDF5 dataset without a host copy or dimension flip.

A leftover empty or truncated .h5 from a crashed write is removed first. A valid HDF5 file is left in place so Legate can overwrite it. The *_legate_vds sidecar is not touched: it holds the payload of a VDS write.

Arguments

  • path: Path to the HDF5 file.

  • dataset: Name of the dataset to write.

  • arr: The array to write.

source