FAQ

What datatypes are supported?

Below is a complete list of types for which h5py supports reading, writing and creating datasets. Each type is mapped to a native NumPy type.

Fully supported types:

Type

Precisions

Notes

Bitfield

1, 2, 4 or 8 byte, BE/LE

Read as unsigned integers

Integer

1, 2, 4 or 8 byte, BE/LE, signed/unsigned

Float

2, 4, 8, 12, 16 byte, BE/LE

Complex

8 or 16 byte, BE/LE

Stored as HDF5 struct

Compound

Arbitrary names and offsets

Strings (fixed-length)

Any length

Strings (variable-length)

Any length, ASCII or Unicode

Opaque (kind ‘V’)

Any length

Boolean

NumPy 1-byte bool

Stored as HDF5 enum

Array

Any supported type

Enumeration

Any NumPy integer type

Read/write as integers

References

Region and object

Variable length array

Any supported type

See Special Types

Other numpy dtypes, such as datetime64 and timedelta64, can optionally be stored in HDF5 opaque data using opaque_dtype(). h5py will read this data back with the same dtype, but other software probably will not understand it.

Unsupported types:

Type

Status

HDF5 “time” type

NumPy “U” strings

No HDF5 equivalent

NumPy generic “O”

Not planned

What compression/processing filters are supported?

Filter

Function

Availability

DEFLATE/GZIP

Standard HDF5 compression

All platforms

SHUFFLE

Increase compression ratio

All platforms

FLETCHER32

Error detection

All platforms

Scale-offset

Integer/float scaling and truncation

All platforms

SZIP

Fast, patented compression for int/float

  • UNIX: if supplied with HDF5.

  • Windows: read-only

LZF

Very fast compression, all types

Ships with h5py, C source available

What file drivers are available?

A number of different HDF5 “drivers”, which provide different modes of access to the filesystem, are accessible in h5py via the high-level interface. The currently supported drivers are:

Driver

Purpose

Notes

sec2

Standard optimized driver

Default on UNIX/Windows

stdio

Buffered I/O using stdio.h

core

In-memory file (optionally backed to disk)

family

Multi-file driver

mpio

Parallel HDF5 file access

What’s the difference between h5py and PyTables?

The two projects have different design goals. PyTables presents a database-like approach to data storage, providing features like indexing and fast “in-kernel” queries on dataset contents. It also has a custom system to represent data types.

In contrast, h5py is an attempt to map the HDF5 feature set to NumPy as closely as possible. For example, the high-level type system uses NumPy dtype objects exclusively, and method and attribute naming follows Python and NumPy conventions for dictionary and array access (i.e. “.dtype” and “.shape” attributes for datasets, group[name] indexing syntax for groups, etc).

Underneath the “high-level” interface to h5py (i.e. NumPy-array-like objects; what you’ll typically be using) is a large Cython layer which calls into C. This “low-level” interface provides access to nearly all of the HDF5 C API. This layer is object-oriented with respect to HDF5 identifiers, supports reference counting, automatic translation between NumPy and HDF5 type objects, translation between the HDF5 error stack and Python exceptions, and more.

This greatly simplifies the design of the complicated high-level interface, by relying on the “Pythonicity” of the C API wrapping.

There’s also a PyTables perspective on this question at the PyTables FAQ.

Does h5py support Parallel HDF5?

Starting with version 2.2, h5py supports Parallel HDF5 on UNIX platforms. mpi4py is required, as well as an MPIO-enabled build of HDF5. Check out Parallel HDF5 for details.

Variable-length (VLEN) data

Starting with version 2.3, all supported types can be stored in variable-length arrays (previously only variable-length byte and unicode strings were supported) See Special Types for use details. Please note that since strings in HDF5 are encoded as ASCII or UTF-8, NUL bytes are not allowed in strings.

Enumerated types

HDF5 enumerated types are supported. As NumPy has no native enum type, they are treated on the Python side as integers with a small amount of metadata attached to the dtype.

NumPy object types

Storage of generic objects (NumPy dtype “O”) is not implemented and not planned to be implemented, as the design goal for h5py is to expose the HDF5 feature set, not add to it. However, objects picked to the “plain-text” protocol (protocol 0) can be stored in HDF5 as strings.

Appending data to a dataset

The short response is that h5py is NumPy-like, not database-like. Unlike the HDF5 packet-table interface (and PyTables), there is no concept of appending rows. Rather, you can expand the shape of the dataset to fit your needs. For example, if I have a series of time traces 1024 points long, I can create an extendable dataset to store them:

>>> dset = myfile.create_dataset("MyDataset", (10, 1024), maxshape=(None, 1024))
>>> dset.shape
(10,1024)

The keyword argument “maxshape” tells HDF5 that the first dimension of the dataset can be expanded to any size, while the second dimension is limited to a maximum size of 1024. We create the dataset with room for an initial ensemble of 10 time traces. If we later want to store 10 more time traces, the dataset can be expanded along the first axis:

>>> dset.resize(20, axis=0)   # or dset.resize((20,1024))
>>> dset.shape
(20, 1024)

Each axis can be resized up to the maximum values in “maxshape”. Things to note:

  • Unlike NumPy arrays, when you resize a dataset the indices of existing data do not change; each axis grows or shrinks independently

  • The dataset rank (number of dimensions) is fixed when it is created

Unicode

As of h5py 2.0.0, Unicode is supported for file names as well as for objects in the file. When object names are read, they are returned as Unicode by default.

However, HDF5 has no predefined datatype to represent fixed-width UTF-16 or UTF-32 (NumPy format) strings. Therefore, the NumPy ‘U’ datatype is not supported.

Exceptions

h5py tries to map the error codes from hdf5 to the corresponding Exception class on the Python side. However the HDF5 group does not consider the error codes to be public API so we can not guarantee type stability of the exceptions raised.

Development

Building from Git

We moved to GitHub in December of 2012 (http://github.com/h5py/h5py).

We use the following conventions for branches and tags:

  • master: integration branch for the next minor (or major) version

  • 2.0, 2.1, 2.2, etc: bugfix branches for released versions

  • tags 2.0.0, 2.0.1, etc: Released bugfix versions

To build from a Git checkout:

Clone the project:

$ git clone https://github.com/h5py/h5py.git
$ cd h5py

(Optional) Choose which branch to build from (e.g. a stable branch):

$ git checkout 2.1

Build the project. If given, /path/to/hdf5 should point to a directory containing a compiled, shared-library build of HDF5 (containing things like “include” and “lib”):

$ python setup.py build [--hdf5=/path/to/hdf5]

(Optional) Run the unit tests:

$ python setup.py test

Report any failing tests to the mailing list (h5py at googlegroups), or by filing a bug report at GitHub.