File objects serve as your entry point into the world of HDF5. In addition to the File-specific capabilities listed here, every File instance is also an HDF5 group representing the root group of the file.
Opening & creating files¶
HDF5 files work generally like standard Python file objects. They support standard modes like r/w/a, and should be closed when they are no longer in use. However, there is obviously no concept of “text” vs “binary” mode.
>>> f = h5py.File('myfile.hdf5','r')
The file name may be a byte string or unicode string. Valid modes are:
r Readonly, file must exist r+ Read/write, file must exist w Create file, truncate if exists w- or x Create file, fail if exists a Read/write if exists, create otherwise (default)
HDF5 ships with a variety of different low-level drivers, which map the logical HDF5 address space to different storage mechanisms. You can specify which driver you want to use when the file is opened:
>>> f = h5py.File('myfile.hdf5', driver=<driver name>, <driver_kwds>)
For example, the HDF5 “core” driver can be used to create a purely in-memory HDF5 file, optionally written out to disk when it is closed. Here’s a list of supported drivers and their options:
- Strongly recommended. Use the standard HDF5 driver appropriate for the current platform. On UNIX, this is the H5FD_SEC2 driver; on Windows, it is H5FD_WINDOWS.
- Unbuffered, optimized I/O using standard POSIX functions.
- Buffered I/O using functions from stdio.h.
Memory-map the entire file; all operations are performed in memory and written back out when the file is closed. Keywords:
- backing_store: If True (default), save changes to a real file
- when closing. If False, the file exists purely in memory and is discarded when closed.
- block_size: Increment (in bytes) by which memory is extended.
- Default is 64k.
Store the file on disk as a series of fixed-length chunks. Useful if the file system doesn’t allow large files. Note: the filename you provide must contain a printf-style integer format code (e.g. %d”), which will be replaced by the file sequence number. Keywords:
memb_size: Maximum file size (default is 2**31-1).
HDF5 has been evolving for many years now. By default, the library will write objects in the most compatible fashion possible, so that older versions will still be able to read files generated by modern programs. However, there can be performance advantages if you are willing to forgo a certain level of backwards compatibility. By using the “libver” option to File, you can specify the minimum and maximum sophistication of these structures:
>>> f = h5py.File('name.hdf5', libver='earliest') # most compatible >>> f = h5py.File('name.hdf5', libver='latest') # most modern
Here “latest” means that HDF5 will always use the newest version of these structures without particular concern for backwards compatibility. The “earliest” option means that HDF5 will make a best effort to be backwards compatible.
The default is “earliest”.
HDF5 allows the user to insert arbitrary data at the beginning of the file,
in a reserved space called the user block. The length of the user block
must be specified when the file is created. It can be either zero
(the default) or a power of two greater than or equal to 512. You
can specify the size of the user block when creating a new file, via the
userblock_size keyword to File; the userblock size of an open file can
likewise be queried through the
Modifying the user block on an open file is not supported; this is a limitation of the HDF5 library. However, once the file is closed you are free to read and write data at the start of the file, provided your modifications don’t leave the user block region.
Filenames on different systems¶
Different operating systems (and different file systems) store filenames with different encodings. Additionally, in Python there are at least two different representations of filenames, as encoded bytes (via str on Python 2, bytes on Python 3) or as a unicode string (via unicode on Python 2 and str on Python 3). The safest bet when creating a new file is to use unicode strings on all systems.
macOS is the simplest system to deal with, it only accepts UTF-8, so using unicode paths will just work (and should be preferred).
Linux (and non-macOS Unix)¶
Unix-like systems use locale settings to determine the correct encoding to use.
These are set via a number of different environment variables, of which
LC_ALL are the ones of most interest. Of special interest is the
locale, which Python will interpret as only allowing ASCII, meaning unicode
paths should be pre-encoded. This will likely change in Python 3.7 with
https://www.python.org/dev/peps/pep-0538/, but this will likely be backported by
distributions to earlier versions.
To summarise, use unicode strings where possible, but be aware that sometimes using encoded bytes may be necessary to read incorrectly encoded filenames.
Windows systems have two different APIs to perform file-related operations, a ANSI (char, legacy) interface and a unicode (wchar) interface. HDF5 currently only supports the ANSI interface, which is limited in what it can encode. This means that it may not be possible to open certain files, and because External links do not specify their encoding, it is possible that opening an external link may not work. There is work being done to fix this (see https://github.com/h5py/h5py/issues/839), but it is likely there will need to be breaking changes make to allow Windows to have the same level of support for unicode filenames as other operating systems.
The best suggestion is to use unicode strings, but to keep to ASCII for filenames to avoid possible breakage.
Unlike Python file objects, the attribute
File.name gives the
HDF5 name of the root group, “
/”. To access the on-disk name, use
File(name, mode=None, driver=None, libver=None, userblock_size, **kwds)¶
Open or create a new file.
Note that in addition to the File-specific methods and properties listed below, File objects inherit the full interface of
- name – Name of file (str or unicode), or an instance of
h5f.FileIDto bind to an existing file identifier.
- mode – Mode in which to open file; one of (“w”, “r”, “r+”, “a”, “w-“). See Opening & creating files.
- driver – File driver to use; see File drivers.
- libver – Compatibility bounds; see Version Bounding.
- userblock_size – Size (in bytes) of the user block. If nonzero, must be a power of 2 and at least 512. See User block.
- kwds – Driver-specific keywords; see File drivers.
Close this file. All open objects will become invalid.
Request that the HDF5 library flush its buffers to disk.
Name of this file on disk. Generally a Unicode string; a byte string will be used if HDF5 returns a non-UTF-8 encoded string.
String indicating if the file is open readonly (“r”) or read-write (“r+”). Will always be one of these two values, regardless of the mode used to open the file.
String giving the driver used to open the file. Refer to File drivers for a list of drivers.
- name – Name of file (str or unicode), or an instance of