The main documentation is organized by documentation flavor. It is impractical to document the entire HDF5 ecosystem in one place, and you should also consult the documentation sets of the many outstanding community projects.įor a first contact with HDF5, the best place is to have a look at the getting started page that shows you how to write and compile your first program with HDF5. It includes specifications and documentation of software and tools developed and maintained by The HDF Group. So ideally I'd like to work in pandas and pyarrow as much as I can.This is the documentation set for HDF5.

But the files are large enough that I'd like to avoid loading them into memory if I can. Is the a straightforward way to read an arbitrary hdf5 file into pandas or pytables? I can load the data using h5py if I need to. Pd.read_hdf(file_path, key='/interval', mode='r')īased on this answer I'm assuming that the problem is related to the fact that Pandas is expecting a very particular hierarchical structure, which is different to the one that the the actual hdf5 file has. ValueError: Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file. Similarly if I try use pd.read_hdf it looks like the file is empty. TypeError: cannot create a storer if the object is not existing nor a value are passed assert not oups()Īnd if I try to access the data I get the following error bar = store.get('/bar') Then the HDFStore object has no keys or groups. If I try to read the data in using HDFStore I am unable to access any of the groups. In this case I'm interested in the group 'bar', which has 3 items in it. If I open the file using h5py the data looks exactly how I would expect. Which I think is related to the way that the files were originally created. I'm struggling to read them into pandas/pyarrow though. I have a bunch of hdf5 files, and I want to turn some of the data in them into parquet files.

Python ffmpeg with hdf5