Background information sources

• HDF5 user's guide
• Another one
• Sample Server

Data type mappings

1. Is DAP supporting 64-bit and 128-bit integer? No. We might if there are many users requesting it, but it's better to not support some types if they are infrequently used than make all of the clients support them. Of course, clients can choose to not support them, but making clients hard(er) to write seems to limit use more than not supporting some data. [jhrg 5/10/07]
2. Is the datatype size in DAP fixed? For example, is byte always 8-bit and Int32 always 32-bit? [ky 2/22/2007] Yes. [jhrg 5/10/07]

Reference to DAP

More about Datatype Mapping

Dataset { 
    Structure {

What's left

Implementation language

Client support

Sample Data

Performance testing

Testsuite development

Aura EOS5 data support

The h5dump header output of h5group.h5,h5_objref.h5 and h5_regref.h5 can be found under ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/Samples-for-dap-enhancement/group/h5group.txt ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/Samples-for-dap-enhancement/references/h5_objref.txt ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/Samples-for-dap-enhancement/references/h5_regref.txt

There two important reasons to map HDF5 group to DAP: 1) We want to well preserve the attribute information of HDF5 Group in this way. Otherwise, the attribute information inside group may not be easily recognized from the client.

2) It will be easy for a future HDF5 client to retrieve the information and rebuild the HDF5 file.

Based on the discussion today, use the example at ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/Samples-for-dap-enhancement/group/h5group.txt

We propose to map the group hierarchy with following information:

Dataset { Structure { Structure { Structure { string h5_comp; }foo2; string h5_array_1; } foo_1;

Structure { string h5_array_1; }foo1_2 }"/"; }h5_group.h5

The string will store the absolute path of the HDF5 dataset(DAP variable) name.

In this way, the attribute information of HDF5 group can be preserved nicely. And the ambiguity caused by HDF5 compound datatype dataset and HDF5 group can be avoided since no HDF5 dataset can have the name "/". However, we do need a careful document for this in the new HDF5 to DAP mapping. Please review this, James. One thing we would like to do is:

Is an empty structure legal? If this is not legal for DAP. Then we have a big problem!!! It is perfectly fine to have an empty group inside HDF5.

Dataset { Structure { }foo3; }; If an empty DAP structure is legal, I think we get this problem solved. Otherwise, you do need to think to create a new object inside DAP to support this; perhaps.

[ky 2007-5-3]

James will define an abstract data type for Group at DAP4. In DAP2, Group will be mapped to structure with a special attribute. [ky 2007-3-8]

Aura EOS5 data support

Aura data support DAP grid data. So we MUST FIND a WAY to map Aura HDF-EOS5 (Grid,Swath,Point) Data to DAP (Grid,Swath,Point) Data using only HDF5 APIs. The problem can be described as follows: HDF-EOS5 data à Using HDF5 library to retrieve all information à Mapping to DAP correctly. For example: (HDF-EOS5 grid) à (HDF5 without geolocation APIs) à DAP Grid correctly All HDF-EOS5 grid geolocation information are put inside an internal HDF5 group called structMetadata. One unclear point is how different projection can be accepted by DAP. Step to tackle this problem: FOR JAMES, PLEASE HELP: How HDF4 handler works with HDF-EOS2 data since swath, grid and point are not new concepts. Please provide us some hints on this. Specifically: which part of code we should read and related documents we can read. [ky 2/22/07]

Several sample HDF5 files that help understand the mapping of group,object reference and data region reference can be found under ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/Samples-for-dap-enhancement

A readme file that describes these files can be found on the parent directory: ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/ [ky]

The sample HDF5 files through OPeNDAP can be found at ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/opendap/. Please read the readme file for more information.

Here is the URL for quick comparisons: http://kagiso.hdfgroup.uiuc.edu:8880/

HDF5DataHandler

The what part of testsuite is still under investigation. The good starting point will be re-using the hdf4 testsuites by converting them into h5 format. The question is how extensive and valid the hdf4 testsuite is.

-- JoeLee - 08 Feb 2007

[ky]

[ky]

I totally agree with you. The unknown performance and interfacing issues make it too risky to go to Python approach. I think I will send out the suggestions to Roberto. If he wants to improve the pydap HDF5 handler, it will be good to NASA Python users. [ky]

[jhrg]

Todo: List the operations for Group

More about Datatype Mapping

Object reference needs to be mapped appropriately to DAP with the suggestions above.

What's left

Sample Data

Performance testing

• What are the criteria of performance related to DAP-HDF5 applications? We need to have at least several areas that are most interested to users to focus on our testings and improvements.

Kent will contact with several potential NASA user groups to ask their opinions.

Reference to DAP

• Python is very clean and easy to use language.
• Both Pydap and Pytables are easy to build in a sense.
• Pydap-HDF5 plugin can serve HDF5 data with limitations.
• Pydap, Pytables continue gaining popularities,even among NASA users
• Both developers are quick to reply
• Pydap-HDF5 plugin is very concise(3 pages in total)
• It is very possible that DODS sequence can be supported by HDF5 via Pytable. This needs some efforts with C++ APIs. The structure of DAP sequence is exactly like struct. However, DAP explicitly demands the sequence to be like a relational database and it depends on the DAP Data handler to *perform the sequence data like a query to the relational database*(called field projection in DAP). Pytables provides the possibility of this promise through its powerful indexing feature. See the selecting value example HowToUse Pytables.

• Pydap is python implementation of DAP, can its performance compete with OPeNDAP?
• Pydap-HDF5 plugin needs more work,

• Pytables doesn't support reference. Unlike Pydap, if we ask Pytables people to

• It involves more packages(Pytables,numpy) than pure HDF5 C++ handler.
• How stable will the pytables, python handler and pydap be?
• Another question added by Mike is that how difficult a C client to use Pydap?

I really think we should help build a python HDF5 handler at least since python is such a promising language.

If you think to add new operators, the HDF5 link operations may give you some hints. It basically links the relations among different groups; groups and datasets.

Since group is a pretty important concept in HDF5, it is better to be supported gracefully in OPeNDAP.

1. Extending PyTables to read more of HDF5
2. Improving efficiency in PyDAP

I've spent some time learning and installing Pydap HDF5 plugin in the past few days. The followings are my thoughts regarding to this issue:

Pros:

1. Python is very clean and easy to use language.
2. Both Pydap and Pytables are easy to build in a sense.
3. Pydap-HDF5 plugin can serve HDF5 data with limitations.
4. Pydap, Pytables continue gaining popularities,even among NASA users
5. Both developers are quick to reply
6. Pydap-HDF5 plugin is very concise(3 pages in total)
7. It is very possible that DODS sequence can be supported by HDF5 via Pytable. This needs some efforts with C++ APIs.

Cons:

1. Pydap is python implementation of DAP, can its performance compete with OPeNDAP??
2. Pydap-HDF5 plugin needs more work, Currently it doesn't support

• HDF5 compound datatype,
• array datatype,
• object reference
• variable length
• some atomic type like char
• group attributes

1. Pytables doesn't support reference. Unlike Pydap, if we ask Pytables people to support reference, extra fundings may be required.
2. It involves more packages(Pytables,numpy) than pure HDF5 C++ handler.
3. How stable will the pytables, python handler and pydap be?

My suggestions:

Cons 1 and 5 are my big concerns. We are taking great risks if we decide to use Python rather than the original C++ handler. However, I also like the Pros. So maybe we can focus on implementing C++ handler but help Roberto improve HDF5 plugin. If possible, persuade Pytables people or advertise other python fans to provide a little funding to improve pytables. I will spend the next few days reading Roberto's HDF5 plug-in and possibly provide the help. James, Feel free to persuade me to change my mind.

I have several Aura sample files and one NPOESS sample file.

1. Interfacing the PyDAP/PyTables HDF5 plugin to the BES
2. Extending PyTables to read more of HDF5
3. Improving efficiency in PyDAP

• The Boost.Python library can be used to embed Python in C++. There's also a SIG with an active mail list.
• There's info about C++/Python mixing at python.org, as well.

PyTables Has a good FAQ.

PyDAP provides an interface to the DAP.

Main topics

Data type mappings

Add Group by subclassing Structure and maybe adding some new operators. This would show up in a DDX as a new type distinct from Structure, but in the DDS/DAS it would show up as a Structure and the client would not be able to use the new operations.

Reference/Link:Map to URL?

Implementation language

C++: Build on the existing HDF5 handler

Python: Use PyDAP and PyTables. What types of HDF5 files could this not read? What capabilities would it support that the regular HDF5 library does not? How to Integrate Python-based handlers into the Server4 BES framework?

Client support

• Ocapi?
• NetCDF CL

+

Data files: We're initially thinking of making the handler work with AURA and NPOS data files. Is this a reasonable place to start? If so, we need sample files. If we're going after other groups of files, then lets get samples of those too.

Testsuite development

Document how to develop testsuites for a data handler

-- JamesGallagher - 17 Nov 2006