To obtain the Szip source code:
ftp://ftp.hdfgroup.org/lib-external/szip/

Szip Compression in HDF Products

Szip compression software, providing lossless compression of scientific data, is provided with HDF software products as of HDF5 Release 1.6.0 and HDF4 Release 2.0.

Comparison of compression performance in tests with HDF4

Size of compressed output [1]

Speed of compression operation [1]

Szip is an implementation of the extended-Rice lossless compression algorithm. The Consultative Committee on Space Data Systems (CCSDS) has adopted the extended-Rice algorithm for international standards for space applications[1,6,7]. Szip is reported to provide fast and effective compression, specifically for the EOS data generated by the NASA Earth Observatory System (EOS)[1]. It was originally developed at University of New Mexico (UNM) and integrated with HDF4 by UNM researchers and developers.

As the graphs to the right illustrate, the primary gain with Szip compression is in speed of processing. Szip also provides some advantage in compression ratio over other compression methods shown here. These results, including the data presented in the graphs below, are from tests conducted by Pen-Shu Yeh, et al. [1], with the HDF4 Szip integration.

Szip and HDF5

Using Szip compression in HDF5: Szip is a stand-alone library that is configured as an optional filter in HDF5. To use Szip with HDF5, the Szip binary library must be downloaded and the HDF5 library must be configured and compiled with Szip support and with the Szip library. Since use of the Szip library is governed by a license (see "Licensing terms" below), The HDF Group (THG) distributes two types of Szip library binaries: one with both the encoder and decoder enabled (license may be required) and one with the decoder only (license-free). Depending on which binary is used, an HDF5 application can create, write, and read datasets compressed with Szip compression, or can only read datasets compressed with Szip.

Applications use Szip by setting Szip as an optional filter when a dataset is created. If the Szip encoder is enabled with the HDF5 library, data is automatically compressed and decompressed with Szip during I/O. If only the decoder is present, the HDF5 library cannot create and write Szip-compressed datasets, but it automatically decompresses Szip-compressed data when data is read.

See this sample HDF5 program for an illustration of the use of Szip compression with HDF5.

Details of the required and optional parameters are provided in the H5Pset_szip entry in the HDF5 Reference Manual.

Software distribution: Starting with Release 1.6.0, HDF5 has been distributed with Szip enabled, making it easier to use Szip compression. As of Release 1.6.4, the software is distributed as follows:

Pre-compiled HDF5 binaries are provided with the Szip encoder and decoder enabled.
Szip source code and two versions of Szip binaries, one with the encoder and decoder enabled and one with only the decoder enabled, will be distributed by THG. (FTP)
Users who cannot legally use Szip encoding should download and install the decode-only Szip Release 2.0 binary.

Szip and HDF4

Using Szip compression in HDF4: Szip is a stand-alone library that is configured as an optional filter in HDF4. To use Szip with HDF4, the Szip binary library must be downloaded and the HDF4 library must be configured and compiled with Szip support and with the Szip library.

Applications use Szip by setting Szip as an optional filter when a dataset is created. Once enabled, data is automatically compressed and decompressed with Szip during I/O.

See this sample HDF4 program for an illustration of the use of Szip compression with HDF4.

Details of the required and optional parameters are provided in the SDsetcompress, SDgetcompress, SDsetchunk, SDgetchunkinfo, and HCget_config_info entries in the HDF4 Reference Manual.

HDF4 Release 2.1 provides substantial bug fixes and includes a few changes in the use of Szip; users should upgrade to HDF4 Release 2.1 as soon as possible. Szip has been extensively tested in this release, but has not been user-tested. Users are encouraged to use Szip and report any problems to THG as soon as possible. For more details see the release notes for Szip in HDF4 Release 2.1.

Software distribution: Starting with Release 2.0, HDF4 has been distributed with Szip enabled, making it easier to use Szip compression:

Szip will be enabled and pre-configured in pre-compiled HDF4 binaries.
Szip source code and Szip binaries will be distributed by THG. (FTP)
Users who cannot legally use Szip encoding should download and build the HDF4 library with the decode-only Szip binary, using Szip Library 2.0.

Licensing terms

The version of Szip distributed with HDF products is free for non-commercial use, which may occur in two sets of circumstances:

Non-commercial users may use the Szip software integrated with HDF products to both encode (compress) and decode (uncompress) data. This applies to educational and research applications.
Commercial users may use the software to decode any data. Further, they may use the software in internal activities that do not involve or result in the development of an Szip-based software product.

Commercial licenses are available for commercial users who wish to distribute an Szip-based software product or engage in commercial uses that are not allowed above. For further licensing information or to view a copy of the Szip copyright statement, see Commercial use terms and the copyright and license notice pertaining to Szip in HDF products.

Further information

See the following materials for further information:

HDF5: The H5Pset_szip entry in the Release 1.6 HDF5 Reference Manual for HDF5 Szip usage information
HDF4: Szip in HDF4 Release 2.1 (PDF)
The SDsetcompress, SDgetcompress, SDsetchunk, SDgetchunkinfo, and HCget_config_info entries in the HDF4 Reference Manual (Release 2.1 or later) for HDF4 Szip usage information
Examples of Szip usage in HDF5 and in HDF4
Szip 2.0 Release Notes (PDF)
An “Szip Performance Study” (PDF)
The Szip source code at ftp://ftp.hdfgroup.org/lib-external/szip/
Commercial use terms and the copyright and license notice pertaining to Szip in HDF products
The HDF/Szip collaborative agreement (PDF)
References to publications discussing Szip or its use

Describes HDF5 Release 1.6.5, and subsequent releases
Describes HDF4 Release 2.1, February 2005, and subsequent releases

Last modified: April 2, 2008