| • |
|
|
|
|||
|
||||
NetCDF (network Common Data Form) is a set of interfaces for array-oriented data access and a freely-distributed collection of data access libraries for C, Fortran, C++, Java, and other languages. The netCDF libraries support a machine-independent format for representing scientific data. Together, the interfaces, libraries, and format support the creation, access, and sharing of scientific data.
NetCDF data is:
The netCDF software was developed by Glenn Davis, Russ Rew, Ed Hartnett, John Caron, Steve Emmerson, and Harvey Davies at the Unidata Program Center in Boulder, Colorado, with contributions from many other netCDF users.
The latest source distribution, which includes the C, Fortran, and C++ interfaces, is available for downloading as a compressed tar file or a gzipped tar file. Installation instructions are available with the distribution or online. The Java interface for netCDF is available separately from the NetCDF Java site.
Binary distributions for some platforms are available from the directory ftp://ftp.unidata.ucar.edu/pub/binary.
Version 4.0 provides compatibility with existing netCDF programs and data, additional data modeling abstractions, and features for use in high performance computing, such as parallel I/O. Support for an enhanced data model is made possible by including the capability to use HDF5 as the data storage layer for the netCDF API.
Features of the netCDF-4 data model include:
Performance benefits of the HDF5-based netCDF-4 format include:
For more information about netCDF-4, see
This release requires the HDF5 1.8.1 release, and the zlib 1.2.3 (or later) release.
Version 4.0 was released on June 12, 2008. The 4.0 release can be obtained as a gzipped tar file or compressed tar file.
Version 3.6.3 of netCDF includes the following features:
For more details, see the release notes.
Version 3.6.3 was released on June 12, 2008. The 3.6.3 release can be obtained as a gzipped tar file or compressed tar file.
Version 3.6.2 of netCDF includes the following features:
Version 3.6.2 also includes some minor bug fixes, and increased testing and portability.
For more details, see the release notes.
Version 3.6.2 was released on March 5, 2007. The 3.6.2 release can be obtained as a gzipped tar file or compressed tar file. (See also the 3.6.2 release of the documentation).
Version 3.6.1 includes some minor bug fixes, and increased testing and portability.
For more details, see the version 3.6.1 release notes.
Version 3.6.1 was released on February 27, 2006. The 3.6.1 release can be obtained as a gzipped tar file or compressed tar file. (See also the 3.6.1 release of the documentation).
Version 3.6.0 includes improved large file support, simplified installation, bug fixes, and portability enhancements to version 3.5. The original netCDF file format is still supported as the default, so files written with version 3.6.0 can be read with previous versions and vice versa. Existing source code requires no code changes.
Version 3.6.0 also supports a new variant of the file format with 64-bit offsets for very large files. Minor code changes are required to create such files by using a non-default flag on creation, and files that use the new 64-bit offset variant will not be readable by previous versions of the netCDF library. Applications built with the version 3.6.0 library will be able to access both formats transparently.
Users are discouraged from using the new format variant unless required for files too large for the classic netCDF format. Third-party software that supports netCDF data access should eventually be upgraded to version 3.6 or later to support both the classic and the new large-file format variant.
The netCDF User's Guides for C, FORTRAN-77, and FORTRAN-90 have been converted from a proprietary format to Texinfo, used in the GNU documentation system. There is also a new Language-neutral User's Guide that documents the netCDF data model and other language-independent concepts. These changes will support better maintenance of the documentation int the future.
Version 3.6 incorporates improved configuration and installation support, so that on most platforms, the netCDF library and utilities can be built without specifying environment variables, and the library will be built with large file support by default, if available. Better Windows support is also available now, included with the standard distribution.
Performance enhancements and bug fixes include a speed-up of Fortran-90 array access interfaces, addition of a "-x" option to ncgen to specify use of fast "no fill" mode for netCDF file creation, fixes to ncdump and ncgen utilities to properly handle large dimensions, and removal of an unintended constraint on record sizes.
For more details, see the version 3.6.0 release notes.
Version 3.6.0 was released on February 27, 2006. The 3.6.0 release can be obtained as a gzipped tar file or compressed tar file. (See also the 3.6.0 release of the documentation).
There are many ways to represent the same information in any general-purpose data model. Choices left up to the user in the case of netCDF include which information to represent as variables or as variable attributes; what names to choose for variables, dimensions, and attributes; what order to use for the dimensions of multidimensional variables; what variables to include in the same netCDF file; and how to use variable attributes to capture the structure and meaning of data. We provide some guidelines in the NetCDF User's Guide (e.g., the section on ``Differences between Attributes and Variables'') and in a new web document ``Writing NetCDF Files: Best Practices'', but we've found that a little experience helps. Occasionally we have decided it was useful to change the structure of netCDF files after experience with how the data is used.
NetCDF files should have the file name extension ".nc". The recommended extension for netCDF files was changed from ".cdf" to ".nc" in 1994 in order to avoid a clash with the NASA CDF file extension, and now it also avoids confusion with "Channel Definition Format" files.
The netcdfgroup@unidata.ucar.edu mailing-list is intended for discussions and announcements about netCDF interfaces, software, and use. The volume of this list varies widely, from one message per month to a dozen messages per day (especially after a new release). A message posted to this mailing-list will be seen by several hundred people, so it's usually not appropriate for asking simple questions about use. Such questions should instead be sent to support@unidata.ucar.edu.
If you would prefer to get only a single daily digest of the postings to the netcdfgroup mailing-list, subscribe instead to the ncdigest@unidata.ucar.edu mailing-list. It's a digested form of the netcdfgroup mailing-list, containing the same messages but appearing at most once per day instead of whenever anyone sends a message to the group.
To subscribe or unsubscribe to either of these mailing lists, use this form.
Here are some example netCDF files.
Discussions of conventions for representing time and handling time-dependent data have been a past topic of discussion on the netcdfgroup mailing list. When the subject comes up, interesting discussions often result, so we've archived past discussions on this subject at http://www.unidata.ucar.edu/software/netcdf/time/.
A summary of Unidata's recommendations is available from http://www.unidata.ucar.edu/software/netcdf/time/recs.html. Briefly, we recommend use of the units conventions supported by the udunits library for time and other units attributes.
Other groups have established more specific conventions that include the representation of time in netCDF files. For more information on such conventions, see the NetCDF Conventions Page at http://www.unidata.ucar.edu/software/netcdf/conventions.html.
The netCDF mailing list has over 500 addresses (some of which are aliases to more addresses) in thirty countries. Several groups have adopted netCDF as a standard way to represent some forms of scientific data.
A somewhat dated description of some of the projects and groups that have used netCDF is available from http://www.unidata.ucar.edu/software/netcdf/usage.html.
A primary reference is the User's Guide:
Rew, R. K., G. P. Davis, S. Emmerson, and H. Davies, NetCDF User's Guide for C, An Interface for Data Access, Version 3, April 1997.
Current online and downloadable documentation is available from the documentation directory.
Other references include:
Brown, S. A, M. Folk, G. Goucher, and R. Rew, "Software for Portable Scientific Data Management," Computers in Physics, American Institute of Physics, Vol. 7, No. 3, May/June 1993, pp. 304-308.
Fulker, D. W., "Unidata Strawman for Storing Earth-Referencing Data," Seventh International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, New Orleans, La., American Meteorology Society, January 1991.
Jenter, H. L. and R. P. Signell, 1992. "NetCDF: A Freely-Available Software-Solution to Data-Access Problems for Numerical Modelers". Proceedings of the American Society of Civil Engineers Conference on Estuarine and Coastal Modeling. Tampa, Florida.
Kuehn, J.A., "Faster Libraries for Creating Network-Portable Self-Describing Datasets", Proceedings of the 37th Cray User Group Meeting, (Barcelona, Spain, March 1996), Cray User Group, Inc.
Rew, R. K. and G. P. Davis, "NetCDF: An Interface for Scientific Data Access," IEEE Computer Graphics and Applications, Vol. 10, No. 4, pp. 76-82, July 1990.
Rew, R. K. and G. P. Davis, "The Unidata netCDF: Software for Scientific Data Access," Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Anaheim, California, American Meteorology Society, pp. 33-40, February 1990.
Rew, R. K. and G. P. Davis, " Unidata's netCDF Interface for Data Access: Status and Plans," Thirteenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, Anaheim, California, American Meteorology Society, February 1997.
Yes, the NetCDF User's Guide contains an appendix that specifies the netCDF file format.
Also, the "NetCDF File Structure and Performance" chapter explains the physical structure of netCDF data at a high enough level to make clear the performance implications of different data organizations.
If users only access netCDF data through the documented interfaces, future changes to the format will be transparent.
We tested the 3.6 release on the following operating systems with various compilers:
The NetCDF Installation and Porting Guide explains how to build netCDF from source on various platforms. Often, it's as easy as running
./configure make test make install
Using netCDF on Windows platforms can be accomplished in several ways.
Note that we have not ported the F90 or C++ APIs to the Windows platform, only the C and F77 APIs. User contributions of ports to F90 windows compilers are very welcome (send them to support-netcdf@unidata.ucar.edu).
On windows, NetCDF consists of a DLL and the ncgen/ncdump executables. The easiest course is to download one of the pre-built DLLs and utilities and just install them on your system.
If you want to use the C API as a DLL, get the pre-built 3.6.1 DLL file.
If you want to use the Fortran API, you may use the pre-built 3.5.1 DLL release. This does not include the 3.6.0 large file feature, so it is possible for other users to create netCDF files of 64-bit offset format which will not be readable to the 3.5.1 version of netCDF. However, netCDF classic format files (the default) are fully compatible between versions.
Some experimental pre-built 3.6.2 DLLs are available for download. We are working on improving the 3.6.2 situation, but for now, try the PGI fortran or f2c fortran, or intel fortran DLLs. These contain a stable 3.6.2 beta release. Please send email to support-netcdf@unidata.ucar.edu if you use these, telling us about your experience.
If you are developing a netCDF-intensive application and want full integration with the Microsoft development environment, you might instead build netCDF from source within Visual Studio. In that case, get the netCDF-3 snapshot and open the netcdf-3.6.2/win32/NET/netcdf.sln file from within Visual Studio. You can build in the usual way, by selecting build solution from the build menu.
The netCDF 3.6.2 release can also use libtool to build shared libraries on various platforms, including Windows. This requires MinGW.
Another alternative for Windows users who have Cygwin (or some other Unix emulator on Windows) is to download the latest release and build as with Unix. The latest release is tested under Cygwin and passes all tests cleanly. (The latest Cygwin does not yet include F90 support, but the daily netCDF snapshot is now testing with gfortran and g95 on cygwin). To build under Cygwin, follow the Unix build instructions in a Cygwin shell. The --enable-shared option to configure will generate the netcdf.dll.
The NetCDF Installation and Porting Guide documents how to use netCDF with Windows.
Windows is a complicated platform to build on. Some useful explanations of the oddities of Windows can be found here:
Once you have the netCDF DLL, you may wish to call it from Visual Basic. The netCDF VB wrapper will help you do this.
If you have an unusual combination of compilers or platform, you may have to set a few environment variables before invoking
./configure make test make install
For details, see the NetCDF Installation and Porting Guide.
You should also check reports we maintain of successful builds in other environments. If you have success with a combination useful to others, please send it for addition to the list.
We make build output from various platforms available for comparison with your output. In general, you can ignore compiler warnings if the "make test" step is successful. Lines that begin with "***" in the "make test" output indicate results from tests. The C and Fortran-77 interfaces are tested extensively, but only rudimentary tests are currently used for the C++ and Fortran-90 interfaces.
If you invoke
ncdump --version
the last line of the resulting output will identify the version
associated with the ncdump utility. You can also call one
of the functions nc_inq_libvers(),
nf_inq_libvers(), or nf90_inq_libvers()
from C, Fortran-77, or Fortran-90 programs to get a version string.
The netCDF installation directory can be set at the time configure is run using the --prefix argument. If it is not specified, /usr/local is used as the default prefix.
For more information see the NetCDF Installation and Porting Guide.
In netCDF 3.6.1 the --enable-64bit option to configure will attempt to build 64-bit versions of the netCDF library. This option was discontinued in the 3.6.2 release.
Users are invited to contribute information for the list below. Send successful build information to <support-netcdf@unidata.ucar.edu>
The following environment variable settings work for us; they are given below as csh commands (sh uses take note!).
setenv CFLAGS "-xarch=v9" setenv CXXFLAGS "-xarch=v9" setenv FFLAGS "-xarch=v9" setenv F90FLAGS "-xarch=v9"
setenv CFLAGS -64 setenv CXXFLAGS -64 setenv FFLAGS -64 setenv F90FLAGS -64
setenv CFLAGS -q64 setenv CXXFLAGS -q64 setenv FFLAGS -q64 setenv F90FLAGS -q64 setenv ARFLAGS -X64 setenv NMFLAGS -X64
setenv CFLAGS +DD64 setenv CXXFLAGS +DD64 setenv FFLAGS "+noppu +DA2.0W" setenv F90FLAGS "+noppu +DA2.0W"
Shared libraries are libraries which can be shared by multiple running applications at the same time. This may improve performance.
For example, if I have a library which provides function foo(), and I have two applications which call foo(), then with a shared library, only one copy of the foo() function will be loaded into RAM, and both programs will use it. With static libraries, each application would have it's own copy of the foo() function.
More information on shared libraries can be found at the following external sites:
The Program-Library HowTo, by David Wheeler.
Starting with version 3.6.2, the netCDF can build shared libraries on platforms which support them, but by default netCDF will build static libraries only. To turn on shared libraries, use the --enable-shared option to the netCDF configure script.
With netCDF version 3.6.2, shared libraries can be built on platforms which support them by using the --enable-shared argument to netCDF configure script.
Users of earlier versions of netCDF can build shared libraries by setting flags correctly during builds.
When you use a static library, the code is copied from the library into your program when the program is built. The library is only needed at build time.
With a shared library the code in the library is not copied into your executable, so the library is needed every time the program is run.
If you write a program that uses the netCDF shared library, the OS will have to find it every time your program is run. It will look in these places:
Directories you specified as shared library locations at build time. Unfortunately this is done differently with different compilers.
Directories specified in the environment variable LD_RUN_PATH at build time.
Directories specified in the OS-specific environment variable for this purpose at run time. (LD_LIBRARY_PATH on Linux and many other Unix variants, LOADLIBS on AIX systems, etc.)
A default list of directories which includes /usr/lib (but don't install software there!), and may or may not contain places you might install netCDF, like /usr/local/lib.
The directories specified in an OS file such as /etc/ld.conf.
By default the netCDF library will be installed in /usr/local/lib. (This can be overridden with the --prefix option to the netCDF configure script).
An external site by Arnaud Deistter has a table of different tools and command line options relating to shared libraries on Linux, Solaris, HP-UX, Tru64, AIX, SGI, Win32, MacOS X, VMS (wow!), and OS/390.
For more information about how do to this in Linux users may find it useful to read this extrernal webpage, some documentation from Caldera, a Linux distributor: Specifying directories to be searched by the dynamic linker.
Yes, but there are significant restrictions on the structure of large netCDF files that result from the 32-bit relative offsets that are part of the classic netCDF format. For details, see NetCDF Classic Format Limitations in the User's Guide.
Large File Support (LFS) refers to operating system and C library facilities to support files larger than 2 GiB. On many 32-bit platforms the default size of a file offset is still a 4-byte signed integer, which limits the maximum size of a file to 2 GiB. Using LFS interfaces and the 64-bit file offset type, the maximum size of a file may be as large as 263 bytes, or 8 EiB. For many current platforms, large file macros or appropriate compiler flags have to be set to build a library with support for large files. This is handled automatically in netCDF 3.6.
More information about Large File Support is available from Adding Large File Support to the Single UNIX Specification.
When the netCDF format was created in 1988, 4-byte fields were reserved for file offsets, specifying where the data for each variable started relative to the beginning of the file or the start of a record boundary.
This first netCDF format variant, the only format supported in versions 3.5.1 and earlier, is referred to as the netCDF classic format. The 32-bit file offset in the classic format limits the total sizes of all but the last non-record variables in a file to less than 2 GiB, with a similar limitation for the data within each record for record variables. For more information see Classic Format Limitations.
The netCDF classic format is also identified as version 1 or CDF1 in reference to the format label at the start of a file.
With netCDF 3.6, a second variant of netCDF format is now supported in addition to the classic format. The new variant is referred to as the 64-bit offset format, version 2, or CDF2. The primary difference from the classic format is the use of 64-bit file offsets instead of 32-bit offsets, but it also supports larger variable and record sizes.
No, version 3.6 of the netCDF library detects which variant of the format is used for each file when it is opened for reading or writing, so it is not necessary to know which variant of the format is used. The version of the format will be preserved by the library on writing. If you want to modify a classic format file to use the 64-bit offset format so you can make it much larger, you will have to create a new file and copy the data to it.
Yes, the 3.6 library and all planned future versions of the library will continue to support reading and writing files using the classic (32-bit offset) format as well as the new 64-bit offset format. There is no need to convert existing archives from the classic to the 64-bit offset format. Even netCDF-4, which will introduce a third variant of the netCDF format based on HDF5, will continue to support accessing classic format netCDF files as well as 64-bit offset netCDF files.
No, we discourage users from making use of the new format unless they need it for very large files. It may be some time until third-party software that uses the netCDF library is upgraded to 3.6 or later versions that support the new large file facilities, so we advise continuing to use the classic netCDF format for data that doesn't require huge file offsets. The library makes this recommendation easy to follow, since the default for file creation is the classic format.
The short answer is that under most circumstances, you should not care, if you use version 3.6.0 or later of the netCDF library. But the difference is indicated in the first four bytes of the file, which are 'C', 'D', 'F', '\001' for the classic netCDF format and 'C', 'D', 'F', '\002' for the new 64-bit offset format. On a Unix system, one way to display the first four bytes of a file, say foo.nc, is to run the following command:
od -An -c -N4 foo.nc
which will output
C D F 001
or
C D F 002
depending on whether foo.nc is a classic or 64-bit offset netCDF file, respectively.
With netCDF version 3.6.2 or later, there is an easier way, using the "-k" option to ncdump to determine the kind of file, for example:
ncdump -k foo.nc
classic
The application will indicate an error trying to open the file and present an error message equivalent to "not a netCDF file". This is why it's a good idea not to create 64-bit offset netCDF files until you actually need them.
Yes, by specifying the appropriate file creation flag you can create 64-bit offset netCDF files the same way on 32-bit platforms as on 64-bit platforms.
With netCDF version 3.6.0 or later, use the NC_64BIT_OFFSET flag when you call nc_create(), as in:
err = nc_create("foo.nc",
NC_NOCLOBBER | NC_64BIT_OFFSET,
&ncid);
In Fortran-77, use the NF_64BIT_OFFSET flag when you call nf_create(), as in:
iret = nf_create('foo.nc',
IOR(NF_NOCLOBBER,NF_64BIT_OFFSET),
ncid)
In Fortran-90, use the NF90_64BIT_OFFSET flag when you call nf_create(), as in:
iret = nf90_create(path="foo.nc",
cmode=or(nf90_noclobber,nf90_64bit_offset),
ncid=ncFileID)
In C++, use the Offset64Bits enum in the NcFile constructor, as in:
NcFile nc("foo.nc",
FileMode=NcFile::New,
FileFormat=NcFile::Offset64Bits);
A new flag, '-v', has been added to ncgen to specify the file format variant. By default or if '-v 1' or '-v classic' is specified, the generated file will be in netCDF classic format. If '-v 2' or '-v 64-bit-offset' is specified, the generated file will use the new 64-bit offset format. To permit creating very large files quickly, another new ncgen flag, '-x', has been added to specify use of nofill mode when generating the netCDF file.
No, there are still some limits on sizes of netCDF objects, even with the new 64-bit offset format. Each fixed-size variable and the data for one record's worth of a single record variable are limited in size to a little less that 4 GiB, which is twice the size limit in versions earlier than netCDF 3.6.
The maximum number of records remains 232-1.
While most platforms support a 64-bit file offset, many
platforms only support a 32-bit size for allocated memory blocks,
array sizes, and memory pointers. In C developer's jargon, these
platforms have a 64-bit off_t type for file offsets,
but a 32-bit size_t type for size of arrays. Changing
netCDF to assume a 64-bit size_t would restrict
netCDF's use to 64-bit platforms.
There are several possible reasons why creating a large file can fail that are not related to the netCDF library:
User quotas may prevent you from creating large files. On a Unix system, you can use the "ulimit" command to report limitations such as the file-size writing limit.
There is insufficient disk space for the file you are trying to write.
The file system in which you are writing may not be configured to allow large files. On a Unix system, you can test this with a commands such as
dd if=/dev/zero bs=1000000 count=3000 of=./largefile ls -l largefile rm largefile
which should write a 3 GByte file named "largefile" in the current directory, verify its size, and remove it.
If you get the netCDF library error "One or more variable sizes violate format constraints", you are trying to define a variable larger than permitted for the file format variant. This error typically occurs when leaving "define mode" rather than when defining a variable. The error status cannot be returned when a variable is first defined, because the last fixed-size variable defined is permitted to be larger than other fixed-size variables (when there are no record variables).
Similarly, the last record variable may be larger than other record variables. This means that subsequently adding a small variable to an existing file may be invalid, because it makes what was previously the last variable now in violation of the format size constraints. For details on the format size constraints, see the Users Guide sections NetCDF Classic Format Limitations and NetCDF 64-bit Offset Format Limitations.
If you get the netCDF library error "Invalid dimension size" for a non-negative size, you are exceeding the size limit of netCDF dimensions, which must be less than 2,147,483,644 for classic files with no large file support and otherwise less than 4,294,967,292.
No, except that 32-bit applications should link with a 32-bit version of the library and 64-bit applications should link with a 64-bit library, similarly to use of other libraries that can support either a 32-bit or 64-bit model of computation.
No, classic files created with the new library should be compatible with all older applications, both for reading and writing, with one minor exception. The exception is due to a correction of a netCDF bug that prevented creating records larger than 4 GiB in classic netCDF files with software linked against versions 3.5.1 and earlier. This limitation in total record size was not a limitation of the classic format, but an unnecessary restriction due to the use of too small a type in an internal data structure in the library.
If you want to always make sure your classic netCDF files are readable by older applications, make sure you don't exceed 4 GiBytes for the total size of a record's worth of data. (All records are the same size, computed by adding the size for a record's worth of each record variable, with suitable padding to make sure each record begins on a byte boundary divisible by 4.)
Utilities available in the current netCDF distribution from Unidata are ncdump, for converting netCDF files to an ASCII human-readable form, and ncgen for converting from the ASCII human-readable form back to a binary netCDF file or a C or FORTRAN program for generating the netCDF file. Software for Manipulating or Displaying NetCDF Data provides a list of other software useful for access, visualization, and analysis of netCDF data and data represented in other forms. Another useful guide to netCDF utilities is available from NOAA's Geophysical Fluid Dynamics Laboratory.
The Scientific Data Format Information FAQ provides a somewhat dated description of other access interfaces and formats for scientific data, including CDF and HDF. A brief comparison of CDF, netCDF, and HDF is available in the CDF FAQ. Another comparison is in Jan Heijmans' An Introduction to Distributed Visualization. John May's book Parallel I/O for High Performance Computing includes a chapter on Scientific Data Libraries that describes netCDF and HDF5, with example source code for reading and writing files using both interfaces.
CDF was developed at the NASA Space Science Data Center at Goddard, and is freely available. It was originally a VMS FORTRAN interface for scientific data access. Unidata reimplemented the library from scratch to use XDR for a machine-independent representation, designed the CDL (network Common Data form Language) text representation for netCDF data, and added aggregate data access, a single-file implementation, named dimensions, and variable-specific attributes.
NetCDF and CDF have evolved independently. CDF now supports many of the same features as netCDF (aggregate data access, XDR representation, single-file representation, variable-specific attributes), but some differences remain (netCDF doesn't support native-mode representation, CDF doesn't support named dimensions). There is no compatibility between data in CDF and netCDF form, and as yet no translation software exists to convert data in one form to data in the other form. For a more detailed description of differences between CDF and netCDF, see the CDF FAQ.
The National Center for Supercomputing Applications (NCSA) developed HDF and makes it freely available. HDF is a data model and extensible data format for self-describing files that was developed independently of netCDF. HDF supports both C and Fortran interfaces, and it has been successfully ported to a wide variety of machine architectures and operating systems. HDF emphasizes a single common format for data, on which many interfaces can be built.
NCSA implemented software that provides a netCDF interface to HDF version 4. With this software, it was possible to use the netCDF calling interface to place data into an HDF4 file.
HDF5 provides a richer data model, with emphasis on efficiency of access, parallel I/O, and support for high-performance computing. The netCDF-4 project is implementing an enhanced netCDF interface on the HDF5 storage layer to preserve the desirable common characteristics of netCDF and HDF5 while taking advantage of their separate strengths: the widespread use and simplicity of netCDF and the generality and performance of HDF5.
Yes, as part of the OPeNDAP framework, developers have implemented a client-server system for access to remote data that supports use of the netCDF interface for clients. The software is available from the OPeNDAP download site. After linking your netCDF application with the OPeNDAP netCDF library, you can use URL's to access data from other sites running an OPeNDAP server. This supports accessing small subsets of large datasets remotely through the netCDF interfaces, without copying the datasets.
The GrADS Data Server provides subsetting and analysis services across the Internet for any GrADS-readable dataset, including suitable netCDF datasets.
Several programs and packages have been developed that convert between GRIB and netCDF data: degrib, CDAT, CDO, and GrADS, among others. The netCDF Java Library (version 2.2 and later) will support reading GRIB data (and some other kinds of data) through a netCDF interface, as if it had been converted to a netCDF dataset.
I have some netcdf files which have data in them and were apparently not properly closed. When I examine them using ncdump they report zero data points, although the size is a few megabytes. Is there a way of recovering them?
Yes, see the archived message for a way to do this.
Yes, the document Known problems with the netCDF Distribution describes reported problems and workarounds in the latest version and some earlier releases.
If you find a bug, send a description to support@unidata.ucar.edu. This is also the address to use for questions or discussions about netCDF that are not appropriate for the entire netcdfgroup mailing list.
A search link is available at the bottom of the netCDF homepage, providing a full-text search of the support questions and answers about netCDF provided by Unidata support staff.
The netCDF distribution comes with interfaces for C, Fortran77, Fortran90, and C++. Other languages for which interfaces are available separately include:
The C-based libraries are not thread-safe. C-based libraries are those that depend on the C library, which currently include all language interfaces except for the Java interface. The Java interface is thread-safe when a few simple rules are followed, such as each thread getting their handle to a file.
It provides all the functionality of the C interface (except for the generalized mapped access of ncvarputg() and ncvargetg()) and is somewhat simpler to use than the C interface. With the C++ interface, no IDs are needed for netCDF components, there is no need to specify types when creating attributes, and less indirection is required for dealing with dimensions. However, the C++ interface is less mature and less-widely used than the C interface, and the documentation for the C++ interface is less extensive, assuming a familiarity with the netCDF data model and the C interface. Recently development of the C++ interface has languished as resources have been redirected to enhancing the Java interface.
It provides all the functionality of the C interface. The Fortran interface uses Fortran conventions for array indices, subscript order, and strings. There is no difference in the on-disk format for data written from the different language interfaces. Data written by a C language program may be read from a Fortran program and vice-versa. The Fortran-90 interface is much smaller than the FORTRAN 77 interface as a result of using optional arguments and overloaded functions wherever possible.
They provide all the functionality of the C interface, using appropriate language conventions. There is no difference in the on-disk format for data written from the different language interfaces. Data written by a C language program may be read from programs that use other language interfaces, and vice-versa.
For clarity, the NetCDF C Interface Guide contains examples which use a function called handle_err() to handle potential errors like this:
status = nc_create("foo.nc", NC_NOCLOBBER, &ncid);
if (status != NC_NOERR) handle_error(status);
Most developers use some sort of macro to invoke netCDF functions and test the status returned in the calling context without a function call, but using such a macro in the User's Guides arguably makes the examples needlessly complex. For example, some really excellent developers define an "ERR" macro and write code like this:
if (nc_create(testfile, NC_CLOBBER, &ncid)) ERR;
where Err is defined in a header file:
/* This macro prints an error message with line number and name of
* test program. */
#define ERR do { \
fflush(stdout); /* Make sure our stdout is synced with stderr. */ \
err++; \
fprintf(stderr, "Sorry! Unexpected result, %s, line: %d\n", \
__FILE__, __LINE__); \
} while (0)
Ultimately, error handling depends on the application which is calling netCDF functions. However we strongly suggest that some form of error checking be used for all netCDF function calls.
NetCDF version 4.0 allows users to use HDF5 as their underlying data format, while continuing to use the familiar netCDF interface. As HDF5 includes data compression, this feature is available to netCDF users.
For more information about netCDF-4, see the netCDF-4 web site.
Another alternative, written and made available by Bill Noon, is a set of modifications to the netCDF 3.3.1 library source that allows transparent access to both compressed and uncompressed netCDF files. A more complete explanation, the source changes, instructions on how to patch the netcdf-3.3.1 source, and some performance numbers are available from the znetcdf site.
In version 4.0 the netCDF API will be extended and implemented on top of the HDF5 data format, for users that have the appropriate HDF5 libraries installed. (Those users without HDF5 will still be able to use netCDF, version 4.0, to create "classic" format netCDF files, as well as 64-bit offset netCDF files.)
NetCDF users with HDF5 will be able to create netCDF-4/HDF5 files with benefits not available with the netCDF format, such as multiple unlimited dimensions. Backward compatibility in accessing old netCDF files will be supported.
The combined library will preserve the desirable common characteristics of netCDF and HDF5 while taking advantage of their separate strengths: the widespread use and simplicity of netCDF and the generality and performance of HDF5.
For more information about netCDF-4, see the netCDF-4 web site.
| Contact Us • Site Map • Search • Terms and Conditions • Privacy Policy • Participation Policy | ||||||
|
||||||
