Reading NetCDF from an HTTP Server

NetcdfFile can open remote files served over HTTP, if the HTTP server allows range requests for a resource, indicated by the Accept-Ranges header (optional), and returns a Content-Length header (required). Therefore, the server must not return the data as "chunked" transfer-coding, such as gzip.

Performance will be strongly affected by the file format and the data access pattern. This is because each request for data will incur one or more file access requests. Each file acccess request becomes an HTTP request, incurring the cost of one round-trip message to the server. Generally, the netcdf-3 format (not netcdf-4) is most likely to give good performance, especially when reading large amounts of non-record data per request. Use OPeNDAP for more consistent performance.

The THREDDS Data Server enables range requests by default. You may have to configure other HTTP servers. Following are our notes on how to do so.

Enabling Accept-Ranges response header in Apache HTTP Server

Make sure mod_headers is either compiled into the httpd binary (from the commandline run: $APACHE_HOME/bin/httpd -l & look for mod_headers) or loaded as a DSO module (look in httpd.conf for mod_headers).

Add the following directive to apply to the web content you need (e.g., it can be at the server-level, vhost-level, <Directory></Directory>, and .htaccess): Header set Accept-Ranges bytes

Example:

    <Directory "/opt/local_htdocs">
       Options FollowSymLinks Indexes
       AllowOverride None
       Order allow,deny
       Allow from all
       Header set Accept-Ranges bytes
    </Directory>

Restart apache
(optional): you can test the result by using the Live Http Headers extension in firefox.

For more information see:

http://httpd.apache.org/docs/2.0/mod/mod_headers.html

This document is maintained by John Caron and was last updated on Sep 22, 2008

				Unidata is a member of the UCAR Office of Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation.
				P.O. Box 3000 • Boulder, CO 80307-3000 USA • Tel: 303-497-8643 • Fax: 303-497-8690