Adding a service to a remotely served dataset
John Caron
caron at unidata.ucar.edu
Mon Jul 9 13:59:19 MDT 2007
no, but it could be added (at least for grid subsetting).
netcdf file writing is quite slow right now, and id like to speed it up before making it widely available.
Ethan Davis wrote:
> John,
>
> Is this capability available with the netCDF subset service as well?
>
> Ethan
>
> John Caron wrote:
>> We made an extension of the WCS Dataset URL format to allow the TDS to
>> serve remote datasets. Identify the dataset by adding the parameter
>> dataset whose value is a URL:
>>
>> http://servername:8080/thredds/wcs?dataset=datasetURL&
>>
>> The URL must be a dataset readable by the NetCDF-Java library,
>> typically an OPeNDAP dataset on another server. It must have gridded
>> data, with identifiable coordinate systems, etc. For example, an
>> OPeNDAP URL might be
>>
>> http://las.pfeg.noaa.gov/cgi-bin/nph-dods/data/oceanwatch/nrt/gac/AG14day.nc
>>
>> This can be served remotely as a WCS dataset with this URL:
>>
>> http://servername:8080/thredds/wcs?dataset=http://las.pfeg.noaa.gov/cgi-bin/nph-dods/data/oceanwatch/nrt/gac/AG14day.nc&
>>
>>
>>
>> This is non-standard WCS; we havent emphasized this feature until we
>> have a chance to review security implications. The WCS service now if
>> off by default, and must explicitly be turned on in threddsConfig.xml,
>> mostly because of this feature. We should probably seperately allow
>> users to enable/disable "remote WCS access".
>>
>>
>> Oliver Newell wrote:
>>> Hi Roy -
>>>
>>> FYI - We're currently experimenting with implementing WCS services,
>>> and the same requirement (wish?) as what you're describing has been
>>> raised. Basically, the need to discover a data set via a cataloging
>>> mechanism (THREDDS, an OGC catalog, a DOD NCES catalog) and then be
>>> able to access it via a WCS that resides somewhere on the network. My
>>> initial take on it is that since WCS data access requests work with
>>> unique identifiers (not URLs), this limits the capabilities of the
>>> service in the sense that the Id-to-resource (data file or db)
>>> mapping is managed locally by the WCS. In other words, the WCS has to
>>> 'know about' the resource in advance of the request, in order to do
>>> the mapping.
>>>
>>> I think the WCS service should support a mode of operation where a
>>> resource's URL is included in the requests to the service, in order
>>> to support the external WCS concept. I think it is still important
>>> to maintain the Id as a separate entity, so it is not just a question
>>> of using a URL for the coverage identifier, though that idea might be
>>> tempting.
>>>
>>> The idea needs flushing out a bit, but it is still early enough in
>>> WCS's development that changes like this can be incorporated into the
>>> spec via the OGC working group (my opinion). It definitely seems like
>>> it would be a useful capability to address the situation you describe.
>>>
>>> -Oliver
>>>
>>> Roy Mendelssohn wrote:
>>>> Hi Glenn:
>>>>
>>>> No problem. But we were interested in the general problem, as there
>>>> are other datasets which also do not have WCS and which we would
>>>> like to add it, rather than trying to get each of them to add it
>>>> (Some just have an OPeNDAP server, not a THREDDS server). Yours
>>>> allowed me to give Ethan a concrete example so he could look at the
>>>> catalogs.
>>>>
>>>> Thanks,
>>>>
>>>> -Roy
>>>> On Jul 9, 2007, at 7:52 AM, Glenn.Rutledge wrote:
>>>>
>>>>> Hello Roy-
>>>>> That particular dataset will be turned on shortly via WCS. That
>>>>> was an oversight. Nice to have users telling us where we need
>>>>> stuff! Glenn
>>>>>
>>>>> Ethan Davis wrote the following on 7/6/2007 6:02 PM:
>>>>>> Hi Roy,
>>>>>>
>>>>>> Looks like you are referencing NCDC catalogs rather than serving
>>>>>> the datasets yourself. So, users would get redirected to the NCDC
>>>>>> server and access the data through the NCDC server. So, as things
>>>>>> stand, NCDC would have to add the WCS service.
>>>>>>
>>>>>> You could point to each NCDC dataset with an NcML wrapper and
>>>>>> actually serve the data from your server rather than redirecting
>>>>>> to NCDC. If you did that, you could add the WCS service. However,
>>>>>> the performance wouldn't be as good as direct from NCDC.
>>>>>>
>>>>>> I'm afraid I don't know about the WCS request for a time range.
>>>>>> So, I'll leave that for others to answer.
>>>>>>
>>>>>> Ethan
>>>>>>
>>>>>> Roy Mendelssohn wrote:
>>>>>>> Hi Ethan:
>>>>>>>
>>>>>>> I am not certain I follow. To be more specific, look at:
>>>>>>>
>>>>>>> http://oceanwatch.pfeg.noaa.gov/thredds/catalog.html
>>>>>>>
>>>>>>> and scroll down to the datasets " Remote test to Nomads THREDDS
>>>>>>> catalog", which is actually on a THREDDS server on an NCDC
>>>>>>> computer. They are remotely served through catalogs. Can we add
>>>>>>> the WCS service on our end, or will it only work if NCDC adds the
>>>>>>> WCS service.
>>>>>>>
>>>>>>> While I have your attention, I have not been able to get a
>>>>>>> straight answer if there is a way to send the WCS request to get
>>>>>>> a time range, rather than a single time, and if so what is the
>>>>>>> syntax for that.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Roy
>>>>>>>
>>>>>>> On Jul 6, 2007, at 11:24 AM, Ethan Davis wrote:
>>>>>>>
>>>>>>>> Hi Roy,
>>>>>>>>
>>>>>>>> If you are actually serving the dataset through your server
>>>>>>>> (which it sounds like you are since you mention existing NcML
>>>>>>>> for the dataset) rather than providing a link in your catalog to
>>>>>>>> the remote server, I believe you can simply add the WCS service
>>>>>>>> to your TDS configuration catalog and serve the data via WCS. It
>>>>>>>> shouldn't require any changes to the NcML, just to the service
>>>>>>>> element in the catalog.
>>>>>>>>
>>>>>>>> Of course, this depends on the dataset meeting the WCS server
>>>>>>>> requirements, e.g., evenly spaced grid.
>>>>>>>>
>>>>>>>> I'm not sure how it would perform. But I'll leave it for John to
>>>>>>>> respond on that.
>>>>>>>>
>>>>>>>> Ethan
>>>>>>>>
>>>>>>>> Roy Mendelssohn wrote:
>>>>>>>>> Hi:
>>>>>>>>>
>>>>>>>>> The question for today is if our TDS is "serving" a dataset
>>>>>>>>> remotely (ie. the remote site has their own TDS and we just
>>>>>>>>> link indirectly to it), can we add a service to our version
>>>>>>>>> that they don't have (just like we can aggregate a remote
>>>>>>>>> dataset that is not aggregated). In particular the remote
>>>>>>>>> dataset only has an OPeNDAP response, but we would like to be
>>>>>>>>> able to respond to both an OPeNDAP and a WCS response.
>>>>>>>>>
>>>>>>>>> If we change our NcML appropriately, will that work?
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>> -Roy
>>>>>>>>>
>>>>>>>>> **********************
>>>>>>>>> "The contents of this message do not reflect any position of
>>>>>>>>> the U.S. Government or NOAA."
>>>>>>>>> **********************
>>>>>>>>> Roy Mendelssohn
>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>> NOAA/NMFS
>>>>>>>>> Environmental Research Division Southwest Fisheries Science
>>>>>>>>> Center
>>>>>>>>> 1352 Lighthouse Avenue
>>>>>>>>> Pacific Grove, CA 93950-2097
>>>>>>>>>
>>>>>>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>>>>>>> voice: (831)-648-9029
>>>>>>>>> fax: (831)-648-8440
>>>>>>>>> www: http://www.pfeg.noaa.gov/
>>>>>>>>>
>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ===============================================================================
>>>>>>>>>
>>>>>>>>> To unsubscribe thredds, visit:
>>>>>>>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>>>>>>> ===============================================================================
>>>>>>>>>
>>>>>>>>
>>>>>>>> --Ethan R. Davis Telephone: (303)
>>>>>>>> 497-8155
>>>>>>>> Software Engineer Fax: (303)
>>>>>>>> 497-8690
>>>>>>>> UCAR Unidata Program Center E-mail:
>>>>>>>> edavis at ucar.edu
>>>>>>>> P.O. Box 3000
>>>>>>>> Boulder, CO 80307-3000
>>>>>>>> http://www.unidata.ucar.edu/
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> **********************
>>>>>>> "The contents of this message do not reflect any position of the
>>>>>>> U.S. Government or NOAA."
>>>>>>> **********************
>>>>>>> Roy Mendelssohn
>>>>>>> Supervisory Operations Research Analyst
>>>>>>> NOAA/NMFS
>>>>>>> Environmental Research Division Southwest Fisheries Science Center
>>>>>>> 1352 Lighthouse Avenue
>>>>>>> Pacific Grove, CA 93950-2097
>>>>>>>
>>>>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>>>>> voice: (831)-648-9029
>>>>>>> fax: (831)-648-8440
>>>>>>> www: http://www.pfeg.noaa.gov/
>>>>>>>
>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --Glenn K. Rutledge
>>>>> Services Team Leader
>>>>> Remote Sensing and Applications Division
>>>>> NOMADS Project Manager
>>>>> National Oceanic and Atmospheric Administration
>>>>> National Climatic Data Center
>>>>> Asheville NC 28801
>>>>> Phone: (828) 271-4097
>>>>> Fax: (828) 271-4328
>>>>>
>>>>> NOMADS: http://nomads.ncdc.noaa.gov/
>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the
>>>> U.S. Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division Southwest Fisheries Science Center
>>>> 1352 Lighthouse Avenue
>>>> Pacific Grove, CA 93950-2097
>>>>
>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>> voice: (831)-648-9029
>>>> fax: (831)-648-8440
>>>> www: http://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>>
>>>>
>>>>
>>>> ===============================================================================
>>>>
>>>> To unsubscribe thredds, visit:
>>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>> ===============================================================================
>>>>
>>>
>>> ===============================================================================
>>>
>>> To unsubscribe thredds, visit:
>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>> ===============================================================================
>>>
>>
>> ===============================================================================
>>
>> To unsubscribe thredds, visit:
>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>> ===============================================================================
>>
>
==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
More information about the Thredds
mailing list