1. It should return completely massaged responses. Given a URL, it should fetch the information at some DAP URL, look up the URL in a mapping DB, fetch the matching AI and merge the URL's information with the AI. Question: Should the URL be passed to the AIS server (in the Query String?) or should the AIS server make a new set of URLs. This would make the AIS Server's URLs look more 'normal' but might also make it less robust because data sources tend to move around. The AIS Server would need to be updated every time the URLs moved. Updatte: Of course, bundling the data URL in the AIS server URL means that the client has to know where the data are, which will also have issues with robustness.

• This is the AIS as we have currently implementented it. This works only for Attributes at present. One key aspect of this architecture is that the combination of data and metadata takes place in the client. Each client needs the AIS software, which eliminates generic clients like web browsers. However, this also means that data which match the metadata are easily retrieved from the 'origin server' without a side trip through the AIS server (which would be acting like a proxy server in this case).
  

• Here's a design where all of the operations are performed by a server. In this design a URL is sent to the AIS server and the server responds by returning the matching Ancillary information and the client merges them or the AIS server handles the merge by dereferencing the URL and merging the information there. This design will work for web browsers, while the other one won't. However, this design results in ugly URLs or requires a more sophisticated configuration. We have a prototype of this but it has weak error handling. One key feature this architecture would more easily support is applying server-side functions to the data (not metadata) returns. This is not supported by our current software, but is conceivable given what we have. This would provide a way to add map variables to data sources (i.e., to handle the case where a variable is 'metadata' and not 'data').
  

* Figure 1. AIS with a data base of mappings between data source and ancillary data URLS:

* Figure 2. AIS with both parts of the merge in the URL:

1. The AIS server is a significantly more powerful component, but there may be performance issues because data are routed through what is effectively a proxy server.
2. The distributed metadata model may not be reliable enough.

I think the first risk should be investigated right away. I think the second risk should be mitigated by design changes in the AIS. The server should return verbose messages about the operations it's performing and any errors it encounters.

-- JamesGallagher - 02 May 2006

1. We need to be able to map the names used in a data source to one or more other names that groups or standards recognize. This means mapping a name like 'u' to 'meridional wind stress.' Question: Is it good enough to do this using attributes, or do we need variable name aliases?
2. We need to add new variables (and their values). In the best of all worlds, the values would come from a file or a function. This argues strongly for the AIS Server architecture with local ancillary information since such a server can control how the CEs are evaluated.

The client-side AIS is nice because it's so clean from a user's perspective. That is, it's obvious what's being done to the objects returned from the data source. But this architecture has a number of drawbacks. First, it does not scale well when the number of mappings increases because the mapping database is built anew for each virtual Connection. A way around this is to use some sort of persistence, with the most obvious being a RDB (e.g., MySQL). But that makes the clients pretty complex to configure. The mapping DB could be cached on disk using techniques similar to the HTTPCache class' method for managing the HTTP cache. So there are ways around the 'persistence problem' of this design. Another issue is that this code does not work in browsers; it only works in libdap-built clients. We'll have to duplicate it in the Java and C libraries, and so will every other implementer if they want their systems to get the benefit of the AIS (which we hope they will...). That's a pretty high price to pay for this design. However, we have a client-side AIS implemented already.

All is not lost. We can use the client-side AIS software to build an AIS server. I think that's where we should go. That way we have the client-side AIS for testing and development and a server for actual deployment. Since a server will probably be stateless, the persistence problem will still need to be solved, but there are a number of ways we can approach it. It might be that after some thought we make the AIS server a Java Servlet, in which case we can make the DB in-memory and constructed when the servlet is loaded into memory. Or we can take the approach that a servlet should use the 'copy HTTPCache' approach. We can keep going with the C++ code base for now and recode in Java once we have something with the correct mix of features. This is appealing since recoding working software is about as easy as development will ever get and we already have a prototype AIS server thing in the Server3.5 code base. This will let us tackle the design problems without mixing them up with a lot of new implementation.

So that leaves some important questions unanswered. What should an AIS Server actually do!

1. It should return completely massaged responses. Given a URL, it should fetch the information at some DAP URL, look up the URL in a mapping DB, fetch the matching AI and merge the URL's information with the AI. Question: Should the URL be passed to the AIS server (in the Query String?) or should the AIS server make a new set of URLs. This would make the AIS Server's URLs look more 'normal' but might also make it less robust because data sources tend to move around. The AIS Server would need to be updated every time the URLs moved.
2. It should return responses marked to clearly indicate where the information came from.
3. It should return decent error messages, since now there are two potential points of failure (the DAP URL might be dead as might be the AI URL).
4. It should provide a web interface for creating AI resources. This will need to support authentication. This might also make it so that the AI URLs are all local, which would boost reliability, although I'm not sure by how much. It would take the design in the directory of a folksonomy, which means that people can contribute much more easily then if they have to set up their own server. The (human) interface will be very important.

In these diagrams I show both the data and the ancillary information (AI?) as remote from the client and on distinct machines. That might not be the case if we create an AIS that supports a folksonomy where users can 'tag' variables and attributes. In that case we might bundle the result with our new server. Some of the lines in these are missing...

• This is the AIS as we have currently implementented it. This works only for Attributes at present.
  
• Here's a slight modification, the file that holds the mapping between URLs and Ancillary resources is now stored on a different machine from the client. This change is modest but allows users to share mappings:
  
• Here's a design where all of the operations are performed by a server. In this design a URL is sent to the AIS server and the server responds by returning the matching Ancillary information and the client merges them or the AIS server handles the merge by dereferencing the URL and merging the data there. This design will work for web browsers, while the others won't. However, this design results in ugly URLs or requires a more sophisticated configuration. We have a prototype of this but it has weak error handling.
  

The AIS

Attributes and Variables

The current AIS implementation is limited to Attributes. Add support for variables. What does that mean exactly?

Possible Architectures

• Client-side, driven by a local or remote configuration file.
• Server-side, using a local (to the server) configuration file.
• Server-side, bundled with a data server.
• Server-side, as a folksonomy, providing a way for users to 'edit datasets.'

These four architectures are not exclusive of each other. One implementation might include all four ideas.

-- JamesGallagher - 13 Dec 2005