Copac Beta : new search urls

As the new Copac beta test interface is now storing users’ search history in a database we needed Copac search urls to be stateless (or RESTful.) If you look at the current Copac urls, you will notice as you navigate through a result set, just how much saved state is encoded in the url. There are references to the session ID and the number of your query within your session.

In the new scheme of things, that is all gone and I believe our search urls are now stateless — that is, all the information needed to display a search result is now encoded in the url. The CGI script serving the url does not have to go delving into a database to work out what to do.

I’ll attempt here to explain the new url scheme and hopefully you will see how it can be used as a machine to machine interface to Copac. I should point out though, that this is describing the beta version and things may change in the future.

So, to perform an author query against the Copac database, all you need is a url like this:

http://beta.copac.ac.uk/search?au=sutter

The above url will perform an author search for “sutter” and will display an HTML rendered page showing the first page of brief records. If you would like the results sorted, then you can add a “sort-order” element to the url as follows:

http://beta.copac.ac.uk/search?au=sutter&sort-order=ti

The above url will sort the query by the record title field. If the result set is too large to sort, then you will be redirected back to the same query without the sort-order.

If you want to view the first full record in a result set, then add an “rn” element to the url:

http://beta.copac.ac.uk/search?au=sutter&rn=1

Similarly, to view the second page of brief records:

http://beta.copac.ac.uk/search?au=sutter&page=2

All the above urls return an HTML display — not what you want for machine to machine communication. So, to get some programmer friendly XML you can add the “format” element to the url:

http://beta.copac.ac.uk/search?au=sutter&page=1&format=XML+-+MODS

The above url returns a page of MODS XML records. A page, by default, is 25 records. If you’d prefer more or less records in a page, then you can set the page size by sending a “Page-size” header with the HTTP request. And, so that you know how large the result is, a “Result-set-size” header is returned with the HTTP response when a “format” is specified in the url.

You can, of course, specify a “sort-order” along with a “format”. You’ll be able to discover the various query fields, sort and format options by delving around the user interface and performing a few queries. I’m not going to document them here and now as it is all still beta and they may change before we go live.

Search results as an Atom feed?

Here’s a few questions for you. Would it be useful to be able to get your Copac search results as an Atom feed? If so, would it help in aggegrating Copac searches with results from other services? Would it make writing widgets for, say, iGoogle or Netvibes, easier? Would you like Copac urls to be RESTful (I hope so, as they will be before long.)

Yesterday I was thinking about the different search result formats we provide and I was wondering if Atom might be useful. Then a conversation I’ve had this morning with some colleagues have made me think an Atom format could be very useful in the areas outlined above. However, I don’t have experience of implementing widgets or working with Feeds, so I thought I’d ask here. Any thoughts, anyone?

Search history & a stateless interface

One of the things I’d like to do for Copac is to re-write the code behind the web based user interface. The current architecture was designed to work with a Z39.50 server and I now consider it to be too complex. This makes it hard to debug when things go wrong and the complexity of it means that things do go wrong.

So, I’d like to move the interface over to a REST based stateless interface that talks dircectly to the database without going through our Z39.50 interface. This should decrease the time to produce a response after a user hits the search button and should be more reliable.

What I wasn’t too sure about, until now, was how we would incorporate Copac’s Search History feature into a stateless, REST based, interface. The answer came to me during the small hours this morning. We can put the searches into the same Atom Publishing Protocol (APP) repository that we plan to use for the Marked List. (The Search History and Marked List would be separate collections within the repository and so wouldn’t be mixed up together.)

The advantages of this are: the user can have an Atom feed of their searches, they can tag and annotate their searches and generally manipulate their search history by deleting and editing entries through APP client software. We might also be able to include searches from other services. I think such a search history would work for any REST based service. So if we can move other Mimas services, such as Zetoc and the Archives Hub over to a REST based interface, then a user could potentially have, in one place, an archive of all the searches they have performed over a number of different services.