Author Archives: neil.mayo

Searching: Results Ranking

Posted on by

The new SUNCAT interface is available at http://suncat.ac.uk/ – this became the main SUNCAT interface in March 2014 but the old interface is still available. This new interface is built on a different platform and will therefore exhibit some differences in behaviour. We have discussed some of these in previous technical blog posts, and updated help and support documentation will also clarify the changes. In this post we will give some attention to how search results are ranked according to their relevance to the search terms.

Relevancy

One of the features of the Solr search server which we use to query the data, is that when we perform a search, the results that we get back include a relevancy score, or rank.

“Relevancy is the quality of results returned from a query, encompassing both what documents are found, and their relative ranking (the order that they are returned to the user).”

The scores are normalised to fall on a scale between 0 and 1, but you don’t need to worry about the scores as we don’t show them to you – we just use them to inform the resulting ordering that we show you. You can read more about relevancy scoring at http://wiki.apache.org/solr/SolrRelevancyFAQ.

By default the returned results are listed in order of relevance, with the most relevant first. This is what is reflected in the position column. Note that while the other sortable columns can be ordered in ascending or descending order, we do not allow the position column to be ordered in ascending order (i.e. from least relevant to most relevant). If you click on the position column header, the results will be ordered in descending order of relevancy.

Results table header

Clicking on the position column orders results by relevance. They may only be ordered in descending order.

We have defined relevancy so that things like punctuation and capitalisation don’t affect a result’s score.

Boosting

Boosting allows us to modify scores; so we give matches on a particular search index (field) more weight than others.

We can boost the importance to the search of a particular search field, or of particular documents when we put them into Solr, or of a particular clause within a query used to search the data. SUNCAT currently performs a variety of boosting:

  • Boost a result (significantly) if the search term matches exactly.
  • Boost a result where the search terms occur close together (within 3 words of each other).
  • When searches are made on the Title Keywords field, results are boosted if the search terms occur in the 245 MARC field (Title Statement), particularly any of the sub-fields $a (Title), $b (Remainder of title), $n (Number of part/section of a work) or $p (Name of part/section of a work).

So for example, searching for “Journal hellenic studies” in the Title Keywords field would produce results including Journal of Hellenic Studies as expected, and also Archaeological reports (which has “Journal of Hellenic Studies” in the Added Title field). However the former would appear higher up in the results because the search terms occur in the main title in the 245 field.

Examples

Here are some sample searches for British trees; first, a search for records with any of the words “British” and “trees”.

Results for "British trees" (any)

Search results for any of the words “British trees”. There are over 36,000.

There are more than 36,000 results. From around the 300th result and towards the end you will see many results which have been returned because they contain the word “British”, and less related to trees. This is possibly not what you were interested in, but much like using a search engine, you can ignore the results at the end, because the most relevant ones are shown to you first. You aren’t forced to make a more accurate search, though you can if necessary.

If we search for records with all (both!) of the words “British” and “trees”, we will get fewer results:

Results for "British trees" (all)

Search results for all of the words “British trees”.

There are only two results that include both words. The Basic search feature uses this interpretation by default, searching for all the specified terms.

You could search for the quoted phrase “British trees” but this produces nothing as the exact phrase does not occur anywhere:

Results for "British trees" (quoted phrase)

There are no search results for the exact phrase “British trees”.

Another aspect which affects the scoring of results is whether a search term has been stemmed. For example, when you enter the word “British”, it will be stemmed so that Solr will look for variations on it, such as “Brit” and “Britain”. Matches on the variations will have less influence over the score than precise matches to “British”.

Conclusion

It can be hard to unravel exactly what causes a particular record to get a higher score than another, because of the variety of factors and weightings that go into its calculation. The relevancy can be affected by the exactness of word matches, by their frequency, by how similar the words in the record are to words in the search term, how close together they are, what fields they appear in, and a variety of other factors it is possible to bring to bear on the scoring algorithm.

In deciding what aspects of the results should be considered most important, it is necessary to make trade-offs. The challenge is to make the results as intuitively sensible as possible, but it is not always possible to infer and reflect the exact intentions of the user – and sometimes particular combinations of boosting and searching on particular fields may give apparently counter-intuitive positioning to some results. Search algorithms are inherently heuristic and are an attempt to provide meaningful results to a simple query. In general, the more accurate and complete the underlying MARC records, the better the resulting scoring will be, much like trying to raise a website’s profile in a search engine.

The Advanced search feature provides more options, and more control over how search terms are interpreted, so that you can really pin down what you are searching for – but the basic search should in most cases provide a quick and effective doorway to the wealth of information in SUNCAT!

Mapping metadata to a common schema

Posted on by

One of the challenges in aggregating metadata is to unify the different schemata of the data sources, to make the data searchable through a single common schema. There are advantages to maintaining the original schema (field names) of each data source so that it can be searched in the same way that would have been possible in its original context (i.e. searching under the same field or fields). However this can complicate searches of the aggregated metadata when each data source represents similar information under a differently-named field – because a search for values on a range of essentially synonymous but differently-named fields across a variety of resources would be more complex to specify. Therefore it makes sense to map similar fields onto a single field, within a schema which is specific to the collection of aggregated data.

This post is about creating a schema for Will’s World and mapping other metadata into it. Despite the technical detail, it might elaborate some common stumbling blocks to be found in mapping disparate information to a single representation.

Developing a Schema

We had to develop a schema for Will’s World (WW) to represent the metadata which is being imported into the text database. This schema has several requirements:

  • Should be capable of representing/unifying metadata from several different sources.
  • Must unify some of the variations on similar pieces of data (for example there is a range of fields dealing with attribution and ownership – including rights/availability/copyright; owner/credit/creator/contributor).
  • Should use existing ontologies as far as is appropriate/possible (for example, Dublin Core, TEI)

Once the schema is designed, it is necessary to define how the metadata we collect from disparate sources will fit into the (probably more restrictive) schema that we have elected to use in our representations. This involves establishing a clear mapping from the source schemata to the target schema. Additionally, we need more than one schema for Will’s World:

  1. A schema for metadata harvested from services, which represents content within those services.
  2. A schema for descriptions of services themselves (including those from which we do not harvest metadata).
  3. As a bonus, we might wish to define one or more schemata for texts relating to Shakespeare.

This post concentrates on the metadata schema, which is the most important and elaborate. We have defined a list of fields with namespaced identifiers, and for each field we define several properties:

  • Internal – whether a field is for internal use only, and is invisible to the user (because, for example, it is meaningless outside of the application).
  • Repeatable – whether a field can appear multiple times. For example, properties like “genre”, “subject” or “author” may be multi-valued (or rather have multiple instances), while we expect a metadata item to have a unique and canonical title (although we accept zero or more alternative titles).
  • Required – whether a field is required (because it acts as an identifier of some sort, or is essential to the meaning/categorisation of the record).

For the purposes of supporting text search, we need to answer a couple of questions for each field:

  1. Should the field be Indexed (searchable through Solr queries)
  2. Should the field be Stored (available to return from Solr for display in the interface)

Schemata

For reference while reading this blog post, you can refer to both the metadata schema and metadata mappings we have developed.

The schema for representing metadata about services is much simpler and records identifying metadata such as id, name, description, URL, provenance and usage/licensing information, and more specific metadata about how to use the service.

Collecting Data

The next stage is to collect, transform and ingest the data into Solr. This is a multi-stage process:

  1. Collect the data – either in a well-structured format like RDF/JSON/XML/CSV, or contained in HTML.
  2. Extract the data (if contained in an unstructured or variable format like HTML).
  3. Transform the data from the source format into a format matching the WW schema.
  4. Transform the WW metadata into a format matching the Solr import document schema.
  5. Ingest the data into Solr, by posting it to the server.

Due to size and memory restrictions, any of the later stages, particularly the transformations and ingest, may involve splitting the documents into multiple (well-formed) documents.

We have to take a variety of approaches to collecting the data, depending on what types are available. Only a relatively small proportion of providers actually make their data available in easy to collect structured formats, while in other cases the data is present directly in HTML pages, or as the result of searches through an HTML web interface.

  1. Scraping data – may involve two stages; in the case that the metadata itself is available directly from known URLs, the first and second steps are combined:
    1. Scrape information from webpages (for example, the results of conducting a search through a search interface)
    2. Extract links from that information and follow them to find the metadata.
  2. Collecting data directly in a structured format.

Depending on the source of the data, it may be processed via scripts which construct structured data from un- or less structured data through a process of parsing and recombining data, or by an XSLT script which performs a simpler and more direct mapping of existing fields to the fields of the schema. This latter may involve many-to-one mappings in addition to direct one-to-one mappings. One-to-many mappings should however be avoided – this would produce data duplication and likely be an indication that the schema was not well defined.

Issues in Schema Design

In designing a metadata schema which is intended to unify metadata from sources with different concerns, we inevitably encounter certain issues which demand to be resolved in a consistent way.

Identifying fields

Naturally we need an identifying field so that every record can be distinguished and uniquely identified. We have two options for this – generate unique id fields (the easiest way would be to ask Solr to impose them), or map external values (from each source’s data) into an id field. The first option requires us to maintain unique identifiers upon updates, when we re-index data. The second option runs the risk of clashes. Identifying tuples or values sourced externally will vary a great deal and there is always the danger of non-uniqueness unless we are very careful about how we define them. It might be best to let Solr create a unique field value for indexing, and record an alternative ‘primary’ identifier from the identifying metadata of each particular data source. This might be considered a unique identifier within the records of the particular source.

Required fields

If the anticipated metadata informing a required field are not present in the source data, a default or “unknown” value must be given. This can be achieved by explicitly producing a default value from a script, or by providing a template in an XSLT transform for each required field, which matches when the source field is not present in a record. Here is an example enforcing the presence of a required dc:title field for a Solr import doc:

<xsl:if test="not(dc:title)">
  <field name="dc:title">Unknown</field>
</xsl:if>

Field values with subelements

What should we do when the source data that we want to insert as a field value incorporates tag structure of its own? We would like to maintain XML substructure where available as part of the value of the field, rather than discarding it. However the Solr import schema does not accept substructure to its “field” elements, so we must enclose them in CDATA tags. This is not easy using XSLT – unless we are willing to write templates to match each of the elements we want to use as subelements in the defined fields of the schema, and encode them appropriately. The solution is to pass the node which contains substructure to a target which wraps it in a CDATA section in the output like this:

<!-- Copy the current element and subelements to the output in a CDATA section -->
 <xsl:template name="xmlsection">
  <!-- Use the current node by default -->
  <xsl:param name="node" select="."/>
  <xsl:text disable-output-escaping="yes"><![CDATA[ <![CDATA[ ]]></xsl:text>
  <xsl:copy-of select="$node"/>
  <xsl:text disable-output-escaping="yes"><![CDATA[]]]]><![CDATA[>]]></xsl:text>
</xsl:template>

Aggregating multiple XML fields

As indicated earlier, there are several cases where we wish to map many source fields from a provider into one Will’s World field as subelements. An example of matching multiple fields at the same level which will contribute to the content of a single field is in National Library of Scotland data, where we want to match and combine the XML elements “summary”, “commentarytitle” and “commentary” into a single “dc:description” field. There are a few options for performing this aggregation; if we know that one of the source fields (say, “summary”) will always be available, we can match on it and then pull in values from the other fields as necessary using xsl:copy-of with a relative path in the select attribute. However, if the “summary” field is missing, the template will not match and we will lose the values of the other fields.

If we cannot be sure that any one of these fields will be available, that is, they are all optional, then we need to match every possible combination. Doing this via a number of similar templates is unnecessarily verbose and will also lead to duplicate output fields, as every one of the fields which is present will match and cause the output field to be written. The solution is to provide an OR-ed list of the possible combinations of these elements, so it will only match once and produce one instance of the output field. In the following simplified example, we want to add elements f1 and f2, which occur at the same level, and either of which may or may not be present, to a single output element – and only do it once. We therefore need to check for the presence of both elements, or of either element without the other:

<!-- Match f1 and/or f2 -->
<xsl:template match="f1[../f2]|f1[not(../f2)]|f2[not(../f1)]">
  <combined-field>
    <xsl:copy-of select="../f1" />
    <xsl:copy-of select="../f2" />
  </combined-field>
</xsl:template>

Note that the XPath expression will become increasingly complicated if we are combining more than two potentially absent elements. In such a situation, it would be desirable to find other ways to do this, to avoid a combinatorial explosion of match expressions.

Issues in Mapping Metadata

Scraping data

We scrape data on the granularity of a page. Sometimes this yields metadata at different levels. For example, the BBC’s “Shakespeare’s Restless World” is a radio programme produced in conjunction with the British Museum, exploring an event or period in history through a particular object from the museum’s collections. The pages describing each show tend to contain metadata about the show, and also metadata about the object around which the show is centred. These can either be mapped to a description of a single entity representing “the programme about the object”, or the metadata can be recorded independently for both the programme and the object.

How to record the various metadata at different levels?

In this case we conceptualise the object as the resource being described; metadata about the programme (title, description, URL to a podcast) will be recorded in a record describing the programme, while supplementary detail (background/object facts, quotes) will be incorporated into the text description (using appropriate tags) and also distributed where possible into a record that describes the object. A page listing objects by play is scraped separately in order to supply associated plays to each programme/object.

Summary

Aggregating a wide range of metadata by mapping its values to the fields of a single common target schema renders it searchable in a unified interface. The mapping effort does however throw up a number of issues, some of which have been described in this post along with potential solutions. In particular:

  • It is necessary to be clear on what one is describing with a particular schema – a record, or a service providing a description of a record.
  • Collection and transformation must be performed carefully, with sensitivity to the requirements of the schema; we cannot rely on receiving or extracting data of the correct type or structure.
  • We should try to maintain as much as we can of the structure of the source metadata, so as to minimise the loss of information inherent in shoehorning data into a different representation.
T-Rex and Utahraptor point out that “Shakespeare needs tons of notes to be readable today” – sometimes information and cultural artefacts need to be mapped into a single common language and supported by supplementary metadata. (Is that a weak segue into a Dinosaur Comics link?)

Creating an XML Schema from a Table Description

I found that developing the schema was most easily done in a spreadsheet or table format, so I wrote scripts to transform that specification (at least partially) into (a) an XML schema and (b) an HTML layout of the same. To produce a schema, we produce elements with minOccurs and maxOccurs attributes. If the field is required, minOccurs is 1, otherwise it is 0. If the field is repeatable, maxOccurs is “unbounded”, otherwise it is 1. Here is the script for generating a schema file from a CSV description of the WW fields.

Disclaimer: this script is quick and dirty, non-validating, and WW-specific. Use it at your own risk; or better, adapt it!

Share

SPARQL – What’s up with that?

Posted on by

The title of this post is intended to convey a touch of bewilderment through use of a phrase from the Cliff Clavin school of observational comedy.

Linked data and SPARQL

In the linked data world, SPARQL (SPARQL Protocol and RDF Query Language) is touted as the preferred method for querying structured RDF data. In recent years several high profile institutions have worked very hard to structure and transform their data into appropriate formats for linked data discovery and sharing, and as part of this, many have produced RDF triple (or quadruple) stores accessible via SPARQL endpoints – usually a web interface where anyone can type and run a SPARQL query in order to retrieve some of that rich linked data goodness.

This is admirable, but I have to admit to having had little success getting something out of SPARQL endpoints that I would consider useful. Every time I try to use a SPARQL facility I find I do better by scraping data from search results in the main interface. I have also increasingly become aware that I am not the only one to find it difficult.

RDF stores are different to relational databases; they are not so amenable to performing a search over the values on a particular field. Nor are they as flexible as text search databases like Solr. Instead they record facts relating entities to other entities. So it is important that as consumers of the data we know what kind of questions make sense and how to ask them in a way that yields useful results for us and does not strain the SPARQL endpoint unduly. If these are not the kind of questions we want to ask then we might need to question the application of SPARQL as the de facto way of accessing RDF triple stores.

I’d like to point out that my aim here is not to complain or to disparage SPARQL in general or anybody’s data in particular; I think it is fantastic so many institutions with large archives are making efforts to open up their data in ways that are considered best practice for the web, and for good reasons. However if SPARQL endpoints turn out to be flawed or inadequately realised, they will not get used and both the opportunity to use the data, and the work to produce it, will be wasted.

Problems with SPARQL endpoints

These are the problems I have commonly experienced:

  • No documentation of available vocabularies.
  • No example queries.
  • No access to unique identifiers so we can search for something specific.
  • Slowness and timeouts due to writing inefficient queries (usually without using unique ids or URIs).
  • Limits on the number of records which can be returned (due to performance limits).

Paraphrasing Juliette Culver’s list of SPARQL Stumbling Blocks on the Pelagios blog, here are some of the problems she experienced:

  • No query examples for the given endpoint.
  • No summary of the data or the ontologies used to represent it.
  • Limited results or query timeouts.
  • SPARQL endpoints are not optimised for full-text searching or keyword search.
  • No link from records in the main interface to the RDF/JSON for the record. (This is mentioned in relation to the British Museum, who provide a very useful search interface to their collection, but don’t appear to link it to the structured data formats available through their SPARQL endpoint.)

Clearly we have experienced similar issues. Note that some of these are due to the nature of RDF and SPARQL, and require a reconception of how to find information. Others are instances of unhelpful presentation; SPARQL endpoints are generally pretty opaque, but this can be alleviated by providing more documentation. With the amount of work it takes to prepare the data, I am surprised by how few providers accompany their endpoints with a clear list of the ontologies they use to represent their data, and at least a handful of example queries. This takes a few minutes but is invaluable to anybody attempting to use the service.

Nature provide the best example I have seen of a SPARQL endpoint, providing a rich set of meaningful example queries. Note also the use of AJAX for (minimal) feedback while query is running, and to keep the query on the results page.

Confusion about Linked Data

A blog post by Andrew Beeken of the JISC CLOCK project reports dissatisfaction with SPARQL endpoints and linked data, and provoked responses from other users of linked data:

“What is simple in SQL is complex in SPARQL (or at least what I wanted to do was) … You see an announcement about Linked Data and don’t know whether to expect a SPARQL endpoint, or lots of embedded RDF.” Chris Keene

“SPARQL seems most useful for our use context as a tool to describe an entity rather than as a means of discovery.” Ed Chamberlain

Chris’ point gives another perspective on linked data in general – what does it mean to provide linked (or should that be linkable) data, and how do we use it? Embedded RDF (RDFa) is good in that it tends to provide structured data in context, enriching a term in a webpage in a way that is invisible by default but that people can consume if they choose to. Ed indicates a fact about RDF as a data storage medium: it is a method of representing facts about entities which are named in an esoteric way; it is not structured in a way that is ideal for the freer keyword searching or SQL-style queries that we are used to.

Owen Wilson suggests the Linked Data Book‘s section 6.3 which describes approaches to consuming linked data, describing three architectural patterns. It looks worth a read to get one thinking about linked data in the right way.

Unique identifiers

“My native English, now I must forego” Richard II, Act 1, Scene 3

One of the tenets of linked data is that each object has a unique identifier. If we are looking for “William Shakespeare” we must use the URI or other identifier that represents him in the given scheme, rather than using the string “William Shakespeare”. It is thus also necessary that we have an easy way to access the unique identifiers that are used in the data, so that we can ask questions about a specific entity without forming a fuzzy, complex and resource-consuming query. The British Museum publicises its controlled terms, that is the limited vocabulary that they use in describing their collection, along with authority files which provide the canonical versions of terms and names, standardised in terms of spelling, capitalisation and so on, and thesauri which map synonymous terms to their canonical equivalents. These terms are used in describing object types, place names and so on, supporting consistency in the collections data. They are all available via the page British Museum controlled terms and the BM object names thesaurus. Armed with knowledge of what words are used in particular fields to categorise or describe entities in the data, and similarly with a list of ids or canonical names for things, we can then start to form structured queries that will yield results.

Shakespeare and British Museum

I have looked in particular at the British Museum’s SPARQL endpoint as an example, as BM is a project partner and because it has several items germane to Will’s World. To start with, the endpoint gives some context; a basic template query is included in the search box, which can be run immediately and which implicitly documents all the relevant ontologies by pulling them in to define namespaces. There is a Help link with some idea of how data is represented and can be accessed/referenced using URIs. All of this is good and I found it easy to get started with the endpoint.

However before long I came up against the problem I’ve had with other endpoints, namely that it is difficult to perform a keyword search, or at least perform a multi-stage search in order to (a) resolve a unique identifier for a keyword or named thing and then (b) retrieve information about or related to that thing. In this case I found a way to achieve what I needed by supplementing my usage of the SPARQL endpoint with keyword searches of the excellent Collection database search – and with some help from the technical staff at BM to resolve a couple of mistakes in my queries, can now harvest metadata about objects related to the person “William Shakespeare”.

It is reassuring to find out I am not alone in having difficulty retrieving and using SPARQL data. I followed Owen Stephen’s blog post about the British Museum’s endpoint with interest. Owen found the CIDOC CRM data model hard to query due to its (rich, but thereby counter-intuitive) multi-level structure. Additionally, he encountered the common issue that it is very difficult to perform a search for data containing or “related to” a particular entity which to start with is represented merely by a string literal such as “William Shakespeare”:

The difficulty of exploring the British Museum data from a simple textual string became a real frustration as I explored the data – it made me realise that while the Linked Data/RDF concept of using URIs and not literals is something I understand and agree with, as people all we know is textual strings that describe things, so to make the data more immediately usable, supporting textual searches (e.g. via a solr index over the literals in the data) might be a good idea.

Admittedly RDF representations and SPARQL are not really intended to provide a “search interface” in the sense to which most users are accustomed. But from the user’s perspective, there must be an easy way to start identifying objects about which we want to ask questions, and this  tends to start with performing some kind of keyword search. It is then necessary to identify the ids representing the resulting objects or records which are of interest. With the BM data this involves mapping a database id, which can be retrieved from the object URL, to the internal id used in the collections.

So what are the right questions?

Structured data requires a structured query – fair enough. However what sort of useful or meaningful query can we formulate when the data, the schema used to represent the data, the identifiers used within the data, are all specified internally? In order to construct an access point into the data, it is helpful to have not just a common language, but a common (or at least public) identifier scheme; canonical ways of referencing the entities in the data, such as “Shakespeare” or “the Rosetta Stone”. Without knowing the appropriate URI or the exact textual form (is it “Rosetta Stone”, “The Rosetta Stone”, “the Rosetta stone”? would we get more results for “Rosetta”?) it is nigh on impossible to easily ask questions about the entity of a SPARQL endpoint.

So how is one supposed to use a SPARQL endpoint? It is not a good medium for asking general questions or performing wide-ranging searches of the data. Instead it seems like a good way to link up records from different informational silos (BM, BL, NLS, RSC…) that share a common identifier scheme. If we know the canonical name of a work (“Macbeth”) or the ISBN of a particular edition, then we can start to link up these disparate sources of data on such entities.

But the variety of translations, the plurality of editions (which will only increase) and other degrees of freedom make it hard to perform an exhaustive analysis/usage of the data. In the case of the BM, who might have unique objects we don’t know we want to see, the way to find them is through keyword search. It seems that only by going first through a search interface or other secondary resource can we identify the items we want to know about and how to refer to them.

What we have in common between different sources is the language or ontologies used to describe the schema (foaf, dc, etc) – but this is syntax rather than semantics; structure rather than content. To echo Ed Chamberlain’s comment, we have access to how data is described, but not so much to the data itself.

 

British Museum data

The approach we will use to harvest British Museum metadata related to Shakespeare is outlined below. It is essentially the same approach that Owen Stephens found workable in his post on SPARQL, and involves reference to a secondary authority (the BM collection search interface) to establish identifiers.

  1. Conduct a search for “Shakespeare” in the collections interface.
  2. Extract an object id from each result. The Rosetta Stone has the id 117631.
  3. Find the corresponding collection id from SPARQL with this query: 
    SELECT * WHERE { 
   ?s <http://www.w3.org/2002/07/owl#sameAs> 
      <http://collection.britishmuseum.org/id/codex/117631> 
}
  4. The result should be a link to metadata describing the object, and the object’s collection id (in this case YCA62958) can be extracted for use in further searches. 
    http://collection.britishmuseum.org/id/object/YCA62958
  5. If there is a result, retrieve metadata about the object from the URL: http://collection.britishmuseum.org/description/object/YCA62958.rdf (or .html, .json, .xml)
  6. If there is no result, scrape metadata from the object’s description page in the collections interface. There is plenty of metadata available, but it is far less structured than RDF, being distributed through HTML.

This last step looks like it will be quite common as many of the Shakespeare-related results are portraits or book frontispieces which have no collection id. I am not sure whether this is an omission, or because they are part of another object, in which case it will require further querying to resolve the source object (if that is what we want to describe).

Another difficulty is that although Owen found a person-institution URI for Mozart, I cannot find one for Shakespeare. There is a rudimentary biography but little else, so we do not have a “Shakespeare” identifier for use in SPARQL searches.

Conclusion

Ultimately I am still finding it non-trivial and a bit hacky to identify, and ask questions about, the real Shakespeare through a SPARQL endpoint.

Click here to view the embedded video.

In summary:

  • SPARQL endpoint providers could provide more documentation and examples.
  • RDF stores allow us to ask structural questions, but semantic questions are much harder without knowing some URIs.
  • It is often necessary to make use of a secondary resource or authority in order to identify the entities we wish to ask about.

Share

“I shall lose my life for want of language”: Shakespeare and digital formats

Posted on by

The Will’s World registry is intended to record three categories of data:

  1. Metadata about services
  2. Metadata harvested or contributed from those services
  3. Annotated plays

The first will describe the services that are available, and will support searching for services that have specific features, and that produce particular types of data about aspects of Shakespeare’s work, history and contemporaries. These data will be contributed by a service during registration; and in the initial stage of the project will be added manually.

The second category will be metadata that we have retrieved from services through queries or scraping from HTML or search interfaces, and will be searchable as an aggregated metadata resource.

The third type of data is different because it involves storing target data rather than just metadata – there will be the actual structure and content of the literary resource. Several questions arise: How will the data be searched, retrieved and visualised? Will it be possible to attach metadata search results to it? Which language will be used to markup the text?

Why store data as well?

Will’s World should be a useful access point for a variety of Shakespeare resources. It should be possible to run a search and retrieve metadata on items from a range of services. What will a user then do with this metadata? The answer to this question is something we don’t know and do not intend to dictate, but one likely use is to link the metadata to the text of the plays. A marked up electronic text can be seen as a backbone which a user might develop with multimedia resources from a number of services, to produce anything from an enriched script for theatre or teaching use, to an online application linking plays to performances, to a mobile phone app. Some examples of what developers and creative types can devise with little more than basic marked-up text were produced at the recent hackday.

What about formats and schemas?

There are a handful of marked up versions of Shakespeare’s plays:

  1. Jon Bosak provides a full set of annotated Shakespeare as an XML test package, which is potentially a great starting point with a basic DTD.
  2. Shakespeare play schema by Susan Kelsch. This appears to be quite comprehensive on representing aspects of Shakespeare plays (including the major play groupings), to the extent that it is not generic enough for any other application. I am not aware of the DTD being put into use in any data.
  3. Open Source Shakespeare (OSS) provides comma-delimited, tilde-demarcated database fields representing the Globe Shakespeare. We converted this into XML using a simple Play-Act-Scene-Paragraph-Line hierarchical schema which provides a simple to understand version of the play structure. The CharID element within Paragraphs indicates the speaker of speech.
  4. Perseus has plays from the Globe Shakespeare, encoded using TEI and available under Creative Commons 3.0.

Jon Bosak’s XML is useful to have, but as he says in his caveat “should not be relied upon for scholarly purposes” and is intended “purely as a learning exercise … a benchmark … and as a resource for testing”. The other schemas are somewhat ad hoc and very Shakespeare specific. It is good that they are tailored closely to their purpose, but it also makes them non-transferrable. Plays of different authors are marked up using subtly or markedly different schemes and become mutually non-comparable, or at least not without difficulty. At the same time it makes it unlikely that such schemas will find a wider audience unless they really describe Shakespeare better than any other option out there.

The XML that we generated from the OSS texts retains typographical elements that should be encoded into TEI, such as square brackets indicating that a line is a stage direction. It fulfilled the requirements of the Culture Hack event and provided a usable baseline of marked up text for the participants, but by its nature it has limited usage unless people find it more generally useful.

The question: is it better to use a schema developed specifically for Shakespeare’s plays (or plays in general), or something more general like TEI?

Literature Markup Languages (LML and XLDL)

Literature Markup Language (along with its predecessor and foundation, Literature Description Language) are attempts to provide an XML markup specifically for literature. It is designed to cover literature at all levels – pamphlets, prose, poetry, plays, criticism. These MLs look promising because they are pitched at a level wider than the works of a specific author, but within a particular field of writing that exhibits a range of common properties – things like figurative language, a variety of structures (some very well defined, such as haiku; others looser) and a range of utterance.

It is encouraging that there are efforts to define the appropriate terminology and structural elements of literature. The LML definition provides for act, scene and genre elements, and the ability to specify the tone of utterances. It is intended to use RDFa to enhance the semantics available in XHTML elements. It should be of much wider application than Shakespeare, though it is not clear whether it is capable of representing complex non-hierarchical structures.

It looks like the creator of LML, Dr Olaf Hoffmann, has thought carefully about the diverse requirements for representing literary works. However it seems an omission that there is no mention of the Text Encoding Initiative (TEI), whose long-lived and widely-used standard makes several recommendations for its application to dramatic and literary texts. I must also admit that I don’t understand why the rationale for LML’s creation is so closely linked with the vector graphics format SVG – I’m not sure of Dr Hoffmann’s use case for marking up text in SVG files.

These schemas have apparently shown little development in the past few years and I am not aware of any projects that have applied the languages.

Is LML/XLDL more appropriate than TEI?

LML contains hard definitions of what it considers the appropriate hierarchical levels of plays to be – namely act, scene, speech and line. While these are more or less universal in drama and are certainly appropriate for Shakespeare’s plays, they are not essential. For example, the act is a typically western construct, and modern literature regularly plays with the structural elements of literature. Thus defining the levels as elements is potentially limiting the application of LML.

TEI does not define scene and act elements, but rather keeps the elements abstract and specifies their role through a type attribute. This allows for more possibilities but can render the markup somewhat verbose and less readable, and subsequent querying more convoluted. TEI is sprawling; it attempts to cover so many possibilities that its flexibility tends to render it too complex and too vague for many applications, and the ways it is applied so disparate that it can still be hard to compare marked up texts like-for-like.

Although TEI has its limitations, it does have provenance and a large and active community.

Text Encoding Initiative (TEI)

Electronic Textual Editing: Drama Case Study: The Cambridge Edition of the Works of Ben Jonson by David Gants provides a good discussion of some of the challenges involved in efforts to markup dramatic texts, and how TEI attempts to handle them. See in particular the section on Encoding Drama. Quotes in this section are taken from this article.

TEI provides a set of guidelines for encoding texts at a semantic level which can be turned into contextually-appropriate representation when necessary. Within those guidelines it describes “encoding strategies designed specifically for drama”. This allows more details to be encoded than the baseline of straight act/scene/line markup, for example the shifting perception of a character. Attributes of a speaker can be modified through the course of the play, characters can share lines. Even Ben Jonson’s plays provide a wide range of complex structures which can be difficult to properly represent. Typographical schemes were devised in the sixteenth century for representing these features on the page, but they can be difficult to transform into a structural representation in XML, which is essentially hierarchical.

The broad hierarchical features common to many plays are pretty straightforward to represent – acts, scenes and lines; speeches and speakers; individuals and chorus. However sometimes these features can overlap or otherwise become complicated. A handful of real-life textual features that TEI aims to support are:

  • The difference between the character/actor speaking and who is perceived to be speaking. For example “plays that deal with English historical subjects will often alter the speech prefix assigned a character as the title and status of that character changes, such as Bolingbroke/Henry IV or Gloucester/Richard III.”
  • Simultaneity and interweaving of utterances. The case study describes markup allowing the reconstruction of a letter (a fictional entity in the imagined world of the play, or a prop on stage) whose contents are read out, distributed over several character utterances with speeches overlapping. With appropriate markup it is then possible to extract objects/structures from the text.
  • Rhyme scheme <rhyme> and meter <met>.
  • Stage directions <stage> and character movement <move>.
  • Poetic sub-structures (stanzas, line groups, verse paragraphs).
  • The rend attribute describes how to render the text of an element, for inline encoding of typographical features.

Although a minimal set of tags tends to get used from any given TEI schema, it can be used to enrich the text with all sorts of supplementary metadata, to provide further semantic and analytical richness to the text (short of providing critical commentary), thus supporting a variety of literary analyses. This is not something we aim to represent, or have the knowledge or resources to produce, but it is one enrichment activity that someone might like to perform given the basic markup.

I’m not convinced about the use of number-suffixed div elements <div0>, <div1> which results in an arbitrary number of new element names. It seems more in the spirit of XML to put these indices in an attribute.

Editions of Shakespeare

There are several editions of Shakespeare, including the following well-known publisher editions:

  • The Arden Shakespeare is described as “the world’s most recognised scholarly edition”. It is commercial and comes at a cost.
  • The Riverside Shakespeare is a long-running series started at the tail end of the 19th century, and includes a scholarly edition. I believe it also established some typographical standards for representation of aspects of the text.
  • The Globe Shakespeare, a 19th century edition produced by Cambridge editors Clark and Wright. Some dispute their inclusion of particular textual variations.

Which of these is the best option for inclusion depends very much on which edition is available in a marked up form. Ultimately it would be great to be able to store several versions (with concordances) and provide parallel access to them with comparison tools.

Requirements

With such a variation in possible digital representations of the plays, we should consider what the purpose is of storing marked up texts, and what the possible applications are.

What is required in an electronic representation?

Searchability, the ability to distinguish between each component and reconstruct any hierarchy that inheres in the text.

Why is it important to agree on a scheme?

There is little need to stress the benefits of having a standardised representation. If a representation scheme is agreed and then adhered to by a number of people, the data of literary artefacts become well defined and their structure can be used predictably. Tools and interpretations can be built up around and from the common format.

What is required of a Shakespeare play schema?

Probably nothing too different to what is required of any other dramatic or literary schema.

Why is there no agreed format?

TEI is the most recognised format; the LML/XLDL effort is interesting but appears not to have been applied to anything as yet.

Conclusion

This post has looked at some of the options out there for Shakespeare mark up and annotation. There are some interesting XML definitions, but apparently a lack of a core set of professionally-annotated texts employing a well-defined schema. We do not have the resources to markup a text ourselves, but we can process whatever editions are available.

The Perseus markup seems to provide a mix of an appropriate edition, a well-established and flexible encoding scheme, and a richness of annotation, and will form the primary text of our registry. It is more complex than other schemas, and so may take some more processing to render for presentation, however it appears worth it to provide as much well-organised information as we can.

In the future, it would be good to see a single encoding scheme become the standard for Shakespeare’s plays, and thereby contribute to a unified effort combining crowd-sourcing, annotation, and a rich diversity of interpretations, transformations, visual representations and unforeseen recombinations of texts, metadata and other resources.

Good Grieffe, More Robot Suits

Finally, a word of warning from T-Rex, about ensuring the long term preservation of Shakespeare in appropriate formats.

 

Share

New Title List Features in version 1.55

Posted on by

The Title List function in your LOCKSS box allows you to report on the holdings of your LOCKSS box or the content available in the LOCKSS network as a whole. The latest release of the LOCKSS software includes some new options, and this post provides a general guide to usage of the Title List feature.

Please see the post View your holdings with LOCKSS titles export for an introduction to the Title List feature.

Title List Main Screen

Title List main screen

Basic Report Options

The main Title List screen presents you with several options for the report, described below. Note that the defaults will produce a standard KBART report, but these options allow you to customise that to better suit your needs.

Scope

This option allows you to specify the scope of the data you want to report on:

  • Available – all titles available in the LOCKSS network. This is all the content LOCKSS users can select from.
  • Configured – the titles that you have configured for collection on your institution’s machine using Add Titles.
  • Collected – the titles that have successfully collected on your machine. A title must be fully collected before it will appear in this list; partial completion is not enough.

Type

This option allows you to filter the titles to include only journals or books if you need to:

  • Journals – show only journal titles
  • Books – show only book titles (those that have an ISBN)
  • All – show all titles

Data Format

This option specifies the format of the data that is output from the reporting tool; the default Title Ranges (KBART) format displays a line for every complete range in a title, while the other options allow you to consolidate these into a single line per title.

The KBART coverage_notes field will contain a description of the complete ranges covered by each row, therefore documenting where the coverage gaps are in the consolidated formats.

  • Title Ranges – show a row for each unbroken range within a title (this is the default KBART format).
  • Titles – show a row for each title, consolidating all the individual unbroken ranges into a single range using the outermost start and end volumes. Note that the single range specified in this format may contain coverage gaps.
  • SFX DataLoader – show a row for each title, with the coverage_notes field listing coverage ranges in the SFX DataLoader format.

Output Format

This option specifies the digital format in which the report will be produced:

  • TSV – tab-separated format
  • CSV – comma-separated format
  • On-screen – the report will be turned into HTML and displayed on-screen

Having selected these options, press the List Titles button to generate the report.

A Title List report showing the customise option

The above shows the top of an HTML report. Notice there is a link to return to the main Title List page, and a button for customising the report you have generated.

Customisation Options

To provide finer-grained control over what gets into your report, there are customisation options allowing you to specify which fields to output and in what order.

Select the Customise Fields button either on the Title List screen or the HTML report screen. You will see these extra options appear at the bottom of the Title List screen:

Title List Customise

Title List customisation options

Coverage Range format

The default KBART output reports each unbroken range of a title on a separate line. Sometimes it is more useful to have a single line per title, with start and end points. The coverage_notes field is therefore used to elaborate which specific ranges are fully available within a title listed in a particular row. This field can display data in a number of formats:

  • Year Ranges – a comma-separated list of ranges, e.g. 1996-2002, 2004-2006, 2009-
  • Year(Volume) Ranges – a comma-separated list of ranges including volume in parentheses e.g. 1996(1)-2002(7), 2004(9)-2006(11), 2009(14)-
  • Year Summary – a concise format showing only the start and end of the whole range, e.g. 1996-
  • Year(Volume) Summary – a concise format showing only the start and end of the whole range, e.g. 1996(1)-
  • SFX DataLoader – this is the format defined by ExLibris for use with their SFX DataLoader utility, e.g. $obj->parsedDate(“>=”,1996,1,undef) && $obj->parsedDate(“<=”,2002,7,undef)

Note that year ranges can have an empty end point, meaning they extend to the present. The summary formats are for convenience and do not indicate where there are coverage gaps. The SFX format represents the same information as the year(volume) format but in a more verbose format.

Field Ordering

The field ordering box lists all the default KBART fields in their default order. You can edit the contents of this box to change which fields appear in the output. The ordering will also be reflected in the output report. On the right-hand side is a list of all the valid field names – be sure to include at least one of the identifying fields or there will be no way to organise the title records!

Note that the first two fields will be used to sort the resulting list.

Omit empty columns

Some columns have nothing in them due to a lack of appropriate data. Select this option to automatically omit such columns and simplify the report.

Reset/Cancel

If you decide you don’t want to customise the output after all, or have made a mistake, you can select either:

  • Reset – to reset the field ordering list to its original values
  • Cancel – to cancel customisation and go straight (back) to the report output

Finally, press List Titles to produce the report, or Hide Customise Fields to skip the customisation and use the three default report configuration options of scope, data format and output format.

Direct URL access to reports

It is possible to access a range of reports directly without manually selecting options on the Title List screen. This is useful for those applications where an automated update of the LOCKSS box’s holdings is required, for example when updating a knowledge base periodically. The default report can be retrieved by requesting the URL

http://lockss.box:8081/Titles?format=tsv

There are five configuration options for direct URL reports, which can be specified as URL parameters. The first four correspond to the basic report options described earlier:

  • format – output format: tsv, csv, html
  • scope – data scope: available, configured, collected
  • type – title type: journals, books, all
  • report – data format: kbart, titles, sfx
  • coverageNotesFormat – format for the coverage_notes field: year, year_volume, year_summary, year_volume_summary, sfx

The bare minimum is a format parameter. So for example a TSV report, in SFX report format, on titles configured in your LOCKSS box, can be retrieved from:

http://lockss.box:8081/Titles?format=tsv&report=sfx&scope=configured

Using the report

There are also several common uses for the reports generated by the Title List feature:

  • Reporting what is available on the LOCKSS network. This can be used to populate a knowledge base in a link resolver such as SFX, or cross-referenced with an institution’s subscriptions to produce a list of what should be added to LOCKSS.
  • Reporting what titles an institution has configured their LOCKSS box to collect. This can be used in institutional reporting on LOCKSS, or cross-referenced with subscriptions and available titles to find out what is missing from your configuration.
  • Reporting what has been successfully collected by LOCKSS. This report can indicate where there might be problems, or be used in institutional reporting.

We plan to produce format options for other link resolver products as necessary, though the standard KBART report should be sufficient for many applications. After the current phase of user interface development, the reports should also be useful in configuring your collection list with less manual effort.

If you have any further use cases for these reports do let us know at edina@ed.ac.uk.

Share

Understanding the Status of Titles

Posted on by

This post is intended to clarify the different classifications you might see for titles in your box, or titles which have been committed to the LOCKSS network.

Available, Configured & Collected

When you use the Title List feature in your LOCKSS box to report on its holdings, there are 3 options available to you that affect the scope of the report:

  • Available will include all titles available in the LOCKSS network. This is all the content LOCKSS users can select from.
  • Configured will report the titles that you have configured for collection on your institution’s machine using Add Titles.
  • Collected will report on the titles that have successfully collected on your machine.

Note that each scope is a subset of the previous one.

Committed Titles

The LOCKSS website provides a spreadsheet listing the publishers who have committed to providing their content in the LOCKSS network. This is available on the page at http://www.lockss.org/community/publishers-titles-gln and is regularly updated.

Note that content that has been committed is not instantly available in the LOCKSS network. It must be scheduled to go through a preparation and testing process that includes (a) writing a plugin which knows how to collect the content (b) testing that plugin to make sure the content collects properly. At this point the content is released to the network – the regular emails detailing new releases are the end result of this process which is perpetually being undertaken by a team of content testers in Stanford.

The spreadsheet is made by combining committed publisher data with a KBART report on Available titles from the Title List. It lists each title with the fields Publisher, Title, ISSN, eISSN, and the following fields indicate the status of each title:

  • Preserved Volumes lists the volumes that are preserved in the LOCKSS network
  • Preserved Years lists the years that are preserved in the LOCKSS network
  • In Progress Volumes lists the volumes that are committed but have not yet cleared the testing process
  • In Progress Years lists the years that are committed but have not yet cleared the testing process

Years and volumes are listed as one or more ranges separated by semi-colons. In some cases the data are not available for one reason or another. The fact that data are missing does not necessarily indicate that the title is not released.

We hope this outline of title statuses will aid institutions in making appropriate use of the reports and data available from LOCKSS boxes and the lockss.org website. Please direct any questions to edina@ed.ac.uk.

Share

LOCKSS User Interface Requirements document

Posted on by

We are pushing on with the preparations for User Interface (UI) development, and are about to get started with the preparatory first stages of coding. The UKLA Members’ Workshop on Thursday will include a presentation on the content of the proposed developments, and also a session for discussion of the requirements that have been identified.

In advance of this we are sharing the draft requirements document, which describes the different types of users of LOCKSS, common use cases which they encounter, the main features identified for implementation, and the work packages which will be undertaken.

Please feel free to record your comments using the link below.

Share