Join Will’s World Online Hack 5-12 Dec 2012

Posted on November 22, 2012 by Muriel Mewissen

Are you interested in Shakespeare? Are you tempted to take part in a hackathon or know someone else who might be? Do you have a great idea for a new app? Do you want to mash your own data with ours? Then get involved in the Will’s World Online Hack! We are pleased to announce that registration for this event is now opened at http://willsworld.edina.ac.uk/wiki.

This hackathon aims first to promote innovative use of the Shakespeare metadata registry built by the Will’s World project to hold metadata describing online digital resources relating Shakespeare,but also to explore an online format for hackathon.

How does an online hack work?

Well, like a traditional hackathon, technical and creative people with different expertise like software developers, graphic designers, domain experts and project managers, get together and collaborate to develop applications and explore concepts. But instead of getting physically together in one location, social media technologies are used to communicate and collaborate online. We used your very useful and positive feedback to our online survey to plan this event.

The event will take place over a week:

Opening session on Wednesday, 5th December, 1pm (BST):
This will be a live and interactive session to present the data, the goal of the event, prize categories, the set of social media tools and technologies to be used during the hack and the Will’s World project itself. Participants will be able to introduce themselves and put forward ideas.

Hack, 24 hours spread over 6 days:
The participants will have six days to form teams, familiarise themselves with the data and code. Participants are free to organise when they spent their 24 hack hours over these six days. They will have the flexibility to work when it suits them. Teams can set their own schedule either for members to work concurrently or consecutively. The Will’s World project team will be on hand throughout to answer any questions and regular interactive drop-in sessions are planned.

Closing session on wednesday, 12th December, 1pm (BST):
This will be a live and interactive session where each team will present their hack either live or as a pre-recorded video and prizes will be awarded.

We are hoping to capture as much as possible of the communication taking place. In particular, the opening and closing sessions will be videoed. All recordings will be shared on the event wiki or the blog.

The use of technology and social media is at the core of this online hack. We will be using a wiki to act as hub to support communication, before, during and after the event. Mailing lists, Google+ hangouts, YouTube, Skype, Twitter, IRC, Github and Dropbox will all help the communication and creativity flow. You will find more information about the event, the data, the technologies and how to take part here.

Register now and receive a goodie bag!

If you fancy taking part in this exciting event and be one of the first pioneering online hackers then please register on the Will’s World Online Hack wiki. Participation is free and the first 50 participants to register will receive a goodie bag!

Will’s World online hack survey results: Your Views!

Posted on November 13, 2012 by Muriel Mewissen

Over the last three weeks we have been drumming up interest for our idea of an online hack event. This twist on the traditional “in person” format has exciting potential to be more flexible and make great use of social media. It seemed like a very attractive idea to us but, we wondered, what did you think?

We drew up a short survey (15 questions) to capture your views, feedback and any experiences that would help us plan a great online hack. We spread the word through this blog, twitter, mailing lists, websites and asked other to do the same.

To date (the survey is still open) we have received 30 replies to the survey and many direct emails with further input. So a BIG THANK YOU to all! We are delighted that you found time to make this hugely valued contribution and we thought that the least we can do is share here what you told us.

A Good idea

In answer to that core question we found that 84% of our survey respondents felt that the online hack was a good idea, of whom: 57% of respondents thought that an online hack was a good idea and would be interested in taking part; a further 27% of respondents felt it was a good idea but they were not sure how it would work.

75% of respondents had attended hack events in the past, and interestingly 3 have already taken part in an online hack.
It is very encouraging to see that most people are supportive of the online format – only 10% would prefer an in-person event. Only one person doesn’t think it will work and another said they wouldn’t be interested in taking part.
Significantly, all three experienced online hackers think it’s a good idea with two of them definitely interested in participating is this hack – this is really encouraging!

Timing – it’s all relative…

Opinions are divided over what format might work best. This is not surprising since most of our respondents had not been to an online hack before so were being asked to speculate on what might work. However, close to half of those who responded favour a week-long drop-in format. Others were split between weekend and weekday days – we had lots of conflicting comments about availability here.

We didn’t ask you where you were based – although we would if we did this again – but from your experience and email addresses we know we have respondents from both sides of the Atlantic which further encourages us that any possible timings and format needs to support an international hack attendance as elegantly as possible.

Participation

We were really pleased to see that you weren’t just being lovely in sharing your views, you were also really up for participating!

50% of respondents said they are definitely interested in taking part in this hack, with an additional 30% a “maybe”, and several others interested but unable to attend on the specific suggested dates in December.
A significant number of people (52%) indicated that they may be able to bring additional data to the hack. However, most note that it would depend on having enough time to prepare it and/or obtain approval for sharing the data.

Social Media Technologies

Social media tools are essential in supporting the communication required by an online hack. Many applications are popular and received support from the participants of the survey, as seen in the graph below:

Knowing what tools you already use means we now feel well informed to choose the right combination of social media and web technologies that will ensure you feel comfortable and familiar with the tools and work for the functionality we think we would need.

We need you… but what do you need?

We also asked you what you might need to be able to take part in an online hack. The main requests were for:

as much data as possible

information on the data available ahead of the event

easy access to the data

access to the APIs ahead of the event

We can definitely see from these responses and our word cloud for this question (below) that the data is crucial!

Team-building and help with communication tools ahead of the event were also highlighted. The importance of time, pizza, publicity, prizes and a greater technology know-how were also mentioned!

Participants

So who are you all?

50% of respondents work in the Higher Education section. A further 18% are freelance and 11% work in the private sector.
Participants were from highly varied background, with different expertise and interests: from experienced developers, to artists, designers, managers, engineers, teachers, students, librarians – the only common characteristics seemed to be a passion for hacking, for Shakespeare or for both!
92% shared their email with us to be kept informed on the developments of our online hack event – thank you! We will be in touch with you soon!

If you’re not one of those who responded but would like to stay up to date on the hack event please either fill in the survey now or drop an email to edina@ed.ac.uk with “Wills World” in the subject line.

I Love Shakespeare

You have shared with us your wishes for playing with data, engaging with communication tools, supporting learning and producing creative material. You have encouraged us in our ambitious vision but warned us of the difficulties too.

Most of all, the word cloud for our additional comments section seems to indicate that you simply love Shakespeare!

Will’s World Online Hack is Coming Soon!

Following the positive responses we have received, we have looked further into the practicalities of organising an online hack event and are delighted to let you know that we will be going ahead with the event in early December! Further details and the official announcement will be out very soon… Watch this space!

Will’s World guest post for myShakespeare blog

Posted on November 9, 2012 by Nicola Osborne

This week we have been continuing to gather feedback on our proposed Will’s World Hack event and begin the process of planning the event based on that feedback. Whilst we work on the Hack event we are also trying to get word out about the project and the event so if you think we’ve missed out a crucial website, mailing list, discussion space, etc. please do let us know. As we have been reaching out we were delighted to have the opportunity to create a guest post for the myShakespeare blog this week.

The myShakespeare is the digital home of the World Shakespeare Festival, part of the London 2012 Cultural Olympiad led by the lovely people at the Royal Shakespeare Company. They have a lovely video explaining what they are about which also highlights the reasons Shakespeare’s work is still so alive in 2012:

Click here to view the embedded video.

We think that’s a great wee take on Shakespeare – perhaps it’s even triggering a few creative ideas for the hack event?

Look out for a new update shortly with our feedback from the survey and more news on the hack. In the meantime we are still interested in hearing your views on what would make a fantastic online Shakespeare Hack event – leave us a comment here or fill in our survey: https://www.survey.ed.ac.uk/willsworldhack/

Mapping metadata to a common schema

Posted on October 30, 2012 by neil.mayo

One of the challenges in aggregating metadata is to unify the different schemata of the data sources, to make the data searchable through a single common schema. There are advantages to maintaining the original schema (field names) of each data source so that it can be searched in the same way that would have been possible in its original context (i.e. searching under the same field or fields). However this can complicate searches of the aggregated metadata when each data source represents similar information under a differently-named field – because a search for values on a range of essentially synonymous but differently-named fields across a variety of resources would be more complex to specify. Therefore it makes sense to map similar fields onto a single field, within a schema which is specific to the collection of aggregated data.

This post is about creating a schema for Will’s World and mapping other metadata into it. Despite the technical detail, it might elaborate some common stumbling blocks to be found in mapping disparate information to a single representation.

Developing a Schema

We had to develop a schema for Will’s World (WW) to represent the metadata which is being imported into the text database. This schema has several requirements:

Should be capable of representing/unifying metadata from several different sources.
Must unify some of the variations on similar pieces of data (for example there is a range of fields dealing with attribution and ownership – including rights/availability/copyright; owner/credit/creator/contributor).
Should use existing ontologies as far as is appropriate/possible (for example, Dublin Core, TEI)

Once the schema is designed, it is necessary to define how the metadata we collect from disparate sources will fit into the (probably more restrictive) schema that we have elected to use in our representations. This involves establishing a clear mapping from the source schemata to the target schema. Additionally, we need more than one schema for Will’s World:

A schema for metadata harvested from services, which represents content within those services.
A schema for descriptions of services themselves (including those from which we do not harvest metadata).
As a bonus, we might wish to define one or more schemata for texts relating to Shakespeare.

This post concentrates on the metadata schema, which is the most important and elaborate. We have defined a list of fields with namespaced identifiers, and for each field we define several properties:

Internal – whether a field is for internal use only, and is invisible to the user (because, for example, it is meaningless outside of the application).
Repeatable – whether a field can appear multiple times. For example, properties like “genre”, “subject” or “author” may be multi-valued (or rather have multiple instances), while we expect a metadata item to have a unique and canonical title (although we accept zero or more alternative titles).
Required – whether a field is required (because it acts as an identifier of some sort, or is essential to the meaning/categorisation of the record).

For the purposes of supporting text search, we need to answer a couple of questions for each field:

Should the field be Indexed (searchable through Solr queries)
Should the field be Stored (available to return from Solr for display in the interface)

Schemata

For reference while reading this blog post, you can refer to both the metadata schema and metadata mappings we have developed.

The schema for representing metadata about services is much simpler and records identifying metadata such as id, name, description, URL, provenance and usage/licensing information, and more specific metadata about how to use the service.

Collecting Data

The next stage is to collect, transform and ingest the data into Solr. This is a multi-stage process:

Collect the data – either in a well-structured format like RDF/JSON/XML/CSV, or contained in HTML.
Extract the data (if contained in an unstructured or variable format like HTML).
Transform the data from the source format into a format matching the WW schema.
Transform the WW metadata into a format matching the Solr import document schema.
Ingest the data into Solr, by posting it to the server.

Due to size and memory restrictions, any of the later stages, particularly the transformations and ingest, may involve splitting the documents into multiple (well-formed) documents.

We have to take a variety of approaches to collecting the data, depending on what types are available. Only a relatively small proportion of providers actually make their data available in easy to collect structured formats, while in other cases the data is present directly in HTML pages, or as the result of searches through an HTML web interface.

Scraping data – may involve two stages; in the case that the metadata itself is available directly from known URLs, the first and second steps are combined:
1. Scrape information from webpages (for example, the results of conducting a search through a search interface)
2. Extract links from that information and follow them to find the metadata.
Collecting data directly in a structured format.

Depending on the source of the data, it may be processed via scripts which construct structured data from un- or less structured data through a process of parsing and recombining data, or by an XSLT script which performs a simpler and more direct mapping of existing fields to the fields of the schema. This latter may involve many-to-one mappings in addition to direct one-to-one mappings. One-to-many mappings should however be avoided – this would produce data duplication and likely be an indication that the schema was not well defined.

Issues in Schema Design

In designing a metadata schema which is intended to unify metadata from sources with different concerns, we inevitably encounter certain issues which demand to be resolved in a consistent way.

Identifying fields

Naturally we need an identifying field so that every record can be distinguished and uniquely identified. We have two options for this – generate unique id fields (the easiest way would be to ask Solr to impose them), or map external values (from each source’s data) into an id field. The first option requires us to maintain unique identifiers upon updates, when we re-index data. The second option runs the risk of clashes. Identifying tuples or values sourced externally will vary a great deal and there is always the danger of non-uniqueness unless we are very careful about how we define them. It might be best to let Solr create a unique field value for indexing, and record an alternative ‘primary’ identifier from the identifying metadata of each particular data source. This might be considered a unique identifier within the records of the particular source.

Required fields

If the anticipated metadata informing a required field are not present in the source data, a default or “unknown” value must be given. This can be achieved by explicitly producing a default value from a script, or by providing a template in an XSLT transform for each required field, which matches when the source field is not present in a record. Here is an example enforcing the presence of a required dc:title field for a Solr import doc:

<xsl:if test="not(dc:title)">
  <field name="dc:title">Unknown</field>
</xsl:if>

Field values with subelements

What should we do when the source data that we want to insert as a field value incorporates tag structure of its own? We would like to maintain XML substructure where available as part of the value of the field, rather than discarding it. However the Solr import schema does not accept substructure to its “field” elements, so we must enclose them in CDATA tags. This is not easy using XSLT – unless we are willing to write templates to match each of the elements we want to use as subelements in the defined fields of the schema, and encode them appropriately. The solution is to pass the node which contains substructure to a target which wraps it in a CDATA section in the output like this:

<!-- Copy the current element and subelements to the output in a CDATA section -->
 <xsl:template name="xmlsection">
  <!-- Use the current node by default -->
  <xsl:param name="node" select="."/>
  <xsl:text disable-output-escaping="yes"><![CDATA[ <![CDATA[ ]]></xsl:text>
  <xsl:copy-of select="$node"/>
  <xsl:text disable-output-escaping="yes"><![CDATA[]]]]><![CDATA[>]]></xsl:text>
</xsl:template>

Aggregating multiple XML fields

As indicated earlier, there are several cases where we wish to map many source fields from a provider into one Will’s World field as subelements. An example of matching multiple fields at the same level which will contribute to the content of a single field is in National Library of Scotland data, where we want to match and combine the XML elements “summary”, “commentarytitle” and “commentary” into a single “dc:description” field. There are a few options for performing this aggregation; if we know that one of the source fields (say, “summary”) will always be available, we can match on it and then pull in values from the other fields as necessary using xsl:copy-of with a relative path in the select attribute. However, if the “summary” field is missing, the template will not match and we will lose the values of the other fields.

If we cannot be sure that any one of these fields will be available, that is, they are all optional, then we need to match every possible combination. Doing this via a number of similar templates is unnecessarily verbose and will also lead to duplicate output fields, as every one of the fields which is present will match and cause the output field to be written. The solution is to provide an OR-ed list of the possible combinations of these elements, so it will only match once and produce one instance of the output field. In the following simplified example, we want to add elements f1 and f2, which occur at the same level, and either of which may or may not be present, to a single output element – and only do it once. We therefore need to check for the presence of both elements, or of either element without the other:

<!-- Match f1 and/or f2 -->
<xsl:template match="f1[../f2]|f1[not(../f2)]|f2[not(../f1)]">
  <combined-field>
    <xsl:copy-of select="../f1" />
    <xsl:copy-of select="../f2" />
  </combined-field>
</xsl:template>

Note that the XPath expression will become increasingly complicated if we are combining more than two potentially absent elements. In such a situation, it would be desirable to find other ways to do this, to avoid a combinatorial explosion of match expressions.

Issues in Mapping Metadata

Scraping data

We scrape data on the granularity of a page. Sometimes this yields metadata at different levels. For example, the BBC’s “Shakespeare’s Restless World” is a radio programme produced in conjunction with the British Museum, exploring an event or period in history through a particular object from the museum’s collections. The pages describing each show tend to contain metadata about the show, and also metadata about the object around which the show is centred. These can either be mapped to a description of a single entity representing “the programme about the object”, or the metadata can be recorded independently for both the programme and the object.

How to record the various metadata at different levels?

In this case we conceptualise the object as the resource being described; metadata about the programme (title, description, URL to a podcast) will be recorded in a record describing the programme, while supplementary detail (background/object facts, quotes) will be incorporated into the text description (using appropriate tags) and also distributed where possible into a record that describes the object. A page listing objects by play is scraped separately in order to supply associated plays to each programme/object.

Summary

Aggregating a wide range of metadata by mapping its values to the fields of a single common target schema renders it searchable in a unified interface. The mapping effort does however throw up a number of issues, some of which have been described in this post along with potential solutions. In particular:

It is necessary to be clear on what one is describing with a particular schema – a record, or a service providing a description of a record.
Collection and transformation must be performed carefully, with sensitivity to the requirements of the schema; we cannot rely on receiving or extracting data of the correct type or structure.
We should try to maintain as much as we can of the structure of the source metadata, so as to minimise the loss of information inherent in shoehorning data into a different representation.

T-Rex and Utahraptor point out that “Shakespeare needs tons of notes to be readable today” – sometimes information and cultural artefacts need to be mapped into a single common language and supported by supplementary metadata. (Is that a weak segue into a Dinosaur Comics link?)

Creating an XML Schema from a Table Description

I found that developing the schema was most easily done in a spreadsheet or table format, so I wrote scripts to transform that specification (at least partially) into (a) an XML schema and (b) an HTML layout of the same. To produce a schema, we produce elements with minOccurs and maxOccurs attributes. If the field is required, minOccurs is 1, otherwise it is 0. If the field is repeatable, maxOccurs is “unbounded”, otherwise it is 1. Here is the script for generating a schema file from a CSV description of the WW fields.

Disclaimer: this script is quick and dirty, non-validating, and WW-specific. Use it at your own risk; or better, adapt it!

Can one desire too much of a good thing?

Posted on October 29, 2012 by Nicola Osborne

An image of spiced syrup cakes – definitely too much of a good thing!

Our blog post title (from As You Like It (Act IV, Scene 1) reflects our desire for your feedback on our Will’s World Registry online hack event. We have had some fantastic comments back already – by email, Google+ and, mainly, via our survey – but we are still hoping for more!

Please do pass along the survey link, https://www.survey.ed.ac.uk/willsworldhack/, to friends, colleagues or any mailing lists or groupd you think may be interested in working with data on Shakespeare.

Whilst we begin the process of analysing surveys and other comments we did want to share a few interesting ideas of new ways to work with Shakespeare data that you have already told us about…

We’ve heard about some fantastic work, led by Faith Lawrence, to programmatically analyse Shakespeare’s work to see if it passes the Bechdel test, originally conceived as a test for movies in one of Alison Bechdel’s regular comic strips. A movie (or cultural object) passes the test if: (1) it has to have at least two women in it, who (2) talk to each other, about (3) something besides a man. Of course for Shakespeare this throws up lots of interesting issues as his plays regularly feature characters cross dressing and playing with their gender and/or identity. Read more in Chris Gutteridge’s blogpost and Faith Lawrence’s paper presented at the Narrative and Hypertext Workshop, Hypertext 2011.

Meanwhile an email with a jokey comment about Christopher Marlowe had us wondering if comparing or analysing the Will’s World Registry with data on any of those who have, at some point, been rumoured to be the real author of Shakespeare’s work might throw up some interesting data… with that in mind we’d welcome any data sets you may have on Christopher Marlowe, Francis Bacon or Edward de Vere…

Wanted poster based on "Christopher Marlowe, dramatist, poet" by Flickr user lisby1 / Lisby Wanted Poster based on "Sr Francis Bacon" by Flickr user Stifts- och landsbiblioteket i Skara / Skara kommun Wanted Poster based on "Edward Vere, Earl of Oxford" by Flickr user lisby1 / Lisby

We’ve also had a number of really thoughtful responses on the practical considerations of running an online hack – some of the risks and opportunities around doing this. If you have any thoughts or concerns on the idea please do leave us comments, complete the survey or email us with any feedback.

In other news… we hear that we have inspired another hack event already! The SPRUCE project folk spotted our tweet about a Will’s World online hack event and decided to set up their own one day remote hackathon, on 16th November, to make file format identification better (crucial for preservation). You can find more information about their #fileidhack on the event wiki.

And finally… if you find yourself in London next week then there is a special Late: Shakespeare beyond the city taking place on Friday 2nd November, from 6pm until 9pm in the British Museum Great Court. A variety of creative interpretations of Shakespeare will be taking part with contributions from the National Theatre and RADA students and creative potential Will’s World Hackers will particularly be interested in the prop-making workshop.

Image credits for this post:

Wanted poster and thumbnail version of wanted poster based on “Christopher Marlowe, dramatist, poet” by Flickr user lisby1 / Lisby; Wanted Poster based on “Sr Francis Bacon” by Flickr user Stifts- och landsbiblioteket i Skara / Skara kommun; Wanted Poster based on “Edward Vere, Earl of Oxford” by Flickr user lisby1 / Lisby

Pondering the Possibility of an Online Hack…

Posted on October 24, 2012 by Nicola Osborne

Over the last few weeks we have been mulling the possibilities of running an Online Hack for the Will’s World project. You can read more about the plans on the Will’s World blog where you will also find a survey that we would love you to complete for us or pass on to developer/hacker/creative colleagues and friends.

As mentioned in our Will’s World post the idea is to try and take the energy and creativity of an online hack event and translate that into something virtual not because we don’t like people, pizza, and coding through the night but because we recognise that type of format isn’t always right for people who may want to take part. That might be because an employer is supportive but can’t release a staff member for a full work day, it may be that they are available but cannot fund travel and accommodation, it might be that they have caring responsibilities that would make an in-person event much harder to fit in, or it may be that the venue isn’t sufficiently accessible for those with different physical abilities.

But there are also lots of other special things that we think an online hack affords. Running something online and with a cunningly chosen time/appreciation of time zones means collaboration across the globe – something the Open Source movement have been up to for years of course. And it can be much easier to start on a new hack when you know what software and hardware you have to hand (not to mention not having to travel with all your tech!), you know your internet connection is reliable and/or you’re used to working with it’s speed, and you have all that quirky personal creative stuff to hand – be it arduino kit, a fine selection of felt tip pens, a monster supply of gluten free brownies, etc.

Will’s World Online Hack is coming soon..

And the reason I wanted to post something about this process on my own blog is that we think this is a pretty innovative idea but one of the particular challenges comes from considering which suite of social technologies will work and combine best to ensure this event has the buzz, the energy and the relevance of an in-person meet up. The survey is part of our approach to finding out what might work but I’d also appreciate any comments here about what you think would work best for real time collaboration?

For instance I’m thinking that Google+ may be an effective and fun tool to try out – particularly for managing multiple video streams – for this event but I haven’t had a good excuse to trial this on such a grand scale before so would love to know others thoughts on how well this works in reality for larger groups of participants. Please do any suggestions or comments on the hack idea either here or via the survey.

Share/Bookmark

Online Hack Event

Posted on October 18, 2012 by Muriel Mewissen

Our Will’s World project is soon coming to an end. While we are busy populating the Shakespeare Registry with great data, we are keen to put this wonderful resource to good use. How can we achieve this in an unusual and fun way?

How about an online hack event!

The idea first came to us as we began planning a traditional hack event – something in person, overnight, possibly featuring pizza. As we started looking at how this would work best we realised there were lots of logistical issues to deal with from finding a suitable venue, to catering to the issue of how participants could find the time to travel and take part.

We also started to think about things that don’t always work so well in in-person hack days… sometimes the software you want to use is sitting on a machine you don’t have with you, sometimes you need to attend to caring responsibilities which just don’t fit into a 24 hour marathon lockdown, and sometimes you just can’t move that meeting or spare the travel time to fly miles away to make that fantastic looking hack day somewhere at the other end of the country.

And that’s when we realised that an online hack event might not only resolve our logistical issues but that the flexibility and potential benefits of an online hack event are also very exciting!

More people can participate:

Participants who wouldn’t be able to travel (whether because of the cost, time, distance, or scheduling conflicts) can easily join in.
The event can be scheduled to allow participants from further afield and different time zones to be included turning a local event into a global hack event.
There is no restriction on the number of participants due to the venue size or cost.

Enable use of familiar tools:

Participants can use their own machine, familiar set up and well-loved applications in their own environment. There is no need to get used to a different technical environment or to first install the tools you can’t do without. That means more time to be creative, to hack, play and collaborate.
Wifi and cable internet connections are also a lot more likely to be fast and reliable – at least something you are used to managing – if they are not being shared intensely by a room full of coders!

Keyboard disassembled and planted with cress

“Prepared keyboard waiting to sprout” by Flickr user wetwebwork

Flexibility in participation:

Participants can choose when and how much efforts they put in to the hack.
Participants can fit their participation around their other commitments.

Promote use of social media technologies:

Many collaboration tools can be used to run the event and connect people: blog, Twitter, wiki, Skype, Google Hangout, videos, websites…
Participants or those who find out about the event later can still share in the event with records of the hacks more easily captured via video, wikis, text chats etc.

Obviously, the critical issue will be to ensure that communication takes place effectively during the hack between people scattered in various locations. Team formation will be interesting – but no weirder than grabbing a coke or a beer and introducing yourself around a room of fellow hackers. And we know that interruptions could be tricky for some participants since they will be occupied with the Will’s World hack from their normal office or home desk. We can see challenges here but we think the benefits could make this a great format for our hack event.

But we want to know what YOU think of the idea…

We have come up with a few scenarios on how this type of online hack could work. We would really appreciate your help in evaluating different formats and communication tools for this event. Please take a moment to tells us what you think and provide us with your feedback by taking this short survey:

https://www.survey.ed.ac.uk/willsworldhack/

Please do feel free to share that survey link with others you think might be interested in this event. We also welcome any comments here as well.

Image based on “Shakespeare’s Globe Theater, Southwark, London” by Flickr User nikoretro/Sheri

One last thing, we are aiming for the online hack to take place during the first week of December. Put it in your diary!

Will’s World at Culture Hack Scotland

Posted on April 30, 2012 by Nicola Osborne

This weekend saw the Culture Hack Scotland event take place at SocietyM in Glasgow. Culture Hack Scotland gathers creative people including developers, designers, artists and creative practitioners and supports them to work together over an inspiring and intense 24 hour period of idea sharing, data hacking and creativity.

The materials we have been scoping for the Will’s World registry seemed like a fantastic potential data source for this event, particularly given that it was taking place in the week of the 448th anniversary of Shakespeare’s birth. We got in touch with the CHS team and after a bit of discussion about what might be appealing to CHS developers and what could be delivered in the time available, it was decided that we would be an official Data Partner for Culture Hack Scotland 2012 providing a marked up version of Macbeth. This would be the full text provided in XML and included various valuable additions to the data that would enable a greater array of possibilities for creative use of the text and/or aggregation with further resources.

Is this a data set I see before me?

Around a week before the event we were able to provide a link to a new data page containing the full text of Macbeth which had been beautifully marked up, including details on characters and locations (including lat/longs added through Unlock), prepared by our software engineer, Neil Mayo. Our colleagues from the Statistical Accounts of Scotland and JISC MediaHub services also made their metadata available (linked from the same page) to enable CHS participants to connect the play to related resources such as historical notes on counties mentioned in the text, or video recordings of key performances.

We sent off the data and it was featured in the weekly update for announcing data provided for the event. There were four of these Data Thursdays in all offering a staged announcement of all of the CHS data, each one announced with the release of data slices enabling developers to have a look and a play with the data ahead of the event. The last Data Thursday included a fantastic mention for the Macbeth data and, as it went live, all of us in the Will’s World team took a deep breath and nervously crossed our fingers that someone would be taking a look, having ideas, and, hopefully building something fantastic…

Who comes here?

Friday evening saw the kick off the event and the gathering of around 100 people including developers of all types, designers, artists, musicians and data owners for the kick off of Culture Hack Scotland. The event opened with an introduction from the Culture Hack Team and an inspiring pair of talks from James Stewart, Technical Architect for data.gov.uk and Brigitta Zics, Head of Digital Media / Convenor of Creative Digital Practice Group for Culture Lab at Newcastle University.

As the evening continued all hack participants were encouraged to share any early ideas they might have to get conversations going and teams forming, and we were delighted to hear at least one possible Macbeth project emerging right away. As the event moved into the team formation and hacking time it became clear that several projects were really excited about the idea of working with the Macbeth data. A number of developers who were looking at working with Macbeth complemented the quality of the XML we had provided for CHS making us ever so proud and extra excited about the possibilities for using this data.

The night has been unruly where we lay

The true business of hacking only really kicked off late in the evening but by midnight a hardcore team of hackers were working, in teams or alone, on a range fascinating ideas and projects. At the 2.30am catch up – a meeting with a lot of coffee and around 30 committed CHS hackers – it became clear that some really creative work was underway: one team had taken a pair of old skinny jeans and were in the process of using Arduino hardware to create a very alternative way to experience The Skinny magazine’s arts listsings; a mobile developer was creating a new interface for Glasgow Museum’s Zoology collections data; one team had already finished a map interface for crowdsourced stories. However for the Will’s World team the most exciting thing was finding out about three projects who had decided to work with the Macbeth data in very different ways: one focusing on the emotions of the text; one building an interactive game; and one looking at how Macbeth and social media could be combines. Indeed Nicola Osborne, who was along at the hack representing Will’s World and EDINA, had gotten involved in one of these projects by creating some Monty Python-esque illustrations for the interactive game team.

Illustrations of characters from SketchyApp

CHS participants carried on working through the night, sustained by never ending supplies of coffee, chocolate and fruit. A few naps were taken but by early Saturday morning – when this CHS piece featuring the Macbeth data appeared in The Herald – even more teams were working away as those who could not make the Friday began to appear, full teams gathered after sleeping in shifts, and the final push to finish off ideas and hacks got under way. The submission deadline for hacks was 4pm ready for a show and tell session of epic proportions with some 29 presentations of over 35 hacks waiting to be showed off.

When shall we three meet again

In the end a total of three teams had worked with the Macbeth XML. They were:

Shakey App

Shakey App is a Massively Multiplayer, RealTime, Macbeth Parlour game. It was designed by the team of Rory Fitzpatrick (@roryf), James Newbery (@froots101), Philip Roberts (@phillip_roberts) & Padmini Ray-Murray (@praymurray). With additional stage design, by Duich McKay and character illustrations by Nicola Osborne.

Shaky App as demonstrated in their Show & Tell Presentation

The way that Shakey App works is that players login and randomly assigned a part to play. They are then prompted with the lines via their phones and the audience around them (who also have to be logged in) can then vote on the best and worst performances by virtually hurling tomatoes in horror or throwing flowers in appreciation. It was demoed live at CHS as pictured above and you can read a full write-up or take a look at the code for the project.

Colouring Macbeth In a Glass Case Of Emotion

Douglas Greenshields (@bedroomation) created this inventive exploration of the Macbeth text through the emotions of the text. The user is presented with a number of extracts from the play and asked about the one that feels most… something… that might be “fearful” or “angry” or “loving”, etc. Each of those emotion questions is coloured a certain way, fear might be green for instance. When you click on the extract that is the most of that emotion the text is coloured appropriately and over time the system builds up an idea of which scenes connect emotionally, how the readers emotion is reflected in their marking up of the text, etc. Click through to the Colouring Macbeth In a Glass Case of Emotion site to have a go and read Douglas’ account of what he has created to find out more.

Macbeth Digital

The team of Marius Ciocanel (@MariusCiocanel), David MacKenzie and Minka Stoyanova decided to create their own Twitter client pulling in the Macbeth text as tweets from each of the characters. “Tweets” are timed to come in at a great reading pace and you can use buttons at the top of the page to switch between scenes.

Screenshot from Macbeth Digital

These projects gave us some fantastic opportunity to think about some really creative uses for Shakespeare data and metadata and we were pleased that colleagues from the Royal Shakespeare Company, who are currently running the incredible World Shakespeare Festival 2012, had also been able to make it to the CHS Show & Tell and see these projects showcased.

We were inspired not only by the Macbeth projects but also by the other hacks showcased at CHS as there were some fantastic ideas from music made from footfall data to ambitious games built from poetry – full details of all of the projects created at the event are now available on the Culture Hack Scotland site. Chatting with developers and designers throughout the event and at the Show and Tell highlighted some really useful reflections and comments around the need for quality data and the opportunities and challenges of aggregating large data sets. We will certainly be bearing in mind those perspectives as we move forwards.

Is that my prize?

Once all of the projects had presented at the Show & Tell the judging took place. We are delighted to say that Shakey App, built on the Will’s World Macbeth data, was awarded the honour of Most Playful Hack and went on to take the Overall Grand Prize Winner!

The team behind the Grand Prize Winner Shakey App.

When the hurlyburly’s done

Now that Culture Hack Scotland has been and gone what happens next?

Well we plan to keep the Macbeth XML available and would love to see any new ways to hack/use/create things from any of the data we made available from CHS. The page will remain live and we will be adding some additional data that complements the Macbeth XML shortly. If you do decide to create something new we would love to hear about it – leave a comment here or drop us an email us via edina@ed.ac.uk.

Culture Hack Scotland cup cake

Culture Hack Scotland 2012 Show & Tell Liveblog

Posted on April 28, 2012 by Nicola Osborne

There’s been a bit of a gap in my liveblogging from CHS 2012. I’ll fill in the gaps in my previous posts soon. But I wanted to share the hacks here so I’ll be liveblogging the hacks as they are shown. It’s all very exciting…

But first, now that we’ve all submitted our hacks, it’s time for a performance by Jonnie Common. And he’s been up 24+ hours playing with archive sound – field recordings from the National Museum of Scotland, Alumni data from Glasgow School of Art, Tramway Footfall Data and Tapestry design data. And it’s pretty cool. Also previewing some tracks from his new album made entirely with noises from his kitchen, including his “Ofen” (apparently all automated call centres think that sounds like “beer cooler”). A really lovely piece of music generated from a base sound of kitchen. And his big finish is a piece made from Tramway footfall data.

Erin and Devon are introducing the Show & Tell with their highlights. Devon says he’s been overwhelmed by the quality of the hacks. The two main things that do it for me are the quality of the hacks – people have actually made some super sophisticated thing, and other thing that amazed me was how many people stayed up the whole night. There were like 30 people at our 2.30am check in! That’s really really special. Thank you to everyone!

James Stewart of Data.gov.uk, Birghitta Zichs researcher at CultureLab, Cath Mainland Chief Exec of the Edinburgh Fringe and Clive, Director of Dundee Contemporary Arts are our judges.

My GI

We’ve been working on two projects – My GI and Uncreative Scotland, exploring the least creative areas of Scotland. We ran out of time for the latter but we could see that project trigger funding etc. The idea is to open up the programming information – curate your own GI by logging in and creating a programme of events with own openings and talks. Break down the hierarchy of the events, ask people to your lag as part of this.

404 Visual

I came here with Interface 3 but did a little sidpromo jet with archive NMS sounds. Basically it’s an audio visual visualiser with several settings. That’s about it.

Yaunne siesnik @amazingrolo

This is a hardware hack. And I also used the NMS field recordings here as I’m a huge fan of field recordings. And I’m really interested in usual ideation of sound, but also the physical inout and output of sound. So this is an output device for sound as lights, using pitch tracking to control the lights. I also hurt my hand recently and this led to me thinking about ways people with limited mobility experience sound. Also sound is hard o do in a public space – too loud and annoying or awkward and problematic. And what. If you can’t hear the sound at all? So I made a bit of software here.

Team 365

Wanted something different. Combined skinny and list data and used sound data to make a soundscape of events. So this is Glasgow tonight. Uses sound cloud data.

Team shaky

Before we get set up we want as many people as possible on twitter/iPads. Massive game. You get a role, you perform, veg or flowers are thrown.

Kyle

I am a student at Glasgow Uni and I worked on a visualisation of the tramway hourly footfall data. Both by hour and by exhibition.

Team banned by google

We didn’t have an idea until late last night, we worked with the creative Scotland data set and data visualisation, we’ve used postcodes and built boundaries and then visualised the amount of spending. We got banned by google or hitting their servers too hard, hence the name.

A short gap so Rohan gives us a thought experiment – imagine the cost of this talent for this many hours!

Douglas

I wanted to do something and Macbeth was a really huge data set and huge amounts of content. It shows a random text, asks you a question about the emotion and then colour codes the text. It builds up the colours over time and responds to the emotion of the reader

Stef & Carolyn

Scottish book trust my favourite place data was something I was looking at on the way to CHS. It’s a lovely idea but it doesn’t look that exciting, it felt like they forgot about the story. So we built storyline using a tile set called stamen o build a painting like map o he stories. We finished at midnight so…

Another hack. Cuts mean artists are not as well supported so this is the idea of communally funding artist time on experimental time. You put money in to fund artists to work for a number of days a month. ? Time and people is the issue for me as part of an arts organisation so the idea here is to give people time for their work.

Jen and paulo

Paulo: This is a hack using chapter titles and the text of the track an novel by Catriona childs. The chapters are named after song titles so you can also lck through to that track. You can also click away, explore other parts of the book. Uses refreshed random images from instagram.

Jen: and we built a second hack to see what’s oing on an hidden where you currently are so you can explore what has been contributed in your area (eg data like “this I the pub from train spotting” )

Macbeth digital

Thought that it would be great to see plays on twitter, a twitter client that runs through the whole of Macbeth, you can see it come in every second or two and skip between scenes

Temas

Field recording hero! This was based on national museum of Scotland. Takes lips automatically, can speed it up. Can increase density. Can also combine some sounds. You can click or play via the keyboard. And you can make your own field recordings

Jim, donny, carol – yarn spinner

This is an easy beautiful and social way o read on a tablet. Swiping sucks. The first book here is Catriona child’s book trackman. It pulls in bits of text and associated images added to the book,can favourite bits of the book, can comment so v good in education, can send to twitter, we added map data and images but users could do that, publishers could do that, and ou can change reading speed etc.

Mitchell

I worked with Glasgow museum zoology data. Metadata from their collection, mashed up with some images from wiki commons. Use data from sample collection to see maps app/google maps. Which museum would have been handy, could add. Read more link. It’s just a nice way to explore the collection. And we’ve added noise too! Although most sound like sheep! But if there was sound info you could build a richer experience, could add user images, build a sort of game for those visiting in person.

Alastair Macdonald

Rohan adds a top fact: Alastair was the first person to sign up to CHS this year!

I was tempted to use the Richard Demarco data – 6000 images from 65 years of the fringe. I wanted to run the images through facial recognition software to see faced and change over time. But far too many images to process. But you can quickly see features like glasses etc. got tons f data back. Only had I’ve to put into exel to see estimated age of subjects – a peak. Early twenties, slightly more women. Glasses wearers by age also. And mood… At 47 everyone is happy! All 6 year olds are surprised! By remember its not a big data set! What’s happening is for a few years images process great, but 2000 and 2007 images seem o be scanned in and messes with data! So that was my experiment… I think it failed.

Kate Ho and interface3 team

We are generally interested in digital storytelling. And we built everything for our game in 2 hours built on Edwin Morgan’s stobhill. It’s from multiple perspectives, it’s a very grim tale about a young URL who had to have an abortion and just as it was about to go into the Incinerator it’s found to be alive. Its told from doctor, porter, parents and boilerman. So it’s an immersive story imagining the hospital as abandoned, has a dark creepy tone like resident evil. You poke around. And move through audio. And it is quite creepy and sinister. We want to develop this on, transported to the highlands where the conception happens in second level. Going to end en route to the incinerator.

Rory (@digitalwestie)

I’ve been working w/Glasgow unesco city of music data set. Over 7000 tracks from different artist, was ruinous about what I might like in there. I use a think called last fm which lets you hare and is over music. And there I a list of festivals in Europe ranked by kast.fm by my taste in music. Thought it would be grea to o something similar. Hence weenies love beats – a mix f Glasgow and last.fm data. So you can look at the data set from your own perspective. Lots more you could do with this data, I added 10,000 tags to this data so think I just scratched the surface.

James baster – flock lights

I used twitter data from everyone here, looked over 9000 tweets around the event. Pulled it together. Was about useful information – contacts, links, possible new connections. For each persons have inormation about their friends, ollwsm conversations to. But just around the hash tag. Can also look at word use to gather by word use and topic. Online now. More data could o in, but what would be useful? And it’s all on GitHub.

Roy, Micheal, tom and jack from Dundee

This team have two products they will be showing. Another hardware hack! We are all associated with Dundee Uni? And we have made the Skinnys jeans! They walk the event frequency of events in Glasgow.

We also Built a device to make words physical Neil I created a visualisation of one of the Edwin morgan poems. Takes each line of a Poem, finds a word, pulls in a random image from flickr. So you get a sort of visual poem. With fun and slightly random effects.

Stef and Katie and Carolyn

This was a project with the Richard demarco data and approached it by looking what was online already. An organisation has been funded to scan in one mans life works really. But the website is kind of getting in the way of the content, hard to get to photographs. so we wanted to highlight key artists and making the images stand out.

Lucy, Chris, Gavin and monkey the dog

This was lucys first coding experiment. It’s a project using arguing. A ducky flashes a light every time a #chscot tweet ones in!

James mail

I am a product designer and I was inspired by the tramway footfall data, this is a thin you could install in a gallery spaces. Releasing ping pong balls or each previous visitor that then mingle with the real visitors. John (jonbca.github) I used footfall data also. Dots bouncing represent people visiting, can see the busy times… And it’s very much a work ian progress.

Textmob

Inspired by poetry and Macbeth we decided to set up flash mob readings! We did this with twitter. You can take part by tweeting #CitizenMob. Then line are assigned to the crowd who can join in the reading

Reading betweet the lines

We did three projects. First was poems of Edwin Morgan told in others tweets who don’t even know they re takin part. Big blue word highlighting show you the connection ian a stream on random tweets. Each is retweeted letting people know to learn more about the poet.

And next we made a generator to make new poems from his work. And it works!

Finally we took university of edinburgh data and it will show the buildings and energy use over time. It animates a beautiful set of colours showing change over time and it looks beautiful but projector isn’t playing ball today.

Festory

We wanted to use edinburgh festivals data – financially worth more than golf! But it means a lot to people who take part so… Festory… We were inspired by GSA alumni archives by ended up building a way to map social memory onto festivals. made it pretty. Can add by show, place, year et. And you can add through the year. And thinking about mapping entire fringe performers. You can check into a show. Want to crowdsource photographs, the performer’s perspective and from the audience and reviews. We had loads of ideas here, lots more work to do!

Creative Scotland – Stephanie Chris et al

My interest is in the creative Scotland data. I came up with 4 broad categories of data – capacity, impact, outcome, investment – or those interested in but outside of arts in stolen – policy makers, donors etc. so you would go in, pull in data on a project and look for how to track and monitor projects. It’s a cultural policy hack!

Ellie Harrison

Suzy invited me here. I felt a bit left out last night. I like to reduce my interaction with tech as much as possible. I was probably in the wrong place. I only got my laptop outat 2pm. And I try to avoid twitter but hard to acaommunicate here without it. So I did a little societal study of twitterholics. So I only spoke to 12 people. Thank you to all in the sample! I also wanted o ask how long each night people tweet by put didn’t know how to visualise this! Leapyearboy tweets least and sleeps least. So here are our sample. Second top is suchprettyeyes/nicola osborne who helped me put this together atbthe last minute. Alastair is our top tweeter! And signed up first!

Smash it – nick street and alex waterston

Last year I made something for stealing from Nms. What better to top it then smashing stuff from the NMS. And I must apologise as I mashed lots of data sets into the concept: Animal reincarnations of dead poets wan you to smash the glass treasures infringe venues for creative Scotland Smash more for more funding! It does work. If you want to play it… You’re mad!

And that ended the Show & Tell portion of the event. Which meant it was time for beer, cupcakes…

Culture Hack Scotland Cupcake

And then, the judges decisions… which were (according to my live tweets):

Commended #chscot most playful app: textmob; and creative Scotland by banned by google. Winner: Macbeth parlour game! Phenomenal!
Recommendation: demarco website by @stef an co. special prize for extra activity also to @stef. & prize for the audio visual player #chscot
Prize mentions also for mist useful the field sound recording hero, and @Alastair’s demarco project. Winner xdesign360 soundscape #chscot
Product hack: commended smash it, winner skinny jeans #chscot
Special award to the fab @kateho team as they did crazy amounts of incredible work in the 24 hours #chscot
Extreme excitement: the fabulour Macbeth parlour game takes top prize! #willdiscover