Digimap adds street names to search results

Until now, Digimap has always used the 1:50 000 Scale Gazetteer to provide search results when using the interface.

Searching for a street name in Digimap Roam

We have now added street names to the search results in Roam, so you can get more detailed results for your search locations. The locations of the streets come from the OS Locator™ gazetteer which we have been working to include since Ordnance Survey made it an OS OpenData™ product. When using the search interface you can now put in a street name and search for it, though we would recommend adding the name of the place the street is located in too, particularly if it is a relatively common street name. The best results in urban areas are still achieved by using postcodes as these can often resolve to a particular part of the street.  However in rural areas where postcodes can be very large, street names can give you the best results.

The search takes all the terms you enter and adds weights to each one; if you enter a full postcode it will just use this to pinpoint your location. If you are not sure your postcode is completely accurate in can be better to leave it out if you have more reliable information. Any street numbers or house names are ignored by the search as these are not stored in the OS Locator or 1:50 000 Gazetteers. The remaining search terms are run against the two gazetteers to produce a list of potential matches.

Be careful when searching for road names that are also places; searching for London Road in Edinburgh will also find Edinburgh Road in London and other possibilities from each city.  The weighting usually puts the best match at the top, though you may need to look a little further down the results list sometimes.

We will continue to work on the search abilities in Digimap to improve the results we return, please let us know if you have any questions:

  • Email: edina@ed.ac.uk
  • Phone: 0131 650 3302

Share

The search for Flight 370

flight370

courtesy of Wayan Vota (https://www.flickr.com/photos/dcmetroblogger/)

As the search for missing Malaysian Airways Flight 370 approaches it’s 5th week, the reliance of Geospatial technology and the skills to analyse large volumes of data are becoming increasingly clear. In this post we will look at some of the geospatial technology and techniques that have been used in the search for Flight 370.

Background

Flight-370 disappeared on the 8th of March 2014 having left Kuala Lumpur en-route for Beijing. There was simply no trace of it. Communications were lost somewhere over the Gulf of Thailand. Speculation quickly rose as to the fate of the aircraft with hijack and rouge pilots being muted as possible explanations.  A catastrophic break-up of the aircraft through an explosion was not ruled out but looked unlikely as this would generally be noticed. Furthermore, there was no sign of debris in the area of Flight 370 last known position.

Data feeds and extrapolation

After a few days, data started turning up that suggested that the plane had stayed aloft for several hours after all communication was lost.  Equipment onboard transmits information such as status updates and diagnostics.  The engineering teams can then monitor the health and performance of components while they are in use.

The engines had sent burst of data every hour and these had been picked up by a satellite operated by Inmarsat. By monitoring the Doppler effect in the received data, Inmarsat was able to chart 2 possible paths; one to the north and the other to the south.  This had never been done before and the innovative use of this data by Inmarsat allowed the rescue effort to be concentrated in 2 distinct areas.

After a bit of tweaking and refining, the Inmarsat scientists were able to discount the Northern corridor and the search re-focused on the Southern corridor, a vast expanse of ocean west of Australia with no suitable landing site.  How they achieved this was really quite clever. They used “truthing data” from other aircraft to monitor the Doppler effect and therefore refine their estimates for Flight 370. They then calculated the speed and altitude of the aircraft and were able to work out roughly where it would have run out of fuel and ditched into the ocean.  This greatly reduced the search area.

Satellite analysis

The search area had been focused to a small section of ocean (ok, so small in this case means the size of Western Europe, but given the size of the Southern Indian Ocean this can be considered to be small).  It was now feasible to start analysing aerial imagery to try and identify debris (given that there was nowhere for the plane to land, on the 24th March Malaysian officials announced that it was beyond reasonable doubt that the plane was lost after ditching in the Southern Indian Ocean). Trawling around to find out what satellites were used was harder than i thought it would be.  Below is a summary of what i found:

  • GAOFEN-1 – a high-resolution optical sensor run by CNSA (Chinese National Space Administration) which was launched in April 2013. Gaofen-1 is equipped with a 2 metre resolution CCD (Charge-Coupled Device), an 8 metre resolution multi-spectral scanner and 16 meter resolution wide-field multi-spectral imager. It is difficult to tell which sensor produced the image below, but from the resolution it looks like it was the 8m res multi-spectral scanner.
chinese satellite

Chinese satellite image of possible debris – Pic from The Guardian/Reuters

  • A French satellite operated by Airbus Defense and Space spotted 122 objects in a cluster. The objects were up to 23m in length and in a cluster. (image released by MOSTI). Airbus Defense and space have a host of satellites run through their Astrium including EnviSAT, CryoSAT, Copernicus, ELISA and Helios 2.
French

Airbus Defence Image

  • Australian AP-3C Orion – Orion aircraft were deployed to likely search areas and scanned the area.  It is likely that the crew were using a combination of electronic surveillance system and just their eyes. This might seem like old-school, but it is an effective method of verification as trained operators can discount or confirm sightings from remote sensing. The aircraft has a long-range and can fly low making it ideal for searching.

Ocean Currents

Why has it taken so long to refine the search area?  Well there are lots of satellites, but only a few of them would have had suitable sensors on-board. Data is collected and beamed back to a receiving centre. The raw data will most probably have to be processed before it can be used for anything.  This takes time.  The search area may well have been narrowed to a chunk of the southern Indian Ocean, but this still represents a huge area, not dissimilar to the size of Western Europe.  Processing and analysing data for such a large area is not easy and will rely on a degree of automation followed by humba verification.

The southern Ocean is a wild place with frequent storms. We can see from above the at optical sensors have been used and these will be unable to penetrate cloud cover. Scientists would have to wait for the satellite to pass over the same area to try and get a better, cloud-free image. The repeat cycle may be anything from 1 day to 10 days or more.

Then you add in the ocean currents.  Anything object floating in the ocean will not be static and could drift by 10′s of kilometres a day. Given that the plane is likely to have crashed 15 days previously, debris could be 100′s of kilometers from the crash site. That is, if it has not already broken up and sunk.  But we can at least model the ocean currents and estimate the potential dispersal of the debris.  The NY Times have some excellent visualisations of both the currents and the wave heights in the southern Indian Ocean during March.  These have been produced by the National Oceanic and Atmospheric Administration and the National Centers for Environmental Prediction through remote sensing data, in-situ data (buoys) and models.  While never 100% accurate, they provide an indication and convey the uncertainty involved in determining a search area.

Locating flight recorders

Once a search area has been identified, the searchers are able to deploy listening devices which locate “pings” emitted by Flight 370′s black box. This is achieved by towing a listening device (TLP-25) back and forth across a wide area.  Pings should be received periodically and the position and strength of these should triangulate the position of the black box. But the sea floor is not flat in this area. It is around 4500m deep with mountains up to 2500m high.  We actually know very little about remote ocean sea beds.  We have limited data collected by ships and most representations come from spaceborne remote sensing data. These are not very accurate and may “miss” large structures (1-2km high) such as seamounts. There is nice overview of ocean mapping on the BBC website.

The difficulties of retrieving debris from deep, remote oceans was highlighted by the search for French Airlines flight 447.  In this case, both black box transmitters failed.

A Chinese ship detected a ping on the 5th April and a day later an Australian ship detected a ping.  But the pings were quite far apart.  The Australian ships detection seemed more consistent and stronger and this was backed up by more detections in the same area on the 8th. It is a slow process, but each detection should help reduce the uncertainty.  The question is, will the batteries in the transponders last much longer?  They are already at the limit of what is expected so time is running out.

Remote Sensing Critical

It is clear that remote sensing technology has been critical in every stage of the search for Flight 370.  It will continue to be so until the plane is found.  It has been used effectively to narrow search areas and discount blind alleys. It is also interesting to note how associated data has been used in ways that it was not intended to locate the plane and praise should be given to the Inmarsat scientists who came up with a novel solution when data and information was scarce.

Articles:

  • The search for Malaysian Airlines Flight 370 – a great article in the New York Times that focuses on the remote sensing data that is being used now that search teams have identified a “likely” crash site in the Southern Indian Ocean.
  • Wikipedia – a growing resource for information about Flight 370
  • Wikipedia – French Airways flight 447
  • NY Times – nice collection of visualisations of ocean conditions in the search area

SUNCAT Redevelopment Feedback Survey Report

We made a preview version of the new service available at UKSG in early April and just closed this off a few weeks ago. During this period we put up an online survey and encouraged emails to the EDINA helpdesk to ask our users for their views on the functionality and ease of use of the redeveloped service. A full report is now available on our website.

Overall, the response was very positive and any significant issues which were raised, EDINA were already aware of and are currently busy trying to resolve for the next release, due this autumn. Over 70% of the respondents reported that the preview service was better than the current service and we hope to improve this figure further as we iron out the glitches.

“Very bold design, clear and simple. A great improvement. Will encourage use of the service.”

We are happy that the vast majority of respondents found that the basic search facility was easy to use, 89%, and that the newly introduced post-search filters were found to be useful to 83% of respondents. Respondents reported that both the pre and post search filters would be good tools for reducing large result sets and filtering out holdings which might not be of interest, e.g. electronic or print holdings.

“Having the basic phrases in the first drop down box allows an easy search but also one that can still be refined without having to go to advanced search. Limiting locations and institutions is useful, especially for us as a public library as we know a number of locations will not lend so we can limit to those that do.”

Some respondents reported issues related to the display of the results list and the advanced search and work is on-going to resolve these. We were already aware that the Exact Title search was a little too exact – requiring exact matching on punctuation and letter case! Further, the relevancy ranking was not always working as expected, so these are another two areas we are currently concentrating on getting optimised.

The feedback received has been extremely useful in confirming areas we had concerns about and so helping us to prioritise immediate fixes for the next release, while also highlighting other interesting suggestions we can investigate for future releases.

SUNCAT Redevelopment: Focus on Post Search Filters

This is the second in a series of blog posts highlighting some of the new features which will be available in the redeveloped SUNCAT. In this post we are going to focus on how the post search filters will improve the new service, enabling the use of faceted browsing to lead users to the most relevant journal titles and holdings for their needs.

Post search filters will be available from the search results display, down the left hand side of the results list, and will provide similar functionality as you will be familiar with via other discovery services and many commercial shopping and travel websites.

At this point we plan to include the following facets:

  • Institution (the institutions holding the journals returned in the current results list)
  • Format (includes electronic and print formats)
  • Subject
  • Author (organisation associated with journals where relevant)
  • Publisher
  • Year (first published)
  • Place of publication
  • Language
Each filter will also display the number of records in the results set associated with it. The five most common filters from each result set will display for most of the facets, with the option to view decreasingly common filters also available. However, in the case of the institutions facet, filters will be displayed alphabetically to allow you to skim down the list for particular institutions, and in the year facet, filters will be displayed in reverse chronological order.

You will be able to combine filters to drill right down to journal titles matching quite specific criteria, but you will then also be able to simply remove filters to widen your results back up again.

An interesting side effect of displaying the filters is that it is now possible to easily see the variation present in the records supplied to SUNCAT, which goes a long way to explaining why not all the records which we would like to match together do! Part of the next step of the development work will be to try and amalgamate some of the filters so that they display in a more manageable and helpful fashion. For example combining year filters containing no numerical characters, such as “||||”, “uuuu” etc. into one “Unknown” filter. Also publication place filters such as “london”, “London.”, “London,” etc. into one filter for “London”.

You can see the faceted browsing in action on the preview service which will be available until Monday 13th May, so please take a look and let us have your comments via the online survey or by email to edina@ed.ac.uk

The following screenshot shows the filters on the preview service:

Feedback on the New Look SUNCAT

As we posted a few weeks ago a preview of the new look SUNCAT service is available at http://test.suncat.ac.uk and a huge thank you to all who have sent us comments so far.

We’ve now put up a short survey and would be very grateful for your input which will help to guide the ongoing development of the service. If you prefer you can also let us have your comments via the EDINA helpdesk at edina@ed.ac.uk

We would also encourage you to circulate details of the preview service and the survey as widely as possible.

Both the survey and the preview will only be available until Monday 13th May, but development work will still be continuing and so we will have more posts to keep you up to date with progress.

Please take a few minutes to let us know what you think!

Preview the New Look SUNCAT

Good morning from UKSG 2013! As promised last month the new look SUNCAT is now available for you to preview at http://test.suncat.ac.uk

The current service will continue as the primary service for the moment, but we would encourage you to take a look, experiment and let us have your feedback on the new service as soon as you get a chance.

The new service is still in development, so there are still issues to be resolved with keeping it up-to-date, searching and relevancy ranking, but to give you a flavour, the new service:

Integrates information from the website and the actual service more closely, with the search box, a map of our Contributing Libraries and a feed of our latest news all on the homepage

• Enables searching limited to a specific library or selection of libraries: you will only see the serials’ holdings of libraries you have specified in your search

• Enables searching limited to a specific location or selection of locations, all the way from Inverness to Exeter: again you will only see the serials’ holdings in locations you have specified in your search.

• Plus you can combine library and location limits to suit your individual search requirements

• Enables searching limited to electronic or print serials only: helpful if you only want to see what might be immediately available online or conversely if want to see only material likely to be available via Inter-Library Loan.

• Enables filtering of search results by library, location, subject, organisation, publisher, date etc.

• Enables more sophisticated searching via the Advanced Search option

• Allows you to choose how many records you want to display on your search results screen

• Provides an alphabetical list of all the libraries holding a particular serial title

• Allows you to quickly see if holdings are print or electronic

• Opens up further information about each library, including directions, contact details, British Library code and when it was last updated in SUNCAT

If you are attending UKSG, please come along to the EDINA stand 26 and we will be happy to demo the service for you and answer your questions in person. However, you can also let us know what you think or if you have any questions or suggestions via the EDINA helpdesk at edina@ed.ac.uk

We hope you like what you see!

JISC MediaHub Embeddable Search Widget – Coming to your website soon!

We are very pleased to announce a brand new embeddable JISC MediaHub Search Widget!

The widget lets you add a search box for the JISC MediaHub service to any website you wish. Which means you could include the search box  on your website, blog, on your institutional portal or VLE, etc.

What does it look like? 

It looks like the search box shown in the top right-hand corner of JISC MediaHub. There are two styles of widget: one has a white background (like JISC MediaHub itself), the other has a black background providing an alternative option to fit with your website design.  A working version is shown below:

What does it do?

The widget allows you to begin a search of JISC MediaHub. Type in your search terms, tick what format you would like the results to appear in, and hit “return”. A new tab or new window will open up with your search results shown – you can then access more detailed filtering of results. Clicking on the “advanced search” link in the widget also triggers a new window to open, taking you directly to our advanced search page.

Where Can I Use the JISC MediaHub Search Widget?

The widget can be used anywhere on the web. You will be prompted to log in from the search results page so this widget can appear on any public or internal institutional pages, virtual learning environments, library blogs, etc.

Where Can I Find the Widget’s Embed Code?

You can access the JISC MediaHub Search Widget embed code from the About page within the JISC MediaHub service here: http://jiscmediahub.ac.uk/about#widget.

How Do I Embed the Widget?

The widget uses a very compact iFrame embed code, so for most web pages you should just be able to paste this code into the HTML and it will just work.

For pages where iFrames have been disabled, we would recommend instead using a JISC MediaHub logo or text link to connect your readers with the service. Full guidelines and logos for linking to JISC MediaHub are available on the JISC MediaHub About Page.

How Can I Comment or Provide Feedback on the JISC MediaHub Search Widget?

Please either leave a comment here on the blog, email the helpdesk (edina@ed.ac.uk) or use the Contact Us page to send us your feedback.

Talk to us about JISC 06/11

Glad to hear that Unlock has been cited in the JISC 06/11 “eContent Capital” call for proposals.

The Unlock team would be very happy to help anyone fit a beneficial use of Unlock into their project proposal. This could feature the Unlock Places place-name and feature search; and/or the Unlock Text geoparser service which extracts place-names from text and tries to find their locations.

One could use Unlock Text to create Linked Data links to geonames.org or Ordnance Survey Open Data. Or use Unlock Places to find the locations of postcodes; or find places within a given county or constituency…

Please drop an email jo.walsh@ed.ac.uk or look up metazool on Skype or Twitter to chat about how Unlock fits with your proposal for JISC 06/11 …

Testing Unlock 3: new API, new features, soon even documentation

This week we are public testing version 3 of Unlock – a fairly deep rewrite including a new simpler API and some more geometrical query functions (searching inside shapes, searching using a buffer). New data – providing a search across Natural Earth Data, returning shapes for countries, regions, etc worldwide. So at last we can use Natural Earth for search, and link it up to geonames point data for countries. We also have an upgraded version of the Edinburgh Geoparser so have date and event information as well as place-name text mining, in Unlock Text.

The new search work is now on our replicated server at Appleton Tower and in a week or two we’ll switch the main unlock.edina.ac.uk over to the new version (keeping the old API supported indefinitely too). Here are notes/links from Joe Vernon. If you do any testing or experimentation with this we’d be very interested to hear how you got on. Note you can add ‘format=json‘ to any of these links to get javascript-useful results, ‘format=txt‘ to get a csv, etc.

‘GENERIC’ SEARCHING

http://geoxwalk-at.edina.ac.uk/ws/search?name=sheffield

http://geoxwalk-at.edina.ac.uk/ws/search?name=wales&featureType=european

http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&name=Marriott&minx=-79&maxx=-78&miny=36&maxy=37&operator=within

NATURAL EARTH GAZETTEER

http://geoxwalk-at.edina.ac.uk/ws/search?name=lake&gazetteer=naturalearth&country=canada

DISTANCE BETWEEN TWO FEATURES

Distance between Edinburgh and Glasgow (by feature ID):

http://geoxwalk-at.edina.ac.uk/ws/distanceBetween?idA=14131223&idB=11153386

SEARCHING WITHIN A FEATURE – ‘SPATIAL MASK’

United Kingdom’s feature ID is: 14127855

Searching for ‘Washington’s within the United Kingdom…

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855

Also, note the difference between searching for within the bounding box of the UK, or adding the ‘realSpatial‘ parameter, which uses the polygon of the feature concerned.

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855format=txt&maxRows=100&realSpatial=no

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855&format=txt&maxRows=100&realSpatial=yes

In this case, it picks up entries in Ireland if using the bounding box rather than the UK’s footprint.

SPATIAL SEARCHING WITH A BUFFER

8 hotels around the Royal Mile
http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&minx=-3.2&maxx=-3.19&miny=55.94&maxy=55.95&operator=within

75 within 2km
http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&minx=-3.2&maxx=-3.19&miny=55.94&maxy=55.95&operator=within&buffer=2000

FOOTPRINTS & POSTCODES

…should still be there:
http://geoxwalk-at.edina.ac.uk/ws/footprintLookup?identifier=14131223
http://geoxwalk-at.edina.ac.uk/ws/postCodeSearch?postCode=eh91pr

IMPLICIT COUNTRY SEARCHING

http://geoxwalk-at.edina.ac.uk/ws/search?format=txt&gazetteer=geonames&featureType=populated place&name=louth
vs
http://geoxwalk-at.edina.ac.uk/ws/search?format=txt&gazetteer=geonames&featureType=populated place&name=louth, uk

TIME BOUNDED SEARCH (still in development)

http://geoxwalk-at.edina.ac.uk/ws/search?name=edinburgh&startYear=2000&endYear=2009

http://geoxwalk-at.edina.ac.uk/ws/search?name=edinburgh&startYear=2000&endYear=2010

Very happy with all this, bringing the Unlock service up to offering something usefully distinctive again, trying to restrain myself from saying (“if X was so easy why don’t we do Y?”)

Unlock in use

It would be great to hear from people about how they are using the Unlock place search services. So you’re encouraged to contact us and tell us how you’re making use of Unlock and what you want out of the service.
screenshots from Molly, Georeferencer
Here are some of the projects and services we’ve heard about that are making interesting use of Unlock in research applications.

The Molly project based at University of Oxford provides an open source mobile location portal service designed for campuses. Molly uses some Cloudmade services and employs Unlock for postcode searching.

Georeferencer.org uses Unlock Places to search old maps. The service is used by National Library of Scotland Map Library and other national libraries in Europe.
More on the use of Unlock Places by georeferencer.org.

CASOS at CMU has been experimenting the Unlock Text service to geolocate social network information.

The Open Fieldwork project has been georeferencing educational resources: “In exploring how we could dynamically position links to fieldwork OER on a map, based on the location where the fieldwork takes place, one approach might be to resolve a position from the resource description or text in the resource. The OF project tried out the EDINA Unlock service – it looks like it could be very useful.”

We had several interesting entries to 2010′s dev8d developer challenge using Unlock:

Embedded GIS-lite Reporting Widget:
Duncan Davidson, Informatics Ventures, University of Edinburgh
“Adding data tables to content management systems and spreadsheet software packages is a fairly simple process, but statistics are easier to understand when the data is visual. Our widget takes geographic data – in this instance data on Scottish councils – passes it through EDINA’s API and then produces coordinates which are mapped onto Google. The end result is an annotated map which makes the data easier to access.”

Geoprints, which also works with the Yahoo Placemaker API, by
Marcus Ramsden at Southampton University.
“Geoprints is a plugin for EPrints. You can upload a pdf, Word document or Powerpoint file, and it will extract the plain text and send it to the EDINA API. GeoPrints uses the API will pull out the locations from that data and send it to the database. Those locations will then be plotted onto a map, which is a better interface for exploring documents.”

Point data in mashups: moving away from pushpins in maps:
Aidan Slingsby, City University London
“Displaying point data as density estimation services, chi surfaces and ‘tagmaps’. Using British placenames classified by generic form and linguistic origin, accessed through the Unlock Places API.”

The dev8d programme for 2011 is being finalised at the moment and should be published soon; the event this year runs over two days, and should definitely be worth attending for developers working in, or near, education and research.