Geospatial API’s and other tools for developers

Since the release of Open data back in 2010, and through the rise of mobiles and the App, the use of spatial data by developers has increased significantly.  Spatial data is no longer the preserve of the geospatial scientists, it is used by mainstream developers who see the value of location. I this post we will look at some of the “free” data and tools that you can use.

Data

OpenstreamAPI:

The Digimap OpenStream service provides access to a Web Map Service (WMS) offering Ordnance Survey OpenData products, including GB Overview, Miniscale, 1:250000 Colour Raster, VectorMap District Raster and OS Streetview.

EDINA aims to provide the latest version of OS OpenData via the OpenStream service. Once registered, you can use the Digimap OpenStream API to do things like:

  • Mashups, combining OS Opendata with maps and data from other sources.
  • adding OS Opendata to Google Earth.
  • Embed maps in your website.
  • provide OS mapping in your own applications.
  • provide OS mapping in a Desktop GIS project.

Openstream viewed in Google Earth

If you want to find out how to use Openstream please check out the folowing useful links:

Advantages – Free. Easy to use and provides regularly updated OS mapping. Available in OSGB and WGS1984 projections
Disadvantages – Not as easy to use in WGS84 as google maps, no satellite imagery available through API.

Google

Most people will be familiar with Google maps, specially if they have a smart phone. What can you say. Global cover (almost) and improving all the time. Packed with detail in urban areas. I suppose the weakness is in rural areas where detail may be sparse. Google maps is enough for many applications but if you may not be as accurate as the data from a national mapping agency, such as the OS.
Google maps is generally available for free in most cases. You have to pay if you use lots of data or if you want the data behind a login.

OpenStreetMap

OpenStreetMap is global community mapping project that wants to make a freely available map of the World. 10 years ago this would have been a crazy idea but today it is almost a reality. It suffers in the same way as Google maps in that it is great in urban areas and less good in rural areas. But, if the detail is not there, just sign up to the cause and add the detail yourself. Your edits will then be available for your, and everyone else’s, map.

Bing

Bing have a mapping stack that you can access through an API in a similar way to Google. Bing was quick to offer a bird’s eye view option for its satellite data and also has OS 1:50 000 mapping available as well as offering detailed street mapping for London. It is worth looking at the data available through the Bing Maps API as it is different to that offered by Google and may better suit your needs.

British Geological Survey

The BGS offer a number of Web Mapping Services (WMS) for developers and GUS users. These include 1:50 000 scale mapping, 1:625 000 mapping (on and offshore), 1:625 000 Hydrogeology, Soil data and contaminated land data. Full information on the use of these datasets and the GetCapabilities statements needed to access them is provided on the BGS website.

API Services

An Application Programming Interface (API) is a protocol intended to be used as an interface by software components to communicate with each other. An API is a library that may include specifications for routines, data structures, object classes, and variables. Thee are a number of API’s that give developers to spatial data and tools to create interactive web maps.

Unlock Text

Unlock Text is a powerful geoparser that can search text hosted on the web in .txt or .html format for references to locations. These locations are then returned ready for use in your results page, web map or any other application.

Unlock

Unlock Text is a RESTful API, you can use it by first setting up a username and password using the POST operation. You can use the GET operation to find out information about this account such as the number of Text documents that have been parsed. POST and GET operations can be used to set up new documents to be parsed and to retrieve the urls of results pages created by the parser.

You will need to use cURL commands; or a REST Console or Client which are available as add-ons for Chrome and Firefox web browsers, to use Unlock Text.

Advantages: Free! There is not really anything else out there to compete with it, unless you pay!

Limitations: Not always a perfect set of results, but then no one else does it perfectly either.  Wikipedia currently blocks it!

Unlock Places

Unlock Places is a place search web service, its API helps developers to find locations for place names. You can also use it to convert postcodes into coordinates, look up electoral boundaries, and to find shapes that overlap or intersect one another.

Unlock Places can provide geographic information describing places – or other features – as points (latitude-longitude coordinates) or larger bounding boxes and more detailed shapes (where possible).

Unlock Places can be used in your web project, app or mobile information service to add a geospatial location on to users’ data.

Advantages: Free

Limitations: Some searches can be a bit slow

National Library of Scotland

NLS

The NLS has a fantastic api which offers access to Historic Maps of the UK. Thus service is great and the mapping is based on out-of-copyright Ordnance Survey maps, dating from the 1920s to the 1940s. Full details of how to implement the map on a webpage or on a mobile app are available on the NLS website.

Google

Google has a host of API’s that allow developers to tap into tools and content delivered by Google. On the maps side of things, there is:

  • Google Maps JavaScripts API v3 – It provides you with the framework to add a number of unique features and content to a Google Maps interface and embed Google Maps in your own web pages. Version 3 of this API is especially designed to be faster and more applicable to mobile devices, as well as traditional desktop browser applications. The API provides a number of utilities for manipulating maps (just like on the http://maps.google.com web page) and adding content to the map through a variety of services. The JavaScript Maps API V3 is a free service, available for any web site that is free to consumers.
  • Mobile API and SDK – Google offers an Google Maps for Android API and a Native iOS SDK for Apple.  Both will allow developers access to google map content and functions such as titl, 3D buildings and Streetview.
The Google maps api has a free use cap of 750k maps a month.  If you think you will exceed this, then you can buy capacity.

Advantages: Google is by far the most popular API for mapping. The public are generally aware of Google Maps and are familiar with both the controls and the content. Mobile Friendly.
Disadvantages: Just because it is the most popular doesn’t mean that it will suit your needs. In particular, the level of details is patchy and may not provide your users with the richness that they need.  For example, you wouldn’t head off into the hills with Google Maps as your main navigational aid. Not entirely free.

Bing Maps API

Bing Maps API offers developers access most of the mapping and satellite images that are available through their main map site.

Bing Maps API as it is different to that offered by Google and may better suit your needs. Did i mention that Bing also does directions? Well it does for car, walking and public transport. The Bing api is has a free option but this has a usage cap and remove the birds-eye view function.  Education/not for profit accounts are available and commercial organisations can sign up to a range of service packages.

Advantages: Alternative mapping stack to Google API and similar user experience. Imagery pretty new.
Disadvantages: User community not as large as Google’s, not free.

Here API

Not heard of Here Maps API? Well Nokia is behind the Here API and that is a good start. Long before Google got serious about delivering spatial data to mobile phones, Nokia were supplying their handsets with mapping pre-loaded. OviMaps, as they used to be known, were pretty good and Nokia have developed them into the Here API service. The mapping is subtly different to Google and uses a pastel pallet that is easy on the eye. In addition to the cartography, there is also relief shaded Digital Terrain Models (DTM) and Maps 3D which is a bit like Google Earth but not quite as easy to use. However, there is a neat 3D option if you have a suitable screen and glasses. Free accounts are capped at 1 million map views a month, slightly higher than the google limit.
Mobile friendly version is available
Advantages: Clean cartography. Looks different to the usual maps. Mobile friendly api
Disadvantages: User community even smaller than Bing maps. Maps 3D not as intuitive as it could be. DTM has artifacts in it that may be annoying. Aerial Imagery not as up-to-date as Bing/Google. Not entirely free.

Openlayers

OpenLayers is an open source JavaScript library for displaying map data in web browsers. It provides an API for building rich web-based geographic applications similar to Google Maps and Bing Maps. OpenLayers is used by the OpenStreetMap project for its slippy map interface which is very similar to Google maps in terms of usability. OpenLayers is used in many web map interfaces including EDINA’s Digimap services.

OpenLayers

The nice thing about OpenLayers is that the user community is both large and very active. This means that you can usually find the answer to the questions that you have when developing an interface. In addition, there are plenty of example maps which allow you to view the code and therefore learn how to implement solutions quite simply.
From Version 2.10, OpenLayers supported built in Navigation and TouchNavigation. Other features such as Map Dragging and Pinch Zoom are also supported. provider the mobile browser support touch events.

Leaflet 

Leaflet

Leaflet is a modern open-source JavaScript library for mobile-friendly interactive maps. It is very similar in style and function to the Openlayers API. It has slightly less features than the Opnelayers API, however this has the advantage of possibly making the API more straightforward. The other biggest, or smallest, advantage that Leaflet has is it’s size. The library is a svelte 64K (Openlayers is over 700k).

One of the trickier aspects of Leaflet is using your own local projection, such as OSGB, rather than the standard Web Mercator.  If you want to do this, then i would suggest reading this blog post which documents the process.

Modest Map

To be added

Map Quest

To be added

100 years after Scott

This week marks the 100 year anniversary of Scott’s ill-fated expedition to the South Pole. I would imagine that you all know who Scott was and what happened on his final attempt to reach the South Pole before Amundsen, if you need a quick refresher then take a look at the excellent article on the BGS website.

Image courtesy of The Scott Polar Research Institute – Image Ref – P2005/5/1704

To mark this anniversary i thought i would compile a list of Antarctic related geospatial resources.  Any excuse to delve back into polar science.

Data:

  • Sharegeo – there are a couple of datasets of interest on Sharegeo such as the 2012 inventory of glaciers and a Global Permafrost map.
  • World Glacier Monitoring Service (WGMS) – compiles the annual inventory of glacier mass balance and maintains a number of other ice related datasets. Not specifically Antarctica focused, but a good global resource.
  • RAMP – Radarsat Antarctic Mapping Project (RAMP) created a high-resolution DEM of Antarctica.  DEMs are available with a 1km, 400m or 200m cell posting and are provided as ARC/INFO and binary grids. The 1Km and 400m grids are also available in ASCII format. RAMP DEM was created and is hosted by the National Snow and Ice Data Center (NSIDC).
  • GLIMS – The GLIMS glacier database is a searchable database of glacier information. GLIMS makes it easier to find out the physical attributes of glaciers around the world.
  • Community Ice Sheet Model – This project, run by the University of Montana, pulls together data that could be used to model the Antarctic ice sheet. It also supply’s code that allows you to run scenario models on the present ice sheet. Data is generally in grid (NetCDF) format.
  • NOAA – NOAA Paleoclimatology gateway provides access to ice core data for Antarctica.  Ice cores are available for a range of locations across the continent and there are some pre-processed comparissons of the asynchorny of Antarctica and Greenland.
  • ESF Research – ESF provides access to data such as global surface seawater dimethylsulfide, biological data (bacterial biomass, bacterial productivity, and UV irradiance data.

Data/Metadata Discovery Portals:

  • National Snow and Ice Data Center – NSIDC is a huge resource and i could spend hours sifting through data.  There is little point trying to list everything NSIDC hold, or make available but highlights include RAMP (see above), searchable database of glacier photos, MODIS imagery and the GLIMS glacier database.
  • Australian Antarctic Data Centre: a massive resource that contains satellite images, maps, ecology, biology and marine data.  Licence restrictions vary depending on the data type and source. In terms of getting your hands on the data itself, the best section to head to is the Data Navigator.  You need to register an account but this is free and takes 2 minutes.
    Some of the other data sources, such as the satellite data is not available to download directly from the AADC, the search facility gives you the metadata that should point you at the data custodian.  I would say that the site could usefully provide links to the data custodians but at least for satellite imagery they don’t appear to do so.
  • British Antarctic Survey (BAS) – BAS have a metadata discovery system which allows ou to explore their data spatially, temporarily and by category.  There are a few datasets in the discovery service but most are just the metadata.
  • SCAR Geoscience Map Catalogue – Jointly hosted with BAS, the SCAR Geoscience metadata catalogue provides a pretty extensive listing of maps of Antarctica.  No preview of the the sheet is provided so this is really a starting point for maps searches as you at least find out what is available, when it was created and at what scale.
  • AMRC & AWS – hosted and maintained by the University of Wisconsin – Madison, this portal contains the latest meteorological and atmospheric observations as well as a selection of satellite images of Antarctica.
  • Cool Antarctica – a fairly comprehensive source of climate data for Antarctica. This site is run by an enthusiastic ex-BAS employee

The list is not exhaustive and I will have to leave it there for this week. I will keep adding to this list, if you find any relevant resources please feel free to add a comment pointing to the resource and i will add it o the list.

 

Image generated using http://kartograph.org/showcase/projections/#stereo

Exploring the Locator OS OpenData set

Fiona Hemsley-Flint had a good look at the OS Locator dataset which is available from the Ordnance Survey Open Data portal. I thought a summary of her findings might be of use to others thinking about how to use this dataset.

Overview

OS Locator contains a list of all the road names in UK, “derived from a number of Ordnance Survey datasets [Meridian2, Road database, Locality dataset, Boundary-Line]. These include the roads database which contains information on road names and road numbers and is the latest generation of Ordnance Survey’s sophisticated and highly detailed geographic data”. OS recommend viewing it on top of mid-scale datasets such as 1:10k & 1:25k Raster and streetview (which is freely available via OS opendata).

Geometries

Each feature is geo-referenced by a centre point and a bounding box (although some of the bboxes are actually line features where the road segment of the feature is horizontal or vertical).
OS Locator names shown on OS map
Figure 1. Multiple occurrences of Ferry Road, differentiated by their locality.

Attribution

The roads have a name and/or a classification, where the classification represents a road number, (e.g. ‘A1′ or ‘B1243′). They also have an associated settlement (town), locality, county/region and local authority; the latter two are derived from Boundary-Line, it is unclear what is used to form the ‘Locality dataset’. Locality and settlement are likely to be the most useful of these attributes when displaying result sets. For roads which cross locality boundaries, a point is assigned for each separate locality, therefore one road may have more than one point associated with it, distinguished by its locality.

Storage

851505 rows of data were added to a development server.
Multiple geometry columns have been added to take into account the different geometries available.
A ‘tsvector’ column has also been added to implement Postgres text search functionality. An example query might be:
select name, classification, locality, settlement from os.locator_nov_10 where search @@ to_tsquery(‘high & street & edinburgh’);

Which returns the following result set:

Name	Classification	Locality	settlement
CORSTORPHINE HIGH STREET		Se Corstorphine	EDINBURGH
HIGH STREET		Musselburgh Central	EDINBURGH
HIGH STREET		Musselburgh North	EDINBURGH
HIGH STREET		Holyrood	EDINBURGH
HIGH STREET	A199	Musselburgh North	EDINBURGH
HIGH STREET	A199	Musselburgh Central	EDINBURGH
NORTH HIGH STREET		Musselburgh North	EDINBURGH
NORTH HIGH STREET	A199	Musselburgh West	EDINBURGH
PORTOBELLO HIGH STREET	B6415	Milton	EDINBURGH
PORTOBELLO HIGH STREET	B6415	Portobello	EDINBURGH
NORTH HIGH STREET	A199	Musselburgh North	EDINBURGH

Overall, the dataset contains a comprehensive list of the roads names within the UK. Decisions will need to be made about how to treat multiple features that actually refer to the same real world road.

The main limitation of this dataset is that it can only be used to show the user the general location of a road – it can’t be used as a precise address gazetteer since it only provides street names with no knowledge of building numbers.

More on the use of Unlock Places by georeferencer.org

Some months back, Klokan Petr Pridal, who maintains OldMapsOnline.org and works with libraries and cartographic institutes across Europe, wrote with some questions about the Unlock Places service. We met at FOSS4G where I presented our work on the Chalice project and the Unlock services.
Petr writes about how Unlock is used in his applications, and what future requirements from the service may be:


It was great to meet you at FOSS4G in Barcelona and discuss with you
the progress related to Unlock and possible cooperation with
OldMapsOnline.org and usage in Georeferencer.org services.

As you have mentioned, the most important thing for us would be to
have in Unlock API/database the bounding boxes (or bounding polygons) for places as direct part of the JSON response.
We need that mostly for villages, towns and cities and for areas such
as districts or countries – all over the world. We need something like
“bounds” as provided by the Google geocoding API.

The second most important feature is to have the chance to install the
service in our servers
– especially in case you can’t provide
guarantees for it in a future.

It would be also great to have chance to improve the service for non-English languages, but right now the gazetteers and text processing is not primary target of our research.

In this moment the Unlock API is in use:

As a standard gazetteer search service to zoom the base maps to a place people type in the search box in our Georeferencer.org service – a
collaborative georeferencing online service for scanned historical
maps. It is in use by National Library of Scotland and a couple of other libraries.

Here’s an example map (you need to register first).

The uniqueness of Unlock is in openness of the license (primarily GeoNames.org CC-BY and also OS OpenData) and also so far very good availability of the online service (EDINA hardware and network?). We are missing the bounding box to be able to zoom our base maps to the correct area (determine the appropriate zoom level). Unlock API replaced Google Geocoder, which we can’t use, because we are displaying also non-google maps (such as Ordnance Survey OpenData) and we are potentially deriving data from the gazetteer database (the control points on the old maps), which is against Google TOS.

In the future we are keen to extend the gazetteer with alternative
historical toponyms
(which people can identify on georeferenced old
maps too), or participate on such work.

The other usage of Unlock API is:

As a metadata text analyzer, in a service such as our
http://geoparser.appspot.com/, where we automatically parse existing
library textual metadata to identify place names and locate the
described maps including automatic approximation of their spatial
coverage (by identifying map scale and physical size in the text and
doing a simple math on top of it). This service is in a prototype
phase only, we are using Yahoo Placemaker and I was testing Unlock Text API
with it too.

Here the huge advantage of Unlock would be primarily the possibility
to add custom gazetteers
(with Geonames as the default one), language detection (for example via Google Language API or otherwise) and also possibility to add into the workflow other tools, such as lemmatizator for particular language – the simplest available via hun/a/ispellu
database integration or via existing morphological rule-based software
such as:

The problem is that without returning the lemmatization of the text the geoparser is almost unusable in non-English languages – especially Slavic
one.

We are very glad for availability of your results and of the reliable
online services you provide. We can concentrate on the problems we
need to solve primarily (georeferencing, clipping, stitching and
presentation of old maps for later analysis) and use your results of
research as a component solving a problem we are touching and we have to practically solve somehow.”


Very glad that Petr wrote at such length about comprehensive use of Unlock. pushing the edges of what we are doing with the service.

We have some work in the pipeline adding bounding boxes for places worldwide by making Natural Earth Data searchable through Unlock Places. Natural Earth is a generalised dataset intended for use in cartography, but should also have quite a lot of re-use value for map search.

Connecting archives with linked geodata – Part II

This is part two of a blog starting with a presentation about the Chalice project and our aim to create a 1000-year place-name gazetteer, available as linked data, text-mined from volumes of the English Place Name Survey.

Something else i’ve been organising is a web service called Unlock; it offers a gazetteer search service that searches with, and returns, shapes rather than just points for place-names. It has its origins in a 2001 project called GeoCrossWalk, extracting shapes from MasterMap and other Ordnance Survey data sources and making them available under a research-only license in the UK, available to subscribers to EDINA’s Digimap service.

Now that so much open geodata is out there, Unlock now contains an open data place search service, indexing and interconnecting the different sources of shapes that match up to names. It has geonames and the OS Open Data sources in it, adding search of Natural Earth data in short order, looking at ways to enhance what others (Nominatim, LinkedGeoData) are already doing with search and re-use of OpenStreetmap data.

The gazetteer search service sits alongside a placename text mining service. However, the text mining service is tuned to contemporary text (American news sources), and a lot of that also has to do with data availability and sharing of models, sets of training data. The more interesting use cases are in archive mining, of semi-unusual, semi-structured sets of documents and records (parliamentary proceedings, or historical population reports, parish and council records). Anything that is recorded will yield data, *is* data, back to the earliest written records we have.


Place-names can provide a kind of universal key to interpreting the written record. Social organisation may change completely, but the land remembers, and place-names remain the same. Through the prism of place-names one can glimpse pre-history; not just what remains of those people wealthy enough to create *stuff* that lasted, but of everybody who otherwise vanished without trace.

The other reason I’m here at FOSS4G; to ask for help. We (the authors of the text mining tools at the Language Technology Group, colleagues at EDINA, smart funders at JISC) want to put together a proper open source distribution of the core components of our work, for others to customise, extend, and work with us on.

We could use advice – the Software Sustainability Institute is one place we are turning for advice on managing an open source release and, hopefully, community. OSS Watch supported us in structuring an open source business case.

Transition to a world that is open by default turns out to be more difficult than one would think. It’s hard to get many minds to look in the same direction at the same time. Maybe legacy problems, kludges either technical, or social, or even emotional, arise to mess things up when we try to act in the clear.

We could use practical advice on managing an open source release of our work to make it as self-sustaining as possible. In the short term; how best to structure a repository for collaboration, for branching and merging; where we should most usefully focus efforts at documentation; how to automate the process of testing to free up effort where it can be more creative; how to find the benefits in moving the process of working, from a closed to an open world.

The Chalice project has a sourceforge repository where we’ve been putting the code the EDINA team has been working on; this includes an evolution of Unlock’s web service API, and user interface / annotation code from Addressing History. We’re now working on the best way to synchronise work-in-progress with currently published, GPL-licensed components from LTG, more pieces of the pipeline making up the “Edinburgh geoparser” and other things…

OpenStream WMS service for ac.uk

A new web mapping service is now available that makes the Ordnance Survery Open datasets available through WMS. The service offered by EDINA is initially available to anyone with an “ac.uk” email and requires a simple email activated registration. Other than that, there are no restrictions on use. The website includes an example mapping app using OpenLayers and isntructions on how to use the service with Google Earth. We also created a mobile app using this datasource, as described in a previous post on the TouchMapLite interface.

the data products available are:

  • GB Overview
  • Miniscale
  • 1:250000 Colour Raster
  • Vector Map District Raster
  • OS Streetview

you can register for the service (initially ac.uk email addresses only) here.

If you would like to find out how you can use Digimap OpenStream with desktop GIS you should read the Go-Geo! Blog.