FOSS4G – a developers review – part4

The 4th and final EDINA developers eye view of FOSS4G 2013.  This one is from Tim Urwin who is the Digimap Service Manager.  Tim has been working at EDINA pretty much from the start of it’s internet mapping adventure and has seen software and toolkits come and go.

Who are you?
My name is Tim and I’m the senior GI Engineer at EDINA in charge of the Data Team and I’m the Operation Service Manager for Digimap. My interest in attending FOSS4G centred around three key components of the Digimap Service: WMS Servers, WMTS options and Database, although I delegated most of the latter to Mike due timetable clashes.

What did you hope to get out of the event?
My aim was to catch up on the latest state and future options of current software used by EDINA services and to find out more about the various open source WMTS options available.

Top 3 things? (Ed – no trains Tim!)

 

  • Chris Tucker’s MapStory keynote was inspirational and well-presented and it is certainly a site I’ll be tracking to see where it heads.
  • Ben Henning’s key note on think before you act for cartography was quite thought provoking.
  • Paul Ramsey’s PostGIS Frenzy talk was as funny as it was informative, and I only caught the re-run. Lots of good information combined with useful tips. (Ed- Paul’s a star, there wasn’t enough room for everyone first time round so he kindly offered to repeat the talk)
  • Honourable mention must go to the Festival of the Spoke Nerd – very, very funny
What will you investigate further?
MapCache and MapProxy WMTS software to replace our existing tile caching option and catch up with all the presentations I couldn’t attend due to timetable clashes. (Ed – remember that all the talks (hopefully) will be available on the FOSS4G YouTube channel when we get them sorted and uploaded)
One closing thought is that it was heartening to see that despite all the professional headaches that Digimap has caused me over the years that our approach to and delivery of the service has been validated as several leading data supply agencies have very similar service architectures. Built with Open Source software at the core, although there with some proprietary components for certain tasks. The primary differences being in WMS and caching software options, although they’ll be closer aligned once we upgrade to a more modern tile caching platform. Now only if we could also have their hardware – they have significantly larger number of servers :)

Oh and as I wasn’t allowed this in my Top 3 – seeing 45108 running again after 16 years of hard work by its custodian group.

FOSS4G – a developers review part 2

Photo – Addy Pope

This is the second part of EDINA’s developer review of FOSS4G 2013.  This time it is Mike Gale who will be providing his opinion on what was presented.

Who are you:

Michael Gale – GIS Engineer / Member of the EDINA’s Data Team. My job is to essentially deal with the vast quantities of GIS data we utilise at EDINA. I translate, modify, split and twist the data we receive into types and formats that our services such as Digimap can then offer to our users. I heavily use the Swiss army knife – GIS command line tools of GDAL/OGR and additionally Safe FME, Shell Scripting, Python & PostGIS.

What you hoped to get out of the event?

To discover the latest and greatest ways to utilise the tools I already use. I was keen to evaluate what advances and benefits PostGIS 2.0 could offer – particularly with 3D data, LiDAR point clouds & pgRouting. Additionally I wanted to discover new ways of integrating Python into my workflows.
Top 3 things you saw at the event (not the food or beer….)

(1) Chris Tucker keynote – MapStory.org

MapStory.org is a new website that empowers a global user community to organise knowledge about the world spatially and temporally. It is essentially a social media platform where people can crowd source geospatial data and create “MapStories” with spatio-temporally enabled narratives. The best way to figure out what that all means is to check out the website!!

(2) Cartopy & Iris – Open Source Python Tools For Analysis and Visualisation – Dr Edward Campbell (Met Office)

Cartopy is a new python mapping library for the transformation and visualisation of geospatial vector and raster data. The library offers the ability for point, line, polygon and image transformations between projections and a way to visualise data with only a few snippets of python code. Iris is a python library that specifically deals with analysing and visualising meteorological and oceanographic datasets, particularly 3D and temporal data.

(3) LiDAR in PostgreSQL with Pointcloud – Paul Ramsey

PostGIS support for LiDAR data has been non-existent until now. Paul Ramsey has created a new spatial data type for PostGIS 2.0 that now offers the ability to import huge amounts of point cloud data, and additionally analyse the information with several new postgis functions. Pretty impressive.

(4) I’ll throw a comedy one in as well: “Up all night to get Mapping”: http://www.youtube.com/watch?v=_EEVYHUQlkU

Editors note: view at your own (ears) risk.

1 thing that you are definitely going to investigate further

The IRIS and Cartopy Python libraries.

Thanks Mike.  I hope to add another couple of review next week.  My overview, with links to as many reviews as i could find, can be found HERE

 

FOSS4G – a developers review part 1

Panos – Edina Developer

As well as being part of the Local organising committee, EDINA sent a number of developers to FOSS4G.  In the first of a series of guest posts we find out what the developers thought of the event and what they will be following up.

First up is Panos. Panos graduated with an MSc in GIS from Edinburgh University 3 years ago and has been working for the geo team at EDINA since.

Who am I and in what I am interested in?

I am Panos and work in EDINA as software engineer. I maintain a service called UK Data Service Support and I am working on a project an EU FP7 project called COBWEB which focuses on mobile GIS development and sensor data. As you can see from my background I am mainly interested on mobile development, GEOSERVER and sensor data frameworks. I managed to attend most of the presentations that have to do with these topics.

What was I expecting?

I was expecting to see some more alternative mobile development solutions from the ones we use here in EDINA (Openlayers, jquery mobile, phonegap) and some more applications on sensor web. I am quite happy that I discovered some new software such as 52North and the fact that other people developed their mobile app with a similar way to us. So, let’s take them one by one:

Mobile development:

  • Most of the projects focused around OpenLayers mobile/leaflet/jquery mobile/sencha touch and phonegap.  EDINA have used a similar blend of technologies in our mobile app, Fieldtip GB. There were many similarities in how they designed their apps, the feedback they received from users, the workflow they followed and the problems they had with touch events on different devices.
  • The outcome is that they would take a similar approach but they would perhaps try an alternative to phonegap.
  • One smart approach they had on visualizing lots of vector data on a small screen was to use MapProxy to merge raster and vector data to deliver a WMS.  The touch event of the users then searches for the closest feature and the app asks for the corresponding WFS returning information for the correct feature.

GEOSERVER:

  • Geoserver 2.4.0 has some new interesting features. The most interesting for me is a monitoring system for checking what kind of users are using the app and what kind of data they are accessing. It’s a nice solution for monitoring the use you have on GEOSERVER and there is even a GUI for it.  I plan to investigate how we might implement this in the UK Data Service Support.

Sensor Web:

  • Unfortunately, the work that has taken place on this is quite limited. It’s mainly about hydrology.
  • North52 (https://github.com/52North/) seems like a promising framework that can adapt to all different scenarios about sensor data. Some people have used for covering the scenario of if someone should go for hiking by considering factors such as birch pollen, meteorology and air quality. This may be useful for COBWEB.

Following up:

I’ll definitely try to investigate the new GEOSERVER functionality and 52North framework in order to see how I can benefit from them in my new projects. I’ll keep you posted with my progress. I would also like to say that these 3 presentations that I watched are not the only one that I found interesting. There are more that are equally interesting such as leaflet, geonode, ZOO project, cartoDB, iris project and cartopy.  You should be able to watch these through ELOGeo in a couple of weeks.

FOSS4G – after the dust settles

Olympics of Geo?

FOSS4G 2013 has been and gone. What can i say, it seemed to go well. It is is hard to tell when you are so involved in organising an event as you notice all the little things that didn’t quite go as intended and you tend to be trying to do a hundred things at the same time. Archaeogeek has written an excellent post about the event from an organisers point of view so i wont repeat that here. Highlights. There are so many to choose from, seeing 200 people make, and then wear, Robin Hood hat at the ice breaker or seeing delegates sitting cross-legged on the floor when all the seats and stairs had already been filled. But here are my top 3:

  • OpenLayers 3 showcase – OpenLayers is awesome and version 3 looks like it will reinforce OpenLayers place as one of the best open source web mapping out there.  New features include map rotation with tilt features “in the pipeline”.
  • QGIS 2.0 Dufour – Quantum GIS is dead, long live QGIS.  The latest version is slicker and packs more features than before. Download it now and start exploring it. You can see some of the cool stuff in this slideshare.
  • Paul Ramsey – the man behind PostGIS did more talks than anyone else, re-running one that was so popular that we couldn’t squeeze everyone in.  His closing Plenary was a call for us to become “open source citizens”.  Certainly one of the most inspirational presentations i have seen in a long time.
  • OK, so this makes it a top 4, but it is a worthy inclusion.  Arnulf Christl winning the Sol Katz award.  Long overdue and a true hero of the OSGeo world.

and the winner is…….

So what’s next?  Well, I hope to post a number of short “reviews” written by people who attended the event which will have their own top 3 lists.  We, the organisers, hope to make all the talks available through EloGeo so that anyone can see what was presented at FOSS4G.  In the meantime, you can scroll through the 4500 tweets from the event if you have the stamina.

FOSS4G 2014 will be held in Portland. Looking forward to it already, just have to work out how to get there……

Other write-ups of the event:

A big thanks to everyone who made this possible, all the LOC team, you know who you are, the volunteers and the staff at the East Midlands Conference Centre. 

FOSS4G 2013 – 5 reasons you should attend

FOSS4G is the annual conference for anyone interested in Free and Open Source Software 4 Geospatial.  FOSS4G 2013 will be held in Nottingham between the 17th and 21st September. So what makes FOSS4G so important and why should you attend?

  1. Network – FOSS4G is the biggest gathering of developers and users of open geospatial software.  There will be over 700 people at the conference. This includes the lead developers on some of the larger open source projects such as OpenLayers and QGIS.
  2. Learn – You’ll learn a lot in a very short period of time.  No matter what your knowledge of open source geo from beginner to expert coder/developer you will learn something new at FOSS4G.  There are workshops for all levels that you can sign up to.
  3. Inspiration – You will be inspired by some of the major names in GIS and data analysis. The list of keynote speakers includes Paul Ramsey (co-founder of PostGIS), Kate Chapman (Acting Director of humanitarian team at OpenStreetMap) and Ben Hennig (Worldmapper Project).  For a full list of Keynote speakers, please refer to the FOSS4G keynote page.
  4. Double the fun – Visit AGI GeoCommunity’13 at the same time. Yes, that’s right FOSS4G and AGI GeoCommunity are happening in the same venue on the same week. This was no accident. GeoCommunity is a great event and the FOSS4G organisers wanted to bring the two audiences together. GeoCommunity’13 runs from the 16th to the 18th September.
  5. Can you afford to miss it?  – What does this mean?  Well, the conference package is quite reasonable given the number and diversity of talks on offer.  £165 for a day pass or £435 for the whole event (3 days and the FOSS4G Gala Night).  FOSS4G was last in Europe back in 2010 and it might not be back until 2017 as it moves between continents. So, if you are based in Europe attending FOSS4G might not be as easy for a number of years.

So, there are 5 pretty good reasons to attend.  I am sure there are many other reasons to come along.  To find out everything that will be going on at FOSS4G please look at the conference website and follow the event on twitter through the #FOSS4G hashtag.

FOSS4G 2013 takes place between the 17th – 21st September 2013 and will be held at the East Midlands Conference Centre, which is situated on The University of Nottingham campus. 

OSGIS 2012 – Day 2

OSGeo_logo

The second day of OSGIS 2012 saw a full day of short paper presentations and a couple of workshops.  The day started with a keynote from Prof. David Martin, University of Southampton.  David is  Director of the ESRC Census Programme and his talk looked at the data that will come out of the 2011 census. It also discussed the future of census programs in the UK.  The take-away points for David’s talk included:

  • Lots of new fields such as “do you intend to remain in UK?”
  • 16th July 2012 – age/sex distribution LADs released
  • Nov 2012 – release to the OA level which will be of interest for Geographers
  • Spring 2013 – multivariate stats and some new stuff like time dependant location data which will be interesting for disaster management/response and answering questions such as “who is where/when?”
  • Access to longitudinal data and data about individuals will still be restricted to secure labs

David made some interesting points including crediting the CDU in Manchester for making the census data far easier to access and analyse.  The data is in excel format and has the crucial area codes which we geographers love.  

He showed some analysis of work place zones which modifies the census units based on where people are during the day (work place) which should make disaster planning more efficient. It was also noted, light-heartedly, that this could be used to determine where to locate your burger van during the week.  

Next up was Ian James, Technical Architect for the Ordnance Survey. Ian’s presentation was on how the OS was embracing the open source opportunity.  The OS now use open source solutions for internal activity and client-facing interfaces.  It took a while to convince the whole organisation that open source solutions were more than capable of handling large and valuable datasets.  It is now clear that some open source solutions are in fact better than their proprietary counterparts.  However, Ian stressed that open source was not free.  There is always a cost associated with software, with open source solutions there is no up-front licence fee, but there is cost associated with training users and administrators or buying 3rd party support.

After coffee, the conference split into parallel strands, I switched rooms to catch certain presentations and my write up will reflect this.  You should be able to watch the presentations on the OSGIS 2012 website.

Matt Walker, Astun Technology demonstrated the open source system Loader, a simple GML loader written in Python that makes use of OGR 1.8.   Matt showed us how Astun were providing TMS/WMS for various clients and how they managed to run it all through Amazon web services.  Top tips from Matt included:

  • Amazon web services are great, you can even have fail-over instances, but be sure to manage your system or risk running up bills quite quickly
  • Use PGDump to increase postgres load times (4x quicker)
  • MapProxy rocks
  • UbuntuGIS makes life easy
Next up was Fernando Gonzalez who presented the possibilities of Collaborative geoprocessing with GGL2.  GGL2 is an evolution of GGL which was a scripting application for GIS.  GGL2 makes scripts much simpler, fewer lines of code makes it easier us humans to read.  GGL2 is available as a plugin for gvSIG and QGIS.  If you want to find out more about GGL2 the look at gearscape.org
EDINA’s Sandy Buchanan gave a demonstration of Cartogrammer which is an online cartogrammer application. It allows users to upload shapefiles and KML files and then create cartograms.  This is very neat and really does remove the technical barrier in producing interesting info-graphics.  The service makes us of ScapeToad and is available as an online service, a widget and an api which can be called from your own website.  We will let you know when it goes live.
Anthony Scott of Sustain gave an excellent presentation on the work he has been doing for MapAction.  If you don’t know what MapAction is or what they do, they provide mapping and GIS services areas that have suffered natural and humanitarian disasters.  Infrastructure is important if aid is to be delivered and this requires knowledge of the what is on the ground at the time, and in some cases, what is left. Take 5 minutes to look at their website and if it sounds like something you would like to support, hit the big red donate button.
Jo Cook, Astun Technology, looked at how you might use open source software and open data to do something useful.  She looked at taking GeoRSS feeds from sites such as NHS Choices and PoliceUK to extract location specific information, link it with other open data and then make this publicly available. According to Jo, you can do quite a lot with very basic python scripting. The last slide of Jo’s presentation has a list of useful resources, seek it out when it is made available on the OSGIS website.
The best presentation prize went to Ken Arroyo Ohori, TU, Delft. Ken demonstrated some code that he had written which fixed overlapping and topologically incorrect polygons.  PPREPAIR looks brilliant and is available in GitHub.  Ken plans to make it into a QGIS plugin when he has time, i think this will be really useful.  Nice aspects include being able to set a “trusted” polygon class which would be assumed to be correct if two polygons intersected.  Ken demonstrated ppgrepair’s capabilities fixing polygons along the Spanish/Portuguese Border. Because two mapping agencies have mapped the border independently, when you combine the two datasets you get horrible overlaps. Ken’s presentation was clear and informative and his ppreapir really does look useful.
The event finished with Steve Feldman of KnowWhere Consulting.  Steve has been working in GIS for many years, but is, by his own admission, not a techie.  He approaches the subject with a business hat on and it is useful to hear this perspective.   Steve reiterated the point that Open Source was not Free software.  It is commercial software with no massive up front lumps sum and no long term contract. You can pay for implementation and support.  You can fund developments that you want, rather than functionality you dont need. Steve suggested that the “Free” was a confusing term, but a member of the audience suggested that Free also related to not being tied to a contract or service provider.  You can opt in and out as you wish.
Feldman

FOSS4G 2013

Steve then took the opportunity to officially launch FOSS4G 2013, which will be held in Nottingham in September next year.  This event will be huge and is definitely one to put in the calendar now and make sure you get along to it.  There will be over 500 delegates from around the world all focused on doing more with open source geospatial tools.  In fact, better than that, volunteer to help at the event.  The local organising committee needs extra people to help make FOSS4G 2013 a success. If you want to help, pledge your support on the pledge page and someone from the loc will get back to you.
 So, another great event.  Thanks to Suchith, Jeremy and their team for making it happen.  OSGIS will not happen in 2013, but FOSS4G will more than make up for it.

 

OSGIS 2012 – Part 1

OSGIS is now in it’s 4th year and has really become one of the main events that brings together users and developers of open source geospatial tools.  The nice thing about OSGIS is that it attracts an even spread of delegates from the commercial, the public and the academic sector.  This cross-sector mixing is, in my opinion, very healthy for the geospatial sector.

Jeremy Morley at OSGIS 2012

Day One of OSGIS 2012 featured workshops where users could get hands on experience of software under the guidance of expert tutors.  The morning session saw an introduction to GeoNetwork, a geospatial catalog service, and and overview of the OSM-GBproject which has made in-roads in topologically correcting OSM data. These workshops are integral to the ethos of OSGIS as they are designed to empower both novices exploring the potential of open source software and the expert users looking to refine their skillset and discuss technical problems.

After lunch I opted to attend the session looking at the educational use of OSGeo Live. For those of you that have not heard of OSGeo Live, it is a bootable DVD which allows you to investigate OSGeo software without having to instal and configure it on your own computer. This is an excellent way to explore the functionality offered by the numerous packages such as uDIG, QGIS, Openlayers and GeoServer.

Barend Kobben of ITC in the Netherlands outlined how OSGeo Live was used in teaching and why it solved many issues.  Increasingly universities are assuming that students will want to use their own laptops rather than relying on open access labs.  This means that the tutors have no control of what computer students will use to complete course work. Supporting multiple operating systems and system configurations is virtually impossible.  Using OSGeo Live removes the necessity to configure systems. Just put the DVD in the computer,reboot and go. Well, most of the time.  Not all computers are set to boot from the DVD drive, users would have to access their BIOS to set their boot sequence. Running the OSGeo Live from a USB stick or on a virtual machine potentially reduces the hassle of dealing with boot sequences.

Jeremy Morley of Nottingham Geospatial Institute echoed Barend’s experiences.  Jeremy had used Oracle VirtualBox and then taken snapshots on a Storage Area Network (SAN) to ensure that students work was backed up.  This looked promising but didn’t scale when 20+ students tried to access the SAN. Unfortunately, the snapshots were tied to a single machine ID, students would have to use the same machine eachtime they accessed their work.  This was not an acceptable solution. Jeremy switched to running OSGeo live from a USBstick and this was an improvement, but again, was not without it’s own issues.  The FAT32 format reduced the usable space on the 8 Gb drives to just under 5Gb and cheaper USB sticks were prone to burning out and failing.  But, the solution was acceptable and Jeremy was able to deliver the course to to the students. Next years course will be refined in light of discovering these issues.

As an aside, Jeremy flagged the potential need for more Geographic Information Systems courses to support the wide and varied technical applications which require in depth knowledge of computing.  There has been a trend of Geographic Information Science courses over recent years where students are taught how to apply GIS to solve scientific problems.  However the maintenance of systems and interfaces which allow data to be published and interacted with is important but forms the base of only a handful of course at the moment.

During the discussion of these two papers, it ws suggested that a Cloud Space to run GIS would be useful, if you could configure what tools you wanted.  A figure of 4Gb was suggested as a reasonable workspace.  This would allow users to analyse data but would have to carefully manage their space.  You could always “do with more space” but you could teach with about 4Gb of space.

The first day closed with a presentation by Jiri Kadlec of Aalto University, Finland. Jiri, by his own admission , was new to open source GIS and set himself the challenge of managing and translating data in differing coordinate systems. Projections and coordinate systems are always a challenge.  The theory is that you should be able to get from any “system” o any other “system” by passing through WGS1984.  Jiri found QGIS to be the bet of the bunch but it was not perfect.  Juiri also put together a neat little projection comparison tool which many of the audience thought would be an excellent aid when teaching students about projections, or for showing representations of land areas in different projections.

The day finished with a drinks reception and a visit to some of the sights of Nottingham. Fortunately, some of the best historic sites just so happen to be pubs and the Jubilee Campus is the site of the old Raleigh bicycle factory.

Historic Site

 

Visiting the English Place Name Survey

I was in Nottingham for OSGIS at the Centre for Geospatial Sciences on Tuesday; skipped out between lunch and coffee break to visit the English Place Name Survey in the same leafy campus.

A card file at EPNS

A card file at EPNS

Met with Paul Cavill, who dropped me right in to the heart of the operation – stacks of index cards in shoe boxes. Each major name has a set of annotation cards, describing different related names and their associations and sources – which range from Victorian maps to Anglo-Saxon chronicles.

The editing process takes the card sets and turns them right into print-ready manuscript. The manuscript then has very consistent layout conventions – capitalisation, indentation. This is going to make our work of structure mining a lot easier.

Another bonus I wasn’t expecting was the presence of OSGB grid references for all the major names. The task of making links becomes a snap – I was imagining a lot of iterative guesswork based on clustering and closeness to names in other sources. (There are four Waltons in the UK in geonames, dozens in the EPNS).

On this basis I reckon the entity recognition will be a breeze, LTG will hardly have to stretch their muscles, which means we can ask them to work on grammars and machine learning recognisers for parts of other related archives within the scope of CHALICE.

Pic_0622_026And we would have freedom in the EDINA team’s time to do more – specifically to look at using the National Map Library of Scotland’s map rectifier tools to correlate the gazetteer with detailed line-drawn maps  also created by the late H. D. G. Foxall. Digitisations of these maps live in the Shropshire Records Office. We must talk with them about their plans (the Records Office holds copyright in the scans).

The eye-opener for me was the index of sources, or rather the bibliography. Each placename variant is marked with a few letters identifying the source of the name. So the index itself provides a key to old maps and gazetteers and archival records. To use Ant Beck’s phrase the EPNS looks like a “decoupled synthesis” of placename knowledge in all these sources. If we extract its structure, we are recoupling the synthesis and the sources, and now know where to look next to go text mining and digitising.

Pic_0622_024So we have the Shropshire Hundreds as a great place to start, as this is where the EPNS are working on now and the volumes are “born digital”. Back at CDDA, Paul Ell has some of the very earliest volumes digitised, and if we find a sample from the middle, we can produce grammar rules that we can be pretty confident will extract the right structure from the whole set, when the time comes to digitise and publish the entire 80+ volume, and growing, set.

But now i’m fascinated by the use of the EPNS derived data as a concordance to so many associated archives documenting historic social patterns. Towards the end of our chat Paul Cavill was speculating about reconstructing Anglo-Saxon England by means of text mining and georeferencing archives – we could provide a reference map to help archaeologists understand what they are finding, or even help them focus on where to look for interesting archaeology.

Paul had been visited by the time-travelling Mormons digitising everything a couple of weeks previously, and will hopefully offer an introduction – i would really, really like to meet them.