Fieldtrip GB – Mapserver 6.2 Mask Layers

By Fiona Hemsley-Flint (GIS Engineer)

Whilst developing the background mapping for the Fieldtrip GB App, it became clear that there was going to have to be some cartographic compromises between urban and rural areas at larger scales; Since we were restricted to using OS Open products, we had a choice between Streetview and Vector Map District (VMD) – Streetview works nicely in urban environments, but not so much in rural areas, where VMD works best  (with the addition of some nice EDINA–crafted relief mapping) . This contrast can be seen in images below.

streetview1VectorMapDistrict1

Streetview (L) and Vector Map District (R) maps in an urban area.

streetview2VectorMapDistrict2

Streetview (L) and Vector Map District (R) maps in a rural area.

In an off-the-cuff comment, Ben set me a challenge – “It would be good if we could have the Streetview maps in urban areas, and VMD maps in rural areas “.

I laughed.

Since these products are continuous over the whole of the country, I didn’t see how we could have two different maps showing at the same time.

Then, because I like a challenge, I thought about it some more and found that the newer versions of MapServer (from 6.2) support something called “Mask Layersâ€�  – where one layer is only displayed in places where it intersects another layer.

I realised if I could define something that constitutes an ‘Urban’ area, then I could create a mask layer of these, which could then be used to only display the Streetview mapping in those areas, and all other areas could display a different map – in this case Vector Map District (we used the beta product although are currently updating to the latest version).

I used the Strategi ‘Large Urban Areas’ classification as my means of defining an ‘Urban’ area – with a buffer to take into account suburbia and differences in scale between Strategi and Streetview products.

The resulting set of layers (simplified!) looks a bit like this:

masking example

Using Mask layers in MapServer 6.2 to display only certain parts of a raster image.

Although this doesn’t necessarily look very pretty in the borders between the two products, I feel that the overall result meets the challenge – in urban areas it is now possible to view street names and building details, and in rural areas, contours and other topographic features are more visible. This hopefully provides a flexibility  for users on different types of field trips to successfully implement the background mapping.

Here’s a snippet of the mapfile showing the implementation of the masking, in case you’re really keen…

#VMD_layer(s) defined before mask

LAYER

…..

END

#Streetview mask layer

LAYER

NAME “Streetview_Mask”

METADATA

…..

END

#Data comes from a shapefile (polygons of urban areas only):

DATA “streetview_mask”

TYPE POLYGON

STATUS OFF

END

#Streetview

LAYER

NAME “Streetview”

METADATA

…..

END

#Data is a series of tiff files, location stored in a tileindex

TYPE Raster

STATUS off

TILEINDEX “streetview.shp”

TILEITEM “Location”

#*****The important bit – setting the mask for the layer*****

MASK “Streetview_Mask”

POSTLABELCACHE TRUE

END

Hacking Mapcache with ImageMagick

To generate tiles for the map stack used by FieldTrip GB we are using 4 Mapserver instances deployed to an OpenStack private cloud. This means we can get all our tiles generated relatively quickly using inexpensive commodity hardware. A problem we have is that the resulting PNG tile images look beautiful but are way too big for users to download to their mobile device in any quantity. So we looked to using Mapserver’s built in JPEG format but our cartographers were not happy with the results. One of my colleagues came up with the bright idea of using ImageMagick to compress the PNG to JPEG instead, and the result (using 75% compression) was much better. We can use the ImageMagick command line  with the following syntax:

convert_png_to_jpeg_delete_png.sh

for var in "$@"
do
echo "converting $var to jpg";
convert $var -quality 75 `echo $var | tr '.png' '.jpg'`;
# rm $var
done

and pipe this script using xargs to traverse an existing cache with the PNG generated tiles.

find . -name '*.png' -print0 |  xargs -0 -P4 ../convert_png_to_jpeg_delete_png.sh

So the cartographers finally relented and we now have much smaller files to download to devices. The only problem is that the script to run the ImageMagick convert takes for ever to run ( well all right – 2 days). It’s not because ImageMagick is slow at compression – it’s super fast. It’s just that the IO overhead involved is huge as we are iterating over  16 million inodes. So our plan of scaling up commodity hardware (4 CPU virtual machine) is failing. A solution is to do the jpeg conversion at the same time as the tile caching – this way you are only dealing with one tile at the point you are writing to the cache – so there is much less overhead.

So it’s time to hack some of the Mapcache code and get ImageMagic to add the above compression just after it writes the PNG to the cache.

This just involves editing a single source file found in the lib directory of the Mapcache source distribution  ( mapcache-master/lib/cache_disk.c ). I’m assuming below you have already downloaded and compiled Mapcache and also have downloaded ImageMagick packages including the devel package.

First of all include the ImageMagick header file

#include  <wand/magick_wand.h>

Then locate the method  _mapcache_cache_disk_set. This is the method where Mapcache actually writes the image tile to disk.

First we add some variables and an Exception macro at the top of the method.

MagickWand *m_wand = NULL ;
MagickBooleanType status;

#define ThrowWandException(wand) \
{ \
char \
*description; \
\
ExceptionType \
severity; \
\
description=MagickGetException(wand,&severity); \
(void) fprintf(stderr,”%s %s %lu %s\n”,GetMagickModule(),description); \
description=(char *) MagickRelinquishMemory(description); \
exit(-1); \
}

Add then right at the end of the method we add the MagickWand equivalent of the convert command line shown above. The compression code is highlighted

if(ret != APR_SUCCESS) {
ctx->set_error(ctx, 500, "failed to close file %s:%s",filename, apr_strerror(ret,errmsg,120));
return; /* we could not create the file */
}

// *******ImageMagick code here ********

ctx->log(ctx, MAPCACHE_INFO, “filename for tile: %s”, filename);
MagickWandGenesis() ;
m_wand=NewMagickWand() ;
status=MagickReadImage(m_wand,filename);
if (status == MagickFalse)
ThrowWandException(m_wand);
// MagickSetImageFormat(m_wand, ‘JPG’) ;
char newfilename[200];
strcpy(newfilename, filename) ;
int blen = strlen(newfilename) ;
if(blen > 3)
{

newfilename[blen-3]=’j’ ;
newfilename[blen-2]=’p’ ;
newfilename[blen-1]=’g’ ;
MagickSetImageCompression(m_wand, JPEGCompression) ;
MagickSetCompressionQuality(m_wand, 75 ) ;
ctx->log(ctx, MAPCACHE_INFO, “filename for new image: %s”, newfilename);
MagickWriteImage(m_wand, newfilename ) ;
}
/* Clean up */
if(m_wand)m_wand = DestroyMagickWand(m_wand);
MagickWandTerminus();

And that’s it. Now just the simple matter of working how to compile it, link it etc.

After a lot of hmm’ing and ah-ha’ing (and reinstalling ImageMagick to more recent version using excellent advice from here ) it meant making the following changes to the Makefile.inc in mapcache src root dir.

INCLUDES=-I../include $(CURL_CFLAGS) $(PNG_INC) $(JPEG_INC) $(TIFF_INC) $(GEOTIFF_INC) $(APR_INC) $(APU_INC) $(PCRE_CFLAGS) $(SQLITE_INC) $(PIXMAN_INC) $(BDB_INC) $(TC_INC) -I/usr/include/ImageMagick
LIBS=$(CURL_LIBS) $(PNG_LIB) $(JPEG_LIB) $(APR_LIBS) $(APU_LIBS) $(PCRE_LIBS) $(SQLITE_LIB) -lMagickWand -lMagickCore $(PIXMAN_LIB) $(TIFF_LIB) $(GEOTIFF_LIB) $(MAPSERVER_LIB) $(BDB_LIB) $(TC_LIB)

Then run make as usual to compile Mapcache and you’re done! The listing below shows the output and difference in compression:

ls -l MyCache/00/000/000/000/000/000/
total 176
-rw-r–r–. 1 root root 4794 Jul 23 13:56 000.jpg
-rw-r–r–. 1 root root 21740 Jul 23 13:56 000.png
-rw-r–r–. 1 root root 2396 Jul 23 13:56 001.jpg
-rw-r–r–. 1 root root 9134 Jul 23 13:56 001.png
-rw-r–r–. 1 root root 8822 Jul 23 13:56 002.jpg
-rw-r–r–. 1 root root 46637 Jul 23 13:56 002.png
-rw-r–r–. 1 root root 8284 Jul 23 13:56 003.jpg
-rw-r–r–. 1 root root 45852 Jul 23 13:56 003.png
-rw-r–r–. 1 root root 755 Jul 23 13:55 004.jpg
-rw-r–r–. 1 root root 2652 Jul 23 13:55 004.png

original PNG tile

converted to JPEG at 75% compression