Raster Management & Optimization¶
jmckenna at gatewaygeomatics.com
- Original Author:
- Last Updated:
Choose the right raster format for your needs¶
The best/optimal data sources for MapServer in terms of speed-of-display is GeoTIFF for rasters.
Here are some other points to consider:
The GeoTIFF image format is the fastest to “decipher”, but once you get beyond a certain point, the disk reading may become slow enough to make it worthwhile to consider other image formats.
For GeoTIFFs larger than 1 GB, ECW images tend to render faster than GeoTIFFs, since decompressing the data (CPU and RAM) is faster than reading the uncompressed data (disk). The downside is that ECW is not open-source, and the licensing is often prohibitive. MrSID is another compressed image format that can be used.
JPEG2000 is a very slow image format, as is JPEG.
Spend time to review GDAL’s associated driver page for your chosen format¶
This is a critical step, as MapServer relies on GDAL for raster data access. Each driver (GDAL format) has its own set of abilities and switches. Find your raster format and review its options here.
Connect to your data through GDAL¶
For data management in MapServer, this should always be one of your first steps. Sometimes desktop GIS programs will display a format in a certain way but your data might not display in MapServer; checking how GDAL reads your raster file, will help you manage and understand the data. Here is an example using gdalinfo to read and return various important metadata regarding a specific raster, in this case a MrSID raster:
gdalinfo ortho_1-1_hc_s_mn163_2019_1.sid Driver: MrSID/Multi-resolution Seamless Image Database (MrSID) Files: ortho_1-1_hc_s_mn163_2019_1.sid ortho_1-1_hc_s_mn163_2019_1.sid.aux.xml Size is 50249, 116781 Coordinate System is: PROJCS["NAD83 / UTM zone 15N", GEOGCS["NAD83", DATUM["North_American_Datum_1983", SPHEROID["GRS 1980",6378137,298.257222101, AUTHORITY["EPSG","7019"]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6269"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4269"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",-93], PARAMETER["scale_factor",0.9996], PARAMETER["false_easting",500000], PARAMETER["false_northing",0], UNIT["metre",1, AUTHORITY["EPSG","9001"]], AXIS["Easting",EAST], AXIS["Northing",NORTH], AUTHORITY["EPSG","26915"]] Origin = (494792.400000000023283,5018068.799999999813735) Pixel Size = (0.600000000000000,-0.600000000000000) Metadata: GEOTIFF_CHAR__GeogAngularUnitsGeoKey=Angular_Degree GEOTIFF_CHAR__GTModelTypeGeoKey=ModelTypeProjected GEOTIFF_CHAR__GTRasterTypeGeoKey=RasterPixelIsArea GEOTIFF_CHAR__ProjectedCSTypeGeoKey=PCS_NAD83_UTM_zone_15N GEOTIFF_CHAR__ProjLinearUnitsGeoKey=Linear_Meter GEOTIFF_NUM__1024__GTModelTypeGeoKey=1 GEOTIFF_NUM__1025__GTRasterTypeGeoKey=1 GEOTIFF_NUM__1026__GTCitationGeoKey=NAD83 / UTM zone 15N GEOTIFF_NUM__2049__GeogCitationGeoKey=NAD83 GEOTIFF_NUM__2054__GeogAngularUnitsGeoKey=9102 GEOTIFF_NUM__2062__GeogTOWGS84GeoKey=0.000000,0.000000,0.000000 GEOTIFF_NUM__3072__ProjectedCSTypeGeoKey=26915 GEOTIFF_NUM__3076__ProjLinearUnitsGeoKey=9001 IMAGE__ENCODING_APPLICATION=GeoExpress 10.0.1.5035 IMAGE__FORMAT=MrSID/MG3 IMAGE__INPUT_FILE_SIZE=30104685840.000000 IMAGE__INPUT_FORMAT=GeoTIFF IMAGE__LTI_ESDK_VERSION=184.108.40.20635.Bob_5011_br win64-vc15/Release_md IMAGE__MODIFICATIONS=COMPRESSED CROPPED EMBEDDED REORDERED-BANDS MOSAICKED REPROJECTED IMAGE__PRINT_DENSITY_UNIT=in IMAGE__PRINT_X_DENSITY=200.000000 IMAGE__PRINT_Y_DENSITY=200.000000 VERSION=MG3 Image Structure Metadata: INTERLEAVE=PIXEL Corner Coordinates: Upper Left ( 494792.400, 5018068.800) ( 93d 3'59.18"W, 45d18'57.98"N) Lower Left ( 494792.400, 4948000.200) ( 93d 3'56.57"W, 44d41' 7.27"N) Upper Right ( 524941.800, 5018068.800) ( 92d40'54.44"W, 45d18'56.45"N) Lower Right ( 524941.800, 4948000.200) ( 92d41' 6.94"W, 44d41' 5.78"N) Center ( 509867.100, 4983034.500) ( 92d52'29.30"W, 45d 0' 2.48"N) Band 1 Block=1024x128 Type=Byte, ColorInterp=Red Min=0.000 Max=228.000 Minimum=0.000, Maximum=228.000, Mean=86.853, StdDev=53.485 Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8 Metadata: STATISTICS_APPROXIMATE=YES STATISTICS_MAXIMUM=228 STATISTICS_MEAN=86.853391304348 STATISTICS_MINIMUM=0 STATISTICS_STDDEV=53.485489960597 STATISTICS_VALID_PERCENT=100 Band 2 Block=1024x128 Type=Byte, ColorInterp=Green Min=0.000 Max=224.000 Minimum=0.000, Maximum=224.000, Mean=81.011, StdDev=45.552 Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8 Metadata: STATISTICS_APPROXIMATE=YES STATISTICS_MAXIMUM=224 STATISTICS_MEAN=81.010782608696 STATISTICS_MINIMUM=0 STATISTICS_STDDEV=45.552372785005 STATISTICS_VALID_PERCENT=100 Band 3 Block=1024x128 Type=Byte, ColorInterp=Blue Min=0.000 Max=211.000 Minimum=0.000, Maximum=211.000, Mean=90.109, StdDev=48.708 Overviews: 25125x58391, 12563x29196, 6282x14598, 3141x7299, 1571x3650, 786x1825, 393x913, 197x457, 99x229, 50x115, 25x58, 13x29, 7x15, 4x8 Metadata: STATISTICS_APPROXIMATE=YES STATISTICS_MAXIMUM=211 STATISTICS_MEAN=90.109217391304 STATISTICS_MINIMUM=0 STATISTICS_STDDEV=48.70795205406 STATISTICS_VALID_PERCENT=100
You can use the extent values returned from gdalinfo (using Lower Left and Upper Right values) to paste into your mapfile’s EXTENT parameter. You can also notice in that summary the PROJCS/AUTHORITY line, which states that this data is currently in the EPSG:26915 projection.
For Windows users, MS4W includes the gdalinfo utility, and all utilities mentioned here.
GeoTIFF supports the creation of “overviews” within the file, which is basically a downsampled version of the raster data suitable for use at lower resolutions. Use the gdaladdo program to add overviews to a GeoTIFF, and MapServer (via GDAL) will automagically choose which downsampled layer to use. Note that overviews significantly increase the disk space required by a GeoTIFF, and in some cases the extra disk reading may offset the performance gained by MapServer not having to resample the image. You’ll just have to try it for yourself and see how it works.
Tileindexes and Internal Tiling¶
Tiling is mostly effective for cases where one commonly requests only a very small area of the image.
A tileindex is how one creates an on-the-fly mosaic from many rasters. This is described in the Tile Indexes MapServer document. That document describes common cases where a tileindex makes sense. In particular, if you have a very large raster and most requests are for a very small spatial area within it, you may want to consider slicing it and tileindexing it.
As an alternative to slicing and mosaicing, TIFFs do support a concept of internal tiling. Like a tileindex, this is mostly effective when the requests are for a small portion of the raster. Internal tiling is done by adding “-co TILED=YES” to gdal_translate, e.g.:
gdal_translate -co TILED=YES original.tif tiled.tif
Learn & Review the various GDAL utilities to manage your rasters¶
GDAL raster commandline utilities are very powerful, and the more processing that you can do offline, the faster and easier it will be for MapServer to display your raster. For example, you might need a colorful ‘shaded relief’ from a raw DEM raster, in which case you could use the gdaldem utility, and then point your MapServer layer to that shaded relief raster. Or you might want to reproject all of your rasters to the desired output/display projection beforehand, by using the gdalwarp utility. Review all of the available GDAL raster utilities here.
Handling your raster LAYERS in the mapfile¶
Please review the notes in the document Mapfile Tuning & Management.
Some remote WMS/OGC services (as WMS actually returns an image from a request sent to a remote server) can be slow or unreliable. Optionally you can use a monitoring service, such as GeoHealthCheck, to keep track of the service’s reliability for you.
You can also consider when the remote WMS layer or remote service should be used. For example, there may be a different WMS server (or a different set of imagery, or even vector outline maps) suitable for drawing the countries or states to orient the user. You could then have the WMS layer come on at a certain scale, or have the layer always available but turned off so the user can choose when to turn it on.