Raster layers

After successfully implementing loading and rendering of a new vector layer in MLT format inside a PMTiles container, it was time to do the same with the raster layers. The idea was to analyze the 4 raster layers bundled with the game (land, relief, population and buildup), decide to use a newer of different dataset (or not), and use some existing industry tool to convert the dataset to a PMTiles container with the tiles in some standard image format. But none of these goals ended up being possible, at all.

Land raster

This was the one that gave me the least trouble. I has already made a note a long time ago to replace the current land raster texture with something both prettier and more functional. The category-palette of the current texture was of no use for the game, and in turn gives some strange colors choices. To replace it I went with the Natural Earth Cross-blended Hypsometric textures, without relief, since I wanted to preserve a separate relief layer. It looks very good combined with the existing relief (tho it needs some adjusting still):

This was the only raster layer I could directly produce with industry tools, but after I loaded it in-game, it highlighted an big problem with the game coordinate projections (more on this later).

Relief raster

For relief the game uses a single channel, 16bpc texture. I didn’t want to change this, but it turns out this kind of format is almost never used for web-oriented tools and formats, even in the mapping ecosystem. And the projection issue from the land raster was still bothering me, so I decided to fall back to my own tool, to control the projection I was using. This significantly increased the time I needed to prepare the raster layers, compared to the vector layer. I adapted my tool to PMTiles and to the new tile definition of 1.19, and used the original DEM dataset (I see no point in improving this layer or adding it more detail). The previous screenshot already uses this layer.

Population and buildup raster

And this where it all exploded and delayed me for at least week. Again I decided to just keep using the existing dataset and my own tool, to not make my life harder, but even after eliminating this variable it was still a pain. The buildup data source is in Mollweide projection and now I needed to properly reproject it, and this is a 10m layer. I won’t bore readers with the eldritch incantations required to perform this task in under 24h using GDAL command line tools, but believe me, it’s not easy.

In the end I spent significantly more time working on the raster layers than on the vector layer, the opposite of how it was in the past map updates.

Fixing the coordinate projection

This game launched in a very rough alpha quality state, and that extended to some internal design choices I still pay the price for today. Geo coordinate transform and projection was kind of YOLO in the first versions (until the “mercator fix” release). That was fixed, but there was still 2 big remnants of the old code: mixed mercator and lon-lat coordinate use, with useless back-and-forth conversions in a lot of places, and the unique to this game “warping projection” used in the map view code, very visible as you get near the poles.

The various imprecisions and code complexity of the useless conversions were bad enough, but the “warping” projection actually made the game incompatible with regular mercator tiles! It is only visible at low zooms, but it’s very noticeable. I didn’t notice it before because when working with vector tiles, every point is transformed with this incorrect formula, but for rasters only the corners are, and they don’t match the vectors. If I wanted to make the game compatible with the web maps ecosystem as I planned, I had to fix this. So I deleted all rendering, UI and view geo transform code and replaced it with correct code, and now the game renders the map just like any other web mercator renderer, and vectors match rasters:

The second task was to vanish all use of lon-lat coordinates in the game, except for interfacing with external formats like GeoJSON export. This was a massive task, taking me longer than a week, and it involved thousands of edits in the code. But now the code is clean, with no useless conversions, everything being expressed in mercator with 64 bit precision. For rendering this is converted down to 32 bit by always picking an on-screen origin point (could be the tile origin, or the center of the screen), and making coordinates relative to that point, which preserves precision. And finally, all on-screen map coordinate transform is done with proper transformation matrices, rather than the earlier ad-hoc formulas I was doing on a misguided attempt to “optimize” (GPUs are very fast).

Map rotation

Since on-screen map rendering is now mediated by generic matrices, implementing rotation is a lot less effort than before, and I did so:

You can now rotate the map using shift-scroll wheel (or horizontal scroll wheel if you have one). It uses acceleration so tiny movements translate into tiny angles, but a quick motion is a fast half circle for example. A huge pet peeve of mine for map viewers which support rotation is going back to 0 degrees, of requiring some widget in a corner or clicking some axis label or similar. I’ve mapped middle click just to this function, and it makes a very agile flow of rotating as needed then immediately going north-up.

Map rotation combined with the lon-lat vanishment means this is an ongoing task, I will keep finding dark corners in some UI where it’s not fully adapted and just assumes axes are always screen-parallel, for awhile.

Map source system

All these efforts made me realize I was trying very hard to be compatible to an industry standard I was not really going to integrate with, because the game core was still designed around a set of bundled map layers. These map layers were now based on industry formats and could be replaced, but this was an Explorer.exe task, not an integrated feature. So it was time to redesign the map system around multiple configurable map sources.

First the internal concept of a “map tile” needed to be changed. Formerly a tile had N layers from the different sources, and conceptually you only needed one tile per tile position. In 1.19 a tile contains the data of only one map source, and the map renderer will consider every source for every tile position, and render as many tiles as needed.

Then the concept of “source” was introduced. A source is a location for a map layer with a single data type and can have its own zoom range and even geo range, independent of all other sources. At the moment there’s no limitation to how many map sources can be added to the game.

To expose this system as a player feature, there’s now a map sources editor in the map panel of the UI:

You can create, delete, reorder and toggle the rendering of your custom map sources in this menu. The default sources are also in the menu, and you cannot delete them, but you can reorder and disable their rendering if desired. For vector map sources that have global metadata (like PMTiles containers) a listing of all the independent vector layers in the source is also displayed, and can be individually toggled, but not deleted or reordered:

The graph area under the listing is LERP filter for various color manipulations. The 4 filters in the screenshot are just an idea, for now the only implemented is the alpha modulator. It is a 2D graph because it allows to simulatenously specify a minimum and maximun value as a function of the current (fractional) zoom level, rather than a singular fixed value.

This is not optional or just me being a turbonerd, this is very much needed to make some map layers usable. The main example is the land-like vector layers, which cover huge areas at some zooms, covering the now much improved land raster texture, which is preferrable. At the same time the land raster becomes too low res when zoomed in, and it’s better tune it down a lot. So the game will specify zoom-alpha filters for these layers. In the industry this is also done, but using map styles. I will look into also supporting this for vector map styles, but if it’s too complex, at least it is already doable at the layer level.

HTTP map sources

All of this work on 1.19 culminated on something I had in mind from the start: the game should support online map data. It should never depend on it, but for extra rendering? Bring it on. So in addition to the local PMTiles container file, I also implemented a new HTTP directory map source.

This source supports both direct raw URLs with the usual “{z}/{y}/{x}” encodings, and TileJSON files. For example, want a satellite layer? You could try the non-commercial layer from eox.at (https://tiles.maps.eox.at/wmts/1.0.0/s2cloudless-2020_3857/default/g/{z}/{y}/{x}.jpg):

But you are not limited to free sources. You can also make an account with a paid map provider and access very high quality data. These providers offer authenticated TileJSON endpoints for use in web maps, and the game is compatible with them. For example here’s Mapbox satellite (aerial) layer:

The layer filter system allows to mix and match data as needed using alpha compositing and ordering, just like Google Maps:

And finally a test I always wanted to try. Since this provider has such a high resolution (in some places), it becomes possible to use the photographed railways as a reference to check the in-game elements dimensions and precision. And it works:

The in-game default rail has standard gauge. In the photo the top rail is iberian gauge and it does not match, but the bottom one is standard gauge, and it matches perfectly. This means the combined systems of texture projection and map coordinate transforms are perfectly matched and in turn perfectly match real life and third party data.

To be clear this is still not a GIS application. It supports exactly one map projection (Web Mercator), with a limited set of raster formats (PNG, JPG, maybe WebP later) and vector formats (MLT, maybe MVT later), and limited set of containers (local PMTiles or HTTP raw URLs with de-facto path encoding or TileJSON endopoints). No multiple projections, no GeoTIFF, no WMTS XML or OGC sources, no SHP, no KML, etc. It is a map renderer for a subset of formats used in web mapping, and that’s it.

Feature-complete but a lot of work left

I don’t plan to add more features to 1.19 but that does not mean it’s done. I’ve pushed hard because I wanted to see the source system, filters and finally the HTTP source capability working together as a MVP project, but now I need to rewind a bit and restart work on many things. For example none of the screenshots show map labels because 1.19 cannot display map labels at all. Or station population is 0 because population sampling is commented out and needs to be updated. There’s many medium size features like these that need more work and attention.