# Architecture How StaticMaps turns options into an image, and why each stage is built the way it is. This is written for people working on the library: it assumes you know Elixir and OTP and have used a slippy map, and explains the map maths from there. For the public API, see [README.md](../README.md). For what is planned next, see [ROADMAP.md](ROADMAP.md). ## The pipeline ``` opts ──▶ Options ──▶ Viewport ──▶ TileGrid ──▶ Fetcher ──▶ Canvas ──▶ encode ──▶ binary validate fit centre which cache, stitch+crop, + normalise and zoom tiles, source markers, where attribution ``` Each stage is a pure function of the one before it, except the fetcher, which is the only stage that does I/O. That split is what makes everything else testable without a network. | Module | Job | Pure? | |---|---|---| | `StaticMaps.Options` | validate and normalise options, look up the source's `info/1` | yes | | `StaticMaps.Projection` | lat/lng ↔ world pixels, latitude clamp | yes | | `StaticMaps.Viewport` | the resolved view, world-to-output rounding, and fitting centre and zoom | yes | | `StaticMaps.TileGrid` | which tiles cover the viewport and where each is placed | yes | | `StaticMaps.Fetcher` | get the bytes for every tile: cache, dedupe, limit, retry | no | | `StaticMaps.TileCache` | behaviour, with an ETS implementation | no | | `StaticMaps.CircuitBreaker` | skips a source that keeps failing | no | | `StaticMaps.Marker` | marker options, how far each marker reaches, the pin's SVG | yes | | `StaticMaps.Canvas` | the only module that calls libvips | yes, given bytes | | `StaticMaps.Plug` | optional HTTP layer over `render/1` | no | | `StaticMaps.Plug.URL` | the signed URL format, written by `path/3` and read by the Plug | yes | | `StaticMaps.Plug.Renders` | joins identical renders and limits each Plug's concurrency | no | ## Coordinates and projection There are three coordinate spaces, and most bugs in prior art come from mixing them up. - **Geographic**: `{lat, lng}` in degrees. What the caller gives us. - **World pixels**: Web Mercator, where the whole world is a square `256 · 2^zoom` pixels on a side, `x` growing east and `y` growing south. All geometry happens here, in floats. - **Output pixels**: the image, `width · scale` by `height · scale`. World pixels become output pixels by subtracting the top-left corner of the viewport and multiplying by `scale`. `StaticMaps.Projection` owns the conversion. Latitude is clamped to ±85.05112878° before projecting, because Mercator `y` goes to infinity at the poles. It never wraps latitude: lat 100 is an invalid input, not lat −80. The `256` in the world size is fixed. It defines what a zoom level means, the convention every slippy map uses, and is independent of the size of the tile images a source serves. Confusing the two is the most common tile-maths bug in prior art; see [Tile size and scale](#tile-size-and-scale). Rounding happens once, at the end, when a position becomes an output pixel, and it always rounds halves up. `Kernel.round/1` rounds halves away from zero, which puts −0.5 and 0.5 two pixels apart and opens a one-pixel seam between neighbouring tiles. Tiles and overlays go through the same function, `Viewport.to_output/2`, so a marker and the street under it can never disagree by a pixel. ## Fitting the viewport `StaticMaps.Viewport` resolves `:center` and `:zoom` when the caller does not give both. 1. Collect what must be visible: the `:bounds` corners and every marker. A `:bounds` with `west > east` crosses the antimeridian, so `east` gets `+ 360` before projecting. 2. Unwrap longitudes onto the shortest arc that covers them all. Markers at 179° and −179° are 2° apart, not 358°, and a box around them has to be contiguous in world pixels. The shortest arc always starts where some marker or bounds edge starts, so trying each of those as the start and keeping the shortest is enough. 3. For each candidate zoom, highest first within the first source's range, project everything to world pixels and grow the box by each marker's icon extent (a pin's tip is its coordinate; the rest of it sits above). 4. Take the first zoom where that box, plus `:padding` on every side and the attribution strip at the bottom, fits in `:size`. If none does, take the lowest. 5. The centre is the middle of that box **in world pixels**, then projected back. Because the box includes the pins, a single pin is centred with its whole icon in view, not with its tip at the centre and its top cut off in a short image. Everything at one point has no size to fit, so it gets zoom 15 instead: close enough to see the street, far enough to see where it is. With a centre but no zoom, the centre stays put and the zoom is the highest at which everything fits around it; with a zoom but no centre, the zoom stays and the centre is the middle of the box at that zoom. Taking the middle in degrees instead is a common mistake. Mercator stretches latitudes unevenly, so a centre halfway between two latitudes in degrees is not halfway between them on the map. For a box spanning many degrees of latitude, the view ends up off-centre and the padding is not respected. ## The tile grid `StaticMaps.TileGrid` turns a viewport into a list of placements: `%{key: {z, x, y}, at: {px, py}}`. - The tile columns and rows are those covering the viewport in world pixels, kept only if their rounded output position still lands on the image. When the view's origin sits on a half pixel, rounding can push a tile that just touches the view entirely off it, and fetching that tile would be wasted. - **Placement uses the unwrapped column.** A view across the antimeridian may need column `-1` or `2^z`. Its position on the canvas is computed from that unwrapped column. Only the fetch key wraps, with `Integer.mod(x, 2^z)`, which is correct for negative numbers where `rem/2` is not. - **Rows outside `0..2^z - 1` are dropped**, not wrapped. Past the poles there is no map; the canvas background shows through. Wrapping them fetches tiles from the opposite pole, and requesting them fetches tiles that do not exist. - **Keys are deduplicated.** At zoom 0 or 1 a wide image may need the same tile in several places. It is fetched once and placed many times. ## Tile size and scale `:zoom` always means the 256-pixel convention above. A source's `info/1` says how its tiles relate to it, and the grid picks the tile zoom and how large each tile is drawn: | Source `tile_size` | `:scale` | Source lists scale 2? | Tiles fetched | Drawn at | |---|---|---|---|---| | 256 | 1 | — | zoom `z` | 256 px | | 256 | 2 | yes | zoom `z`, high-density | 512 px, native | | 256 | 2 | no | zoom `z` | 512 px, scaled up | | 512 | 1 | — | zoom `z − 1` | 512 px | | 512 | 2 | — | zoom `z − 1` | 1024 px, scaled up | A 512-pixel tile at zoom `z − 1` covers the same ground as four 256-pixel tiles at zoom `z`, which is why a 512 source is asked for one zoom lower. Its minimum usable `:zoom` is therefore one higher than its own minimum. Every fetched tile is checked against the size this table predicts. A mismatch is an error naming the source, because it means `info/1` is wrong, and silently scaling would hide that. ## Fetching `StaticMaps.Fetcher` gets bytes for a list of tile keys. For each key it: 1. looks in the tile cache; 2. joins an in-flight fetch of the same key if another render already started one; 3. otherwise waits for a slot under the source's concurrency limit and calls the source's `fetch/2` in its own task, so a source that raises or hangs fails that tile and nothing else. The concurrency limit is **per source and shared across all renders**, not per render. A per-render limit of 2 is no limit at all when fifty renders run at once, and OSM's policy is about the total load you put on them. A source here is its module together with its options, so two `Source.Template` servers do not throttle each other. `Source.OSM` allows two concurrent requests; other sources default to eight. A `Source.Template` with several mirrors still has one limit, shared by all of them. Giving each mirror its own limit looks natural but cannot hold during failover: while one mirror is down, its tiles move to the others, which then carry their own share and the dead mirror's at once, twice their limit, just when they are the only servers left. One limit for the source means no mirror ever sees more than `:max_concurrency` requests. A render that gives up waiting withdraws its requests: queued fetches nobody is waiting for are dropped, and late replies are discarded rather than left in the caller's mailbox. The same happens when a caller process dies mid-render. `StaticMaps.Source.HTTP` implements `fetch/2` for URL-based sources: - One `User-Agent` for everything: `StaticMaps/ (+)`. Headers a source adds are merged with it, never replace it. - A connect timeout and a receive timeout on every request. There is no request without a deadline. - Retries, with exponential backoff, on transport errors, `408`, `429`, `500`, `502`, `503` and `504`. `Retry-After` is honoured up to a limit, 10 seconds by default; a server asking for a longer wait fails the tile instead of stalling the render. A `404` is not retried; it means the tile does not exist. - The response must be `200` with an image content type, and must be a complete image of the expected size, before it counts as a tile. Anything else is an error and is not cached. Completeness is checked from each format's end marker, because libvips decodes a truncated PNG or JPEG without complaint and fills in the missing rows. - Subdomains, when a source uses them, are chosen from `x + y`, so a tile always has the same URL and HTTP caches along the way can do their job. A render whose tiles all come back is complete. Otherwise the render fails with `%StaticMaps.TileError{}`, or, under `on_tile_error: :fill`, the missing placements are dropped and the background shows through. There is no third case where some errors are swallowed and others crash. ## Fallback sources and the circuit breaker `:source` may be a list. `render/1` tries each source in order, building a separate tile grid for each, since sources can differ in tile size. The first source whose tiles all come back is drawn. Tiles are never mixed between sources: providers draw different maps, and a fallback tile in the middle of the primary's map is a visible patch with the wrong attribution. Mixing is only safe between mirrors of one source, which is why mirrors live inside `Source.Template` and fallback lives in `render/1`. If every source fails, a render with one source returns its `%TileError{}`, and a render with several returns `%StaticMaps.UnavailableError{}` listing each attempt. Under `on_tile_error: :fill`, the first source that was tried is drawn with its gaps. `StaticMaps.CircuitBreaker` counts consecutive failed renders per source, in an ETS table. Only failures that say the source is unhealthy count: connection errors, timeouts, `5xx`, `408`, `429`, `401` and `403`, responses that are not images, truncated images, and crashes. A `404` or a tile of the wrong size does not; otherwise one area of the map with no tiles would take a working source offline for every render. At the threshold, 5 by default, the source is open: renders skip it for the cooldown, 30 seconds by default. When the cooldown ends the next render tries it again. Success clears the count; another failure opens it for a further cooldown. Without the breaker, a source that is down costs every render a full round of timeouts and retries before the fallback is even tried. ## The tile cache `StaticMaps.TileCache` is a behaviour: `get/1`, `put/2`, and nothing else the pipeline depends on. Freshness, revalidation and validation live in `StaticMaps.TileCache` itself, so an implementation only stores and evicts. The default implementation is an ETS table owned by a process in the library's supervision tree. Only sources that implement the optional `fetch_cacheable/3` callback are cached. Every HTTP source does; a source that reads tiles from local storage gains nothing from a copy in memory. A source whose `info/1` says `cache: false` is skipped entirely: its tiles are never stored, never answered from the cache, and never served stale. Several commercial providers forbid server-side caching in their terms, so this is per source; the rest of the sources in a render are still cached. - **Key**: `{source_module, hash(source_opts), z, x, y, scale}`. The options are hashed so an API key in them never sits in a cache key or a log line. - **Freshness**: `Cache-Control: s-maxage`, else `max-age`, less any `Age` a CDN in front of the server reports; else `Expires`; else 7 days. `no-cache` is stored but revalidated every time; `no-store` is not stored. An unreadable `Expires`, such as the common `Expires: 0`, counts as already expired. - **Where it is checked**: a fresh tile is answered in the rendering process before anything is queued, so a cache hit never waits for a slot behind slow fetches. - **Revalidation**: an expired entry keeps its `ETag`, `Last-Modified` and lifetime, so the next fetch is a conditional request, and a `304` renews it without a download. Many servers send a bare `304`; the entry then keeps the lifetime it was stored with, rather than a `no-cache` tile turning into a week-long one. If the server cannot be reached, the expired entry is served rather than failing the render. - **Failure**: a cache that raises, such as the ETS table while its owner restarts, is a miss. Renders carry on fetching; they never fail because of the cache. - **Validation**: a response is stored only after it decodes as a complete image of the expected size, so a bad tile is never cached for a week. - **Size**: a byte limit, 64 MB by default. When a put goes over it, the least recently used entries are evicted down to 90% of the limit, so eviction does not run on every put. Reads go straight to the ETS table, but puts and eviction go through its owner process, so the byte count is only changed in one place and cannot drift. A cache that never evicts is an outage waiting for a busy week. A disk-backed implementation writes to a temporary file and renames it into place, so a reader never sees half a tile. ## Compositing `StaticMaps.Canvas` is the only module that talks to libvips, through [`vix`](https://github.com/akash-akya/vix), which ships precompiled libvips binaries. Keeping it in one module means a change of imaging library touches one file. 1. Create a canvas of the **output** size, filled with `:background`. 2. Insert each tile at its placement. libvips clips anything outside the canvas, so stitching and cropping are one step, and the full mosaic of tiles is never built. A tile with transparency is flattened against `:background` first, and a tile fetched at scale 1 for a scale 2 render is scaled up to its drawn size. 3. Draw markers, in order. The default pin is an SVG built from a template, and SVG icons are loaded as they are; libvips renders both at the output scale, so edges are anti-aliased and a `scale: 2` pin is drawn at twice the size, not enlarged. A label and a colour are escaped before going into the SVG, so no marker option can inject markup. A marker is drawn once for every copy of the world it lands on. 4. Draw the attribution strip: text on a translucent box, in a bottom corner, wrapping onto more lines when the image is too narrow. If libvips draws nothing, because no font is installed, the render fails: a map without its credit must not pass as finished. 5. Encode. A JPEG has no transparency, so the background colour matters there. Overlays are composited onto the tiles with an alpha channel added for the purpose, then the canvas goes back to three bands, so every image has the same shape whatever was drawn on it. All drawing uses one draw order: tiles, then markers, then attribution. Attribution is always on top, so nothing a caller draws can cover it. Fitting has to know the strip's height before anything is drawn, so `StaticMaps.Options` measures the strip of every source the render might use at the requested width, and passes the tallest to `Viewport.fit/1` as a band along the bottom that nothing is fitted into. Measuring every source matters because a fallback's credit may wrap onto more lines than the primary's. The fitted box must fit above the band, and the centre moves down by half the band so the box sits in the middle of the space above it. ## The Plug `StaticMaps.Plug` is described from the outside in the README. Inside, a request goes: ``` GET /maps/.?&sig= │ ├─ not GET or HEAD ──────────────────────────────────────────── 405, no-store ├─ HMAC over ".?", hash == sha256(query) ── fail ─▶ 403, no-store ├─ If-None-Match == weak ETag of (query, Plug config, version) ─▶ 304 (no render, no fetch) ├─ decode, resolve named icons, enforce limits ──────────────── fail ─▶ 400, no-store ├─ join an in-flight render of the same map, or wait for this Plug's render slot ├─ render(opts ++ on_tile_error: :fail) ──────────── tiles fail ─▶ 503, no-store └─ 200, Cache-Control from :cache plus no-transform, ETag, Content-Type ``` The URL format lives in `StaticMaps.Plug.URL`, which `path/3` writes and the Plug reads, so the two cannot drift apart. The query is canonical, with a fixed parameter order and number format, so one map has one URL. The signature covers the file name as well as the query, and it always comes last, so the Plug can split it off the raw query string and check exactly the bytes that were signed. The hash in the file name makes the path alone identify the map, for CDNs that ignore query strings. The ETag is computed from the file name, the signed query and the Plug's own configuration, so the `304` path costs one HMAC and one comparison. The file name is in it because the PNG and JPEG of one map share a query but not their bytes; the ETag also keys the renders, so without it one format's request could receive the other's image. That is the point of a validator: a revalidating browser should cost nothing. The configuration is in it because the same URL renders differently from a different source, attribution or icon; the library version is in it so a release that draws differently replaces what caches hold. `StaticMaps.Plug.Renders` runs the renders. It works like the fetcher: requests for the map already rendering wait for that render, and each mounted Plug has its own limit and queue, keyed by its configuration. A single limit shared by every mount would be set by whichever asked last, so one mount's `:max_concurrency` would override another's. The Plug sends the whole image for `HEAD` as for `GET`; the server drops the body and keeps the `Content-Length`, so a `HEAD` reports the real size. The `:attribution` and `:source` options come from the Plug's configuration, never from the query string. Anything a request can set, a stranger can set, signature or not, once they have one valid URL to copy. ## Testing strategy - **No network in the test suite.** Tests use a synthetic source whose tiles are solid colours derived from `{z, x, y}`, each labelled with its key. A tile in the wrong place or from the wrong wrap is visible at a glance in a failing image. - **Projection and grid maths are property-tested**: round trips through the projection, every output pixel covered by exactly one placement, keys always inside `0..2^z - 1`. Antimeridian, pole and zoom 0 cases are explicit examples as well. - **Images are compared by decoded pixels with a tolerance**, never by bytes. PNG bytes change with the zlib and libvips versions, and a byte comparison fails on an upgrade while telling you nothing about the map. - **The Plug is tested with `Plug.Test`**: signature rejection, the 304 path not calling the fetcher, error responses carrying `no-store`, and concurrent identical requests producing one render. ## Lessons from prior art Before this design was written, five existing implementations were read closely. Each decision above answers something they got wrong or right. | Decision | What it avoids | |---|---| | Clamp latitude, drop rows past the poles | Wrapping `y` to the opposite pole (komoot, StephanGeorg); wrapping latitude so 100° becomes −80° (komoot) | | Place from the unwrapped column | Tiles across the antimeridian landing off-canvas because the offset was computed after wrapping (quilt) | | Centre in world pixels, not degrees | Off-centre views and ignored padding on tall bounds (komoot, StephanGeorg) | | `tile_size` and `scales` declared by the source | One tile size hard-coded (256 in four of them, 512 in quilt), or tile size confused with scale (komoot, StephanGeorg) | | One tile-failure policy, `:fail` by default | Tiles silently dropped (StephanGeorg, Luzifer); some errors swallowed while others crash (flopp); one failure retried six times and the image lost (quilt) | | Failures sent `no-store` | A map with a hole in it cached for a day as a success (Luzifer) | | Retry `429`/`5xx`, never `404` | Retrying the permanent error and giving up on the transient ones (quilt) | | Timeouts on every request | Requests with no deadline at all (komoot, StephanGeorg, Luzifer) | | Concurrency limited per source, globally | One goroutine per tile (Luzifer); unbounded `Promise.all` (quilt) | | Identifying `User-Agent` that callers cannot remove | No `User-Agent` at all (StephanGeorg); a generic one replaced by any custom headers (komoot) | | Tile cache honours HTTP headers and has a size limit | Disk caches that never expire and never evict (flopp, Luzifer) | | Validate a tile before caching it | An HTML error page saved as a tile | | Attribution drawn by default, not settable from a request | No attribution at all (komoot, StephanGeorg, quilt); anyone can turn it off with a query parameter (Luzifer) | | Sources are modules | Any caller making the server fetch any URL (Luzifer's overlays) | | `render/1` is a pure function of its options | Mutable builders holding render state, unsafe to share between processes (komoot, StephanGeorg, flopp) | | Explicit `0` handled as a value | `zoom: 0` treated as "not given" (komoot, StephanGeorg) | | Pixel-tolerance image tests, no network | Tests hitting live tile servers (StephanGeorg), byte-equal PNG assertions that never ran (quilt), a test file that cannot import (komoot), no tests (Luzifer) | Some things they got right, and this design keeps them: - Fitting that counts marker icon extents, not just marker points (all three fitting implementations). - Drawing overlays anti-aliased, whether through SVG (StephanGeorg) or supersampling (komoot). - Handing the renderer a plain `tile → bytes` function rather than an HTTP client (flopp). It is why `Source` has a `fetch/2` callback, and why a source reading tiles from a file fits the same behaviour as a tile server. - Putting the renderer's version in the cache identity (Luzifer), which is why the Plug's ETag includes the library version. - Stitching and cropping in one operation on an output-sized canvas (quilt).