OpenStreetMaps Extract Transform Load (ETL) procedure
The WMF maps stack use OSM to render static and dynamic maps. Imposm3 is the tool behind the ETL procedure that loads data into a PostgreSQL DB schema that conforms with the maps applications' needs.
- Postgres 11
- PostGIS 3.1+ (this version allows the use of the function ST_GeomAsMVT to generate vector tiles with Tegola)
Imposm3 mapping and DB schema
The database schema is defined in a configuration file that is available in the operations/puppet repository.
Import OSM data
In order to import OSM data, we have a production-ready script that needs to be executed with root access and the latest planet dump from OSM, with a caveat, always choose a specific revision to execute the script instead of the one tagged with latest, here is an example of the command to be executed:
sudo -s screen -DR # always execute with screen because the script takes several hours to finish imposm-initial-import \ -d 210906 \ -x webproxy.eqiad.wmnet:8080
The script will take care of the whole procedure, but if you want to know how the process works and execute it locally, please refer to the guide present in maps-config
As part of the Maps 2.0 work, it was introduced the concept of SQL functions to query the layers of the map. See the SQL functions that can be accessed.
Updating Layers Functions
When changing a SQL function behavior that introduces significant changes or needs to be easily rolled-back, we decided to add version suffix in the function name and updating the applications usage of the functions accordingly. This means that, the next changes in public.layer_admin will be applied in public.layer_admin_v2.