Maps/Beta Cluster setup

Setting up a Kartotherian/Tilerator instance in the Beta Cluster presents some difficulties due to the limited amount of storage and memory available. This page documents the process.

Note: The loading instructions here are for the osm-bright.tm2 style. The "new" schema/styles/sources (ClearTables/Brighmed/Meddo) have their own loading process.

TODO: Automate this process along the lines of the process-osm-data.sh script written for the new styles.

Launch new instance

Note: this will probably not go smoothly.

1. Create a new WMCS instance, naming it deployment-mapsXX (where XX is the next numeric value after the existing instances). For example, deployment-maps03 and deployment-maps04 currently exist; the next instance would be deployment-maps05. Use the m1.large flavor.
2. Change the Puppetmaster to deployment-puppetmaster04 (see instructions on the bottom of the task description for T195686)
3. Add the maps-admins group: sudo groupadd -g 1000 maps-admins
4. In Horizon, Apply the following Puppet classes and get Puppet running cleanly:

role::labs::lvm::srv
profile::redis::master
profile::maps::cassandra
profile::maps::apps
profile::maps::osm_master

5. Add the maps security group in Horizon.

Download map data

Even with extra storage, WMCS instances offer far less storage space than is needed to import the full planet data. Instead, download extracts for one or more individual countries from https://download.geofabrik.de/ into /srv/downloads.

Set up the Postgres database

Create the loading schema with sudo -u postgres psql -d gis -c 'CREATE SCHEMA IF NOT EXISTS loading; GRANT ALL ON SCHEMA loading TO osmupdater;'
Grant default permissions to the rendering users with sudo -u osmupdater psql -d gis -c 'ALTER DEFAULT PRIVILEGES IN SCHEMA public, loading GRANT SELECT ON TABLES TO kartotherian, tilerator, tileratorui;'

Import data into Postgres

1. Find the Postgres password for the osmupdater user with sudo -u osmupdater crontab -l and save it to ~/osmupdater_pass.
2. Import the data.

Example: sudo -u osmupdater PGPASSWORD="$(< ~/osmupdater_pass)" osm2pgsql --create --slim -C 4096 --number-processes 1 --hstore <data file>.osm.pbf -H localhost -U osmupdater -d gis
In WMCS it can be difficult to get this to complete before being killed by the OOM killer. In the example command line above, the cache size is increased to 4GB (from a default of 800MB) and the number of processes is reduced to 1, in attempt to reduce the memory load.
When setting up deployment-maps04, using the --flat-nodes option (which reportedly saves on space and memory usage) resulted in the process dying with Bus error.
Detailed documentation is available at https://switch2osm.org/loading-osm-data/

3. Import shapefiles and custom functions as documented on https://github.com/kartotherian/osm-bright.tm2source#install:

sudo -u osmupdater curl -O http://data.openstreetmapdata.com/water-polygons-split-3857.zip
sudo -u osmupdater unzip water-polygons-split-3857.zip && rm water-polygons-split-3857.zip
cd water-polygons-split-3857
shp2pgsql -I -s 3857 -g way water_polygons.shp water_polygons | sudo -u postgres psql -Xqd gis

4. Update permissions for the newly imported shapefiles: sudo -u postgres psql -d gis -c 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO kartotherian, tilerator, tileratorui;'

5. Import custom functions as documented on https://github.com/kartotherian/osm-bright.tm2source#install

cd /srv/deployment/kartotherian/deploy/node_modules/postgis-vt-util
sudo -u postgres psql -Xqd gis -f lib.sql
cd ../@kartotherian/osm-bright-source
sudo -u postgres psql -Xqd gis -f sql/admin.sql
sudo -u postgres psql -Xqd gis -f sql/functions.sql
sudo -u postgres psql -Xqd gis -f sql/create-indexes.sql
sudo -u postgres psql -Xqd gis -f sql/names.sql
sudo -u postgres psql -d gis -c 'select populate_admin();'

6. Revoke permissions for users to create arbitrary functions on the public schema (see CVE-2018-1058).

sudo -u postgres psql -c 'REVOKE CREATE ON SCHEMA public FROM PUBLIC;'

Set up Cassandra users

Find the file /usr/local/bin/maps-grants.cql and manually perform the user creation steps listed there to set up the kartotherian, tilerator, and tileratorui users.

Warning: Do not actually run maps-grants.cql or manually perform the ALTER KEYSPACE system_auth WITH REPLICATION step. This command will result in your Cassandra installation requiring a quorum it's unable to achieve, rendering the database unusable.

Create the map data keyspace

See Maps/Keyspace_Setup.

Generate tiles

With tileshell or tileratorui, generate the desired tiles. See https://github.com/kartotherian/tilerator/blob/master/README.md for detailed instructions.

Example:

node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js --config /etc/tilerator/config.yaml -j.fromZoom 0 -j.beforeZoom 8 -j.generatorId gen -j.storageId v3 -j.deleteEmpty -j.saveSolid -j.zoom=0