Maps/v2/Common tasks

From Wikitech
< Maps‎ | v2

Common maps tasks

Access map nodes

ssh maps1009.eqiad.wmnet

Deploy new code

Kartotherian

Package the code

Currently, kartotherian main code is available in the master branch.

In order to deploy kartotherian you need to package the deploy repo locally and submit to gerrit, kartotherian uses service-runner to run a Docker container for packaging, the following script assumes you have both the kartotherian repo and the deploy repo at the same directory level: 

cd ../kartotherian
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull origin master
git status
git --no-pager log --decorate -n 1
rm -rf node_modules/
npm i service-runner --ignore-scripts
cp config.test.yaml config.yaml # Needed just to infer some packaging attributes

cd ../deploy
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull
git --no-pager log --decorate -n 1
git submodule update --init
git status

# Change to the kartotherian source directory
cd ../kartotherian
./server.js build --deploy-repo --force

NOTE: In order for the script to work you need to first merge the changes to kartotherian/package repo and then build the deploy package.

Submit changes to production with scap

After the code is reviewed and merged, it should be available at deployment.eqiad.wmnet 

ssh deployment.eqiad.wmnet
cd /srv/deployment/kartotherian/deploy
git pull && git submodule update --init
git log -n 1
# deploy to eqiad
scap deploy --environment eqiad -l 'maps1*' `git log --pretty=format:'%s' -n 1`
# deploy to codfw
scap deploy --environment codfw -l 'maps2*' `git log --pretty=format:'%s' -n 1`

Tegola

Configuration changes

Tegola

The tegola config is maintained using helm in the deployment charts repo and populated by the helm values per env:

Kartotherian

Kartotherian is using scap for deployment. The configuration is maintained here:

Postgres

Postgres config is maintained using puppet. Here are the relevant sections:

Imposm

OSM import process is configured using puppet. Here are the relevant sections:

EventGate

Enable/disable import process

In order to disable the import process you need to switch the boolean value to false and run puppet:

  • profile::maps::osm_master::disable_replication_cron: false

Run a planet import

When running a planet import, make sure that the following checklist is followed:

  • disable disable_replication_service and disable_tile_generation_cron in hieradata
  • execute initial-imposm-import script using the following command
# change -d accordingly when performing the import with the most updated values
sudo -s
screen -DR
imposm-initial-import -d 210906 -x webproxy.eqiad.wmnet:8080
  • Post a log message in IRC #wikimedia-operations !log maintenance: trigger full planet re-import for maps eqiad
  • monitor the full planet import
    • Is there any disk space issue?
    • Did the script finished properly?
    • Are the logs sufficient?
  • monitor and check the APIs, in case of rollback, run: sudo -u osmupdater imposm-rollback-import
  • REINDEX DATABASE gis;
  • Monitor and check the APIs, if needed rollback with the following command
  • Deploy data to the public schema by running:
sudo -u osmupdater imposm-deploy-import
  • re-enable the replication service and check for its health
  • re-enable the tile-generation cron and check for its health
  • remove imposm backup to free disk space
  • check if sql functions and indexes are applied (if not apply them manually)
  • re-enable the replication service and check for its health
  • re-enable the tile-generation cron and check for its health
  • clean-up old data in backup
    • Post a log message in IRC #wikimedia-operations !log <your_username>@maps1009 imposm-removebackup-import (T305845)
imposm-removebackup-import

Pooling/depooling nodes

TODO

Get a DB shell

> ssh maps1005.eqiad.wmnet

> sudo su

> su - postgres

> psql

postgres=# \c gis

Run an example query

Select all landuse geometries included in the bounding box:

  • SRID 3857 (the one we use in our layers config)
  • Box: left=2632692 bottom=4572168 right=2648887 top=4581940

This bounding box is a rectangle that includes the center of Athens, Greece.

gis=# SELECT osm_id, ST_AsMVTGeom(geometry, ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857)) AS geom FROM layer_landuse(ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857), 9);

Reindex DB

WARNING this operation might take even ~10 hours to be completed

From a pgsql shell, connect to the `gis` DB and run:

gis #= REINDEX DATABASE gis;

Example HTTP requests

Kartotherian

Get the raster tile x=0 y=0 z=0 for source osm-intl curl https://maps.wikimedia.org/osm-intl/0/0/0.png

Tegola

Tegola currently is only used internally. In order to reach out for the current example we will ssh to the deployment node (eq. in eqiad)

> ssh deployment.eqiad.wmnet

To get the endpoint information for different maps and layers:

> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/capabilities | jq .

To get the vector tile x=0 y=0 z=0 for map osm

> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/maps/osm/{z}/{x}/{y}.pbf

EventGate

Post an event: 

curl -X POST -d '
{
    "\$schema": "/maps/tile_change/1.0.0",
    "meta": {
    "dt": "$(date --iso-8601=s)"
    "stream": "maps.tile_expiration"
  },
"tile": "1/0/2",
"state": "expired"
}' https://eventgate-main.svc.eqiad.wmnet:4492/v1/events/

Example tegola commands

  • Run server
    • tegola -c </path/to/config> serve
  • Cache a tile
    • tegola -c /path/to/config cache seed tile-name 0/0/0
  • Purge a tile from cache
    • tegola -c /path/to/config cache purge tile-name 0/0/0

Clean-up Kafka queue

Reset the queue: 

kafka-commit-last-message --topic "eqiad.maps.tiles_change" --broker kafka-main1001.eqiad.wmnet:9092 --group-id "poppy-eqiad.maps.tiles_change"

Verify that there are no messages to be consumed from last offset 

kafka-consume-messages --topic "eqiad.maps.tiles_change" --broker kafka-main1001.eqiad.wmnet:9092 --offset earliest --group-id "poppy-eqiad.maps.tiles_change"

List tiles on Openstack Swift

cd /root
source .tegola_credentials
swift -A $ST_AUTH -U $ST_USER -K $ST_KEY list tegola-swift-container

Load balance maps requests

In order to perform safe switchovers in maps we have developed a simple load balancer for kartotherian.

To configure the load balancer you should edit kartotherian config templates with the following code:

Enable load mirroring 

# Switch requests to a secondary source
switch:
  uri: mirror://
  params:
    source: { ref: tm2source }
    secondarySource: { ref: tegola }
    enableMirror: true
    loadBalancer:
      bucketSize: 1000 # milliseconds
      totalBuckets: 10 # number of buckets in the histogram
      threshold: 0.1 # 10% of load to the mirrored source

Enable load switch 

# Switch requests to a secondary source
switch:
  uri: switch://
  params:
    source: { ref: tm2source }
    secondarySource: { ref: tegola }
    enableSwitch: true
    loadBalancer:
      bucketSize: 1000 # milliseconds
      totalBuckets: 10 # number of buckets in the histogram
      threshold: 0.1 # 10% of secondary source load will be returned in the responses

Here's a real world example where kartotherian is configured to mirror 100% of the requests to eqiad tegola.

Check tile pregeneration status

Currently the only way we have to check for pregeneration status is through tile pregeneration logs.

Here are the links to logstash:

Codfw is running every day at 00:00

Eqiad is running every day at 12:00

To filter for errors we can just query for "errors".

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Maps/v2/Common_tasks&oldid=2035988"