User:MSantos/Maps/Tile storage

Setting up Cassandra tile storage

On first start of a Cassandra node, it is necessary to clean the data directory rm -rf /srv/cassandra/*. See Cassandra documentation for more details.
Also remember to touch /etc/cassandra/service-enabled before starting Cassandra on each node. See https://phabricator.wikimedia.org/T214166

To initialize the first Cassandra node, we need to add the local node to the list of seeds by manually editing /etc/cassandra/cassandra.yaml and restarting cassandra:

  seed_provider:
    # Addresses of hosts that are deemed contact points.
    # Cassandra nodes use this list of hosts to find each other and learn
    # the topology of the ring.  You must change this if you are running
    # multiple nodes!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
        # seeds is actually a comma-delimited list of addresses.
        # Ex: "<ip1>,<ip2>,<ip3>"
        # Omit own host name / IP in multi-node clusters (see
        # https://phabricator.wikimedia.org/T91617).
       - seeds: 10.64.16.42,10.64.32.117,10.64.48.154 # '''add local node here to initialize the first Cassandra node'''

change the cassandra super user password to match the one configured in private repo using cqlsh:

 cqlsh <maps1001.eqiad.wmnet> -u cassandra
 Password: cassandra
 ALTER USER cassandra WITH PASSWORD '<password>';

Setup of user access / rights for cassandra

 cat /usr/local/bin/maps-grants.cql | cqlsh <maps1001.eqiad.wmnet> -u cassandra

Setup replication of Cassandra system_auth according to documentation.
Very important point to note is there's a catch when setting up replicator factor for system_auth keyspace on cassandra. We recently discovered that increasing replication factor according to docs causes outage on cassandra. See T214434 and T157354. Also this: Incident documentation/20190122-maps.

Initial creation of cassandra keyspace: To prevent accidental modification of schema, Tilerator source configuration does not allow to create schema by default. The sources file used by tilerator / kartotherian is configured in /etc/(kartotherian|tilerator|tileratorui)/config.yaml, look for the sources: key. This is a reference to a sources file in the kartotherian / tilerator source directory. For example /srv/deployment/tilerator/deploy/src/sources.prod2.yaml.

The easiest way to create a new keyspace is to run Tilerator with a custom sources file, which instructs tilerator to create the missing keyspace. For example, create a temporary file, e.g. /home/yurik/my-source-file with the following configuration (change v4 with the keyspace declared in the sources configuration file):

v4:
  uri: cassandra://
  params:
    keyspace: v4
    cp: {var: cassandra-servers}
    username: {var: cassandra-user}
    password: {var: cassandra-pswd}
    # These parameters are only used if keyspace needs to be created:
    repfactor: 4
    durablewrite: 0
    createIfMissing: true
    copyInfoFrom: {ref: gen}

And run this bash script:

node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js \
  # Use TileratorUI configuration (including password variables)
  --config /etc/tileratorui/config.yaml \
  # Use this sources file instead of the one in the config file
  --source /home/yurik/my-source-file

Tileshell will not exit, so ^C it after it reports "done".

Bulk Copying

The fastest way to copy a large number of tiles from one source to another is to use a large number of parts and specify saveSolid=true (skips solid tile detection). E.g. to copy all z16 tiles from v3 to v4, use src=v3 dst=v4 zoom=16 parts=60 saveSolid=true