Jump to content

Add a wiki

From Wikitech
(Redirected from Add a language)
This guide may be incomplete. Some cases don't need Apache configs or a docroot. Look at the existing similar wikis to see what is required.

This page documents the process for adding a new wiki. This includes new languages on sister projects, and wikis for committees, chapters etc.

Preparation

The following steps need to be completed before the wiki database may be created.

The Phabricator-maintenance-bot can do some of the following steps.

Notify

  • Create a Phabricator task for the wiki creation (if one doesn't exist already).
  • Create (if not already) a sub-task "Prepare and check storage layer for <new wiki>".
    • It should be tagged with #DBA, #wmcs-kanban and #Data-Services.
  • Notify the Operations list. In particular, it needs to be made clear whether the wiki should be public, or private. If public, ops will arrange for the wiki to be replicated to Cloud Services. If private, ops will need to add the wiki to $private_wikis in operations/puppet.git:/manifests/realm.pp.
  • IMPORTANT: If the wiki is a regular public wiki to appear on Cloud Services - you can continue. If the wiki is private and should not be replicated to Cloud Services DO NOT CONTINUE UNTIL YOU HAVE THE OK FROM OPS/DBAs to check no private data is leaked. There are mechanisms in place to prevent that by default, but those should be manually checked.

DNS

First of all, ensure the relevant domain names exist for the new wiki. Make the following changes in a commit for the operations/dns.git repo and submit to Gerrit for review.

  • If it is a language project, ensure the language code is present in /templates/helpers/langlist.tmpl. This is shared between all sister projects. If another sister project has the same language already, then this has probably been done already.
  • If it is a subdomain of ".wikimedia.org" domain (chapter wiki or special wiki)
  • Merge the change in Gerrit and run authdns-update.
  • Query the DNS servers to make sure it has been correctly deployed. See DNS#HOWTO for details.
  • For new languages, there is also a need to regenerate zones. Run on ns0, ns1 and ns2: authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsdctl reload-zones

Apache configuration

Apache configuration is located in the operations/puppet.git repo.

  • Common configuration:
    • For a new language project, this step is usually not needed as shared configuration already covers it.
    • For a new chapter wiki, add it to mediawiki.yaml under wikimedia-chapter
  • After the change is merged in Gerrit, deploy the configuration change and (if needed) gracefully restart app servers. See Application servers/Runbook#Deploying config for details.
  • If there are additional domains that should point to the same wiki, add it to redirects/redirects.dat.

MediaWiki pre-install configuration

In a local checkout of operations/mediawiki-config.git, run

php multiversion/bin/manage-dblist prepare <db> <lang> <family> <privacy>

Where:

  • <db> is the database name to be created.
  • <lang> is the language code, which will be added to langlist if it isn't there already.
  • <family> is the site group or project, e.g. wikipedia for Wikipedia, wikimedia for a chapter wiki, or special for a special wiki.
  • <privacy> is either public, private or fishbowl.

With MediaWiki installed at a location specified in the MW_INSTALL_PATH environment variable, and the WikimediaMaintenance extension checked out somewhere, update interwiki.php:

php $MW_INSTALL_PATH/extensions/WikimediaMaintenance/dumpInterwiki.php > wmf-config/interwiki.php

For special wikis or other non-standard domains:

  • Set $wgServer, $wgCanonicalServer
  • Add the subdomain to the list in MWMultiVersion::setSiteInfoForWiki
  • Add the domain to $wgLocalVirtualHosts. This is necessary to make cross-wiki HTTP requests work (used by ie. cross-wiki notifications).

For new languages:

  • Add the language to InterwikiSortOrders.php

Commit the above changes, upload to Gerrit, and get a CR+1 so that they are ready to deploy.

MediaWiki post-install commit

In a separate commit on top of the pre-install commit, run:

php multiversion/bin/manage-dblist activate <db>

This will add the wiki to all.dblist. Get a CR+1 for this change also.

Optional MediaWiki configuration

Some configuration can be deferred until after install, or the defaults may be good enough.

  • Update the configuration files (located in /wmf-config). Some may have sufficient defaults but the following are often wiki-specific in InitialiseSettings.php:
    • $wgLogos (see below), $wgSitename, $wgExtraNamespaces, $wgLocaltimezone.
    • groupOverrides.
    • Ensure $wgCategoryCollation is set to the appropiate sorting for this wiki's language. If you are not sure, ask on the task.
    • If an extension needs to be enabled, this is usually done by setting a wmg-variable to true in InitialiseSettings along with custom configuration there. Some extensions' configuration are located in a dedicated files in the /wmf-config directory.

Projects, in particular Wikipedias, usually have a localized logo.

Check the following:

  • The project has a full localized logo in Commons (image + wordmark). For Wikipedia, the filename is usually File:Wikipedia-logo-v2-LANGUAGE.svg, for example File:Wikipedia-logo-v2-fr.svg for French. If there is no localized logo, ask the project's editor to provide the localized text and get the logo created and uploaded to Commons.
  • The project has versions of the logo in all the necessary sizes and resolutions uploaded. If not, create, upload, and configure them according to the instructions at Wikimedia site requests#Change the logo of a Wikimedia wiki.
  • The project has a localized wordmark without the image. This is shown at the bottom of the mobile site and in some other places. This is not necessary if the name of the project is completely identical to the default English name, but necessary if it's not identical. For Wikipedia, the filename is usually File:Wikipedia-wordmark-LANGUAGE.svg, for example File:Wikipedia-wordmark-fr.svg for French.

Language configuration

When adding a new language:

  1. Make sure that the wiki's content language code appears in the language-data repository (file data/langdb.yaml; if you're adding it, make sure to follow the instructions in README).
  2. Make sure that the wiki's content language code is fully configured to be supported by core MediaWiki. The code must appear in the following files:
    • includes/languages/data/Names.php (precisely this code, and not a variant)
    • The file languages/messages/MessagesAbc.php must exist (replace Abc with the language code).
  3. If the language is written from right to left, make sure it is configured as such in the following files:
    • In the core MediaWiki repository, the file languages/messages/MessagesAbc.php must explicitly say $rtl = true; near the top.
    • In the MobileFrontend extension repository: src/mobile.languages.structured/rtlLanguages.js
    • In the Android app repository: app/src/main/java/org/wikipedia/util/L10nUtil.java (the RTL_LANGS list)
    • In the iOS app repository: Wikipedia/Code/MWLanguageInfo.m (the rtlLanguages list)
  4. Check whether the language code appears in any extra configuration for Wikibase and Wikidata. If it appears in any of the following files, it should be removed (verify with Wikidata developers):
  5. Check that usernames in the writing system of this language are possible and aren't filtered out. If they are blocked, add support for this writing system to AntiSpoof. Example patch.

For a new language project, make sure the language has been approved by the Language committee at Requests for new languages on Meta-Wiki. Usually, the Phabricator task will contain a link to the approval page. For all other wikis, go directly to #Install.

IMPORTANT: For Private Wikis

  • Private wiki databases must not be replicated to the Cloud Services DB MySQL instances!
    • Before creating a database for a private wiki, make sure to add the db name to the puppet global array $private_wikis in operations/puppet.git:/hieradata/common.yaml.
    • Deploy this config change with puppet and manually restart the Prelabsdb-db MySQL instance (Sanitarium) on the server that will house this wiki's db (most likely s5).
    • If you need help with this, please ask a member of the DBAs for help. This is very important.
  • If you enable local upload for a private wiki, you must get the SRE team to check the permissions on the swift containers for your new wiki (you risk the contents being world-readable otherwise) - refer them to Swift/How To#Checking / Fixing container ACLs for private wikis for details. If you have disabled local upload (i.e. Local file uploads: disabled), then this isn't necessary, as no swift containers should exist.

WikimediaMessages

Translatable project name

Add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediaprojectnames/en.json and qqq.json. The message keys should include the wiki database name (WIKI_DBNAME below) and official "human readable" name (WIKI_NAME below) as follows:

en.json
Key Message
project-localized-name-WIKI_DBNAME WIKI_NAME

For example, "project-localized-name-enwiki": "English Wikipedia",

qqq.json
Key Message
project-localized-name-WIKI_DBNAME {{ProjectNameDocumentation|url=WIKI_URL|name=WIKI_NAME|language=WIKI_LANG}}

For example, "project-localized-name-enwiki": "{{ProjectNameDocumentation|url=https://en.wikipedia.org|name=English Wikipedia|language=en}}",

Cross-wiki (a.k.a. interwiki) search result title

If this is a new language Wikipedia, add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediainterwikisearchresults/en.json and qqq.json in the list of search-interwiki-results messages.

Wikipedia mobile apps

If you are creating a new Wikipedia, add an entry for the new language.

For the Android app, you must run the following two scripts, which will update the list of languages known to the app (including the corresponding language codes, autonyms, English names, etc.): 

$ cd scripts
$ ./generate_wiki_languages.py
$ ./make-templates.py

This should update several .java and .xml files that you may then submit in a pull-request to our repository.

(TODO: Is there anything to do for the iOS app?)

Install

Database creation

Deploy the pre-install mediawiki-config commit with scap backport and then run

mwscript-k8s -f -- extensions/WikimediaMaintenance/addWiki.php --wiki=$db

Although that's currently broken as reported at T341553#10369931 so instead maybe run

mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=$db

Update sites by running this on the maintenance server:

foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol=https

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering of many wikis!

Deploy the post-install mediawiki-config commit.

RESTBase

RESTBase is a service providing a RESTful API for the projects' wikis. This service is in a process of deprecation, below is an outline for the current process.

Before proceeding with these steps please reach out to the #platform-engineering team directly.

To enable it to serve the new wiki as well, create a task tagged with #platform_engineering, then create a patch for mediawiki/services/restbase/deploy adding its domain to the appropriate section in RESTBase's configuration, and link that patch with the phab task.

After the patch is merged, a restart of the RB service across the cluster is required, which can be done by ops or restbase-admins/restbase-roots.

cxserver

For project families in which the ContentTranslation is installed (as of 2018—only Wikipedia projects): Add the language code to the ContentTranslation registry - mediawiki/services/cxserver repository, file config/languages.yaml (included by config.dev.yaml and config.prod.yaml).

Once merged to master, ping the ContentTranslation developers to deploy the change. That requires to sync repositories, ie to update mediawiki/services/cxserver/deploy to match mediawiki/services/cxserver. See: https://gerrit.wikimedia.org/r/c/303763/ for an example commit.

Wiki Replicas

These steps are necessary to make the wiki available in the Wiki Replicas.

Before maintain views: DBAs' work

First, the DBAs need to run this on both sanitariums (eqiad and codfw) and all cloud replicas of that section (clouddb10xm): 

root@db1154:~# redact_sanitarium.sh -d gpewiki -S /run/mysqld/mysqld.s5.sock | mysql -S /run/mysqld/mysqld.s5.sock gpewiki

Then: 

root@db1154:~# check_private_data.py -S /run/mysqld/mysqld.s5.sock | mysql -S /run/mysqld/mysqld.s5.sock

And then run check_private_data.py -S /run/mysqld/mysqld.s5.sock to make sure there is no private data. Also wait for some edits to be done in the wiki and run it again to make sure triggers work fine.

Once that's done, create the ${wiki}_p database and granted access to labsdbuser in the cloud replicas (otherwise a bug in MariaDB will cause the scripts to fail). This what DBAs need to do on each cloud replicas (not sanitariums): 

$ sudo -i mysql --skip-ssl
mysql:root@localhost [(none)]> create database ckbwiktionary_p;
mysql:root@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `ckbwiktionary_p`.* TO 'labsdbuser';

Maintain views

When the task is ready for WMCS, run the cookbook sre.wikireplicas.add-wiki on the any of the cluster management hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet). The following example is for smnwiki and it's task. Replace those values with the wiki database name you are adding and the correct task ID: 

$ sudo /usr/local/bin/secure-cookbook sre.wikireplicas.add-wiki --task-id T264900 smnwiki
START - Cookbook sre.wikireplicas.add-wiki
Generating views...
Adding DNS
Finalizing meta_p
Added views for new wiki: smnwiki T264900
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0)
The "Adding DNS" step takes quite a while to complete. Do not panic!

If you are adding more than one wiki that day or several tickets have been opened to add wikis, you may find that the DNS step creates all of them at once. After that has been done, it doesn't need to be done again. You can save yourself lots of time by using the --skip-dns option on the command line so it doesn't sit there scanning for new wikis.

If in doubt you can always use the --dry-run option on the secure-cookbook command so that you just see what it would do. For example: 

$ sudo secure-cookbook --dry-run sre.wikireplicas.add-wiki --task-id T260551 thankyouwiki
DRY-RUN: Executing cookbook sre.wikireplicas.add-wiki with args: ['--task-id', 'T260551', 'thankyouwiki']
DRY-RUN: START - Cookbook sre.wikireplicas.add-wiki
DRY-RUN: Generating views...
DRY-RUN: Executing commands ['/usr/local/sbin/maintain-replica-indexes --database thankyouwiki', '/usr/local/sbin/maintain-views --databases thankyouwiki'] on 4 hosts: labsdb[1009-1012].eqiad.wmnet
DRY-RUN: Adding DNS
DRY-RUN: Executing commands ['source /root/novaenv.sh; wmcs-wikireplica-dns --aliases'] on 1 hosts: cloudcontrol1003.wikimedia.org
DRY-RUN: Finalizing meta_p
DRY-RUN: Executing commands ['/usr/local/sbin/maintain-meta_p --databases thankyouwiki'] on 4 hosts: labsdb[1009-1012].eqiad.wmnet
DRY-RUN: Added views for new wiki: thankyouwiki T260551
DRY-RUN: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0)

Manual method in case Maintain views fail

Once it is replicating to the clouddb* servers (e.g. clouddb1013), run the following 2 commands on each replica server: 

localhost:~$ ssh clouddb10xx.eqiad.wmnet
clouddb10xx:~$ sudo /usr/local/sbin/maintain-replica-indexes --database $wiki --debug
clouddb10xx:~$ sudo /usr/local/sbin/maintain-views --databases $wiki --debug

Note the section for use with the wikireplica dns next 

clouddb10xx:~$ grep $wiki /usr/local/lib/mediawiki-config/dblists/s*.dblist* | grep -o '\w[0-9]'
:# Should return a shared, like s3, s5, etc

From a cloud control host, add the wikidb alias in the Wiki Replicas service name: 

localhost:~$ ssh cloudcontrol1003.wikimedia.org
cloudcontrol1005:~$ sudo -i
cloudcontrol1005:~$ source novaenv.sh
cloudcontrol1005:~$ /usr/local/sbin/wmcs-wikireplica-dns --aliases --shard <sN>
:# Use the shard arg only if you know the shard (which you can get on a replica server as noted above)
:# If the shard is s3 or a full rebuild is done it will take quite a while to run

Insert a new row in meta_p.wiki for the new wiki by running the following on each of the replica servers that host the s7 instance. 

localhost:~$ ssh clouddb10xx.eqiad.wmnet
clouddb10xx:~$ sudo /usr/local/sbin/maintain-meta_p --database $wiki

Finish up for either method

Before resolving the ticket, log into a Toolforge bastion as yourself and run: 

localhost:~$ ssh login.toolforge.org
tools-sgebastion-07:~$ sql $wiki
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 470351380
Server version: 10.1.43-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [$wiki_p]> select * from page limit 2;

If you get an error from the select * from page limit 2; statement, you may have missed a step or need to do some troubleshooting. You should just get a couple records.

Post-install

General announcement in Tech/News

Add an entry to m:Tech/News so that people know the new wiki exists, and can adjust their tools/bots/etc. as appropriate.

Data Engineering

Most wikis are added to the Pageviews definition, which means that the log events are extracted from the Webrequest stream and saved as distinct datasets.

Exceptions to this include:

  • private wikis
  • Wikimania conference wikis
  • Other special cases such as usability,outreach,login,vote,strategy etc.

If you are unsure about whether or not the wiki should be included,check with the Data Engineering team.

In order to add the new wiki to the pageview definition, submit a change proposal to the analytics/refinery repository, adding the wiki to static_data/pageview/allowlist/allowlist.tsv.

Incubator

If there's something to import (as is often the case in new language projects), someone will do so. Their process is described at Incubator:Importing from Incubator (logged at Incubator:Site creation log).

After any import Incubator is completed, Inform the community and make a Phabricator task for removing old interwiki links and migrating them to Wikidata (For example T134991 for edits such as d:Special:Diff/336584053 and jam:Special:Diff/12330). You can do it by yourself using interwikidata.py in pywikibot. 

python3 pwb.py scripts/interwikidata.py -lang:LANGCODE -clean -start:! -always

Tell wikistats Cloud VPS project to add the wiki

If the wiki is a public wiki, create a Phabricator task (subtask of the main task to create the wiki) with the tag "VPS-project-wikistats" and just ask for the wiki to be added. (What needs to be done can be seen f.e. in https://phabricator.wikimedia.org/T140970#2531977)

Tell Pywikibot project to add the wiki

Create a Phabricator task (subtask of the main task to create the wiki) with the tag "pywikibot" and just ask for the wiki to be added.

Mobile apps

Wikipedia has mobile apps for Android and iOS. A Wikipedia in a new language must work in the app after the import from the Incubator is complete. Report a bug under the apps tags in Phabricator if any of the following doesn't work:

  • You are supposed see the language in the user preferences.
    • Android: Settings -> Wikipedia language.
    • iOS: Settings -> My languages
  • You see the language in the interlanguage links list. Find an article that exists in the new Wikipedia and in English. Go to the English Wikipedia, tap the 文Α icon at the bottom and find the article in the list. In particular:
    • The article name must appear.
    • The language name (autonym) must appear.
    • The item must be findable using the language name in English and the autonym. (If you don't know how to type the autonym, try pasting the autonym or ask somebody who writes in that Wikipedia.)
    • Tapping the item must show the article.

Portal

Add the new site to the wikimedia/portals repository:

  • If the language is right-to-left and the site is a Wikipedia, make sure it appears in src/wikipedia.org/rtl-languages.json.
  • Check that the language name is written in translatewiki and exported to the appropriate json file under l10n/.

After creating the wiki and deploying the portal changes, test that the language name appears correctly in the wikipedia.org portal.

Kiwix

Tell Kiwix to add support for the new wiki. Kiwix is an external project, but it works quite closely with Wikimedia to provide regularly-update offline versions as ZIM files. To ask the adding of a wiki, file an issue on the https://github.com/openzim/zim-requests GitHub repository. Boilerplate for Wikipedia: 

Website URL: https://CODE.wikipedia.org
License: CC-BY-SA 3.0
Desired ZIM Icon –png (URL or attach one): https://drive.farm.openzim.org/wikipedia_all/favicon-48x48.png
Language (ISO 639-3): kcg
Is this a MediaWiki?: yes

Comments:

  • Make sure to correct the language code in the URL.
  • For non-Wikipedia projects, make sure that you use the correct URL.
  • If it's the first wiki in the language, make sure to mention the three-letter language code, the English name of the language and the autonym, so that Kiwix developers can add the language to their database.
  • If the default license ever changes from CC-BY-SA 3.0 to CC-BY-SA 4.0 (or something else), write it there, and update the documentation here.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Add_a_wiki&oldid=2250338"