Jump to content

Add a wiki

From Wikitech
(Redirected from Add a language)
This guide may be incomplete. Some cases don't need Apache configs or a docroot. Look at the existing similar wikis to see what is required.

This page documents the process for adding a new wiki. This includes new languages on sister projects, and wikis for committees, chapters etc.

Preparation

The following steps need to be completed before the wiki database may be created.

The Phabricator-maintenance-bot can do some of the following steps.

Notify

  • Create a Phabricator task for the wiki creation (if one doesn't exist already).
  • Create (if not already) a sub-task "Prepare and check storage layer for <new wiki>".
    • This sub-task is usually created automatically by Maintenance_bot
    • It should be tagged with #DBA.
  • Notify the Operations list. In particular, it needs to be made clear whether the wiki should be public, or private. If public, ops will arrange for the wiki to be replicated to Wiki Replicas. If private, ops will need to add the wiki to $private_wikis in operations/puppet.git:/manifests/realm.pp.
  • IMPORTANT: If the wiki is a regular public wiki to appear on Wiki Replicas - you can continue. If the wiki is private and should not be replicated to Wiki Replicas DO NOT CONTINUE UNTIL YOU HAVE THE OK FROM OPS/DBAs to check no private data is leaked. There are mechanisms in place to prevent that by default, but those should be manually checked.

DNS

First of all, ensure the relevant domain names exist for the new wiki. Make the following changes in a commit for the operations/dns.git repo and submit to Gerrit for review.

  • If it is a language project, ensure the language code is present in /templates/helpers/langlist.tmpl. This is shared between all sister projects. If another sister project has the same language already, then this has probably been done already.
  • If it is a subdomain of ".wikimedia.org" domain (chapter wiki or special wiki)
  • Merge the change in Gerrit and run authdns-update.
  • Query the DNS servers to make sure it has been correctly deployed. See DNS#HOWTO for details.
  • For new languages, there is also a need to regenerate zones. Run on ns0, ns1 and ns2: authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsdctl reload-zones

Apache configuration

Apache configuration is located in the operations/puppet.git repo.

  • Common configuration:
    • For a new language project, this step is usually not needed as shared configuration already covers it.
    • For a new chapter wiki, add it to mediawiki.yaml under wikimedia-chapter
  • After the change is merged in Gerrit, deploy the configuration change and (if needed) gracefully restart app servers. See Application servers/Runbook#Deploying config for details.
  • If there are additional domains that should point to the same wiki, add it to redirects/redirects.dat.

MediaWiki pre-install configuration

In a local checkout of operations/mediawiki-config.git, run

php multiversion/bin/manage-dblist prepare <db> <lang> <family> <privacy>

Where:

  • <db> is the database name to be created.
  • <lang> is the language code, which will be added to langlist if it isn't there already.
  • <family> is the site group or project, e.g. wikipedia for Wikipedia, wikimedia for a chapter wiki, or special for a special wiki.
  • <privacy> is either public, private or fishbowl.

With MediaWiki installed at a location specified in the MW_INSTALL_PATH environment variable, and the WikimediaMaintenance extension checked out somewhere, update interwiki.php:

php $MW_INSTALL_PATH/extensions/WikimediaMaintenance/dumpInterwiki.php > wmf-config/interwiki.php

For special wikis or other non-standard domains:

  • Set $wgServer, $wgCanonicalServer
  • Add the subdomain to the list in MWMultiVersion::setSiteInfoForWiki
  • Add the domain to $wgLocalVirtualHosts. This is necessary to make cross-wiki HTTP requests work (used by ie. cross-wiki notifications).

For new languages:

  • Add the language to InterwikiSortOrders.php

Commit the above changes, upload to Gerrit, and get a CR+1 so that they are ready to deploy.

MediaWiki post-install commit

In a separate commit on top of the pre-install commit, run:

php multiversion/bin/manage-dblist activate <db>

This will add the wiki to all.dblist. Get a CR+1 for this change also.

Optional MediaWiki configuration

Some configuration can be deferred until after install, or the defaults may be good enough.

  • Update the configuration files (located in /wmf-config). Some may have sufficient defaults but the following are often wiki-specific in InitialiseSettings.php:
    • $wgLogos (see below), $wgSitename, $wgExtraNamespaces, $wgLocaltimezone.
    • groupOverrides.
    • Ensure $wgCategoryCollation is set to the appropiate sorting for this wiki's language. If you are not sure, ask on the task.
    • If an extension needs to be enabled, this is usually done by setting a wmg-variable to true in InitialiseSettings along with custom configuration there. Some extensions' configuration are located in a dedicated files in the /wmf-config directory.

Projects, in particular Wikipedias, usually have a localized logo.

Check the following:

  • The project has a full localized logo in Commons (image + wordmark). For Wikipedia, the filename is usually File:Wikipedia-logo-v2-LANGUAGE.svg, for example File:Wikipedia-logo-v2-fr.svg for French. If there is no localized logo, ask the project's editor to provide the localized text and get the logo created and uploaded to Commons.
  • The project has versions of the logo in all the necessary sizes and resolutions uploaded. If not, create, upload, and configure them according to the instructions at Wikimedia site requests#Change the logo of a Wikimedia wiki.
  • The project has a localized wordmark without the image. This is shown at the bottom of the mobile site and in some other places. This is not necessary if the name of the project is completely identical to the default English name, but necessary if it's not identical. For Wikipedia, the filename is usually File:Wikipedia-wordmark-LANGUAGE.svg, for example File:Wikipedia-wordmark-fr.svg for French.

Language configuration

When adding a new language:

  1. Make sure that the wiki's content language code appears in the language-data repository (file data/langdb.yaml; if you're adding it, make sure to follow the instructions in README).
  2. Make sure that the wiki's content language code is fully configured to be supported by core MediaWiki. The code must appear in the following files:
    • includes/languages/data/Names.php (precisely this code, and not a variant)
    • The file languages/messages/MessagesAbc.php must exist (replace Abc with the language code).
  3. If the language is written from right to left, make sure it is configured as such in the following files:
    • In the core MediaWiki repository, the file languages/messages/MessagesAbc.php must explicitly say $rtl = true; near the top.
    • In the MobileFrontend extension repository: src/mobile.languages.structured/rtlLanguages.js
    • In the Android app repository: app/src/main/java/org/wikipedia/util/L10nUtil.java (the RTL_LANGS list)
    • In the iOS app repository: Wikipedia/Code/MWLanguageInfo.m (the rtlLanguages list)
  4. Check whether the language code appears in any extra configuration for Wikibase and Wikidata. If it appears in any of the following files, it should be removed (verify with Wikidata developers):
  5. Check that usernames in the writing system of this language are possible and aren't filtered out. If they are blocked, add support for this writing system to AntiSpoof. Example patch.

For a new language project, make sure the language has been approved by the Language committee at Requests for new languages on Meta-Wiki. Usually, the Phabricator task will contain a link to the approval page. For all other wikis, go directly to #Install.

IMPORTANT: For Private Wikis

  • Private wiki databases must not be replicated to Wiki Replicas!
    • Before creating a database for a private wiki, make sure to add the db name to the puppet global array $private_wikis in operations/puppet.git:/hieradata/common.yaml.
    • Deploy this config change with puppet and manually restart the Prelabsdb-db MySQL instance (Sanitarium) on the server that will house this wiki's db (most likely s5).
    • If you need help with this, please ask a member of the DBAs for help. This is very important.
  • If you enable local upload for a private wiki, you must get the SRE team to check the permissions on the swift containers for your new wiki (you risk the contents being world-readable otherwise) - refer them to Swift/How To#Checking / Fixing container ACLs for private wikis for details. If you have disabled local upload (i.e. Local file uploads: disabled), then this isn't necessary, as no swift containers should exist.

WikimediaMessages

Translatable project name

Add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediaprojectnames/en.json and qqq.json. The message keys should include the wiki database name (WIKI_DBNAME below) and official "human readable" name (WIKI_NAME below) as follows:

en.json
Key Message
project-localized-name-WIKI_DBNAME WIKI_NAME

For example, "project-localized-name-enwiki": "English Wikipedia",

qqq.json
Key Message
project-localized-name-WIKI_DBNAME {{ProjectNameDocumentation|url=WIKI_URL|name=WIKI_NAME|language=WIKI_LANG}}

For example, "project-localized-name-enwiki": "{{ProjectNameDocumentation|url=https://en.wikipedia.org|name=English Wikipedia|language=en}}",

Cross-wiki (a.k.a. interwiki) search result title

If this is a new language Wikipedia, add a message with the wiki name to extensions/WikimediaMessages/i18n/wikimediainterwikisearchresults/en.json and qqq.json in the list of search-interwiki-results messages.

Wikipedia mobile apps

If you are creating a new Wikipedia, add an entry for the new language.

For the Android app, you must run the following two scripts, which will update the list of languages known to the app (including the corresponding language codes, autonyms, English names, etc.): 

$ cd scripts
$ ./generate_wiki_languages.py
$ ./make-templates.py

This should update several .java and .xml files that you may then submit in a pull-request to our repository.

(TODO: Is there anything to do for the iOS app?)

RESTBase

RESTBase is a service providing a RESTful API for the projects' wikis. This service is in a process of deprecation, below is an outline for the current process.

The RESTBase configuration should be deployed before installation of the wiki.

  • Create a patch for mediawiki/services/restbase/deploy adding its domain to the appropriate section in RESTBase's configuration.
  • Get it merged and deployed.

After the patch is merged, a restart of the RB service across the cluster is required, which can be done by ops or restbase-admins/restbase-roots.

Install

Database creation

Deploy the pre-install mediawiki-config commit with scap backport and then run

mwscript-k8s -f -- extensions/WikimediaMaintenance/addWiki.php --wiki=$db

Update sites by running this on the maintenance server:

foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol=https

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering of many wikis!

Deploy the post-install mediawiki-config commit.

cxserver

For project families in which the ContentTranslation is installed (as of 2018—only Wikipedia projects): Add the language code to the ContentTranslation registry - mediawiki/services/cxserver repository, file config/languages.yaml (included by config.dev.yaml and config.prod.yaml).

Once merged to master, ping the ContentTranslation developers to deploy the change. That requires to sync repositories, ie to update mediawiki/services/cxserver/deploy to match mediawiki/services/cxserver. See: https://gerrit.wikimedia.org/r/c/303763/ for an example commit.

Wiki Replicas

These steps are necessary to make the wiki available in the Wiki Replicas.

DBAs' work

First, the DBAs need to sanitize the data and set up the grants. This is described at MariaDB/PII.

Creating views

This part is handled by the Wikimedia Cloud Services team and is documented at Portal:Data_Services/Admin/Wiki_Replicas#Adding_a_wiki

Post-install

General announcement in Tech/News

Add an entry to m:Tech/News so that people know the new wiki exists, and can adjust their tools/bots/etc. as appropriate.

Data Engineering

Most wikis are added to the Pageviews definition, which means that the log events are extracted from the Webrequest stream and saved as distinct datasets.

Exceptions to this include:

  • private wikis
  • Wikimania conference wikis
  • Other special cases such as usability,outreach,login,vote,strategy etc.

If you are unsure about whether or not the wiki should be included,check with the Data Engineering team.

In order to add the new wiki to the pageview definition, submit a change proposal to the analytics/refinery repository, adding the wiki to static_data/pageview/allowlist/allowlist.tsv.

Incubator

If there's something to import (as is often the case in new language projects), someone will do so. Their process is described at Incubator:Importing from Incubator (logged at Incubator:Site creation log).

After any import Incubator is completed, Inform the community and make a Phabricator task for removing old interwiki links and migrating them to Wikidata (For example T134991 for edits such as d:Special:Diff/336584053 and jam:Special:Diff/12330). You can do it by yourself using interwikidata.py in pywikibot. 

python3 pwb.py scripts/interwikidata.py -lang:LANGCODE -clean -start:! -always

Tell wikistats Cloud VPS project to add the wiki

If the wiki is a public wiki, create a Phabricator task (subtask of the main task to create the wiki) with the tag "VPS-project-wikistats" and just ask for the wiki to be added. (What needs to be done can be seen f.e. in https://phabricator.wikimedia.org/T140970#2531977)

Tell Pywikibot project to add the wiki

Create a Phabricator task (subtask of the main task to create the wiki) with the tag "pywikibot" and just ask for the wiki to be added.

Mobile apps

Wikipedia has mobile apps for Android and iOS. A Wikipedia in a new language must work in the app after the import from the Incubator is complete. Report a bug under the apps tags in Phabricator if any of the following doesn't work:

  • You are supposed see the language in the user preferences.
    • Android: Settings -> Wikipedia language.
    • iOS: Settings -> My languages
  • You see the language in the interlanguage links list. Find an article that exists in the new Wikipedia and in English. Go to the English Wikipedia, tap the 文Α icon at the bottom and find the article in the list. In particular:
    • The article name must appear.
    • The language name (autonym) must appear.
    • The item must be findable using the language name in English and the autonym. (If you don't know how to type the autonym, try pasting the autonym or ask somebody who writes in that Wikipedia.)
    • Tapping the item must show the article.

Portal

Add the new site to the wikimedia/portals repository:

  • If the language is right-to-left and the site is a Wikipedia, make sure it appears in src/wikipedia.org/rtl-languages.json.
  • Check that the language name is written in translatewiki and exported to the appropriate json file under l10n/.

After creating the wiki and deploying the portal changes, test that the language name appears correctly in the wikipedia.org portal.

Kiwix

Tell Kiwix to add support for the new wiki. Kiwix is an external project, but it works quite closely with Wikimedia to provide regularly-update offline versions as ZIM files. To ask the adding of a wiki, file an issue on the https://github.com/openzim/zim-requests GitHub repository. Boilerplate for Wikipedia: 

Website URL: https://CODE.wikipedia.org
License: CC-BY-SA 3.0
Desired ZIM Icon –png (URL or attach one): https://drive.farm.openzim.org/wikipedia_all/favicon-48x48.png
Language (ISO 639-3): kcg
Is this a MediaWiki?: yes

Comments:

  • Make sure to correct the language code in the URL.
  • For non-Wikipedia projects, make sure that you use the correct URL.
  • If it's the first wiki in the language, make sure to mention the three-letter language code, the English name of the language and the autonym, so that Kiwix developers can add the language to their database.
  • If the default license ever changes from CC-BY-SA 3.0 to CC-BY-SA 4.0 (or something else), write it there, and update the documentation here.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Add_a_wiki&oldid=2256087"