From Wikitech
Jump to navigation Jump to search
Toolforge tools
Crystal Clear app package utilities.png CommTech
Description Hosts Community Tech's test wiki and bot
Maintainer(s) Community Tech (View all)
License MIT License
Issues Open tasks · Report a bug

This page is for documentation related to the commtech VPS project.

CommTech Wiki

CommTech Wiki is a MediaWiki-Vagrant instance used for internal testing of MediaWiki features. It lives on the VPS instance commtech-2.commtech.eqiad.wmflabs. The wiki installation is located in the /srv/mediawiki-vagrant directory on that server.

Community Tech bot

This section documents bot tasks that live on the commtech-bot.commtech.eqiad.wmflabs instance.

Creating a new instance

  1. Create a new instance (m1.medium or larger)
  2. Install PHP along with some dependencies (using a Ondřej Surý Debian package).
    sudo apt-get install apt-transport-https lsb-release ca-certificates
    sudo wget -O /etc/apt/trusted.gpg.d/php.gpg
    echo "deb $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
    sudo apt-get update
    sudo apt-get install -y php7.2 php7.2-cli php7.2-common php7.2-curl php7.2-json
    sudo apt-get install -y zip unzip php7.2-zip
  3. Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.:
    sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
  4. Create the bot directory with sudo mkdir /var/bot
  5. Create the commtech-bot user and give it ownership of the bot directory.
    sudo useradd -M commtech-bot 
    sudo usermod -L commtech-bot
    sudo chown commtech-bot:commtech-bot /var/bot

Popular Pages bot

After creating an instance (see above), do the following:

  1. Clone the repo in the bot directory:
    sudo su commtech-bot # Make sure you're the bot user
    cd /var/bot
    git clone
  2. Follow the installation steps at
  3. Edit the crontab for commtech-bot with crontab -e then add the following to the bottom:
    */30 * 3-11 * * /usr/bin/flock -n /tmp/fcj.lockfile /var/bot/popularpages/checkReports.php en.wikipedia
    0 */2 3-9 * * php /var/bot/popularpages/generateIndex.php en.wikipedia
    The first entry continually runs the main bot task ever 30 minutes for most of the month (it takes that long!). Using flock ensures there's only ever one instance of the bot running at a time. The second entry refreshes the index page every two hours so that users can monitor progress of the bot.

Note that if you make any modifications to the /var/bot directory as any other user than commtech-bot, you must restore ownership to commtech-bot. For instance to pull in the latest changes from master, you could use:

cd /var/bot
sudo git pull origin master
sudo chown commtech-bot:commtech-bot .


The NSFW instance is at commtech-nsfw.commtech.eqiad.wmflabs. This is a NSFW classifier service, using Yahoo!'s open source algorithm, wrapped in a Docker container.

See for installation instructions.

To query the service, make a POST request with cURL:

$ curl -d 'url='


Creating a new instance

  1. Create a new m1.small instance.
  2. Run:
    sudo apt-get update
    sudo apt-get install -y apt-transport-https lsb-release ca-certificates
    sudo apt-get install -y apache2 libxml2-dev
  3. Create the WikiWho credentials file with sudo htpasswd -c /etc/apache2/passwd-nospecial "[username]" "[password]"
  4. Make note of the hash for the password with echo -n "WhoWroteThat:[password]" | base64 replacing [password] with the password for the WikiWho account.
  5. Create /etc/apache2/sites-available/wikiwho.conf as root with the contents:
    <VirtualHost *:80>
            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog ${APACHE_LOG_DIR}/access.log combined
            <Directory />
                    AuthUserFile /etc/apache2/htpasswd
                    AuthName "Identify"
                    AuthType Basic
                    require valid-user
            RequestHeader set Authorization "Basic [hash]"
            SSLProxyEngine on
            ProxyPass /
            ProxyPassReverse /
            RewriteEngine On
            RewriteCond %{HTTP:X-Forwarded-Proto} !https
            RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
    replacing [hash] with the output of step #4.
  6. Run:
    sudo a2enmod rewrite
    sudo a2ensite wikiwho
    sudo a2enmod proxy proxy_http proxy_ajp proxy_wstunnel rewrite deflate headers proxy_balancer proxy_connect proxy_html xml2enc ssl
    sudo service apache2 reload
    sudo service apache2 restart