Tool:CommTech

Toolforge tools
CommTech
Website	https://commtech.wmflabs.org
Description	Hosts Community Tech's test wiki and bot
Maintainer(s)	Community Tech (View all)
License	MIT License
Issues	Open tasks · Report a bug

This page is for documentation related to the commtech VPS project.

CommTech Wiki

URL: https://commtech.wmflabs.org
License: MIT (MediaWiki Vagrant), GPL-2.0+ (MediaWiki)

CommTech Wiki is a MediaWiki-Vagrant instance used for internal testing of MediaWiki features. It lives on the VPS instance commtech-2.commtech.eqiad.wmflabs. The wiki installation is located in the /srv/mediawiki-vagrant directory on that server.

Community Tech bot

This section documents bot tasks that live on the commtech-bot.commtech.eqiad.wmflabs instance.

Creating a new instance

Create a new instance (m1.medium or larger)

Install PHP along with some dependencies

sudo apt update
sudo apt install -y php php-cli php-common php-curl php-json zip unzip php-zip php-mysql php-dom
sudo apt install composer cron

Create the bot directory with sudo mkdir /var/bot

Create the commtech-bot user and give it ownership of the bot directory.

sudo useradd -M commtech-bot 
sudo usermod -L commtech-bot
sudo chown commtech-bot:commtech-bot /var/bot

Popular Pages bot

Source: https://github.com/wikimedia/popularpages
License: MIT

After creating an instance (see above), do the following:

Clone the repo in the bot directory:

sudo su commtech-bot # Make sure you're the bot user
cd /var/bot
git clone https://github.com/wikimedia/popularpages.git

Follow the installation steps at https://github.com/wikimedia/popularpages

Edit the crontab for commtech-bot with crontab -e then add the following to the bottom:

# enwiki
*/10 * 4-11 * * /usr/bin/flock -n /tmp/fcj.lockfile php /var/bot/popularpages/bin/checkReports.php en.wikipedia
0 */2 4-11 * * php /var/bot/popularpages/bin/generateIndex.php en.wikipedia

# arwiki
*/10 * 3-6 * * /usr/bin/flock -n /tmp/fcj-arwiki.lockfile php /var/bot/popularpages/bin/checkReports.php ar.wikipedia
0 */2 3-6 * * php /var/bot/popularpages/bin/generateIndex.php ar.wikipedia

The first entry continually runs the main bot task ever 30 minutes for most of the month (it takes that long!). Using flock ensures there's only ever one instance of the bot running at a time. The second entry refreshes the index page every two hours so that users can monitor progress of the bot.

Note that if you make any modifications to the /var/bot directory as any other user than commtech-bot, you must restore ownership to commtech-bot. For instance to pull in the latest changes from master, you could use:

cd /var/bot
sudo git pull origin master
sudo chown commtech-bot:commtech-bot .

NSFW

URL: https://nsfw.wmflabs.org
Source: https://github.com/rahiel/open_nsfw--
License: https://github.com/rahiel/open_nsfw--/blob/master/LICENSE.txt

The NSFW instance is at commtech-nsfw.commtech.eqiad.wmflabs. This is a NSFW classifier service, using Yahoo!'s open source algorithm, wrapped in a Docker container.

See https://github.com/rahiel/open_nsfw-- for installation instructions.

To query the service, make a POST request with cURL:

$ curl -d 'url=https://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Jimmy_Wales_in_August_2006.jpg/319px-Jimmy_Wales_in_August_2006.jpg' https://nsfw.wmflabs.org
0.006064464803785086

WikiWho

URL: https://wikiwho.wmflabs.org

Creating a new instance

Create a new m1.small instance.

Run:

sudo apt-get update
sudo apt-get install -y apt-transport-https lsb-release ca-certificates
sudo apt-get install -y apache2 libxml2-dev

Create the WikiWho credentials file with sudo htpasswd -c /etc/apache2/passwd-nospecial "[username]" "[password]"
Make note of the hash for the password with echo -n "WhoWroteThat:[password]" | base64 replacing [password] with the password for the WikiWho account.

Create /etc/apache2/sites-available/wikiwho.conf as root with the contents:

<VirtualHost *:80>
        ServerName wikiwho.wmflabs.org
        ServerAdmin tools.wikiwho@tools.wmflabs.org

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <Directory />
                AuthUserFile /etc/apache2/htpasswd
                AuthName "Identify"
                AuthType Basic
                require valid-user
        </Directory>

        RequestHeader set Authorization "Basic [hash]"

        SSLProxyEngine on
        ProxyPass / https://api.wikiwho.net/
        ProxyPassReverse / https://api.wikiwho.net/

        RewriteEngine On

        RewriteCond %{HTTP:X-Forwarded-Proto} !https
        RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
</VirtualHost>

replacing [hash] with the output of step #4.

Run:

sudo a2enmod rewrite
sudo a2ensite wikiwho
sudo a2enmod proxy proxy_http proxy_ajp proxy_wstunnel rewrite deflate headers proxy_balancer proxy_connect proxy_html xml2enc ssl
sudo service apache2 reload
sudo service apache2 restart

Pageviews

URL: https://pageviews.wmcloud.org

Creating a new instance

Create a new m1.small instance.

Install Apache, PHP and some dependencies:

sudo apt update && sudo apt -y upgrade
sudo apt -y install php php-common
sudo apt -y install php-cli php-fpm php-json php-xml php-mysql php-intl php-zip php-curl php-apcu
sudo apt -y install apache2 libapache2-mod-php zip unzip default-mysql-client

Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.
```
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
```

Clone the repository, first removing the html directory created by Apache.

cd /var/www && sudo rm -rf html
sudo git clone https://github.com/MusikAnimal/pageviews.git
cd /var/www/pageviews

Become the root user with sudo su root
Run sudo composer install --no-dev -o
Run sudo cp config.sample.php config.php and edit config.php accordingly (see inline comments).
Make sure that all the files in the repo are owned by www-data.
```
sudo chown -R www-data:www-data .
```

Create the web server configuration file at /etc/apache2/sites-available/pageviews.conf with the following:

<VirtualHost *:80>
        DocumentRoot /var/www/public_html/
        ServerName pageviews.wmcloud.org

        CustomLog ${APACHE_LOG_DIR}/access.log pageviews
        ErrorLog ${APACHE_LOG_DIR}/error.log

        <Directory /var/www/public_html/>
                Options Indexes FollowSymLinks
                AllowOverride None
                Require all granted
        </Directory>
</VirtualHost>

Enable the mod-rewrite Apache module, and enable the web server configuration.

sudo a2dismod mpm_event
sudo a2enmod php7.4
sudo a2enmod rewrite
sudo a2ensite pageviews
sudo service apache2 reload

(Re)start Apache:
```
sudo service apache2 restart
```
Moving forward, you should use sudo service apache2 graceful to restart the server.
[TODO] Add symlinks (this should probably go in version control)