Tool:CommTech

From Wikitech
Jump to navigation Jump to search
Toolforge tools
Crystal Clear app package utilities.png CommTech
Website https://commtech.wmflabs.org
Description Hosts Community Tech's test wiki and bot
Maintainer(s) Community Tech (View all)
License MIT License
Issues Open tasks · Report a bug

This page is for documentation related to the commtech VPS project.

CommTech Wiki

CommTech Wiki is a MediaWiki-Vagrant instance used for internal testing of MediaWiki features. It lives on the VPS instance commtech-2.commtech.eqiad.wmflabs. The wiki installation is located in the /srv/mediawiki-vagrant directory on that server.

Community Tech bot

This section documents bot tasks that live on the commtech-bot.commtech.eqiad.wmflabs instance.

Creating a new instance

  1. Create a new instance (m1.medium or larger)
  2. Install PHP along with some dependencies (using a Ondřej Surý Debian package).
    sudo apt-get install apt-transport-https lsb-release ca-certificates
    sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
    echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
    sudo apt-get update
    sudo apt-get install -y php7.2 php7.2-cli php7.2-common php7.2-curl php7.2-json
    sudo apt-get install -y zip unzip php7.2-zip
    
  3. Install composer by following these instructions, but make sure to install to the /usr/local/bin directory and with the filename composer, e.g.:
    sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
    
  4. Create the bot directory with sudo mkdir /var/bot
  5. Create the commtech-bot user and give it ownership of the bot directory.
    sudo useradd -M commtech-bot 
    sudo usermod -L commtech-bot
    sudo chown commtech-bot:commtech-bot /var/bot
    

Popular Pages bot

After creating an instance (see above), do the following:

  1. Clone the repo in the bot directory:
    sudo su commtech-bot # Make sure you're the bot user
    cd /var/bot
    git clone https://github.com/wikimedia/popularpages.git
    
  2. Follow the installation steps at https://github.com/wikimedia/popularpages
  3. Edit the crontab for commtech-bot with crontab -e then add the following to the bottom:
    */30 * 3-11 * * /usr/bin/flock -n /tmp/fcj.lockfile /var/bot/popularpages/checkReports.php en.wikipedia
    0 */2 3-9 * * php /var/bot/popularpages/generateIndex.php en.wikipedia
    
    The first entry continually runs the main bot task ever 30 minutes for most of the month (it takes that long!). Using flock ensures there's only ever one instance of the bot running at a time. The second entry refreshes the index page every two hours so that users can monitor progress of the bot.

Note that if you make any modifications to the /var/bot directory as any other user than commtech-bot, you must restore ownership to commtech-bot. For instance to pull in the latest changes from master, you could use:

cd /var/bot
sudo git pull origin master
sudo chown commtech-bot:commtech-bot .

NSFW

The NSFW instance is at commtech-nsfw.commtech.eqiad.wmflabs. This is a NSFW classifier service, using Yahoo!'s open source algorithm, wrapped in a Docker container.

See https://github.com/rahiel/open_nsfw-- for installation instructions.

To query the service, make a POST request with cURL:

$ curl -d 'url=https://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Jimmy_Wales_in_August_2006.jpg/319px-Jimmy_Wales_in_August_2006.jpg' https://nsfw.wmflabs.org
0.006064464803785086