Help:Toolforge/My first Pywikibot tool

From Wikitech
Jump to navigation Jump to search

Warning Caution: This page may contain inaccuracies. It is currently being edited and redesigned for better readability. For further information, please see T134495.

.

Pywikibot is a Python library which automates work on MediaWiki sites. Use this guide to create your own tool based on Pywikibot and host it on the Toolforge hosting enviornment.

Request access to Toolforge

Before creating your tool, you need to create a Wikimedia developer account and request access Toolforge:

  1. Create a Wikimedia developer account from the Toolforge admin console. This account is used for most developer activities including editing this wiki, accessing code repositories, and accessing the bug tracking system. Take note of your LDAP username and your UNIX shell username:
  2. Fill out an access request to join the Toolforge project. You will receive a reply on your Wikitech talk page.

Accessing Toolforge for the first time

Once your request to access Toolforge is approved, you can set up ssh access to Toolforge severs:

Generate a new SSH key for your account

  1. From a Linux terminal, you can generate a new ssh key using the ssh-keygen command:
    ssh-keygen -t rsa -C "your_email@youremail.com"

    Select a passphrase for your ssh key and press Enter. The command will create two files in your ~/.ssh directory:

    ~/.ssh/id_rsa : identification (private) key
    ~/.ssh/id_rsa.pub : public key
    
  2. Add your public key to your Wikimedia developer account in the SSH Keys page of the Toolforge admin console. Copy the contents of ~/.ssh/id_rsa.pub into the New SSH Key field and click Add SSH key.

Log into Toolforge with ssh

After you add your public key to your Wikimedia developer account, you can use your private key to log into the Toolforge servers. From a Linux command line, use ssh to log in with your UNIX shell username:

ssh -i ~/.ssh/id_rsa <unix shell username>@login.toolforge.org

Create your Pywikibot Tool account

In addition to your own Toolforge account, you also need to create a Tool acoount for your Pywikibot tool.

  1. Navigate to the Create Tool Account page in the Toolforge admin console.
  2. Enter a Unique Tool Name for your tool account.
    Do not prefix your tool name with tools. as this prefix will cause errors.

Note: If you only recently received access to the tools project, you may get an error about appropriate credentials. Log out and back in to fix the issue.

Within a minute or two, Toolforge creates the Tool account and grants you access. If you were logged in through ssh when you created the Tool account, you must log off and log in again

Access your Pywikibot tool

After you create your Pywikibot's tool account, you can log into your tool project with SSH.

$ ssh <unix shell username>@login.toolforge.org

From the command line, switch to your tool account:

become <toolname>

You should see the command prompt change to:

tools.<toolname>@tools-bastion:~$

Now, you can install Pywikibot.

Install Pywikibot

For most purposes, using the centralized 'core' files is recommended. The shared files are available at /data/project/shared/pywikibot/stable, and steps for configuring your tool account are provided below. The configuration files themselves are stored in your tool account in the $HOME/.pywikibot directory, or another directory, where they can be used via the -dir option (all of this is described in more detail in the instructions).

If you are a developer and/or would like to control when the code is updated, please see Installing Pywikibot locally for instructions.

To set up your Tools account to use the shared 'core' framework:

1. Become your tool-account

maintainer@tools-login:~$ become toolname

2. In your home directory, create (or edit, if it exists already) a '.bash_profile' file:

nano .bash_profile

and include the following line:

export PYTHONPATH=/data/project/shared/pywikibot/stable:/data/project/shared/pywikibot/stable/scripts

The path should be on one line, though it may appear to be on multiple lines depending on your screen width. When you save the .bash_profile file (CTRL+X), your settings will be updated for all future shell sessions.

3. Import the path settings into your current session:

tools.tool@tools-login$ source .bash_profile

4. In your home directory, create a subdirectory named '.pywikibot' (the '.' is important!) for bot-related files:

tools.tool@tools-login$ mkdir $HOME/.pywikibot
example of configuration for commons.wikimedia.org

5. Configure Pywikibot.

To create configuration files, use the following command and then follow the instructions. You may also use an existing configuration file (e.g., 'user-config.py') that works on another system by copying it into your .pywikibot directory:

tools.tool@tools-login$ python3 /data/project/shared/pywikibot/stable/pywikibot/scripts/generate_user_files.py

6. Test out your setup. In general, all jobs should be run on the grid, but it's fine to test your setup on the command line. You should see the following terminal output (or something similar):

tools.tool@tools-login$ python3 /data/project/shared/pywikibot/stable/pywikibot/scripts/version.py
Pywikibot: [https] r-pywikibot-core.git (1db1f28, g15095, 2021/05/31, 14:35:28, stable)
Release version: 6.3.0
requests version: 2.12.4
  cacerts: /etc/ssl/certs/ca-certificates.crt
    certificate test: ok
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516]

Note that you do not need to run scripts using pwb.py, but run scripts directly, e.g., python3 /data/project/shared/pywikibot/stable/pywikibot/scripts/version.py. Setting PYTHONPATH means that you no longer need the pwb.py helper script to make, say, import pywikibot work. Anyway the pwb.py helper script has additional advantages like ignoring typing mistakes for script names, script path redirection, dependency checks, see pwb script documentation.

If you need to use multiple user-config.py files, you can do so by adding -dir:<path where you want your user-config.py> to every python command. To use the local directory, use -dir:. (colon dot).

For more information about Pywikibot, please see the Pywikibot documentation. The pywikibot mailing list (pywikibot at lists.wikimedia.org) and IRC channel (#pywikibot connect) are good places to go for additional help. Other useful information about using the centralized 'core' files is available here: User:Russell Blau/Using pywikibot on Labs

Warning Caution: Script path for Pywikibot framework utility scripts (generate_family_file.py, generate_user_files.py, shell.py, version.py) has been changed in core (master) branch with release 7.0.0. To use them the path is /data/project/shared/pywikibot/core/pywikibot/scripts/<script_name> or it can be invoked by the pwb.py wrapper script. See also: https://doc.wikimedia.org/pywikibot/master/utilities/index.html


Choose a license

Pywikibot comes with an MIT LICENSE file. Make sure to choose a license for your tool early on. Also see, Help:Tool_Labs/Developing#Licensing_your_source_code

Documentation

Set up a webpage for your tool

You can add webpage for your tool under https://TOOLNAME.toolforge.org. For example, https://my-first-pywikibot-tool.toolforge.org. Once you set up a webpage for your tool, the Tool list will link to it.

To set up webpage for you tool:

  1. Log in with your Tool account.
  2. Create a ~/public_html directory.
  3. Create ~/public_html/index.html.
  4. Start the web service:
$ webservice start

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Receive mail announcements about critical changes
Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Track work tasks and report bugs
Use the Phabricator workboard #Cloud-Services for bug reports and feature requests about the Cloud VPS infrastructure itself
Learn about major near-term plans
Read the News wiki page
Read news and stories about Wikimedia Cloud Services
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)