Help:Toolforge/Pywikibot

Toolforge

• Cloud Services overview
• Toolforge user docs

Get started

• Quickstart: set up and get access
• How Toolforge works
• Rules you must follow
• Tutorials

Build and run tools

• Navigate tool accounts and files
• Kubernetes backend
• Run a web service
• Schedule and manage jobs
• Language-specific details:
  • Python
  • Pywikibot
  • Node.js
  • PHP
  • ...more languages/frameworks
• Use Redis for caching
• Index content with Elasticsearch

Access shared storage and databases

• Access shared storage and public wiki dumps
• Access wiki replicas
• Access replica search indices
• Manage tool databases

Share and maintain tools

• Set up version control and code review
• Develop successful tools
• Find and share tools on Toolhub
• Deprecate a tool
• Contribute to Toolforge

Get help

• How and where to get help
• Troubleshooting

Useful links

• Toolforge admin docs
• List of tools
• Toolforge Admin Console (toolsadmin)

[]

The Pywikibot framework is a Python library and collection of scripts to automate work on MediaWiki sites.

Prerequisites

To effectively use Pywikibot on Toolforge, you need:

• Access to a tool account on Toolforge. See Help:Toolforge/Quickstart to learn how to set it up and use. Note that to use Toolforge you need a beginner-level understanding of Linux terminal, SSH (see Internet Technologies/SSH), and Bash (see Bash Shell Scripting).
• Basic familiarity with Python and Pywikibot. You don't need to know how to program in Python but it helps to understand how to run Python scripts, and manage virtual environments and package dependencies. To learn Python, see Non-Programmer's Tutorial for Python 3 or The Python Tutorial in Python documentation. The latter has a section on virtual environments and packages. For information about Pywikibot, see Manual:Pywikibot.

Installation and setup

See also: mw:Special:MyLanguage/Manual:Pywikibot/Installation

1. Become your tool-account

   MAINTAINER@tools-sgebastion-10:~$ become TOOLNAME
   

2. Create a setup script (e.g., pwb_venv.sh) that creates a virtual environment and installs dependencies.

   tools.TOOLNAME@tools-sgebastion-10:~$ touch $HOME/pwb_venv.sh
   tools.TOOLNAME@tools-sgebastion-10:~$ chmod ug+x $HOME/pwb_venv.sh  # make the script executable
   

   You need to install from git if you want to use the Pywikibot scripts.

Pywikibot installation source	Example
PyPI	pwb_venv.sh #!/bin/bash # clean-up directories if they already exist rm -fdr pwbvenv # create a virtual environment python3 -m venv pwbvenv # activate it source pwbvenv/bin/activate # install dependencies pip install --upgrade pip setuptools wheel pip install pywikibot[mwoauth,mysql] # update as needed pip install ... # any other dependencies here
git	pwb_venv.sh #!/bin/bash # clean-up directories if they already exist rm -fdr pwbvenv $HOME/pywikibot-core # create a virtual environment python3 -m venv pwbvenv # activate it source pwbvenv/bin/activate # clone Pywikibot git clone --recursive --branch stable "https://gerrit.wikimedia.org/r/pywikibot/core" $HOME/pywikibot-core # install dependencies pip install --upgrade pip setuptools wheel pip install $HOME/pywikibot-core[mwoauth,mysql] # update as needed pip install ... # any other dependencies here

3. Run the setup script using the jobs framework.

   tools.TOOLNAME@tools-sgebastion-10:~$ toolforge-jobs run setup-venv --command $HOME/pwb_venv.sh --image python3.11 --wait
   INFO: job 'setup-venv' completed
   

4. Configure Pywikibot
   • Interactively:

     tools.TOOLNAME@tools-sgebastion-10:~$ webservice python3.11 shell  # start an interactive Kubernetes shell
     tools.TOOLNAME@shell-#########:~$ source pwbvenv/bin/activate  # activate the virtual environment
     (pwbvenv) tools.TOOLNAME@shell-#########:~$ pwb generate_user_files  # run the generate_user_files.py Pywikibot script
     

     user-config.py and (optionally) user-password.py are generated by following the prompts.
   • Manually: See the manual on MediaWiki wiki: user-config.py, BotPasswords, and OAuth.
5. You may test the installation from an interactive shell.

   tools.TOOLNAME@tools-sgebastion-10:~$ webservice python3.11 shell  # start an interactive Kubernetes shell
   tools.TOOLNAME@shell-#########:~$ source pwbvenv/bin/activate  # activate the virtual environment
   (pwbvenv) tools.TOOLNAME@shell-#########:~$ pwb version  # run the version.py Pywikibot script
   Pywikibot: pywikibot/__init__.py (, -1 (unknown), 2023/01/21, 21:32:22, UNKNOWN)
   Release version: 8.0.0
   setuptools version: 66.1.1
   mwparserfromhell version: 0.6.4
   wikitextparser version: n/a
   requests version: 2.28.2
     cacerts: /data/project/TOOLNAME/pwbvenv/lib/python3.11/site-packages/certifi/cacert.pem
       certificate test: ok
   Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
   [GCC 10.2.1 20210110]
   PYWIKIBOT_DIR: Not set
   PYWIKIBOT_DIR_PWB: /data/project/TOOLNAME/pwbvenv/lib/python3.11/site-packages/pywikibot/scripts
   PYWIKIBOT_NO_USER_CONFIG: Not set
   Config base dir: /data/project/TOOLNAME
   Usernames for family 'commons':
           commons: Example bot
   

Create jobs

Main article: Help:Toolforge/Jobs framework

Example of running the Pywikibot version.py script once:

tools.TOOLNAME@tools-sgebastion-10:~$ toolforge-jobs run pywikibot-version --command "$HOME/pwbvenv/bin/pwb version" --image python3.11

Example of running your own script on a schedule:

tools.TOOLNAME@tools-sgebastion-10:~$ toolforge-jobs run my-cool-job --command "$HOME/pwbvenv/bin/python3 $HOME/my-cool-script.py" --image python3.11 --schedule "1 2 * * 3"

Additional parameters for the job could include, for example, additional memory allocation (--mem MEM), job restart after being finished (--continuous), etc.

OAuth

Pywikibot can use OAuth v1.0a as the method of authenticating with wikis. To configure this behavior for Wikimedia wikis, follow the instructions in Manual:Pywikibot/OAuth/Wikimedia.

Caution:

When using OAuth as the authentication method, be sure to set the correct permissions on the user-config.py file as described in Manual:Pywikibot/OAuth. This will keep your OAuth credentials more secure.

To learn more about OAuth in general, see OAuth/For Developers.

Next steps

Now that you know how to run your bot on Toolforge, read the script development and maintenance tips described in Help:Toolforge/Developing successful tools. These tips will help you to license, publish, and support your bot over time.

If you are looking for ideas on what types of scripts you can run using Pywikibot, see Manual:Pywikibot/Scripts. If you want to write your own script, see Manual:Pywikibot/Cookbook.

If you want to teach others how to run bots on Toolforge, see the How to host a bot on Toolforge - workshop materials.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

• Chat in real time in the IRC channel #wikimedia-cloud ^connect, the bridged Telegram group, or the bridged Mattermost channel
• Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

• Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
• Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)