Help talk:Toolforge/Running Pywikibot scripts (advanced)/Archives/2023

Please do not post any new comments on this page. This is a discussion archive See current discussion or the archives index.

Imagine a world in which every single tutorial works fine

Hello. I am emijrp. You may remember me from such bots as Emijrpbot, with over 50 million edits. I have used Wikipedia toolservers since the old days, remember the Toolserver located in Germany? That was back in the 2000s. Hehe, the server crashed while replicating the databases, poor server. Well, I have followed this tutorial and I can't install the damn thing. It fails. The step 3 never ends. When I do it manually, it fails when doing "pip install $HOME/pywikibot-core[mwoauth,mysql]". If I try to launch a shell "webservice python3.9 shell", it says forbidden. Everytime you migrate the tools to a new scheme, you make things worse. Goodbye, I will launch my bots from my machine. Thanks. Emijrp (talk) 08:21, 9 April 2023 (UTC)

Inability to run archivebot.py from k8s

Hi all, today, I've spent an hour or so debugging archivebot.py from Pywikibot's collection in k8s. It was failing with:

>>> [[cs:Diskuse s wikipedistou:Jan Myšák]] <<< ***
Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:Jan Myšák]]
12 thread(s) found on [[cs:Diskuse s wikipedistou:Jan Myšák]]
Processing 12 threads
24 thread(s) found on [[cs:Diskuse s wikipedistou:Jan Myšák/Archiv/2023]]
Archiving 7 thread(s).
ERROR: Error occurred while processing page [[cs:Diskuse s wikipedistou:Jan Myšák]]
Traceback (most recent call last):
  File "/data/project/urbanecmbot/pwb/scripts/archivebot.py", line 846, in process_page
    archiver.run()
  File "/data/project/urbanecmbot/pwb/scripts/archivebot.py", line 796, in run
    comment = i18n.twtranslate(self.site.code,
  File "/data/project/urbanecmbot/venv/lib/python3.9/site-packages/pywikibot/i18n.py", line 801, in twtranslate
    raise pywikibot.exceptions.TranslationError(fill(
pywikibot.exceptions.TranslationError: No English translation has been defined for TranslateWiki key
"archivebot-archive-summary". It can happen due to lack of i18n
submodule or files or an outdated submodule. See
https://www.mediawiki.org/wiki/Manual:Pywikibot/i18n ***

Unfortunately, https://www.mediawiki.org/wiki/Manual:Pywikibot/i18n doesn't provide any useful info, as it expects users to run Pywikibot from their git repo, rather than from a Python virtual environment (and the first option doesn't really work in k8s because of missing PWB-required packages in the base python image). This happens because ~/venv/lib/python3.9/site-packages/pywikibot/scripts/i18n misses the JSON files for all the scripts, even though the i18n module is present in the clonned ~/pywikibot-core.

I managed to workaround the problem by manually doing cp -r ~/pywikibot-core/scripts/i18n/* ~/venv/lib/python3.9/site-packages/pywikibot/scripts/i18n, but I'm fairly certain that (a) this is not an expected step (hence not amending the docs, merely recording here) (b) this will break when I update Pywikibot.

Does anyone know how to fix this properly? Urbanecm (talk) 21:58, 18 June 2023 (UTC)

@Urbanecm: I just did a fresh install from a fresh git clone and tested as below. I was not able to reproduce the error. Would you provide the command that you are running?

[22:19:24]tools.jjmc89-bot-dev@shell-1687558757:~> rm -fdr $HOME/pywikibot-core
[22:19:36]tools.jjmc89-bot-dev@shell-1687558757:~> rm -fdr pwbvenv
[22:19:46]tools.jjmc89-bot-dev@shell-1687558757:~> python3 -m venv pwbvenv
[22:20:07]tools.jjmc89-bot-dev@shell-1687558757:~> source pwbvenv/bin/activate
(pwbvenv) [22:21:48]tools.jjmc89-bot-dev@shell-1687558757:~> git clone --depth 1 --recursive --branch stable --shallow-submodules "https://gerrit.wikimedia.org/r/pywikibot/core" $HOME/pyw
ikibot-core
Cloning into '/data/project/jjmc89-bot-dev/pywikibot-core'...
remote: Counting objects: 595, done
remote: Finding sources: 100% (595/595)
remote: Getting sizes: 100% (507/507)
remote: Compressing objects: 100% (5072676/5072676)
remote: Total 595 (delta 115), reused 219 (delta 87)
Receiving objects: 100% (595/595), 1.86 MiB | 3.90 MiB/s, done.
Resolving deltas: 100% (115/115), done.
Updating files: 100% (556/556), done.
Submodule 'i18n' (https://gerrit.wikimedia.org/r/pywikibot/i18n) registered for path 'scripts/i18n'
Cloning into '/data/project/jjmc89-bot-dev/pywikibot-core/scripts/i18n'...
remote: Counting objects: 4092, done
remote: Finding sources: 100% (4092/4092)
remote: Getting sizes: 100% (3456/3456)
remote: Compressing objects: 100% (1204669/1204669)
remote: Total 4092 (delta 979), reused 1900 (delta 635)
Receiving objects: 100% (4092/4092), 886.37 KiB | 3.14 MiB/s, done.
Resolving deltas: 100% (979/979), done.
Submodule path 'scripts/i18n': checked out 'a2371dafd220c2839f5b21b6d504c50ca8cd7209'
(pwbvenv) [22:23:09]tools.jjmc89-bot-dev@shell-1687558757:~> pip install --upgrade pip setuptools wheel
Requirement already satisfied: pip in ./pwbvenv/lib/python3.9/site-packages (20.3.4)
Collecting pip
  Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 21.6 MB/s
Requirement already satisfied: setuptools in ./pwbvenv/lib/python3.9/site-packages (44.1.1)
Collecting setuptools
  Downloading setuptools-68.0.0-py3-none-any.whl (804 kB)
     |████████████████████████████████| 804 kB 38.6 MB/s
Collecting wheel
  Downloading wheel-0.40.0-py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 141 kB/s
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 44.1.1
    Uninstalling setuptools-44.1.1:
      Successfully uninstalled setuptools-44.1.1
  Attempting uninstall: pip
    Found existing installation: pip 20.3.4
    Uninstalling pip-20.3.4:
      Successfully uninstalled pip-20.3.4
Successfully installed pip-23.1.2 setuptools-68.0.0 wheel-0.40.0
(pwbvenv) [22:24:44]tools.jjmc89-bot-dev@shell-1687558757:~> pip install $HOME/pywikibot-core[mwoauth,mysql]
Processing ./pywikibot-core
  Preparing metadata (setup.py) ... done
Collecting mwparserfromhell>=0.5.2 (from pywikibot==8.1.2)
  Using cached mwparserfromhell-0.6.4-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (178 kB)
Collecting requests>=2.21.0 (from pywikibot==8.1.2)
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 997.5 kB/s eta 0:00:00
Requirement already satisfied: setuptools>=40.8.0 in ./pwbvenv/lib/python3.9/site-packages (from pywikibot==8.1.2) (68.0.0)
Collecting mwoauth!=0.3.1,>=0.2.4 (from pywikibot==8.1.2)
  Using cached mwoauth-0.3.8-py3-none-any.whl (13 kB)
Collecting PyMySQL>=0.9.3 (from pywikibot==8.1.2)
  Downloading PyMySQL-1.0.3-py3-none-any.whl (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.7/43.7 kB 2.5 MB/s eta 0:00:00
Collecting PyJWT>=1.0.1 (from mwoauth!=0.3.1,>=0.2.4->pywikibot==8.1.2)
  Downloading PyJWT-2.7.0-py3-none-any.whl (22 kB)
Collecting oauthlib (from mwoauth!=0.3.1,>=0.2.4->pywikibot==8.1.2)
  Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Collecting requests-oauthlib (from mwoauth!=0.3.1,>=0.2.4->pywikibot==8.1.2)
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting six (from mwoauth!=0.3.1,>=0.2.4->pywikibot==8.1.2)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting charset-normalizer<4,>=2 (from requests>=2.21.0->pywikibot==8.1.2)
  Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5 (from requests>=2.21.0->pywikibot==8.1.2)
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from requests>=2.21.0->pywikibot==8.1.2)
  Downloading urllib3-2.0.3-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB 5.3 MB/s eta 0:00:00
Collecting certifi>=2017.4.17 (from requests>=2.21.0->pywikibot==8.1.2)
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB 6.0 MB/s eta 0:00:00
Building wheels for collected packages: pywikibot
  Building wheel for pywikibot (setup.py) ... done
  Created wheel for pywikibot: filename=pywikibot-8.1.2-py3-none-any.whl size=613113 sha256=3b594c48a9d94dbd573af509c4cf9477b2696a227e8e7ccfd594db75958f7b2e
  Stored in directory: /data/project/jjmc89-bot-dev/.cache/pip/wheels/e3/1f/6a/9f0be8eee50f85385d0f30a1ab0eda8278cc40cee7206c31ec
Successfully built pywikibot
Installing collected packages: urllib3, six, PyMySQL, PyJWT, oauthlib, mwparserfromhell, idna, charset-normalizer, certifi, requests, requests-oauthlib, pywikibot, mwoauth
Successfully installed PyJWT-2.7.0 PyMySQL-1.0.3 certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 mwoauth-0.3.8 mwparserfromhell-0.6.4 oauthlib-3.2.2 pywikibot-8.1.2 requests-2.31.0 requests-oauthlib-1.3.1 six-1.16.0 urllib3-2.0.3


(pwbvenv) [22:35:28]tools.jjmc89-bot-dev@shell-1687558757:~> pywikibot-core/pwb.py archivebot -simulate Archivace -page:'Diskuse s wikipedistou:Jan Myšák' -lang:cs


>>> [[cs:Diskuse s wikipedistou:Jan Myšák]] <<<
Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:Jan Myšák]]
5 thread(s) found on [[cs:Diskuse s wikipedistou:Jan Myšák]]
Processing 5 threads
Only 0 (< 1) threads are old enough. Skipping

Execution time: 1 seconds
(pwbvenv) [22:35:48]tools.jjmc89-bot-dev@shell-1687558757:~> pywikibot-core/scripts/archivebot.py -simulate Archivace -page:'Diskuse s wikipedistou:Jan Myšák' -lang:cs


>>> [[cs:Diskuse s wikipedistou:Jan Myšák]] <<<
Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:Jan Myšák]]
5 thread(s) found on [[cs:Diskuse s wikipedistou:Jan Myšák]]
Processing 5 threads
Only 0 (< 1) threads are old enough. Skipping

Execution time: 1 seconds

— JJMC89 (T·C) 22:48, 23 June 2023 (UTC)

@JJMC89@Urbanecm I followed Help:Toolforge/Pywikibot and had the same issue. The instructions recommend to setup the venv via "toolforge-jobs run setup-venv --command $HOME/pwb_venv.sh --image python3.11 --wait", not from the shell. Mpaa (talk) 13:37, 2 November 2023 (UTC)

It does not matter how the venv is created.

When running from shell, this is used:

• /data/project/shared/pywikibot/stable/pywikibot/i18n.py, which will give this json file /data/project/shared/pywikibot/stable/scripts/i18n/archivebot/en.json

when running as toolforge job, this one is used:

• /data/project/wikisource-bot/pwbvenv/lib/python3.11/site-packages/pywikibot/i18n.py and no json file is found as they are searched relative to the path of i18n.

If the workaround is not applied, the error above appears. Mpaa (talk) 21:08, 2 November 2023 (UTC)

The shared pywikibot should not be involved in any way. I'd guess you are modifying your PATH or PYTHONPATH somewhere. Running in k8s shell or toolforge-jobs should make no difference if you aren't changing the environment in only one of them. — JJMC89 (T·C) 18:06, 4 November 2023 (UTC)

ModuleNotFoundError

@JJMC89: Hello. Sorry to bother you again. To add a dependency (lxml), I reinstalled the pywikibot a few hours ago. (pwbvenv) tools.TOOLNAME@shell-#########:~$ pwb version and tools.TOOLNAME@tools-sgebastion-10:~$ toolforge-jobs run pywikibot-version --command "$HOME/pwbvenv/bin/pwb version" --image python3.11 are working as expected. But when I am trying to run my script(s) through cron/yaml, I am getting the following error (in .err file):

Traceback (most recent call last):
  File "/data/project/kiranbot4/enwiki/mostrefs/scripts/temp.py", line 1, in <module>
    import pywikibot
  File "/data/project/shared/pywikibot/stable/pywikibot/__init__.py", line 49, in <module>
    from pywikibot.bot import (
  File "/data/project/shared/pywikibot/stable/pywikibot/bot.py", line 112, in <module>
    from pywikibot import config, daemonize, i18n, version
  File "/data/project/shared/pywikibot/stable/pywikibot/version.py", line 26, in <module>
    from pywikibot.comms.http import fetch
  File "/data/project/shared/pywikibot/stable/pywikibot/comms/http.py", line 45, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

I am following all the steps correctly, the only difference is following lines in the "pwb_venv.sh" file:

# install dependencies
pip install --upgrade pip setuptools wheel
pip install $HOME/pywikibot-core[html,mwoauth,mysql] pandas aiohttp aiofiles defusedxml pyexiv2 fuzzywuzzy lxml

I am not sure what is causing the ModuleNotFoundError. I had updated the dependencies similarly two-three weeks ago (fuzzywuzzy), and everything was working fine. Kindly let me know if you need more information from me. Regards, —usernamekiran (talk) 19:28, 12 November 2023 (UTC)

@usernamekiran: requests is installed in the venv.

$ /data/project/kiranbot4/pwbvenv/bin/pip freeze | grep requests
requests==2.31.0
requests-oauthlib==1.3.1

The shared pywikibot should not be involved in any way. I'd guess you are modifying your PATH or PYTHONPATH somewhere for that to be showing up in the traceback. Also, make sure your jobs yaml file is using the same image you used to create the venv. — JJMC89 (T·C) 17:32, 13 November 2023 (UTC)

@JJMC89: Thanks, this worked when I tested it in "traditional" yaml file. I updated image: docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest to image: docker-registry.tools.wmflabs.org/toolforge-python311-sssd-base:latest. I had thought of that before you had mentioned it. I had tried it in the recommended yaml file format at Help:Toolforge/Jobs_framework#Loading_jobs_from_a_YAML_file. Somehow, my file had image: tf-python39 which had thrown me off guard (not sure where did that tf come from, but it worked fine). After updating the image, now everything is everything fine. Thanks a lot for your help again. See you around. —usernamekiran (talk) 19:35, 13 November 2023 (UTC)