Help talk:Toolforge/Archives/2013

Please do not post any new comments on this page. This is a discussion archive See current discussion or the archives index.

Best practice hints

What I miss are best practice hints for two things:

• How to publish code? Should we all use github? Or is gerrit the better way? A link to a small tutorial would be nice.

• How to document a tool and how to communicate with the users. I can say, we use often a page inside Wikipedia because this was the easiest way for normal user to come in contact with us, we had an english version and mostly a version in our native language (German). Is it better to use the wikitech-wiki or the Mediawiki-wiki? Which wiki use "Single User Login".

Everybody should be free to go other ways but it would help if the majority use one way. --kolossos (talk) 10:04, 31 May 2013 (UTC)

How to add and modify a description for an project

I want to add a description at the list of https://tools.wmflabs.org/ for one of my tools. How can I do this? --kolossos (talk) 10:04, 31 May 2013 (UTC)

See Nova Resource:Tools/Help#Tool Labs's landing page. Anomie (talk) 13:18, 31 May 2013 (UTC)

Thanks.--kolossos (talk) 15:27, 31 May 2013 (UTC)

GUI tool for databasework

At the hackathon in Amsterdam it was mentioned to be possible to use an external GUI tool to interact with the database that runs on your own machine rather then to use PHPMyAdmin. What settings should be used for that?

Henna (talk) 11:21, 23 June 2013 (UTC)

Basically, it's the same settings as at Toolserver, except that you replace enwiki-p.userdb.toolserver.org with enwiki.labs (and username@nightshade.toolserver.org with username@tools-login.wmflabs.org). --Tim Landscheidt (talk) 12:21, 23 June 2013 (UTC)

300-350M? Really?

I was tempted to add this to Nova Resource:Tools/Help#Why am I getting errors about libgcc_s.so.1 must be installed for pthread_cancel to work?, but then I thought it might be too much detail so I decided to put it here for others to decide about.

Experiments in July 2013 with "do nothing for 60 seconds" scripts in various languages gave the following results:

Language	h_vmem needed	Code
C	5M	#include <unistd.h> int main(void){ return sleep(60); }
Lua	14M	local clock = os.clock local t0 = clock() + 60 while clock() < t0 do end
Perl	20M	sleep 60;
Python	30M	from time import sleep; sleep(60);
PHP	350M	sleep(60);
NodeJS	750M	var e = new Date().getTime() + 60000; while (new Date().getTime() <= e) {}

Of course, real scripts in most of these languages will load various modules/libaries and consequently have higher memory requirements. PHP may be somewhat of an exception here, as its 350M already includes a large number of such extensions (although any additional extensions or libraries (e.g. from PEAR) loaded would still increase the number). Anomie (talk) 02:21, 21 August 2013 (UTC)

I also tried a simple Java program. But I must have been doing something wrong, because it complained that it didn't have enough memory until I gave it 3.2G. Anomie (talk) 02:21, 21 August 2013 (UTC)

Java (un)helfully allocates most of its heap on startup; you can control that behavior with the -Xms and -Xmx command-line options. — Coren/Marc ^(talk) 22:55, 22 August 2013 (UTC)

Specifying 32M for both of those options makes it slightly better. Now it "only" needs 1.2G. Anomie (talk) 13:11, 23 August 2013 (UTC)

With jamvm, I can get a max heap space of 300M with "just" 750M. Shrinking the heap space doesn't help much anymore. Growing the availible space to 2750M allowed me to run with a max heap space of 2300M, so that at least seems to scale. Jamvm is installed on all nodes, and is called with -jamvm. Martijn Hoekstra (talk) 21:04, 15 October 2013 (UTC)

Public tool database?

Can I create a database as a tool user that is visible to other tools? On toolserver, we did that by adding "_p" to the database name. Should be documented, even if it's "you can't do that here". --Magnus Manske (talk) 15:47, 22 August 2013 (UTC)

It should indeed. You have two choices:

1. We do keep to the "_p" convention, by default every database user has "SELECT on %_p.*" and so can read databases named that way; or
2. Since users have all privileges on databases they are allowed to create (username__%) with grant option, you can grant the appropriate to any or all users with a GRANT statement.

— Coren/Marc ^(talk) 22:52, 22 August 2013 (UTC)

Does 20% of this page really need to be about pywikipedia?

I know that pywikipedia is a popular tool, but it seems excessive for a fifth of this general help page to be about using it. At the moment and not counting the "TOCright" template, 16417 of 85463 bytes of wikitext (19.2%) and 230 of 1172 lines of wikitext (19.6%) are in the "Pywikipedia" section. Anomie (talk) 13:08, 3 September 2013 (UTC)

Naming of user-created databases

Do the user-created databases really have to start with some "random" string like "p50252g21636"? Why? Couldn't the database names be made more user-friendly, like using the normal user name? svick (talk) 12:34, 26 September 2013 (UTC)

Because usernames are not unique across projects. — Coren/Marc ^(talk) 20:02, 15 October 2013 (UTC)

So what about something like projectname_username? I think that would still be a big improvement over p50252g21636. Svick (talk) 20:21, 15 October 2013 (UTC)

Potential for disallowed characters in the project or user names, probably. "p50252g21636" is more or less projectid_userid. Anomie (talk) 13:04, 16 October 2013 (UTC)

The biggest issue is Mysql's hard-coded username length limit. — Coren/Marc ^(talk) 16:13, 29 December 2013 (UTC)

Getting help

Hi. I am not sure where to post this so please let me know if there is a more appropriate place. As described in more detail here, I have gotten a Wikipedia template filler to work "internally" from the command line on the Tools Lab server, but I am having trouble accessing from an external web browser. I would be very grateful if someone with knowledge of perl and Tools Lab could help me. Thanks. Boghog (talk) 20:11, 27 October 2013 (UTC)

The above mentioned problem is the inability of the perl cgi script to access a local library even though the library in question has world read and execute privileges. There is no mention of this on the help page. Any suggestions on how to get this to work would be greatly appreciated. Thanks. Boghog (talk) 12:02, 28 October 2013 (UTC)

OK, I think I have solved most the problems. The key was to add the following to the cgi script. This made debugging the library much easier:

use CGI::Carp 'fatalsToBrowser';

Adding this single line gave some very useful feedback that indicated despite using cpanm for the installs, there were a still a few unfulfilled dependencies. After installing these missing modules, the script finally worked. I still need to get url and isbn options to work, but at least the most commonly used options now work. The link for running the tool is here: citation-template-filling. Boghog (talk) 15:57, 1 November 2013 (UTC)

MySQL Workbench

MySQL Workbench doesn't work for me the way described. I always get the following error message: "Bad authentication type, the server is not accepting this type of authentication.". I'm using the same private key file which works for normal SSH access. Maybe this is something like http://bugs.mysql.com/bug.php?id=61633 . I wish I could just use phpMyAdmin... I'm really no friend of managing databases on command line. --APPER (talk) 19:07, 31 October 2013 (UTC)

I now did the SSH tunneling using my SSH client and used MySQL Workbench only for the SQL stuff. --APPER (talk) 13:47, 2 November 2013 (UTC)

submitting git update on a directory

Hello,

I would like to run a git update everyday, but can't find how to submit it. My Pywikipedia repository is "/data/project/herculebot/pywikibot-compat/". I would like to run "git pull" on this directory with qsub. How to do so ?

--Hercule (talk) 21:27, 26 November 2013 (UTC)

Schedule a cronjob to run a command along the lines of cd /data/project/herculebot/pywikibot-compat/ && /usr/local/bin/jsub -cwd -N git-pull /usr/bin/git pull. I haven't tested it, but that should be about right. If you really want to use qsub instead of jsub, see /usr/local/bin/jsub and work out what qsub command it winds up using. Anomie (talk) 14:53, 27 November 2013 (UTC)

Impossible to submit Java program

Hello,

I'm unable to run my Java script using jsub.

In the terminal, logged on tools.login I can run the following command line with no problem :

java -Xmx256M -Xms256M -jar /data/project/herculebot/Wikipedia.jar

When I edit my crontab, I add the following line :

54 * * * * cd /data/project/herculebot/ && /usr/local/bin/jsub -once -mem 256m -quiet -N Cacographies -j y java -Xmx256M -Xms256M -jar /data/project/herculebot/Wikipedia.jar

I always have the same error on my Cacographies.out file : Error occurred during initialization of VM Could not reserve enough space for object heap Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I tried to give more memory, with -mem 768m, but it's not better.

Can someone help me to solve this issue ?

Thanks

--Hercule (talk) 21:01, 1 December 2013 (UTC)

To close this here as well: This was discussed on the mailing list, and the suggestion there was to increase the memory requested massively with 2 GByte apparently enough to satisfy Java :-). --Tim Landscheidt (talk) 02:49, 11 December 2013 (UTC)

add permissions around take to docs?

from IRC just now:

<brainwane> I'm having trouble "take"ing a file 
<brainwane> $ take /home/brainwane/biographies.txt
<brainwane> /home/brainwane/biographies.txt: you must own the containing directory
<brainwane> where do I put this file so my tools acct can "take" it?
<brainwane> petan: any ideas?
<andrewbogott> tools accounts usually have their own home directory
<Coren> brainwane: In any directory your tool account owns.
* chippy has quit (Remote host closed the connection)
<andrewbogott> in, I think… /data/project/<toolname>
<brainwane> ok, so I should have scp
<Coren> I.e. mv it before you take it.  :-)
<brainwane> I tried....
<Coren> Ah, your own home doesn't give permission to your tool.  :-)
<brainwane> $ mv biographies.txt /data/project/missing-from-wikipedia/public_html/missing-from-wikipedia/
<brainwane> mv: cannot create regular file `/data/project/missing-from-wikipedia/public_html/missing-from-wikipedia/biographies.txt': Permission denied
* dr0ptp4kt has quit (Quit: dr0ptp4kt)
<Coren> brainwane: Ah!  Your permissions are too strict then.
<brainwane> oh?
<Coren> brainwane: You probably want your tool's home to be g+w anyways; otherwise you can't properly have more than one maintainer without a lot of trouble.
<brainwane> g+w? groupwrite?
<Coren> brainwane: from your tool account, "chmod -R g+w ~"
<brainwane> got it. done
<Coren> That'll give your tool maintainers write access to the tool, and that includes yourself.  :-)
<brainwane> cool. was that in the docs and I missed it?
<Coren> That's normal unixy stuff which we mostly don't cover in the labs-specific docs.
<brainwane> :/
<Coren> It would be good if we could find a nice tutorial we could link to though.
<brainwane> I've been using Unix since 1999 and I missed this.
<brainwane> I presumed that creating a tool account would automatically give me as maintainer the necessary permissions.
<Coren> brainwane: It does; but if you did things like use some scp clients or unpacked tarballs with restrictive permissions, it'll override.

Sumana Harihareswara, a volunteer (talk) 23:30, 3 December 2013 (UTC)