Nova Resource Talk:Tools/Help

From Wikitech
Jump to: navigation, search

What I miss are best practice hints for two things:

  • How to publish code? Should we all use github? Or is gerrit the better way? A link to a small tutorial would be nice.
  • How to document a tool and how to communicate with the users. I can say, we use often a page inside Wikipedia because this was the easiest way for normal user to come in contact with us, we had an english version and mostly a version in our native language (German). Is it better to use the wikitech-wiki or the Mediawiki-wiki? Which wiki use "Single User Login".

Everybody should be free to go other ways but it would help if the majority use one way. --kolossos (talk) 10:04, 31 May 2013 (UTC)

How to add and modify a description for an project

I want to add a description at the list of for one of my tools. How can I do this? --kolossos (talk) 10:04, 31 May 2013 (UTC)

See Nova Resource:Tools/Help#Tool Labs's landing page. Anomie (talk) 13:18, 31 May 2013 (UTC)
Thanks.--kolossos (talk) 15:27, 31 May 2013 (UTC)

GUI tool for databasework

At the hackathon in Amsterdam it was mentioned to be possible to use an external GUI tool to interact with the database that runs on your own machine rather then to use PHPMyAdmin. What settings should be used for that?

Henna (talk) 11:21, 23 June 2013 (UTC)

Basically, it's the same settings as at Toolserver, except that you replace with enwiki.labs (and with --Tim Landscheidt (talk) 12:21, 23 June 2013 (UTC)

300-350M? Really?

I was tempted to add this to Nova Resource:Tools/Help#Why am I getting errors about must be installed for pthread_cancel to work?, but then I thought it might be too much detail so I decided to put it here for others to decide about.

Experiments in July 2013 with "do nothing for 60 seconds" scripts in various languages gave the following results:

Language h_vmem needed Code
C 5M

#include <unistd.h>
int main(void){ return sleep(60); }

Lua 14M local clock = os.clock local t0 = clock() + 60 while clock() < t0 do end
Perl 20M sleep 60;
Python 30M from time import sleep; sleep(60);
PHP 350M sleep(60);
NodeJS 750M var e = new Date().getTime() + 60000; while (new Date().getTime() <= e) {}

Of course, real scripts in most of these languages will load various modules/libaries and consequently have higher memory requirements. PHP may be somewhat of an exception here, as its 350M already includes a large number of such extensions (although any additional extensions or libraries (e.g. from PEAR) loaded would still increase the number). Anomie (talk) 02:21, 21 August 2013 (UTC)

I also tried a simple Java program. But I must have been doing something wrong, because it complained that it didn't have enough memory until I gave it 3.2G. Anomie (talk) 02:21, 21 August 2013 (UTC)
Java (un)helfully allocates most of its heap on startup; you can control that behavior with the -Xms and -Xmx command-line options. — Coren/Marc (talk) 22:55, 22 August 2013 (UTC)
Specifying 32M for both of those options makes it slightly better. Now it "only" needs 1.2G. Anomie (talk) 13:11, 23 August 2013 (UTC)
With jamvm, I can get a max heap space of 300M with "just" 750M. Shrinking the heap space doesn't help much anymore. Growing the availible space to 2750M allowed me to run with a max heap space of 2300M, so that at least seems to scale. Jamvm is installed on all nodes, and is called with -jamvm. Martijn Hoekstra (talk) 21:04, 15 October 2013 (UTC)

Public tool database?

Can I create a database as a tool user that is visible to other tools? On toolserver, we did that by adding "_p" to the database name. Should be documented, even if it's "you can't do that here". --Magnus Manske (talk) 15:47, 22 August 2013 (UTC)

It should indeed. You have two choices:
  1. We do keep to the "_p" convention, by default every database user has "SELECT on %_p.*" and so can read databases named that way; or
  2. Since users have all privileges on databases they are allowed to create (username__%) with grant option, you can grant the appropriate to any or all users with a GRANT statement.
— Coren/Marc (talk) 22:52, 22 August 2013 (UTC)

Does 20% of this page really need to be about pywikipedia?

I know that pywikipedia is a popular tool, but it seems excessive for a fifth of this general help page to be about using it. At the moment and not counting the "TOCright" template, 16417 of 85463 bytes of wikitext (19.2%) and 230 of 1172 lines of wikitext (19.6%) are in the "Pywikipedia" section. Anomie (talk) 13:08, 3 September 2013 (UTC)

Hosted jQuery etc.

Does Tool Labs offer hosted resources like jQuery? Tools like Pathoschild's important [1] and friends started using e.g. [2] on which means agreeing to [3] -> [4] -> [5] i.e. to send all data to Google for any use. I block them so it's no big privacy issue for me, but it would be nice to fix. --Nemo 09:12, 10 September 2013 (UTC)

Can we just load the appropriate files from or Anomie (talk) 17:16, 10 September 2013 (UTC)
That would be ideal, but are they "free access"? Maybe they are and Pathoschild just didn't think about it. --Nemo 17:25, 10 September 2013 (UTC)
As jQuery is available under the MIT licence, why pull it from other hosts at all and not just host it locally so that the tool developer has full control over it? After the first use of a tool by a user there shouldn't be a difference in caching. --Tim Landscheidt (talk) 01:24, 27 October 2013 (UTC)
There are a number of very good reasons to prefer to avoid having umpteen different copies of jQuery (and others). We can't use the one in bits (it'd introduce a great deal of complications related to the resource loader) but it'd make a great deal of sense to have one available for all tools on our own infrastructure. I'll try to see if we can rely on our own caching servers for that but if not, I'll make a repository for shared browser objects like this for Tools. — Coren/Marc (talk) 16:12, 29 December 2013 (UTC)

Naming of user-created databases

Do the user-created databases really have to start with some "random" string like "p50252g21636"? Why? Couldn't the database names be made more user-friendly, like using the normal user name? svick (talk) 12:34, 26 September 2013 (UTC)

Because usernames are not unique across projects. — Coren/Marc (talk) 20:02, 15 October 2013 (UTC)
So what about something like projectname_username? I think that would still be a big improvement over p50252g21636. Svick (talk) 20:21, 15 October 2013 (UTC)
Potential for disallowed characters in the project or user names, probably. "p50252g21636" is more or less projectid_userid. Anomie (talk) 13:04, 16 October 2013 (UTC)
The biggest issue is Mysql's hard-coded username length limit. — Coren/Marc (talk) 16:13, 29 December 2013 (UTC)

Getting help

Hi. I am not sure where to post this so please let me know if there is a more appropriate place. As described in more detail here, I have gotten a Wikipedia template filler to work "internally" from the command line on the Tools Lab server, but I am having trouble accessing from an external web browser. I would be very grateful if someone with knowledge of perl and Tools Lab could help me. Thanks. Boghog (talk) 20:11, 27 October 2013 (UTC)

The above mentioned problem is the inability of the perl cgi script to access a local library even though the library in question has world read and execute privileges. There is no mention of this on the help page. Any suggestions on how to get this to work would be greatly appreciated. Thanks. Boghog (talk) 12:02, 28 October 2013 (UTC)

OK, I think I have solved most the problems. The key was to add the following to the cgi script. This made debugging the library much easier:
use CGI::Carp 'fatalsToBrowser';
Adding this single line gave some very useful feedback that indicated despite using cpanm for the installs, there were a still a few unfulfilled dependencies. After installing these missing modules, the script finally worked. I still need to get url and isbn options to work, but at least the most commonly used options now work. The link for running the tool is here: citation-template-filling. Boghog (talk) 15:57, 1 November 2013 (UTC)

MySQL Workbench

MySQL Workbench doesn't work for me the way described. I always get the following error message: "Bad authentication type, the server is not accepting this type of authentication.". I'm using the same private key file which works for normal SSH access. Maybe this is something like . I wish I could just use phpMyAdmin... I'm really no friend of managing databases on command line. --APPER (talk) 19:07, 31 October 2013 (UTC)

I now did the SSH tunneling using my SSH client and used MySQL Workbench only for the SQL stuff. --APPER (talk) 13:47, 2 November 2013 (UTC)

submitting git update on a directory


I would like to run a git update everyday, but can't find how to submit it. My Pywikipedia repository is "/data/project/herculebot/pywikibot-compat/". I would like to run "git pull" on this directory with qsub. How to do so ?

--Hercule (talk) 21:27, 26 November 2013 (UTC)

Schedule a cronjob to run a command along the lines of cd /data/project/herculebot/pywikibot-compat/ && /usr/local/bin/jsub -cwd -N git-pull /usr/bin/git pull. I haven't tested it, but that should be about right. If you really want to use qsub instead of jsub, see /usr/local/bin/jsub and work out what qsub command it winds up using. Anomie (talk) 14:53, 27 November 2013 (UTC)

Impossible to submit Java program


I'm unable to run my Java script using jsub.

In the terminal, logged on tools.login I can run the following command line with no problem :

java -Xmx256M -Xms256M -jar /data/project/herculebot/Wikipedia.jar

When I edit my crontab, I add the following line :

54 * * * * cd /data/project/herculebot/ && /usr/local/bin/jsub -once -mem 256m -quiet -N Cacographies -j y java -Xmx256M -Xms256M -jar /data/project/herculebot/Wikipedia.jar

I always have the same error on my Cacographies.out file : Error occurred during initialization of VM Could not reserve enough space for object heap Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.

I tried to give more memory, with -mem 768m, but it's not better.

Can someone help me to solve this issue ?


--Hercule (talk) 21:01, 1 December 2013 (UTC)

To close this here as well: This was discussed on the mailing list, and the suggestion there was to increase the memory requested massively with 2 GByte apparently enough to satisfy Java :-). --Tim Landscheidt (talk) 02:49, 11 December 2013 (UTC)

add permissions around take to docs?

from IRC just now:

<brainwane> I'm having trouble "take"ing a file 
<brainwane> $ take /home/brainwane/biographies.txt
<brainwane> /home/brainwane/biographies.txt: you must own the containing directory
<brainwane> where do I put this file so my tools acct can "take" it?
<brainwane> petan: any ideas?
<andrewbogott> tools accounts usually have their own home directory
<Coren> brainwane: In any directory your tool account owns.
* chippy has quit (Remote host closed the connection)
<andrewbogott> in, I think… /data/project/<toolname>
<brainwane> ok, so I should have scp
<Coren> I.e. mv it before you take it.  :-)
<brainwane> I tried....
<Coren> Ah, your own home doesn't give permission to your tool.  :-)
<brainwane> $ mv biographies.txt /data/project/missing-from-wikipedia/public_html/missing-from-wikipedia/
<brainwane> mv: cannot create regular file `/data/project/missing-from-wikipedia/public_html/missing-from-wikipedia/biographies.txt': Permission denied
* dr0ptp4kt has quit (Quit: dr0ptp4kt)
<Coren> brainwane: Ah!  Your permissions are too strict then.
<brainwane> oh?
<Coren> brainwane: You probably want your tool's home to be g+w anyways; otherwise you can't properly have more than one maintainer without a lot of trouble.
<brainwane> g+w? groupwrite?
<Coren> brainwane: from your tool account, "chmod -R g+w ~"
<brainwane> got it. done
<Coren> That'll give your tool maintainers write access to the tool, and that includes yourself.  :-)
<brainwane> cool. was that in the docs and I missed it?
<Coren> That's normal unixy stuff which we mostly don't cover in the labs-specific docs.
<brainwane> :/
<Coren> It would be good if we could find a nice tutorial we could link to though.
<brainwane> I've been using Unix since 1999 and I missed this.
<brainwane> I presumed that creating a tool account would automatically give me as maintainer the necessary permissions.
<Coren> brainwane: It does; but if you did things like use some scp clients or unpacked tarballs with restrictive permissions, it'll override.

Sumana Harihareswara, a volunteer (talk) 23:30, 3 December 2013 (UTC)

Python files in public_html

How are they handled? mod_python? wsgi? I am unable to create a working hello-world page. --Chricho (talk) 18:03, 1 January 2014 (UTC)

Ah, it is run using cgi, too… --Chricho (talk) 18:22, 1 January 2014 (UTC)

Docs inaccuracies?

The Tools/Help page says that the public_html directory is at ~/public_html. Well, that is not correct, it is in fact at /data/project/mytool/public_html, so you need to "become" the tool for it to be in your home directory. It would be nice if this were clarified at the first location in the doc where public_html is mentioned. Thanks! Oleg Alexandrov (talk) 05:53, 2 January 2014 (UTC)

~ is the abbreviation for an (any) account's home directory (as ~account is an abbreviation for a specific account's home directory). Tool accounts have their home directories at /data/project/$TOOL, so ~/public_html is the subdirectory of the tool's home directory.
The first (:-)) mention of public_html even says:
Note that individual tool accounts have both a ~/public_html/ and a ~/cgi-bin/ directory in the home directory for storing Web files.
If you feel that the documentation could be improved by a clarification, be bold! :-) It's probably easier for you to find a wording that is easier understandable than for someone who is already used to the terminology. --Tim Landscheidt (talk) 16:15, 3 January 2014 (UTC)

Access databases from PHP

How do I access databases from PHP? Also, not sure if it should, but /data/svwiktionary/ doesn't contain a password to use (I might have deleted it a long time ago if it did).

I'm interested in both the replicas and using my own db. I haven't yet created my own db, but intend to at some point. skalman (talk) 10:33, 25 January 2014 (UTC)

I had apparently reset the password in /data/svwiktionary/ to something of my choice, so I incorrectly assumed that the "correct password" was lost. However, it seems like it works with the password if I use the password that's in there. skalman (talk) 16:32, 25 January 2014 (UTC)

PHP upgrade

Currently PHP 5.3.10 is installed. Would it be possible to upgrade to 5.5? I don't have any hard dependencies, but I'm lazy and it would be nice to use the shorter array syntax, especially for nested arrays. Furthermore, version 5.3 will be unsupported by July this year. skalman (talk) 22:24, 25 January 2014 (UTC)

A newbie issue: WinSCP and Putty

When using Toolserver, I was used to open two windows - a Putty console and a WinSCP connection. I did the same into Labs with much frustration by obvious mismatch of logged user, since when using WinSCP I am "alebot", while while using Putty i "become itsource". Files were owned by one user, or another one, a simple and obvious evidence for unix skilled users but an enigma for persistent newbies as I am getting frustrating messages when trying to edit anything using the wrong user.

The solution of such an enigma was so obvious that no one of skilled users I asked for help gave me its banal solution.... I had to run a ls -l command to grab the question. --Alex brollo (talk) 09:11, 3 March 2014 (UTC)