Help talk:Toolforge/Archives/2016

Please do not post any new comments on this page. This is a discussion archive See current discussion or the archives index.

Hosted jQuery etc.

Does Tool Labs offer hosted resources like jQuery? Tools like Pathoschild's important and friends started using e.g. on ajax.googleapis.com which means agreeing to -> -> i.e. to send all data to Google for any use. I block them so it's no big privacy issue for me, but it would be nice to fix. --Nemo 09:12, 10 September 2013 (UTC)

Can we just load the appropriate files from bits.wikimedia.org or bits.beta.wmflabs.org? Anomie (talk) 17:16, 10 September 2013 (UTC)

That would be ideal, but are they "free access"? Maybe they are and Pathoschild just didn't think about it. --Nemo 17:25, 10 September 2013 (UTC)

As jQuery is available under the MIT licence, why pull it from other hosts at all and not just host it locally so that the tool developer has full control over it? After the first use of a tool by a user there shouldn't be a difference in caching. --Tim Landscheidt (talk) 01:24, 27 October 2013 (UTC)

There are a number of very good reasons to prefer to avoid having umpteen different copies of jQuery (and others). We can't use the one in bits (it'd introduce a great deal of complications related to the resource loader) but it'd make a great deal of sense to have one available for all tools on our own infrastructure. I'll try to see if we can rely on our own caching servers for that but if not, I'll make a repository for shared browser objects like this for Tools. — Coren/Marc ^(talk) 16:12, 29 December 2013 (UTC)

The static now hosts a variety of free software shared resources including Bootstrap and jQuery. --BryanDavis (talk) 05:56, 3 February 2016 (UTC)

Reorganisation

This page has grown organically, and I think a few issues have crept up on us:

• The page is so large that finding information is very difficult.
• A lot of useful information is separate, inconsistent, and would be messy to merge in (like Getting started with Flask, Stub Python app, and Accessing Tool Labs with PuTTY and WinSCP).
• The page is a confused mixture of core concepts, getting started, see-also links, advanced howtos, and language-specific content.
• The page often makes unjustified assumptions (e.g. Make the tool translatable assumes you're using PHP).
• And it's still missing a huge amount of information — how to create a node.js/Java/Python/etc app, adding tools to the tool directory, using bigbrother, using the Labs SQL Optimizer, etc.

To address this, I'd like to rename this page to "Help:Tool Labs", break it apart into more specific subpages, pull in useful separate info, and expand missing documentation. This page would become an index that looks something like this:

Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers. Tool Labs is a part of the Labs project.

Introduction

• What is Tool Labs? — learn the core concepts (rationale, features, architecture & terminology).
• Rules — learn about the rules that apply to all tools we host (licensing, privacy, resource usage, etc).

Getting started

• Getting started with Tool Labs — get started as a Tool Labs developer (request access, SSH login, create or join a tool instance).
• Getting started with...
  • Java — create a basic Java application using Tomcat.
  • node.js — create a basic node.js application using the webgrid.
  • PHP — create a basic PHP application with translations using webservice.
  • Python — create a basic Python tool using Flask and MVC.
• Accessing the databases — querying Wikimedia SQL databases, creating user databases, and using Redis.
• Publishing your tools — add your tools to the public directory so others can find them.

How to

• Source control and issue tracking — using Git, Gerrit, and Phabricator.
• Advanced access to your tools — using SFTP, SSH , port tunnelling, PuTTY and WinSCP, etc.
• Scheduling — schedule automated tasks or apps using Cron or the webgrid.
• Mail — sending and receiving email.
• Moving a tool from Toolserver to Tool Labs.
• FAQ.

I think this format would be easier to maintain and use, would make it easier to add missing information, and would let us go into more depth on specific pages.

Any thoughts or objections? —Pathoschild 08:33, 09 November 2014 (UTC)

Let me point out that "assumes you're using PHP" is incorrect: it's just assumed that Intuition can and should be made to work with whatever one is using.

Splitting and reorganising is nice, but I hope you don't mean to create 15 separate pages! For a gradual approach, I suggest that you move the page and reshuffle its sections across different parent sections (but keeping current section titles so that links don't break), and after that split out as much content as possibly to existing pages like Help:Git and others. Then it will be easier to see how much content is left and how to reorganise it. --Nemo 19:58, 9 November 2014 (UTC) P.s.: A half-page index can and should be on Help:Contents; one outdated index is enough.

I do think we should split the page; the scope is too extensive to cover with a single page. We can use supersections instead if that's preferred, but I think that would be less effective. —Pathoschild 20:29, 09 November 2014 (UTC)

I moved the page since that seems uncontroversial. —Pathoschild 03:09, 10 November 2014 (UTC)

I don't particularly like the split-up. Yesterday I was looking for the syntax of .lighttpd.conf, and I had to go to a sub-page (that I needed to add to my watchlist) to see the file mentioned. But even there a search for the bit that I remembered came up empty because the advice I was looking for was hidden in a collapsed "Extended instructions" section.

I think the overarching problem is that the page(s) address a very wide range of audiences. Some pieces seem to have a total novice in mind, while others present concepts that you need very advanced skills to understand. --Tim Landscheidt (talk) 17:41, 13 November 2014 (UTC)

I think this page has long outgrown a single page that answers all questions. I'm still trying to wrap my head around what is (and isn't) here to figure out a better organization. I think those that have complained before about not wanting multiple pages may have just trained themselves to use ctrl-f instead of the site search. --BryanDavis (talk) 06:05, 3 February 2016 (UTC)

Definitely. With C-f I can iterate over all matches, and if all matches are contained in one page, once I reach the top one again, I'm done. With multiple pages I have to repeat the process for each chunk of information that is on a different page. With C-f, I can amend the search term on the fly and easily try different spellings or synonyms ("Was it 'hostname' or 'host name'?"). If despite that C-f comes up empty on a single page, I can be reasonably sure that the information is not available, while with multiple pages and MediaWiki's search it might be missing, but it might also be "hidden".

The biggest problem I see with this documentation is that there is not sufficient feedback by the users, either in form of comments or – be bold! – edits. A few years ago, WMF hired an external writer to reorganize it, recently another staffer polished it, now you have put it on your agenda, and everybody seems to have to work in total darkness of what a new user or beginner might look for with the risk that afterwards the effort is lauded, but there is no objective metric to assess progress. I'm guilty of that myself in other open source projects where I don't complain about or improve bad documentation, instead taking it as is, but in an ecosphere of people used to sticking {{Confusing}} on articles or just fixing them themselves, it would be very nice and appreciated if they behaved the same way here. --Tim Landscheidt (talk) 05:35, 6 February 2016 (UTC)

@Tim Landscheidt: You present reasonable points about the utility of C-f for refreshing one's memory of a previously discovered fact. In an earlier post you said "I think the overarching problem is that the page(s) address a very wide range of audiences." which may be getting more to the issue that I'm interested in solving. A full analysis has not been performed and published yet on the survey of Tool Labs users that was done in September 2015. The question "Tool Labs documentation is easy-to-find" is one that Yuvipanda and I have discussed a bit however. The responses to that question were nearly equally split between "agree/strongly agree" and "disagree/strongly disagree" (48/54). Those raw response counts are missing a deeper analysis however of what might differentiate the two groups. I'm hoping to spend some time "soon" on preparing a more complete analysis of the survey results and one thing I'm interested doing in that analysis is looking for correlations in areas such as this one where there is a stark divide between the respondents. It may turn out that when the results are further divided by number of years of experience, number of tools built, preferred programming language, or some other demographic criteria that a more clear pattern emerges to guide us to making informed changes to the documentation and its organization.

I fully agree with the second paragraph of your response. The only sustainable way for Tool Labs to grow is for the users of the Tool Labs services to help each other solve problems and learn. Looking to Wikimedia Foundation staff and a small number of highly engaged volunteers to answer all questions and maintain all documentation while also maintaining and supporting the underlying platforms and services will not scale. --BryanDavis (talk) 17:59, 6 February 2016 (UTC)

"Can I delete a tool?"

@Merlijn van Deen: I assume your edit reflects a policy decision, but not a technical limitation? --Tim Landscheidt (talk) 14:13, 27 April 2016 (UTC)

A bit of both. The 'rmtool' tool in misctools is undeployed, undocumented (as far as Nova_Resource:Tools/Admin goes, at least, although there is a small man page and the code is not too hard to follow) and unmaintained (it still refers to /data/project/.system/webservers which hasn't existed for a while). 'rmtool' also doesn't 'delete' the tool, as the service group will continue to exist. So... I don't feel comfortable announcing 'admins can delete your tool' as long as we don't actually have a documented protocol for that. valhallasw (Merlijn van Deen) (talk) 07:29, 28 April 2016 (UTC)

Eh, I removed rmtool with gerrit:239522 last September?! I have documented (my) protocol now at Nova Resource:Tools/Admin#Deleting a tool. --Tim Landscheidt (talk) 10:29, 28 April 2016 (UTC)

I guess I had an old checkout of that repo in my home directory -- that explains why it's undeployed :-). Thank you for documenting your protocol! valhallasw (Merlijn van Deen) (talk) 11:24, 28 April 2016 (UTC)

Python is different versions with different modules. How run on grid and with scheduling?

I need to run a bot on grid with scheduling. It made under the python version 3.4, and normally works in the tool home via command python3 myscript.py. But on grid installed v3.2, without basic modules like 'request', 'urllib.parse', etc, and I get problem with utf-8, like on obsolete version 2.7. (Errors with utf-8, although that the files in utf-8, have # - * - coding: utf-8 - * -, have the prefix of strings 'u' like u'not english a string'.)

If in console of Tool labs run:

• python -V, result: 2.7.6
• python3 -V, result: 3.4.3

For grid run (according Help:Tool_Labs/Developing#Setup_job_submission and Help:Tool Labs/Grid):

• jsub -N job_name python -V, result error (job_name.err): Python 2.7.3
• jsub -N job_name python3 -V, result error (job_name.err): -bash: /usr/bin/python3.4: No such file or directory

If I try run myscript.py for display version:

import sys
print (sys.version_info)

then it say that on grid set v3.2, and wrote errors said above (no modules, utf-8).

How to run a bot on grid scheduled? --Vladis13 (talk) 10:27, 20 August 2016 (UTC)

Asked on Phabricator:T143473. --Vladis13 (talk) 10:49, 20 August 2016 (UTC)

Link to redis example

seems that the code in https://github.com/yuvipanda/SuchABot does not use redis anymore Shavtay (talk) 21:37, 28 December 2016 (UTC)

Shavtay It looks like the Redis consumer code from SuchABot was moved to https://github.com/yuvipanda/gerrit-to-redis. I'll update the link in the docs. --BryanDavis (talk) 01:23, 29 December 2016 (UTC)