Help:Toolforge/Dumps

Toolforge
Toolforge technical documentation Toolforge administrator documentation Help and communication List of tools Terms and conditions Glossary Technical documentation resources
Getting started Cloud VPS and Toolforge at a glance About Toolforge Toolforge quickstart guide How-tos, tutorials and walkthroughs Create a Wikimedia developer account Access Toolforge instances with PuTTY and WinSCP Troubleshooting Toolforge
Developing tools Tool accounts How to develop tools Develop Mono/.NET tools and bots Develop tools in Java Develop tools in Python Develop tools using Pywikibot framework Web servers for tools (Kubernetes) Grid engine
Additional documentation Version control in Toolforge Elasticsearch for Toolforge Redis for Toolforge Dumps and Toolforge Database

This page contains information about dumps and Toolforge

Dumps generated by Wikimedia projects

Toolforge has access to a directory storing the dumps generated by Wikimedia projects: public Wikimedia datasets.

Recent dumps

The most recent two dumps can be found in:

/public/dumps/public

This directory is read-only, but you can copy the files to your tool's home directory if necessary. Ideally you can find (or build!) a library that can be used to read data from the dumps without decompressing them. See meta:Data dumps/Other tools for some examples.

Older dumps

• Can be manually downloaded from the Wikimedia downloads server, or from mirrors which may have better bandwidth.
• /public/dumps/pagecounts-raw contains some years of the pagecount/projectcount data.

Dumps in general

• Portal:Data Services#Wikimedia_Dumps
• meta:Data dumps
• https://dumps.wikimedia.org
• Operational runbooks for maintaining the Dumps generation system

Communication and support

We communicate and provide support through several primary channels. Please reach out with questions and to join the conversation.

Communicate with us

	Connect	Best for
Phabricator Workboard	#Cloud-Services	Task tracking and bug reporting
IRC Channel	#wikimedia-cloud connect	General discussion and support
Mailing List	cloud@	Information about ongoing initiatives, general discussion and support
Announcement emails	cloud-announce@	Information about critical changes (all messages mirrored to cloud@)
News wiki page	News	Information about major near-term plans
Blog	Clouds & Unicorns	Learning more details about some of our work