Help:Toolforge/Elasticsearch

Toolforge

• Toolforge technical documentation
• Toolforge administrator documentation
• Help and communication
• List of tools
• Terms and conditions
• Rules
• Glossary
• Technical documentation resources

Getting started

• Cloud Services, Cloud VPS, and Toolforge overview
• About Toolforge
• Toolforge quickstart guide
• How-tos, tutorials and walkthroughs
• Create a Wikimedia developer account
• Access Toolforge instances with PuTTY and WinSCP
• Troubleshooting Toolforge

Developing tools

• Tool accounts
• Developing successful tools
• Develop tools in Java
• Develop tools in Mono/.NET
• Develop tools in Python
• Develop tools using Pywikibot framework
• Develop tools in Rust
• Web servers for tools
• Jobs framework
• Raw Kubernetes jobs (using API)

Backends

• Kubernetes (default)
• Grid engine (Deprecated, Don't use)

Additional documentation

• Version control in Toolforge
• Elasticsearch for Toolforge
• Redis for Toolforge
• Dumps and Toolforge
• Database
• Contributing

[]

Overview

This page contains information about local Elasticsearch services in Toolforge. For information about the replica of the Wikimedia CirrusSearch Elasticsearch indices available from Toolforge and Cloud VPS instances, see Help:CirrusSearch elasticsearch replicas.

About Elasticsearch

Elasticsearch is a full-text search system built on Apache Lucene. It can be used to index and search data stored as JSON documents.

Elasticsearch is the technology used to power Wikimedia's CirrusSearch system.

Elasticsearch for Toolforge

An Elasticsearch version 7 cluster for all tools is available at http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud, on the non-standard port 80.

This Elasticsearch cluster is a shared resource. All documents indexed in it can be read by anonymous users from within Toolforge. Write access is needed to create new indexes, and a password is needed to store or update documents. However, see addition notes on the talk page.

Read-only access

The Elasticsearch servers allow anyone to read any of the indexes that it contains. This access is limited to other hosts in the Toolforge project (e.g. the OGE job grid, Kubernetes containers, and the bastion servers).

The Elasticsearch service is available on port 80 at http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud

Note: The default Elasticsearch port (9200) is not used.

Write access

Elasticsearch does not offer multi-tenant access control in its open source version.

PUT, POST, or DELETE requests sent to the Elasticsearch servers require HTTP Basic Authentication using a username and password specific to each tool.

Requests for write access can be made by filing this Phabricator task.

When credentials have been created they will be placed in /data/project/$TOOL/.elasticsearch.ini. Access requests are currently processed manually and may take a few days to be fulfilled.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

• Chat in real time in the IRC channel #wikimedia-cloud ^connect, the bridged Telegram group, or the bridged Mattermost channel
• Discuss via email after you subscribed to the cloud@ mailing list

Receive mail announcements about critical changes

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Learn about major near-term plans

Read the News wiki page

Read news and stories about Wikimedia Cloud Services

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)