EE Dashboard

NOTE: This dashboard is no longer available. Contact the Wikimedia Editing team to find out their plans for it. The data this dashboard visualized is still available at:

https://analytics.wikimedia.org/datasets/archive/public-datasets/

For example, for the English Wikipedia, echo data is here: https://analytics.wikimedia.org/datasets/archive/public-datasets/enwiki/echo/ . And, WikiLove data is here: https://analytics.wikimedia.org/datasets/archive/public-datasets/enwiki/wikilove/

EE Dashboard is a Cloud VPS instance hosting dashboards related to Editor Engagement and powered by Limn, a d3-based data visualization tool.

Anatomy of a dashboard

Currently, there are only two pages to view on ee-dashboard:

To view stats for other languages, change 'en' to a different language code.

Each page on a dashboard can contain an arbitrary number of tabs and each tab can host multiple graphs. The layout of each page is defined by a JSON file that lives in a GitHub repository. This file defines all the tabs on the page and which graphs should be displayed under each tab.

Getting access to the ee-dashboard instance

To be able to deploy changes to the dashboard layout and the source code, you will need a labs account and set up agent forwarding. Once this is done, you can ssh -A <username>@bastion.wmflabs.org to get access to bastion and from there ssh into the instance where ee-dashboard is hosted (currently this is limn1). You also need to be added as a member of the project, you can contact the Analytics team to request SSH access to limn1.

How to create new graphs and dashboards

The best way to modify a dashboard is to install local instance of Limn and set it up to serve data from the ee-dashboard configuration.

1. Install Limn

Limn is the software that runs the ee-dashboard. Make sure you have nodejs and npm installed.

git clone https://github.com/wikimedia/limn.git
cd limn
npm install
npm i -g coco
npm start

From now on, we'll assume that your local Limn instance lives in ~/git/limn.

2. Clone limn-editor-engagement-data

The ee-dashboard configuration is hosted in the limn-editor-engagement-data git repository.

git clone https://github.com/wikimedia/limn-editor-engagement-data.git

We'll assume from now on that you cloned this repository into ~/git/limn-editor-engagement-data. You'll see that this repository hosts three types of configuration files:

dashboards: hosts dashboard definitions
datasources: hosts metadata about data sources
graphs: hosts individual graph definitions.

For your local Limn instance to be able to load these files, you need to configure it as follows:

coke -v ~/git/limn/var -d ~/git/limn-editor-engagement-data/ -t eee link_data

If you get the error "no Cokefile", make sure you are running the command from within the limn directory.

You should now be able to point your browser to http://localhost:5000 and open dashboards such as http://localhost:5000/dashboards/enwiki-features.

3. Create a new data file

To create a new timeseries plot you first need to generated a CSV or TSV data file and register it as a data source. The data file can live on any publicly accessible web server (for example analytics.wikimedia.org/datasets). For n distinct series, the data file needs to be formatted as follows:

date,field_1,...,field_n
2013-06-11,value_1,...,value_n 
2013-06-12,value_1,...,value_n 
2013-06-13,value_1,...,value_n

The date field must be a valid ISO 8601 UTC date (2013-06-11) or timestamp (2013-06-11 00:00:00). Data files are typically generated by scripts run by cron on stat1, Wikimedia's data private crunching machine.

4. Register the data file as a data source

Once your data file is being generated and stored in a publicly accessible location, you'll need to register it as a new data source on ee-dashboard. To do this, go to the data source view, sign in using your WMF credentials, click "Add New Data Source", fill in the form with the full URI where the data file is hosted, and click Save. The Unique ID you assign the data should include the name of the project's database, for example "echo_notifications_by_user_group_enwiki".

Each data source will be stored as a JSON file under ~/git/limn/var/data/datasources. Once you are done, you can move all the data sources under limn-editor-engagement-data.

cd ~/git/limn/var/data/datasources
mv *.json eee/

5. Create new graph

After you have registered the data source, go to Create Graph view. Enter in the same Unique ID that you used for the data source and give the graph a name and description, then click Save.

Each graph is stored as a JSON file under ~/git/limn/var/data/graphs. Once you are done, you can move all the graph definitions under limn-editor-engagement-data.

cd ~/git/limn/var/data/graphs
mv *.json eee/

You can fine-tune individual graph settings by modifying the corresponding JSON file. The file can also accessed from the UI, by appending .json?pretty to the URL of a graph, e.g. http://ee-dashboard.wmflabs.org/graphs/enwiki_pc_logging_actions_daily_perc.json?pretty.

6. Add the graph to a dashboard

To add a graph to a given dashboards you need to edit the dashboard configuration file and add the graph under the appropriate tab:

{
    "id": "metrics",
    "headline": "English Wikipedia",
    "subhead": "editor engagement metrics",
    "tabs": [
        {
            "name": "Registrations",
            "graph_ids": [
                "enwiki_daily_reg",
                "enwiki_hourly_reg_y2y",
                "enwiki_daily_reg_mobile"
            ]
        },
        {
            "name": "Edits",
            "graph_ids": [
                "enwiki_revtags"
            ]
        }
    ]
}

Dashboards are stored in ~/git/limn/var/data/dashboards. When you're done, make sure all dashboard definitions are moved under limn-editor-engagement-data.

cd ~/git/limn/var/data/dashboards
mv *.json eee/

7. Commit all changes

Once you are happy with your changes, commit them, merge them into master (if you worked on a separate branch) and push them to the limn-editor-engagement-data remote.

8. Deploy on ee-dashboard

To deploy these changes, log on to the ee-dashboard instance on Labs (by ssh'ing into labs: ssh -A <username>@bastion.wmflabs.org and from there into limn1: ssh limn1). For simple changes you should be able to deploy by just pulling these changes from the git repo into /var/lib/limn/ee-dashboard/.

Need help?

For help with the deployment process or Limn configuration files, you can contact Dan Andreescu from the Wikimedia Foundation Analytics team.