Portal:Toolforge/Sustainability Scoring Framework Proposal

This page outlines a proposed scoring framework for evaluating the sustainability of the Toolforge platform. The goal is to provide a structured, transparent way to assess and improve key aspects of the platform over time.

This version focuses on three foundational categories — Observability, Platform Access & Deployment, and Source Code Availability — which were identified through previous admin surveys and platform usage insights as critical areas for long-term sustainability.

Each category is weighted to reflect its strategic importance and broken down into clear, measurable subcategories. The framework is designed to evolve while maintaining a fixed 100-point model for comparability and clarity.

We're now seeking feedback from Toolforge and WMCS admins before finalizing this proposal. Please leave your thoughts, questions, or suggestions on the talk page linked above.

Your input is vital to ensuring this framework aligns with both community priorities and technical realities.

Proposed Scoring Framework

This framework evaluates platform sustainability across three essential categories:

• Observability
• Platform Access & Deployment
• Source Code Availability

Each category is weighted and scored using binary, percentage, or scaled metrics, with a fixed total score of **100 points**.

Each category (and its subcategories) can be independently evaluated as a percentage. For example: With a score of 24 out of 30 points in Observability, that translates to 80% completion for that category. Simultaneously, Observability contributes 30 points toward the total 100-point sustainability score.

Proposed Weighted Categories

1. Observability — 30 Points

Emphasizes transparency and early detection of issues.

• Centralized Logging (20)
  • Centralized log access
  • Retention/rotation policies
• Metrics Dashboard (5)
  • Dashboards with CPU, memory, request metrics
• Platform Abuse Monitoring (5)
  • Misbehavior detection (e.g., abuse, crypto mining)

2. Platform Access & Deployment — 40 Points

Focuses on reducing friction for developers using Toolforge.

• CLI Access (10)
  • Tools manageable via command line interface
• WebUI Coverage (10)
  • % of operations executable via Web interface
• API Coverage (10)
  • % of common functions exposed via platform API
• SSH-Free Workflow (10)
  • Tools deployable without requiring SSH access

3. Source Code Availability — 30 Points

Ensures transparency and maintainability through open source.

• Public Code Availability (20)
  • % of tools with code publicly available
• Policy Enforcement (5)
  • Adherence to public source code requirements
• Automation (5)
  • Automated tools to detect source availability

Scoring Methodology

Metrics are categorized as:

• Binary Metrics (Yes/No) — e.g., "Is centralized logging implemented?"
• Percentage Metrics — e.g., "What % of tools have public source code?" Formula: (Raw % * Max Points) / 100
• Partial/Scaled Matrix — e.g., "rating 0–3" Formula: (Score / Max Scale) * Max Points

Normalization Example

• Binary → "Yes" on a 10-point item = 10
• Percent → 60% on a 10-point item = 6
• Partial → 2 out of 3 on a 6-point item = 4

This approach ensures fairness and comparability across scoring types. This example is reflected in the matrix/table below.

Evolution of the Score – Fixed 100-Point Model

Goal

Maintain a 100-point system over time for clarity and consistency.

Why It Matters

• Comparability — Year-over-year tracking (e.g., 62/100 → 78/100)
• Clarity — Easy to communicate performance via percent complete
• Simplicity — Great for reporting dashboards and stakeholder updates

How to Evolve While Keeping the 100 Points

1. Reallocate Weights as priorities change
2. Adjust or Replace Subcategories to reflect platform evolution
3. Keep Historical Snapshots for time-series reporting

Category	Subcategory	Max Points	Scoring Type	Definition of Done	Raw Score	Max Scale (if Partial)	Normalized Score
Observability	Centralized Logging	15	Binary	Logs accessible through centralized UI or API.	0		0
Observability	Metrics Dashboard	7	Partial	Basic CPU/memory/request metrics available per tool.	0		0
Observability	Abuse Monitoring	8	Binary	Automated detection of crypto mining, resource abuse, etc.	0		0
Platform Access & Deployment	CLI Access	10	Binary	CLI access available without SSH.	0		0
Platform Access & Deployment	WebUI Coverage	10	Percent	% of operations that can be performed in Web UI.	0		0
Platform Access & Deployment	API Coverage	10	Percent	% of Toolforge functions supported by API.	0		0
Platform Access & Deployment	SSH-Free Workflow	10	Binary	Deployment possible without SSH.	0		0
Source Code Availability	Policy Enforcement	5	Binary	Public source code policy enforced.	0		0
Source Code Availability	Public Code Availability	20	Percent	% of tools with publicly viewable source and metadata.	0		0
Source Code Availability	Automation	5	Binary	Automated public source code detection in place.	0		0
	TOTALS	100					0