We list and describe activity metrics. For different situations, the metrics have different meanings. Each activity metric can be used in association with other metrics in the creations of value-oriented health metrics. We have a template for the metrics to be used for new detail metric pages.
|Age of Community||Time since repository/organization was registered; or Time since first release.
“Results showed that the age of the project played a marginally significant role in attracting active users, but not developers. We attribute this differential effect of age on users and developers to the fact that age may be seen as an indicator of application maturity by users, and hence taken as a positive signal, whereas it may convey more ambiguous signals to developers.” (Chengalur-Smith et al., 2010, p.674) (Grewal, Lilien, & Mallapragada, 2006)
|All Licenses||List of licenses|
|Apache Maturity Model||The Apache Project Maturity Model provides guidelines for assessing the maturity of a project|
|Blogposts||Number of blogposts that mention the project|
|Bug Age||Age of known bugs in issue tracker
Use label for determining bugs?
|Bugs after Release||Number of bugs reported after a release|
|Bus Factor||The number of developers it would need to lose to destroy its progress. Alternatively: Number of companies that would have to stop support.|
|CII Best Practices Badge||The CII Best Practices Badge Program provide maturity self-certification: passing, silver, and gold.|
|Code Modularity||Modular code allows parallel development, which Linus Torvalds drove for Linux (OSLS Torvalds)
(Baldwin & Clark, 2006)
|Commit Bias||Diversity metric: acceptance rate (and time to acceptance) differences per gender, ethnicity, etc…|
|Community Activity||Contribution Frequency
(Contribution = commit, issue, comment, …)
|Contribution Acceptance||Ratio of contributions accepted vs. closed without acceptance|
|Contribution Age||Time since last contribution
Gives a sense of how active the community is. (Contribution = commit, issue, comment, …)
|Contribution Diversity||Ratio of code committed by contributors other than original project initiator
Contributions are going up beyond the core team
|Contributor Activity||Activity level of individual contributors|
|Contributor Breadth||Ratio of non-core committers (drive-by committers)
Can indicate openess to outsiders
|Contributor Diversity|| Ratio of contributors from a single company over all contributors
Also described as: Maintainers from different companies. Diversity of contributor affiliation.
|Contributors||Number of contributors|
|Decision Distribution||Central vs. distributed decision making
Governance model, scalability of community
|Dependency Depth||Number of projects included in code base + number of projects relying on focal project (recursive)
Indicator about centrality in open source Dependency network
|Distribution of Work||How much recent activity is distributed?|
|Downloads||Number of downloads
! beware: downloads might be skewed by builders
Used as measure for 'success' (Grewal, Lilien, & Mallapragada, 2006)
|Forks||Number of forks|
|Gatherings||Number of face-to-face/in-person meetings per year
Resets contentious issues; Resolve tensions; Avoid longstanding grudges
|Issue Comments||Number of Comments per Issue|
|Issue Response Rate|| Time between a new issue is opened and a maintainer responds
Also called: bug response rate. The maintainer is believed to not “pile on” but try to solve an issue.
|Issues Open||Number of open issues|
|Issues submitted/closed||Issues submitted vs. issues closed
|Job Postings||Number of job postings that mention the project as a preferred or required skill|
|Known Vulnerabilities||Number of reported vulnerabilities
Could be limited to issue-tracker or extended vulnerability databases (e.g. CVE)
|Language Bias||Diversity metric: Bias against gender, ethnicity, … in use of language (maybe use sentiment analysis)|
|License Conflict||Does the project contain incompatible licenses|
|License Count||Number of licenses|
|License Coverage||Number of files with a file notice (copyright notice + license notice)|
|License Declared||What license does the project declare|
|Non-Source Contributions||Track contributions like running tests in test environment, writing blog posts, producing videos, giving talks, etc…|
|Onion Layers||Distance between onion model layers (users, contributors, committers, and steering committee)
Rule of thumb: factor of 10x between layers. (OSLS'17 Node.js keynote)
|Path to Leadership||A communicated path from lurker to contributor to maintainer. (or. track members: time from user to maintainer/leader)
Rational: If active contributors are not included in leadership decisions they might lose interest and leave. (Focus on least likely contributor)
|Project Life Cycle||Community assigned label
Some communities have a project life cycle for example: proposal, incubaton, active, deprecated, end of life (Source: Hyperledger).
|Pull Request Comments||Number of comments per pull request|
|Pull Request Discussion Diversity||Number of different people discussing each pull request|
|Pull Request made/closed||Pull requests made vs. pull requests closed
Encompasses number of pull requests rejectet
|Pull Requests Open||Number of open pull requests
Might be more telling than total pull requests
|Relative Activity|| I sum up the activities (GH issues+comments, GH pull requests+comments and GH commits) for the project members and for the non-project members, then I create a ratio of the two.
Compare the activity between committers-as-a-group and contributors-as-a-group. It easily shows when a project is not yet popular, or when a project is not paying attention to its users. I also feel that a balance between the two groups is essential; ie) a project with a lot more contributor than committer activity is one that is failing to 'recruit' committers quickly enough.
|Release Maturity||Ratio of major and minor releases|
|Release Note Completeness||Number of functionality changes and bug fixes represented in release notes vs. release.
Good for users, also shows diligence of community
|Release Velocity||Time between releases
Regular releases are a reliability metric
|Reopened issues||Rate of issues closed but discussion continues or issues that were closed and re-opened|
|Repository Size||Overall size of the repository or number of commits|
|Retrospectives||Existence of after release meetings
Collect lessons learned, improve processes, recognize contributors
|Rewards||Rewards, shout-outs, recognition, and mentions in pull-requests or change logs - might improve contribution levels|
|Roadmap||Existence and quality of roadmap
Best Practice: community engagement and scalability (might not be automatically computable)
|Role Definitions||Existence and quality of role definitions
Governance related. Relates to “Path do Leadership”
|Size of Code Base||Lines of code|
|Stack Overflow||Several metrics: # of questions asked, response rate, number of responding people that have verified solutions|
|Stars||Number of stars (GitHub)|
|Time to Contributor||Time to becoming a contributor|
|Transparency||Number of comments per issue
Discussion is occuring openly - could also indicate level of agreement
|Unity||Rivalry or unity of community (sentiment analysis?)|
|Update Age||Time since last update|
|Update Rate||Number of updates over period x|
|Update Regularity||How consistently and frequently are updates provided.|
|Use of Acronym||Frequency of acronyms used
Specialized language can be a barrier for new contributors.
|User Groups||user groups perform a variety of crucial marketing, service support, and business-development functions at the grassroots level
(Bagozzi & Dholakia, 2006)
|Watchers||Number of watchers (GitHub)|
|YouTube Videos||Number of Youtube videos that mention or specifically deal with the project (e.g. tutorials)|
The following section are a collection of random thoughts that might be helpful in a future discussion.
This includes reasons why metrics are considered for other reasons This section collects notes on what possible goals might be.
These categories are meta-concepts that cannot be assessed with a single metric and the combination of metrics will be different depending on the context.
We have heard other classifications that we simply list here.
Ideas for these classifications is to 1. generate a uniform classification and through conversations merge the different classifications. 2. create mappings of the indicators to the different classifications