How do open projects signal their “openness” to the outside community? This is a really hard question, particularly because nowadays “open” has become a buzzword that doesn’t just signal a project’s position to the community, but is also used as a marketing term to increase support, users, or resources.
I was thinking about this the other day, so decided to take to twitter:
I really wish we had better nomenclature to distinguish between "we make our code publicly available but have little interest in helping you use it" vs "we run an open project/community and want to help you become developers"— Chris Holdgraf (@choldgraf) October 22, 2018
I was surprised at how much this question resonated with people. Here are a few highlights from the (very interesting) conversation that came out of that question.
Some discussion threads
Wishes vs. reality
Tal immediately brought up a really important point: many projects want to be inclusive and welcoming to others, but they don’t have time to do so.
I think the problem with that distinction is that lots of (most?) people _want_ to help people use/contribute to a codebase, but don't have the time/resources to. I would approach this as a measurement rather than nomenclature problem: can we devise good metrics of responsivity?— Tal Yarkoni (@talyarkoni) October 22, 2018
I think this is an important distinction, and something that should be signaled clearly. One the one hand, if a person generally wants others to contribute to the project, then they’re some degree of openness higher than a project that actively discourages this.
On the other hand, running open projects does take work, and a project that says “well I’d like to be open but can’t commit the time to do it” also isn’t that open in practice. No hard feelings there, but I think that the goal of defining a “degree of openness” isn’t to signal a value judgment on the people related to the project, but on the project itself. If you really want to grow an open community around a project, you need to dedicate time and resources to the community itself, not just the technical pieces of the tool.
Metrics of openness
That leaves open the question: “how do we measure the practical openness of a project, rather than just what it says?”. A few folks mentioned that the CHAOSS project does a lot of work in this gneeral space:
CHAOSS defines standards for metrics to collect about communities. They don’t necessarily say what others should do with those metrics, so perhaps that’s on the open community to define for themselves.
Personally, I’d love to see more tooling that makes it possible to scrape activity statistics from open repositories. Tal and others suggested a few things:
- time to initial response to new issues (maybe separated by new vs. old contributors)
- inequality coefficient for contributor commits
- number of unique organizations/email domains in contrbutors
- use of positive/welcoming language
- explicit roles defined, and pathways towards working more with the community
I’d love to see more thoughts along these lines. If we could define a collection of metrics around openness, it’d paint a much more rich picture than simply “does this project have a permissive license.”
There was also a specific metric around governance that’s worth highlighting:
This work comes to mind:— Georg J.P. Link (@GeorgLink) October 23, 2018
Laffan, L. (2012). A new way of measuring openness: The open governance index. Technology Innovation Management Review, 18--24. https://t.co/zkVhC3RjCD
The paper linked above is a study that investigated “open governance” in a number of open-source mobile projects. It’s an interesting exploration of the ways that decision-making is made (and signaled) in several projects. Perhaps unsurprisingly, they conclude that “more open” projects are most-likely to be successful in the long term (with a few exceptions).
Finally, apparently there’s also a “badge” to signal the status of a repository (is it active, vaporware, abandoned, etc):
I found https://t.co/FtbJmYXmfa after searching for what people have done to indicate the level of support a project receives in terms of a lifecycle status. Not quite user-centric but I found it interesting (and open to PRs itself)— Peter Parente (@parente) October 24, 2018
I’d love to see more of these semi-automated signals to help guide the open source community in deciding what projects to adopt and contribute to. As more and more people do their work online and in the open, it also creates a challenge of sifting through the noise to make the most of your (limited) time and energy. Having better metrics like these will make these decisions easier.
Mozilla’s archetypes of open projects
One of the most fascinating links I found was Mozilla’s “archetypes of open projects” document:
Open Source Archetypes: A framework For Purposeful Open Source https://t.co/ydJI7eaiPm— Justin Kiggins (@neuromusic) October 22, 2018
Briefly, this is an internal document that Mozilla made public. It attempts to define the different kinds of open projects that exist. Importantly, it also explains the value propositions of each, how it can be used strategically within an organization, and how it supports (or doesn’t) an open community around it.
I added some thoughts about how Project Jupyter fits into these archetypes on the Jupyter governance research issue and I’d love to think more about how these archetypes fit into the pre-existing open communities that are out there. If anybody wants to brainstorm how these archetypes fit into the scientific open community, I’d love to chat :-)
On that note, I want to give a brief shout-out to Mozilla in general, which has either conducted or sponsored a bunch of interesting work in open projects. For example, they have a whole wiki dedicated to working openly:
Importance of ethnography:
A final note on the importance of ethnography:
See @CHAOSSproj for discussion around metrics. However the reason you do not see these kinds of metrics widely implemented is metrics are insufficient to tell the story when it comes to the complexity of people + community. See @epicpeople_org for Ethnography + mixed quant/qual.— Auggy (@mmmpork) October 23, 2018
For all of my talk about metrics above, I’ve come to appreciate that numbers are never sufficient to describe the complexities of a community or group. Over the last several years at the Berkeley Institute for Data Science, I’ve had the pleasure of working with several ethnographers who have shared their perspective on how to study communities. Semi-automatically-calculated numbers can be a great way to see relatively coarse-level view of a community, but if you really wany to understand what’s going on, you need to dig in there, conduct qualitative interviews, operate in the community, and create some stories that back up (or not) the quantitative data that you collect. We’d all be better off if there were more ethnographers in our respective communities <3.
OK, that’s enough for now - I hope these links are useful and I’ll try to update them over time if I hear of some new projects along these lines. If you have any suggestions, feel free to leave ‘em in the comments!