Rebooting Peer Review as Relational Infrastructure

Intro

There’s a particular loneliness that shows up in open source right now. Generative AI is creating so much to read: code, docs, posts, explanations. It’s disconcerting, not least because so much of it feels non-connective — a message with no sender.

pyOpenSci was no exception. An influx of submissions made triage hard. Finding editors and reviewers was a chronic struggle. Infrastructure designed to make people’s lives easier went unused because those people didn’t know it existed. The community had recently come together to co-create a generative AI policy, but now had to apply those rules alone. pyOpenSci was full of amazing, passionate individuals, but they were largely struggling with these challenges alone.

This post documents how I’ve spent the past two months trying to rebuild connection.

Start with listening

One of the first things I did was get on long calls with the Editors in Chief (EiCs) — Carter and Kylen — and with Eliot, who was handing off the peer review lead role as I came in. I wanted to hear how peer review worked from the people most involved, especially how it felt: what was joyful, what was draining, what they hoped for, what stressed them out.

What they told me became the roadmap for the reboot. Finding and tracking editors was a messy shared problem nobody was sure how to tackle. Shared infrastructure existed that people didn’t know about, or that didn’t meet their needs, so private spreadsheets filled the gaps. Applying the new generative AI policy felt different to different people, and there was no clear place to bring up issues with it. There were community spaces people could bring these problems to, but they were almost all asynchronous text channels, most of them with dozens or hundreds of other community members.

Carter had an idea that turned out to be crucial: recurring face-to-face check-ins. This became our monthly peer review call. Having more of that time sparked creativity in a way Slack threads hadn’t, and more of a sense of being all in this together.

Listening wasn’t a prelude to the “important” work. It was the most important work.

Tools follow conversation

The early EiC calls didn’t only surface feelings. They surfaced mismatches between the work people were doing and the docs and tools meant to support it — sometimes because people didn’t know something existed, sometimes because what existed didn’t meet the need. The editorial dashboard is a good example. It was helpful in theory, but too often out of date, so someone had built a separate spreadsheet. Through conversation we fixed the disconnect: give EiCs a GitHub workflow (and access to that workflow) so they can update the dashboard themselves whenever they need to.

A smaller example of the same pattern: once EiCs and editors understood the infrastructure and how to update it, it became obvious that EiC workflow really benefited from being able to click from a dashboard list straight into the linked submission issue. We asked for the feature. It got added. That’s relational infrastructure producing better technical infrastructure.

Relational friction is good friction

This focus on listening extended to our onboarding process. pyOpenSci regularly needs to onboard both reviewers and editors, but the work of editing is a somewhat harder job — it requires experience reviewing, but also a willingness to be a kind of project manager, checking in on review status, recruiting reviewers, checking in to make sure everything gets done. Together, the EiCs and I decided the best way to onboard potential new editors would be doing videocalls — helping create that human connection right from the beginning.

Having a call with every potential editor might seem like a lot of work, but again, it’s the important work. I like to distinguish between two kinds of friction: administrative friction, and relational friction.

Administrative friction is annoying and should be automated away if possible. Relational friction is the stuff that makes life worth living.

(As an aside — it isn’t always clear what’s administrative friction and what’s relational friction! I used to help newcomers to open source make contributions by pairing on linting Python code. It was a great way for us to bond, and for them to learn some Python. The advent of linters automated away that friction, for better or worse.)

We’re continuing to improve the onboarding process, as well as other processes involved in peer review, like finding and assigning reviewers, applying the Gen AI policy, etc. As we do that, I’m always keeping an eye on what we’re automating away. If it’s adminstrative friction, great! That clears up more time and energy for things like community calls. But if it’s relational friction, I almost always advocate for keeping it.

Places to be human together

The monthly peer review call Carter suggested starts with icebreakers. It’s a small thing, but it helps to see each other as people first. We do get work done together — unstick collective problems, check what’s falling through cracks — but the call is also a place to ask for help, ask naive questions, and just hang out with your pyOpenSci friends.

Once you make room for face-to-face connection, it starts to multiply. Akrita, one of the editors, organized and ran a community info call to onboard new people. It was well attended and very appreciated. Recently another community member, Carol, suggested a virtual docs sprint. People loved the idea. We’re starting to establish a cultural pattern of noticing where async isn’t enough and asking for a space to share time together.

Bringing the community into the AI research

pyOpenSci has grant funding to research how AI is changing open source. Leah Wasser, Daniel Chen and I have been leading development on a survey, focus group questions, and a landscape analysis of AI policy. It’s real research, but it’s also the struggle our community is living through.

So we’re bringing the community into the research. We used a community info call to walk through focus group and survey questions with the community. We also shared copies of the instruments for people to leave feedback on as we refine them. Bringing people into the research process isn’t only about getting better data. It’s about empowering people to participate in the research that impacts them.

It also says: we care enough about your experience with AI enough to formally study it. You’re not alone in your struggles. The ways AI is making your work harder (or stranger, or lonelier) are worth listening to, discussing, and addressing together.

Where we are

Two months in, the reboot is ongoing. Onboarding is still iterating. AI decisions are still evolving. Peer review is a living system. But connection is more intentional than it was. Editors are coming in through real outreach and real conversations. EiCs have more of what they need — including trust made concrete through tool and platform access. The community is helping lead the AI research, not just subject to it. People keep inventing places to be together.

In a time when generative AI is severing our connections to each other, rebooting peer review has meant choosing the opposite on purpose: focusing on the relational infrastructure that keeps us connected. Because connection is what matters most — for good peer review, and for a vibrant community.