Lessons learned: 100 interviews about numerical JavaScript
How much of what a project believes about its users is actually based on evidence?
We ran a hundred customer-discovery interviews in seven weeks earlier this year, reaching out to anyone we could think of who might have something to say about high-performance numerical and scientific computation in JavaScript (which is the thing stdlib exists to do), as well as those with knowledge about sustaining open-source ecosystems. We also searched the Journal of Open Source Software (JOSS) for work involving JavaScript—a remarkably short list—and wrote to the people behind the papers.
So, the sample is what two people's address books could reach, plus what one literature search could find. Our networks largely held people we already knew, which shaped the corpus.
Projects overestimate how well they know their users. It's in the corpus over and over—maintainers confident about what users need, users describing something adjacent but not the same when you ask them directly with no maintainer in the room. Which is not a maintainer failing, or a developer failing. Expertise and certainty aren't correlated the way we'd like them to be, and Tavris and Aronson have the bleak version of it in Mistakes Were Made (But Not by Me): “training does not increase accuracy; it increases people's confidence in their accuracy.”[1] That's also how a wrong assumption persists—nothing in the loop is set up to catch it, so it just keeps being the thing everyone knows. We ran the interviews, found the pattern, and still catch ourselves in it.
What we did, and what the numbers can't say
The interviews were the "customer-discovery requirement" of the NSF I-Corps program, which stdlib went through as part of its POSE award. The count (100) and the timeline (seven weeks) came with it. Two of us doing them.
Recruiting a hundred people that fast doesn't leave much room for a sampling strategy, and two of our three channels were just us. My side of that reach was R and data visualization; Athan's was scientific Python and numerical JavaScript.
The third channel wasn't a network. To find academics actually using JavaScript in a lab, we used a mix of GitHub repo surveying and JOSS as an index—papers citing JavaScript—and cold-emailed the authors we found that way. Most had no idea who either of us were, but were generous enough to give us their time.
The interviews were also confidential, which is a routine research condition rather than a dramatic one. A brief survey runs alongside this, because the gap in the corpus is a specific gap and the people who could close it are, by definition, not in it.
Four major themes
1. The ecosystem doesn't answer is this right
People arriving at JavaScript from R or Python don't say I can't do the thing I want to do in this language. They say I can't tell whether this is right. The problem they're describing isn't capability, it's epistemic: there's no obvious mechanism in the JavaScript ecosystem for distinguishing a numerically sound implementation from one that merely runs and returns a number of roughly the right magnitude.[2]
Several people named this before we'd asked anything about stdlib, which is what makes it their framing rather than ours. The named counterexample, repeatedly, was CRAN—the Comprehensive R Archive Network that runs checks before a package is listed, and where the listing itself carries information.[3] In the eyes of those used to the CRAN model, npm (the predominant package manager for Node.js) appears to be an open field of sorts. Things grow in it. Nobody inspects them.
So people reach for the proxies that exist. Stars. Weekly installs. Download counts. At least one person described using install counts as a quality signal while being entirely clear-eyed that it isn't one, because no signal of statistical validity is on offer and something has to bear the load. Popularity is what you use when correctness isn't measurable from the outside.
A large share of the people opening pull requests against stdlib aren't using it in their own work—filling in the ecosystem canvas is where we had to write that down.[4] And the actual grounds for trusting the library are the reference implementations the tests check against, the fixtures generated from them, the benchmarks. All of it stays invisible to the people standing closest to it. Those grounds are inspectable, which is what open source has instead of trust us; being able to read the source is how trust here gets earned rather than granted. Reading it is also a cost, unevenly payable, which is most of what documentation conventions exist to lower. What people have in the meantime is what they've been able to watch: how a maintainer handles a bug report, what a review looks like, behavior over time. That's real evidence and it's rational to use. It just isn't transferable—you can't hand it to your team lead, and sooner or later you'll be asked to.
A project looking at its own trusting users is looking at the wrong evidence. What it sees is people relying on it. What it tends to read that as is our quality is legible from the outside. What it's often actually seeing is the people who got close enough to watch it work. And the people for whom the quality wasn't legible didn't file an issue saying so—they used something else, or went back to R, and the project never saw them at all.
More than one person described building a small solo tool that achieved unexpectedly large downstream use with essentially no feedback reaching them. That's silent adoption, and stars and downloads can't detect it at all—they count use, not whether any of it ever gets back to the person who built the thing.
Trust us, we're careful asks for exactly the thing the finding says is unavailable. It also misplaces the work: robustness and reliability aren't a claim you make about the product, they're a key part of the product. So the job isn't to ask anyone for trust—it's to show that what we do lines up with the things they already trust.
stdlib already has the material for that, built deliberately and for engineering reasons—to know the implementations are correct—rather than to demonstrate anything to anyone. The expected values in the special-function test fixtures aren't hand-written, and they aren't captured from stdlib's own output—they're generated by external reference implementations and committed next to the tests with the generator script included. Close to two hundred packages check against Julia; the rest mostly against NumPy and SciPy, Boost.Math, and R.[5] You don't have to believe us about any of that. You can check that we match what you'd have used anyway.
2. AI is the densest theme, and it doesn't converge
Roughly thirty of the interviews touch AI-assisted contribution.
We went in with a prior, and the prior was wrong. We expected sentiment to sort by seniority—junior contributors enthusiastic, senior maintainers alarmed, some tidy generational story. It doesn't. Senior maintainers of substantial projects show up alarmed and enthusiastic in roughly equal measure, and neither that story nor its inverse holds.
Across people who disagree about everything else here there was one near-universal claim: AI has lowered the cost of submitting a contribution much further than it's lowered the cost of evaluating one. That asymmetry is the one thing nearly everyone sees. What they profoundly disagree about is whether it's a crisis, a nuisance to route around, or a net win.
Three registers are clear in the corpus, and they're different arguments:
- There's alarm about quality—volume up, signal down, review time is finite.
- There's alarm reframed as security, which is a different claim entirely: a contribution pipeline without process isn't just noisy, it's a risk surface, and treating it as a code-quality problem understates it.
- And there's a rarer fatalist register that doesn't really argue—the technology arrived, it's here, and whether it should be isn't a live question.
A project's posture with respect to newcomers, described in Do you WANT contributors?, is a separate matter from the register in which it lands.
There may be a fourth: that code review was never only quality control, it was how a project handed its judgment to a new person, and something that produces acceptable diffs without that exchange has broken the part that mattered even when the diff is fine.
Two positions invert the dominant frame entirely. One imagines AI as the filter—the triage layer that protects a maintainer's attention rather than the flood that overwhelms it. The other argues that a project should deliberately optimize its own legibility to models as a discoverability strategy, on the reasoning that training cutoffs are why newer libraries are structurally invisible to the tool an increasing number of people ask first.
Sentiment may sort instead by which side of the submit/review transaction someone spends their day on; that hasn't been checked against every interview that touches AI (and may have legitimately changed over the subsequent months).
3. Who showed up, and what they came from
Twenty-six of the hundred people we interviewed had contributed to stdlib. Seventy-four hadn't. That split matters because it means this isn't contributors imagining what users want. It's users, or prospective users.
Within it, one relationship is about as strong as a hundred conversations can show: whether someone has contributed to stdlib predicts what language they compute in. The contributor group is JavaScript- and C/C++-native at roughly three-quarters. Not one of the twenty-six contributors arrives primarily from Python. Exactly one arrives primarily from R, and that applied-statistics-to-Node path is the origin story of stdlib's own precursor project—which isn't evidence that R people find their way to stdlib, it's evidence that one of them founded it. Meanwhile, half the non-contributor group is Python- or R-primary.
Both groups came out of the same biased pool, which is what makes the comparison usable. Whatever our networks did to the absolute numbers, they did to both columns—so the difference between them survives the sampling problem even though neither column on its own does.
Two negative findings sat alongside it, and we predicted neither. MATLAB is primary for nobody. Two people name it as a language they left. Julia is primary for nobody either, showing up only as a reference implementation people consult when they want to check what a correct answer looks like. Both of those were carrying weight in our persona assumptions, and neither earned it.
The obvious story to tell here is that stdlib's written material addresses JavaScript developers, its talks address scientists arriving from Python and R, and the contributor population is what happened. The data is consistent with that. It doesn't establish it, and there's a sufficient alternative explanation sitting right there: seventeen of the twenty-six contributors arrived by way of Google Summer of Code (GSoC)—some in the program, others contributing in the hope of getting into it—and either way the sorting is by project.
4. The most valuable parts don't scale
Ask people what made their contributor experience valuable and the same answer comes back again and again: a specific human paid individualized attention to them. Eighteen interviews say so, which makes it consistent rather than dominant—onboarding generally turns up in roughly twice as many. The Stakeholder Journey is its own subject; among the maintainers and program managers here who took the question up, it was near-unanimous that individualized attention doesn't scale, and nobody claimed to have solved it. That's a real tradeoff, not a tooling gap. Three things came up reliably about why.
First, the good-first-issue pipeline isn't the solved problem it looks like from outside: one large project reports that its organic supply of genuine beginner-appropriate issues ran out, and that beginner tasks now have to be manufactured. That complicates any assumption that onboarding gets easier as a project matures.
Second, mentorship labor includes a category that no onboarding framing counts. Sustaining the motivation of a contributor who has lost interest isn't review and it isn't documentation, and it doesn't appear on any dashboard. It's emotional work, and the one person who named it described it as the biggest stumbling block they had.
Third, an actively evolving codebase imposes a contributor cost that maintainers don't always register. Conventions shift; the shift is rarely documented as a shift. This showed up independently from more than one contributor inside the same project, which is about as clean a demonstration of the visibility asymmetry as we got.
Onboarding and mentorship read as big themes partly by construction. GSoC is a mentorship-heavy program, and it brought in a substantial part of the contributor group. The theme is real. Its rank is an artifact of who we talked to.
What we still can't answer
We don't know whether JavaScript developers actually have the ndarray comprehension gap we've assumed they have. Our own framing has been that people arriving from JavaScript need what is an ndarray explained while people arriving from Python need the subtleties. Half of that has evidence. On the other half, all hundred interviews say nothing at all—not a little, nothing. That could mean the people who bounced off it were never reachable by this recruitment channel, or it could mean it was a maintainer's assumption about users that we've been carrying as a fact. This corpus can't separate those.
There's a lot we didn't reliably ask. Which programming language someone had come from is one instance: twelve of the hundred can't be classified on the axis that produced one of our strongest findings. Migration friction is another: it came up as the two-language problem—one language, namely Python or R, for analysis, and JavaScript for visualization—not in a form we could count.
And then there's a limit on the whole corpus that no amount of re-reading fixes, and this one isn't about our sample: it's about what asking does. Nobody was hiding anything—people said plainly when they weren't sure, and more than one of them worked out mid-conversation that shorthand was all they had. That's the thing. A question hands you a slot shaped like a position, and what's available to fill it is shorthand, because that's what anybody can actually retrieve about a choice like this.
The summary here runs the operation a second time. A hundred conversations became four themes, and a theme is a position that's been made comparable to other positions—which is why it can be summarized, and it's paid for out of exactly the particularity that made any one answer worth having. So the sections above are the shape of what a hundred people say when they're asked. That's a real thing to know. It isn't the shape of what they know.
A project's confidence about its users and a project's evidence about its users may be close to unrelated quantities. Trust is where that gap opens widest. The trust a project can see is the trust it earned in view of people close enough to watch, and everyone else's assessment happens somewhere the project can't reliably see. The distrust never shows up at all—it looks like somebody quietly using something else.
There's a short survey (11 questions). If the language you compute in, your discipline, or your path to numerical computing on the web isn't represented above, that's the hole.
stdlib is an open source software project dedicated to providing a comprehensive suite of robust, high-performance libraries to accelerate your project's development and give you peace of mind knowing that you're depending on expertly crafted, high-quality software.
If you've enjoyed this post, give us a star 🌟 on GitHub and consider supporting the project. Your contributions and continued support help ensure the project's long-term success and are greatly appreciated!
Acknowledgments
This work was supported in part by the National Science Foundation under Award No. 2449410.
Disclaimer: Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Carol Tavris and Elliot Aronson. Mistakes Were Made (But Not by Me): Why We Justify Foolish Beliefs, Bad Decisions, and Hurtful Acts. Third edition, Houghton Mifflin Harcourt, 2020, chapter 5, "Law and Disorder." ISBN 978-0-358-32961-9. ↩︎
Of course, as with all of open source, you could always read the source code. But, in practice, this is least likely to be helpful for those used to doing their numeric and scientific analysis in other languages. ↩︎
CRAN Repository Policy. https://cran.r-project.org/web/packages/policies.html. Packages must pass
R CMD checkon the maintained platforms before and after being listed, which is the mechanism the interviewees were reaching for—not the curation judgment, but the existence of any automated gate at all. ↩︎"The Open-Source Ecosystem Canvas." https://blog.stdlib.io/open-source-ecosystem-canvas/. The Community Members cell records the phrase the canvas forced into the open—usually non-users—next to casual contributors. ↩︎
Counts of
test/fixtures/<generator>/directories under@stdlib/math/base/special, taken against thedevelopbranch on 2026-08-26: Julia 198, Python 23, C++ 20, R 11, C 3, Wolfram 2. Each generator is a script calling an established library in that language, committed alongside the fixtures it produced. ↩︎