Measuring Open Access Impact

Question

In strict terms to open access - meaning no paywall - green or gold open access. What is the best way to try and measure impact of open versus closed access to articles?

As articles are unique I'm finding it difficult to report potential impact between closed versus open. In just look at the "access" metric I can see that open articles are downloaded and view at a higher rate. However, it's hard to say that it is because it is an open article and not just one that is a higher interest or need.

We do have access to Altmetrics - which is great - it would be good to know if I can use this data to compare to closed research in any meaningful way.

Thank you!

Answer 1

I think your question is actually a statistics question. You know the impact of an article, but you want to know "what would the impact have been if it had been published differently?" (counterfactual).

A common tool to answer this question is randomization: For example, let's say you want to compare open access versus closed access. What you would do, is you'd randomly publish articles in one or the other access format. This concept is usually used in clinical trials for example, where patients are randomly assigned to a treatment. In the end they compute a simple statistical model with treatment as a covariate to compute the treatment effect.

Of course you can't do that. You can't do a randomized study for paper publishing. In this case you need Causal Inference methods. Causal inference is quite a big field. A good entry is the coursera course: https://www.coursera.org/learn/crash-course-in-causality

You were saying that you already have the data, right? I'd be happy to help if you are interested.