Looking at the top X% of a population

Recently, I came across an article that claims shorter titles of scientific articles tend to result in a greater number of citations, implying concise titles are better.

While this sounds plausible, the article uses only the top 2% of articles, ranked by the number of citations.

My questions is whether this is a valid approach to answer the research question: Does title length affect the number of citations of an article? Can't this only help explain a (small) difference in citations, given your article is already in the top 2% cited articles?

I don't understand what justifies this approach to the problem. What kind of bias does sampling from an upper quantile introduce in general and can any claim be made about the 98% with less (<16) citations?

The advantage of short paper titles
Adrian Letchford, Helen Susannah Moat, Tobias Preis
R. Soc. open sci. 2015 2 150266; DOI: 10.1098/rsos.150266. Published 26 August 2015

Same authors, similar strategy:
The advantage of simple paper abstracts
Adrian Letchford, Tobias Preis, Helen Susannah Moat
Journal of Informetrics, Volume 10, Issue 1, 2016, Pages 1-8, ISSN 1751-1577; https://doi.org/10.1016/j.joi.2015.11.001.

Why did they only look at the top 2% citations?

For pretty much any distribution that looks concave, the datapoints in the upper quantiles will usually span a much larger range than those near the middle as the distribution tapers off. The probability distribution over the number of citations is probably a heavy-tailed distribution so the above must hold true. Since the objective of the study you cite was to quantify the influence of a variable (title length) on the number of citations, it makes sense that they would focus on the data near the tail where the sensitivity can be easily revealed. If instead, they had used the middle quantile, they wouldn't see much variation in the number of citations to quantify any effect.

Is their conclusion valid?

In general, whether or not it is okay to restrict your focus on the upper quantile of the population depends on the relationship between the variables your are attempting to model. Let $x$ and $y$ be the predictor and response variables such that:

$${y} = {f}(x) + \epsilon$$

for some function $f$ and independent noise $\epsilon$. Let $y_{(p)}$ denote the subset of all $y$ that lie in the neighbourhood of the $p$th percentile of the distribution $P(y)$, and $x_{(p)}=f^{-1}(y_{(p)})$ be the corresponding set of values of $x$.

Now suppose you choose two different quantiles around $p$th and $q$th percentiles of the population and use linear regression to model the relationship as $y_{(p)} = \beta_p x_{(p)}$ and $y_{(q)} = \beta_q x_{(q)}$ respectively. We have:

$$\beta_p=\frac{dy}{dx}\Bigr|_{\substack{x=x_{(p)}}}=f'(x)|_{\substack{x=x_{(p)}}}$$ $$\beta_q=\frac{dy}{dx}\Bigr|_{\substack{x=x_{(q)}}}=f'(x)|_{\substack{x=x_{(q)}}}$$

If $f(x)$ is linear in $x$, then it wouldn't matter which neighbourhood you choose. For instance, if $f(x)=x$, then $\beta_p = \beta_q = 1$ and both quantiles will be representative of the full population.

However, if $f(x)$ is non-linear, then the conclusions you draw could depend drastically on the sample you choose. For example, if $f(x)=1/x$, then:

$$\beta_p=-\frac{1}{x_{(p)}^2}=-y_{(p)}^2$$ $$\beta_q=-\frac{1}{x_{(q)}^2}=-y_{(q)}^2$$

So $|\beta_p|$ >> $|\beta_q|$ if $y_{(p)}$ > $y_{(q)}$ yielding drastically different results for different quantiles of the population.

So, unless the true relationship happens to be linear, it would be wrong to draw general conclusions about the full population based on a biased sample.

• Thank you for your thoughts, however, my concern is mainly about the inference on articles in general, using only this part of the total population. Commented Oct 27, 2017 at 5:54
• I have expanded my answer to address your main question. Hopefully, I have answered it now. Commented Oct 28, 2017 at 21:29
• Thank you for your effort, I really like the visual explanation! I will accept your answer. Commented Oct 29, 2017 at 4:00

My questions is whether this is a valid approach to answer the research question: Does title length affect the number of citations of an article?

It would be a valid approach to answering the research question 'Does the title length affect the number of citations of top 2% cited articles'.

But it's problematic to generalize this result to evaluate the effect for 'articles in general' -- because the population that is being sampled (top 2% articles) is probably quite different to the population of 'articles in general'.

• (+1) This is precisely my concern. However, I would like to know if there is some way to demonstrate that the sampled portion is probably different, as you state. Commented Oct 27, 2017 at 5:53
• I think to demonstrate whether the sampled proportion is 'probably different' (or not), one would have to do another study. Without that, we don't know -- they might be different, or not. Commented Oct 27, 2017 at 5:56