# How to convert discrete data into a continuous curve

I am interested in drawing graphs to show the distribution of letters throughout a word. This would be in the form of a graph with the x-axis being a continuous scale from 0 to 1 and the y-axis being frequencies. For example I would expect the letter 'Q' to have a distribution that is higher for small x and decreases as x increases since words are more likely to begin with 'Q' than end with it.

I have a dataset of 1 million words that I've mined from famous novels in R and I have a list for each letter of every position it appeared in as a proportion of the words it was in e.g. 3/8 if it was the third letter of an eight letter word.

I am unsure of how to convert this data into a smooth curve showing their distributions. I have ideas that are accurate (just counting the number of occurrences of each value) but don't look smooth (since they'll be a massive jump at x = 0.5 and due to the influence of common 2-letter words). I also have ideas that are smooth (placing down normal distribution curves at each point and then summing) but don't feel accurate or valid in any sense. What method would be the happy medium of this?

• You might be interested in Baayen's Word Frequency Distributions. (Check your local library or ILL… I found it an approachable text.) Commented Jan 12, 2018 at 22:45

I second some of the caution advised in using this approach. That being said, you can certainly force the distributions for each letter to appear smooth. One option in R is using the ggplot function stat_density where you can specify a bandwidth bw= that governs the degree of smoothing. Of course smoothing reduces your information, and you will have to decide the appropriate trade off between smoothing and information loss when visually representing your distributions.

Using the iris data in R here is one approach that you can build from.

library(ggplot2)
data(iris)

g1<-ggplot(aes(x=Petal.Width), data = iris)
g2<-g1+stat_density(aes(x=Petal.Width), fill="#F8766D", bw = .0975,
iris[iris$Species=="versicolor",], alpha = .75) g3<-g2+stat_density(aes(x=Petal.Width), fill="#00BFC4", bw = .17, iris[iris$Species=="virginica",], alpha = .75)
g4<-g3+stat_density(aes(x=Petal.Width), fill="#80FF00", bw = .17,
iris[iris$Species=="setosa",], alpha = .75) g5<-g4+scale_x_continuous(limits = c(-.5,3.5)) g5 Which returns the following plot. Note that some non-allowable values are depicted when using this smoothing approach as petal width cannot actually be negative (just one downside of "over"-smoothing). Of course if I reduce the bandwidth, the distributions lose some of their nice smooth qualities, but they do become more accurate in their representation of the data. g1<-ggplot(aes(x=Petal.Width), data = iris) g2<-g1+stat_density(aes(x=Petal.Width), fill="#F8766D", bw = .03, iris[iris$Species=="versicolor",], alpha = .75)
g3<-g2+stat_density(aes(x=Petal.Width), fill="#00BFC4", bw = .06,
iris[iris$Species=="virginica",], alpha = .75) g4<-g3+stat_density(aes(x=Petal.Width), fill="#80FF00", bw = .03, iris[iris$Species=="setosa",], alpha = .75)
g5<-g4+scale_x_continuous(limits = c(-.5,3.5))
g5

One good rule in smoothing would be to ensure your final graphic does not include values that are not allowed on the scale. And remember the choice of the degree of smoothing is subjective.

• You can use the group argument to simplify the ggplot code, additionally I don't think there is a good reason to create a new object for each layer of the plot; for example ggplot(aes(x=Petal.Width, group = Species, fill = Species), data = iris) + geom_density( alpha = .75, bw = .17, col = NA) + scale_x_continuous(limits = c(-.5,3.5)) Commented Jan 12, 2018 at 23:21
• No disagreement. It was just some old code I had in which I wanted to make things explicit about the choices of bandwidth for each group. The goal was not coding efficiency but rather clarity in the illustration. Commented Jan 12, 2018 at 23:23
• Fair enough - Apologies for being overally perfectionist with the code ! Commented Jan 12, 2018 at 23:29
• No apology necessary. Paying attention to efficient coding is increasingly important for the modern day applied statistician. Commented Jan 12, 2018 at 23:32

It's not clear why you want to describe this totally discrete data with a continuous distribution. I think you'd be better off by looking at letter frequency at each position, where each position is the letter index (e.g. 3), and not the proportional location of the letter (e.g. 3/8). That proportional index has problems like you mentioned, with 2-letter words only having values at 0.5 and 1, 4-letter words only having values at 0.25, 0.5, 0.75, and 1, and so on. On top of that, values at 0.5 are the beginning of the word for 2-letter words, but halfway through the word for 10-letter words. You can see that 0.5 doesn't as have much meaning as the positional index if you're concerned about word beginning or endings.

Instead, look at all words that have at least two letters and calculate frequency at position 2. Then look at all words that have at least 3 letters, and calculate frequency at position 3. Now each positional distribution sums to 1, and accurately takes into account only words that are actually that long. By taking proportions, you get rid of those big bumps that represent certain-length words.

• I see your point but I do feel like the method I am suggesting offers more interesting information about the distribution of letters. So yeah, I could split them up into individual length words and draw a histogram for each but what's the point in that? The 2,3,4 letter words tell me nothing, as you said, and the longer words are rarer so I don't have much data to work with plus they may not be representative of the whole language. What I really want is a way to combine the whole thing Commented Jan 12, 2018 at 19:47