2
$\begingroup$

I want to analyze the performance of two classifiers. For that I have a dataset with 30000 observations that each have an independent variable interactions and a binary response variable accurate, where the former describes the amount of information available about the observation and second describes whether the classifier predicted the label correctly. There is a strong relationship between interactions and accurate that I want to model using LOESS.

More specifically, I am interested in a subset of the data containing only the observations with the 25% lowest values for interactions. I am using the following code in R, where cnn_loess and svm_loess contains the 25% lowest of 30000 observations for respectively a CNN and a SVM classifer.

loess <- rbind(cnn_loess, svm_loess)
p <- ggplot(loess, aes(x=n_int, y=acc)) + stat_smooth(aes(colour=type), lty=1, method = "loess", span = 1, size=1, se=TRUE)
p <- p + scale_x_continuous(name='interactions') + scale_y_continuous(name='gender prediction accuracy')
p <- p + coord_cartesian(xlim = c(0, 750), ylim = c(0.5,0.85))

However, this results in the plot below, where the confidence interval at the right side is bigger than it should be. Perhaps, more importantly, the lines and confidence intervals are likely changed as compared to if I had used LOESS on the entire dataset and just only plotted a subset of the graph.

I have tried plotting the entire dataset, but I run out of memory. I've also tried using a lower value for the parameter span than 1, which allows me to plot the dataset, but gives a plot, which is not usable. So how do I best plot the LOESS for my subset of the data? I do not want to manipulate the results in any way.

enter image description here

UPDATE: Based on help from Brent Kerby I've isolated the memory issue to calling predict(..., se=TRUE). It works fine without standard errors, but I need them.. Any ideas?

I'm using R version 3.1.3 (September 2015).

x <- seq(0, 30000)
df <- as.data.frame(x)
df$y <- 1 - df$x*(1/30000)

loess_mod <- loess(y ~ x, df)
loess_pred <- predict(loess_mod, x, se=FALSE) # works fine
loess_pred <- predict(loess_mod, x, se=TRUE) # crash with memory error ("Error: cannot allocate vector of size 6.7 Gb")

df$fit <- loess_pred$fit
df$ucl <- loess_pred$fit + 1.96 * loess_pred$se.fit
df$lcl <- loess_pred$fit - 1.96 * loess_pred$se.fits

p <- ggplot(data=df, aes(x=x, y=fit))
p <- p + geom_smooth(aes(ymin = lcl, ymax = ucl), data=df, stat="identity")
p
$\endgroup$
1
$\begingroup$

In this case, rather than using stat_smooth to perform modeling, prediction, and plotting, you can perform these three steps separately using loess, predict, and geom_smooth, which will allow you greater control. You can provide the full dataset to loess, restrict to the desired range of interactions values when calling predict, and then display the results using geom_smooth with the option stat="identity" to prevent it from applying a second statistical transformation (See the last example here.)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! That sounds interesting. However, I am worried that I would still have the overloaded memory problem. I guess of loess, predict and geom_smooth, it is loess that is the memory-intensive one and with this method I'll still be providing the full dataset to loess. $\endgroup$ – pir Oct 6 '15 at 22:00
  • $\begingroup$ For large datasets, loess is time-intensive, but it requires little memory beyond that occupied by the original dataset itself. I've had no problem using these three functions in turn on a dataset of 100000 points; even for that amount of data the memory usage is negligible. I'm not sure what is causing the overloaded memory in your case, but breaking it down into these three steps may make it easier to diagnose. $\endgroup$ – Brent Kerby Oct 6 '15 at 22:41
  • $\begingroup$ Did you also use span=1? $\endgroup$ – pir Oct 7 '15 at 5:37
  • $\begingroup$ Yes, I used span=1 and didn't have any problem. $\endgroup$ – Brent Kerby Oct 7 '15 at 14:18
  • $\begingroup$ Sadly, I still get the problem. But at least it's not been isolated to predict when se=TRUE. Any idea how to let me compute standard errors without memory issues? $\endgroup$ – pir Oct 14 '15 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.