1
$\begingroup$

I have a set of data points where X axis represents the percentage of disk space, and Y axis represents the memory used by an application. In general, higher the disk space, the higher memory it'll use.

Sample data points (disk space percentage, memory used): (10%, 1MB), (20%, 1.5MB), (25%, 1.2MB), (60%,10MB)

From this data, I'm interested in finding out top N outlier samples which are using less disk space but more memory, when compared to other data points.

What statistics are suitable for this purpose? Is it even possible to apply a model without knowing what's the normal memory usage for a given disk space on an average?

$\endgroup$
  • 2
    $\begingroup$ It sounds like you might be interested in the thread at stats.stackexchange.com/questions/213 $\endgroup$ – whuber Jan 8 '19 at 20:39
  • $\begingroup$ will look at it. Edited the question a bit. Is there any other information I can add to the question to make it more clear? $\endgroup$ – RandomQuestion Jan 8 '19 at 21:27
  • $\begingroup$ Either an illustration (as a scatterplot) or a clearer description of what it looks like could help, because there are various different ways to approach this situation, depending on what that plot looks like and what kind of "outliers" you mean. (There are some subtle distinctions when you view one variable as being "independent" and the other as "responding" to its value.) $\endgroup$ – whuber Jan 8 '19 at 23:48
1
$\begingroup$

There are a number of statistics and functions for defining outliers.

Assuming a normal distribution, you can standardize your data according to the sample mean and sample standard deviation. Assign a z-score to each observation.

Given that you have a bivariate data set, and that you want to know the outliers conditional on disk space, I would recommend doing a linear regression.

If you have access to the data, I would recommend doing a residual analysis in R.

Build a linear model using memory as a function of disk space. The linear model works best under assumptions of normality of variables. Then, you're looking for large positive residuals. (Positive because you're looking for outliers that use a lot more memory.)

library(car)
set.seed(1234)

#generating data
disk <- c(rep(.01, 25), rep(.05, 5), rep(.1, 5))
memory <- rnorm(35, 5, 1)
mydata <- as.data.frame(cbind(disk, memory))
model <- lm(memory ~ disk, data=mydata)

###residual plot
par(mfrow=c(2,2)) #better ordering of graphs
plot(model, which = 1:4)

###Order data by largest residual
residuals <- model$residuals
mydata2 <- as.data.frame(cbind(disk, memory, model$residuals))
colnames(mydata2) <- c("disk", "memory", "residuals")
mydata2[order(-residuals),] 

Here is the residual plot, qqplot, cooks distance and scale-location.

Residual Plot

And here is the output after sorting the data based on descending residual value (so the top values are the largest outliers.

Table of Positive Residuals

If you're looking for specific statistics, I would consider using Shapiro-Wilk for normality of the distribution, or the outlierTest() function from the car package.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.