Skip to main content
4 of 5
added tag
gung - Reinstate Monica
  • 147.5k
  • 89
  • 406
  • 717

Data cleansing in regression analysis

I've got a dataset for Temperature & KwH and I'm currently performing the regression below. (further regression based on coeffs is performed within PHP)

# Some kind of List structure..
UsageDataFrame  <- data.frame(Energy, Temperatures);

# lm is used to fit linear models. It can be used to carry out regression,
# single stratum analysis of variance and analysis of covariance (although
# aov may provide a more convenient interface for these).
LinearModel     <- lm(Energy ~ 1+Temperatures+I(Temperatures^2), data = UsageDataFrame)

# coefficients
Coefficients    <- coefficients(LinearModel)

system('clear');

cat("--- Coefficients ---\n");
print(Coefficients);
cat('\n\n');

Pastebin http://pastebin.com/Nxdfvzqk

The issue comes with our data, we can't ensure there isn't random communication failures or just random errors. This can leave us with values like

Temperatures <- c(16,15,13,18,20,17,20);
Energy <- c(4,3,3,4,0,60,4)

Temperatures <- c(17,17,14,17,21,16,19);
Energy <- c(4,3,3,4,0,0,4)

Now as humans we can clearly see that the 60 for Kwh is a mistake based on the temperature, however we have over 2,000 systems each with multiple meters and each in different locations all over the country.. and with different levels of normal Energy usage.

A normal dataset would be 48 values for both Temperatures & Energy per day, per meter. In a full year its likely we could have around 0-500 bad points per meter out of 17520 points.

I've read other posts about the tawny package however I've not really seen any examples which would me to pass a data.frame and it process them through cross analysis.

I understand not much can be done, however big massive values surely could be stripped based on the temperature? And the number of times it occurs..

Since R is maths based I see no reason to move this into any other language.

Please note: I'm a Software Developer and have never used R before