# Using statistics to prove that an upgrade worked

suppose I sell wind turbines. A customer asks me to upgrade one of the wind turbines (turbine A) in a wind farm. The upgrade is performed, and the customer wants some "proof" that performance (say, power output) has improved. Now, I have power output measurements for the turbines in the wind turbine before and after the upgrade. However, just comparing average power output of turbine A before and after the upgrade is not sufficient proof, because the power output strongly depends on wind speed. Thus, variations in the power output may simply be due to variability in wind speed before and after upgrading. So, some way of controlling for enviromental conditions variability is needed. An approach may be to compare averages on very long periods of time, but I'd rather not tell the customer that in order to know if the upgrade works or not, she/he needs to keep measuring power output for an year. Idea:

• make a regression of the power of turbine A, on the powers of the other turbines, before upgrade. I store the regression residuals. These residuals ($\epsilon_i=\hat{y}_i-y_i$) have zero mean (a property of linear regression).
• predict the power of turbine A after the upgrade, using the same model as before (I don't refit the model). Now, residuals don't need to have a 0 mean. On the contrary, I'd expect the mean to be negative, if the upgrade works. My reasoning is: wind affect all turbines in (approximately) the same way, so if the only change before and after upgrade is random variation in wind speed, then regression residuals will still be close to 0. But if instead turbine A has improved, and all the others are still the same, then the power output from turbine A should be higher, on average, than that predicted by the regression.
• I can perform a two-sample t-test, where the two samples are residuals before and after the upgrade. The null hypothesis is that the mean residual before upgrade (0) is equal to or smaller than the mean residual after upgrade (one-sided t-test). If I get a very low p-value, I have significant evidence against the null, and I reject it -> upgrade worked.

What do you say? Does it sound reasonable? Most importantly, is the sign in the my one-sided test correct, or am I proving the wrong hypothesis? Of course, I'd check that the regression has good predictive performance, using cross-validation. Do residuals need to be normally distributed for this to work? I don't think so: what I need to be (approximately) normally distributed is the sample mean of the two samples of residuals. This I could check with bootstrap, for example.

• Is your customer really this statistically literate? I'd think that a scatterplot of the old and new turbines' output against wind speed would be striking visual evidence: if the new turbines really are better, they should cluster away from whatever trend (linear or otherwise) the old turbines follow. – Sycorax Oct 13 '15 at 18:51
• He doesn't have data for power output of the new turbine for many windspeeds ("but I'd rather not tell the customer that in order to know if the upgrade works or not, she/he needs to keep measuring power output for an year.") – bdeonovic Oct 13 '15 at 19:00
• @bdeonovic you're right. For user777, some customers are quickly getting more statistically literate than manufacturers - of course I can't make names, but I know of at least one competitor who had issues because it had to settle a similar issue with a customer. – DeltaIV Oct 13 '15 at 19:10
• @DeltaIV I think your approach sounds very reasonable. I was a bit confused when I read through it because typically residuals are defined as $(y_i-\hat{y}_i)$, but as long as you are consistent it should be fine. If you define them as I prescribed then you would expect the residual of your new data point with upgrade to be postive, which I think sounds better i.e. your turbine is producing MORE power than expected – bdeonovic Oct 13 '15 at 19:16
• One of the things you will find is that not all years are created equal. If you assume that - it will bite you (hard/expensive) in the butt. for most places you can get historic weather data online. Anyone with a (few hundred grand invested in a) wind turbine should have a basic weather station nearby. There are ~5% to 10% to be gained by pro-active aiming the turbine - no (big dollar) hardware required. – EngrStudent Oct 13 '15 at 19:50