Finding a function for data points with disturbances

I have a question regarding how to find an equation for data points that have disturbances. For example, I have the following data:

• P = 366 measured values
• T = 366 measured values
• t = [ 1 : 366 ], representing the days of the year (index)

So at each t (day), we have value of P and corresponding value of T. When drawing P and T vs. t, it shows a continued curve for P with disturbances. These disturbances are caused by the change of T and it is clear that P is mainly affected by T, meaning that: P = f(T)

This figure is shown here:

This figure shows P vs. T:

My aim is to find this relation between P and T or in another words: P = f(T). What I have tried and think so far is first to smooth the curve P and from the smoothed points and smoothed curve, some function may be established. Then, including the effect of disturbances (up and down) by some function, may be exponential.

Could you please guide me how can I manipulate this problem to get a final model P = f(T).

• Your proposed model misses some important aspects. This would be clear if you were to redraw the second figure (the scatterplot) in a way that distinguishes time, because the upper envelope occurs between 160 and 290 days and the lower envelope at other times. Evidently $P$ depends on time $t$ and so does $T$. Separate out these dependencies, writing $P(t) = g(t) + \delta(t)$ and $T(t) = f(t) + \epsilon(t)$. The task is to identify $f$ and $g$ under the supposition that $\delta$ depends on $\epsilon$.
– whuber
Commented Sep 19, 2011 at 14:32

$P$ (power) depends on $T$ (temperature) as well as $t$ (time) in these natural, obvious ways:

1. Power is the sum of a base consumption (for operating electrical and gas equipment such as stoves and refrigerators in a consumer setting), consumption for lighting, and for heating and cooling.

2. Base consumption may vary but the variation is likely to be relatively small and can be handled with additional factors (such as a vacation/weekend schedule) or assumed to be randomly varying around a constant level.

3. Lighting needs depend on the length of the night and should be just about directly proportional to the length.

4. The heating and cooling consumption depends on the amount of time a building falls below a heating threshold or above a cooling threshold.

(4) is a complex relationship. Consider heating; the analysis of cooling is similar. As the external temperature varies, the internal temperature will vary, too, but with a lag in time and with a dampening effect. Heating is needed only when the internal temperature would fall beneath the heating threshold $h$, and then only to supply enough heat to bring the building up to the temperature $h$. Mathematically we can conceive of $T$ as an instantaneous function of time $t$. The hypothetical internal temperature of the building $S(t)$ would depend on previous temperatures. A good model for this would be the linear operator

$$S(t) = \int_{-\infty}^{t}\kappa(t-u) T(u) du$$

given by some (unknown) kernel $\kappa$. The value $\kappa(x)$ gives the amount to weight $T(t-x)$ in estimating the internal temperature $S(t)$. The amount of heating needed at time $t$ is then proportional to the difference between the desired temperature and the temperature the building otherwise would attain,

$$(h-S(t))^+ = \min(0, h - S(t)).$$

This part of the model is clearly apparent in both plots. Cooling is required only at times in the middle of the first plot, when temperatures are high. When temperatures are lower, there is no cooling burden. The variation in power (blue) is small; essentially the power is constant on these days. Once the temperature (red) exceeds 21 or so, power consumption starts growing roughly in proportion to the amount of exceedance. After day 345, temperatures once again fall below 21 consistently, eliminating the cooling load, and once again power reaches its minimum. This behavior is what squashes the bottom left of the scatterplot (second figure) into a vertically narrow interval.

If we ignore the problem of estimating the kernel $\kappa$ (which amounts to assuming the house is uninsulated), the model we need to fit would be something in the form

$$\int_{t-1}^{t}P(x)dx = \int_{t-1}^{t}\left(\beta_0 + \beta_{l}f(x) + \beta_h(h-T(x))^+ + \beta_c (T(x) - c)^+ + \varepsilon(x)\right)dx.$$

Here, $\beta_0$ is a parameter representing the base load, $\beta_{l}$ is a lighting load coefficient, $f(x)$ is the length of night as a function of the day $x$, $\beta_h$ is a heating load coefficient, $h$ is either known (from a thermostat setting) or is a parameter representing the heating threshold, $\beta_c$ is a cooling load coefficient, and $c$ is either known or is a parameter representing the cooling threshold, with $c \ge t$. The random variation not otherwise accounted for is represented by a stochastic process $\varepsilon(x)$. Over the long term its mean should be zero.

If we further neglect diurnal temperature variation (which is a poor idea when recording power over longer periods, such as weeks or months, but is may be a decent approximation for daily power usage), then we can approximate these integrals by their integrands, getting something like

$$P(t) = \beta_0 + \beta_{l}f(t) + \beta_h(h-T(t))^+ + \beta_c(T(t) - c)^+ + \delta(t).$$

Here, $\delta(t)$ is the daily average of the $\varepsilon(x)$ for $x$ between $t-1$ and $t$ plus some additional small amounts due to this point approximation of the integrals. (Those additional small terms will roughly be proportional to the amount of variation in the integrands--large in the spring and fall, smaller in the summer and winter. That gives useful clues about how to refine this model if it comes to that.) It's likely these random terms are not identically distributed--their variance is probably related to the temporal rate of change of the temperature--and they also are likely to be serially correlated. These possibilities will encourage us to check the model residuals carefully after fitting it and perhaps to use time series techniques in the fitting, but they don't change the basic nature of the deterministic terms that appear.

The night-length function $f(t)$ follows an annual cycle and should either be obtained from data for the location, an astronomical calculation, or can be estimated as a linear combination of sines and cosines; either way, it's nonlinear and will not be accurately estimated as a polynomial in $t$. The most important terms, $(h-T(t))^+$ and $(T(t) - c)^+$, are sharply non-linear functions of the observed temperature $T(t)$. Because of this, it would be a good idea to incorporate them as given (and to investigate what the values of $h$ and $c$ might be) rather than attempt to accommodate them merely by throwing a few extra powers of $T(t)$ into a linear regression. Because of these important nonlinearities, beware of following the computationally expedient approach of modeling $P$ as a polynomial in $T$ and $t$: it is unlikely to work well and even if you throw in enough terms to get a decent fit, you can have little confidence that it will continue to fit well in the future.

We can obtain meaningful information from the parameters: $\beta_c$ is the marginal cost of cooling per additional degree of outside temperature above $c$ and $\beta_h$ is the marginal cost of heating for each lower degree of outside temperature below $h$. If the thresholds $c$ and $h$ are not known, they can be estimated (using maximum likelihood, for instance), and thereby provide extremely useful indicators of how aggressively the building is being cooled or heated. For instance, I found through an analysis of monthly utility bills that my value of $h$ is $15$ degrees Celsius, which is essentially the average thermostat setting across three heating zones in my house, and that the old heating system cost almost $2$ dollars per day per degree Celsius, which is $\beta_h$ converted from units of power into the cost of that power. (This made it easy to decide to replace it with a high-efficiency system.)

This analysis sketches out a path anyone can take for characterizing the relationship between power consumption and temperature in a building. Simply follow it backwards: start with the simplified final model and a crude fitting procedure (like least squares). If it's sufficiently accurate, stop. If not, analyze the residuals to identify which approximations may have introduced the greater errors, then track backwards through this analysis to see how to improve those approximations and make them more accurate. In principle, a dataset covering a sufficiently long period could suffice even to estimate the kernel $\kappa$, which would give interesting and potentially valuable information about the thermal characteristics of the building's insulation.

• Great effort whuber and very useful information,thanks. Actually, the effects of weekends,vacations,people activities have been put in consideration in other analysis. My real data were measured hourly during the year meaning that I have 24*366=8784 points. In the above figures, P and T are the average point for each day.For each day, average point for P and T were calculated and plotted.T is the weather temperature only, it is not the cooling or heating temperature.Also,can we forget about the characteristics of P and T and deal with them as points Y and X and study their figures only, thanks Commented Sep 20, 2011 at 8:23

Have you tried some simple multiple linear regression? For example,

$$P = \beta_0 + \beta_1 T + \beta_2 T^2 + \beta_3 t + \beta_4 t^2 + \beta_5 t \times T$$

where $\beta_i$'s are coefficients to be estimated. In the model I've included squared terms and an interaction to try and capture the non-linear aspects of your model.

• Ok. I applied MLR and I got the following: store2.up-00.com/Sep11/jub39934.jpg So, since the coeff. of the last four terms are very small, can we neglect them and does this mean that P is affected mainly by T not t ? Also, When can we say that the results are good and satisfactory ? Can we add more terms like exp,cos...etc to improve the results to be more accurate. What I understanded from MLR is like 'playing' with terms, i.e adding and removing some terms until we reach to good results.. thanks csgillespie for your help. Commented Sep 19, 2011 at 13:56
• Adding and removing terms should be theory-guided, as you don’t want to discover accidental relations. Commented Sep 19, 2011 at 14:16
• Take a look at the plots: it is clear that simply modeling $P = f(T)$ will not be effective, because the relationship between $P$ and $T$ obviously varies over time. Thus linear regression won't work.
– whuber
Commented Sep 19, 2011 at 14:33
• @whuber: The first figure clearly shows the P and T depend on time. However, predicting P with just T will get you close to your goal, i.e. if you know T, you will be able to predict P fairly well. By having additional terms in the model, I hoped to mope up the remaining variation. Commented Sep 19, 2011 at 15:33
• To make the picture clearer, P is the power consumption, T is the temperature and t is the time in days. all data were measured in a daily basis. Power is mainly affected by temperature which is also has a relation with time i.e summer and winter months and days and we can say that time is temperature and temperature is time. thanks Commented Sep 19, 2011 at 16:18