I have the following four graphs for which I have to compute the slope of the straight portion of the graph (it returns the value of Young's Modulus of a material). I initially received a 4 datasets with a lot of noise but I reduced the noises by fitting an equivalent 13-degree polynomial curve using np.poylfit.

Next step is to calculate the slope of the straight portion of the curve (highlighted with red rectangular boxes). I tried using $\chi$ squared test to fit a linear line but it is very slow and to make it fast I have to specify the program to calculate only until 1000 points (approximate value by which the linear portion ends).

Can anyone suggest me a more generic and faster way to find the slopes of the highlighted segments?

graph_sample1 graph_sample2 graph_sample3 graph_sample4

  • 3
    $\begingroup$ If you know the portion that you want to fit the line to, you can simply run an OLS of Stress on Strain. Why do you want to use $\chi^2$ in any way? $\endgroup$ – Stephan Kolassa Feb 11 at 8:28
  • $\begingroup$ There can be many graphs so I cannot analyze each and every graph and assume the straight portion of the graph and run the test. Even for OLS, I think I will need to specify what portion of the curve I need to fit right? $\endgroup$ – Aman Sharma Feb 11 at 8:34
  • $\begingroup$ You do not want to use a 13th degree polynomial under any circumstances! It sounds like you might be trying to estimate the limiting slope as the strain drops to zero. Would that be the case? $\endgroup$ – whuber Feb 11 at 14:03
  • $\begingroup$ @whuberNo, I didn't want to find the limiting slope. The average slope over the linear portion is sufficient for my calculations. And you're right I shouldn't use a 13-degree polynomial. It was deviated by a lot from my actual data so I used convolution to get a smooth curve. $\endgroup$ – Aman Sharma Feb 11 at 14:30
  • $\begingroup$ That's interesting, because it departs from the usual physicist's definition of Young's Modulus. You are, in effect, looking for an average over an interval of strains. Unfortunately that interval is not (yet) well-defined: we now need some physical criterion for determining the upper endpoint of that interval. Sure, statistical procedures can detect when that endpoint might have been reached, but they cannot a priori characterize it in a physically meaningful way. Could you clarify your definition? $\endgroup$ – whuber Feb 11 at 19:16

Based on your comments, it sounds like your problem is how to determine the initial segment of your data that we can "reasonably" fit a straight line to. I would suggest a simple local search strategy.

Start with a proportion $p_0=1$. For each iteration $i$, fit a straight line to the initial $p_i$ proportion of your data. Assess which proportion $q_i$ of your observations lies above the fitted line. If $q_i$ is "too large", set $p_{i+1}$ smaller than $p_i$. If $q_i$ is "too small", set $p_{i+1}$ larger than $p_i$. Iterate.

The regression lines should be very fast to fit. You might want to suppress the intercept, i.e., force the line to go through the origin. In that case, you only have to estimate the slope, and this is easily done by

$$ \beta = \frac{\sum_{j=1}^{p_iN} x_jy_j}{\sum_{j=1}^{p_iN} x_j^2}, $$

assuming that your $N$ data points are listed in increasing order of $x_j$. (You could even use an "online" method for updating the slope estimate, but that may be more complicated than the saving in runtime would warrant.)

For updating $p_i$, you could use some variant of bisection search.

Finally, to determine whether $q_i$ is "good", you should experiment a little. Of course, we are aiming for $q=\frac{1}{2}$ if your initial data are symmetrically distributed around the straight line. However, they may not be - it's hard to figure this out based on the full plots you are giving. So in the case of asymmetric distributions in this initial segment, you may want to aim for some other value of $q$. You will need to experiment a little.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.