1
$\begingroup$

I have the below data. I want to fit a non-decreasing curve for this data.

But, althouhg its grağh seems so regular, I couldn't find a suitable form of function. What kind of function may be suitable for the data.

I will be very glad for any help. Thanks a lot.

1   0,732624193
2   1,200108125
3   1,68811084
4   2,24460301
5   2,639797804
6   3,129340857
7   3,59578095
8   4,075671738
9   4,499706618
10  4,938809576
11  5,357368819
12  5,775935121
13  6,152748142
14  6,542909935
15  6,917884709
16  7,318374759
17  7,706887198
18  8,024342485
19  8,386742193
20  8,759473838
21  9,096697903
22  9,442161132
23  9,776189454
24  10,04685275
25  10,44737472
26  10,73286187
27  11,07501659
28  11,34340416
29  11,61706181
30  11,91219663
31  12,19997992
32  12,46160562
33  12,64493511
34  12,87722319
35  13,09525623
36  13,33787142
37  13,57410201
38  13,78410808
39  13,96788738
40  14,19974498
41  14,40126277
42  14,63074118
43  14,91062441
44  15,20221408
45  15,42637581
46  15,63486103
47  15,82855433
48  15,97395587
49  16,16364445
50  16,34061186
51  16,46713061
52  16,60904203
53  16,77117154
54  16,93843089
55  17,15597228
56  17,3065981
57  17,4533185
58  17,57796846
59  17,72241579
60  17,86958431
61  18,08499505
62  18,22227803
63  18,34658856
64  18,4970054
65  18,62637386
66  18,75224401
67  18,82904471
68  18,92908889
69  19,01882395
70  19,12175476
71  19,21370787
72  19,29834989
73  19,36350613
74  19,48827906
75  19,54979572
76  19,63089052
77  19,72275485
78  19,77992057
79  19,85039546
80  19,93103915
81  20,03867987
82  20,13624479
83  20,16631002
84  20,27487258
85  20,36274989
86  20,41147668
87  20,50257231
88  20,51992914
89  20,55399805
90  20,59546524
91  20,6024585
92  20,65952735
93  20,72986288
94  20,76867903
95  20,85908945
96  20,94532696
97  20,96286849
98  21,00325095
99  21,06156886
100 21,08856307
101 21,12465402
102 21,17790176
103 21,17790176
104 21,22095988
105 21,24269251
106 21,28008352
107 21,31208712
108 21,33563444
109 21,3535785
110 21,37156164
111 21,38304028
112 21,39212703
113 21,42192738
114 21,42192738
115 21,43789942
116 21,47142078
117 21,51139773
118 21,53466036
119 21,54447502
120 21,5572884
121 21,5572884
122 21,59681046
123 21,6070952
124 21,6070952
125 21,6070952
126 21,6507205
127 21,66113255
128 21,6718944
129 21,6718944
130 21,6718944
131 21,6718944
132 21,72442301
133 21,74277312
134 21,74277312
135 21,74277312
136 21,74277312
137 21,74277312
138 21,74277312
139 21,75454642
140 21,78481196
141 21,80021715
142 21,80021715
143 21,80021715
144 21,82023616
145 21,82023616
146 21,84075406
147 21,85645695
148 21,8723175
149 21,88529702
150 21,88529702
151 21,93374357
152 21,94666325
153 21,94666325
154 21,94666325
155 21,96344402
156 21,96344402
157 21,96344402
158 21,96344402
159 21,96344402
160 21,96344402
161 21,96344402
162 21,96344402
163 21,96344402
164 21,96344402
165 21,98154907
166 21,9994693
167 21,9994693
168 21,9994693
169 21,9994693

Where its graph is like that

enter image description here

$\endgroup$
2
$\begingroup$

My equation search found that an inverse Harris yield density model:

y = x / (a + b*pow(x, c))

with fitted parameters of

a = 1.9953497744180733E+00

b = 4.6314864228655921E-03

c = 1.3877857961620943E+00

yields R-squared = 0.9997 and RMSE = 0.0988

enter image description here

$\endgroup$
1
$\begingroup$

I've worked with similar curves some time ago. Fitting $y=a(1-e^{-bx})$ and $y=a\ln x +b$ separately and averaging them yielded very good fits in my case. Since the curves cannot be cast to linear model, you need to use some sort of nonlinear curve fitting tool. curve_fit in scipy.optimize can be useful if you're using Python. Instead of trying to fit a complex model, averaging two/or more basic models provides good convergence rates. Since these curve-fitting libraries apply some sort of gradient-descent approach, local minima might occasionally bother you. In such cases, resorting to the differenced version of the above curve (i.e. $x_n-x_{n-1}$), fitting exponential decay, or inverse-polynomial decay (or again averaging them) and applying cumsum in the end might also be a good approach (only if you can't fit with nonlinear fitting tool).

EDIT: Applying the methodology to this data: (first save the data to a text file having a whitespace between the columns as separator):

from scipy.optimize import curve_fit
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

tmp = pd.read_csv("<PATH>", sep = " ", header = None)
tmp.columns = ["ind", "data"]

def f1(x,a,b): return a*(1-np.exp(-b*x))
def f2(x,a,b): return a*np.log(x) + b

x, y = tmp.ind.values, tmp.data.values

popt1,_ = curve_fit(f1, x, y, p0 = np.random.random(2))
popt2,_ = curve_fit(f2, x, y, p0 = np.random.random(2))

f1v = f1(x,*popt1)
f2v = f2(x,*popt2)

fig, ax = plt.subplots(2,2,figsize=(18,9))
ax[0][0].plot(x,y); ax[0][0].legend(["real"]); ax[0][0].grid()
ax[0][1].plot(x,f1v,x,y); ax[0][1].legend(["exp","real"]); ax[0][1].grid()
ax[1][0].plot(x,f2v,x,y); ax[1][0].legend(["log","real"]); ax[1][0].grid()
ax[1][1].plot(x,(f1v+f2v)/2,x,y); ax[1][1].legend(["avg","real"]); ax[1][1].grid()
plt.show()

Output

Here, $y=a(1-e^{-bx})$ fit best to the data. In my cases, the right end of the curve was having a small trend (instead of complete saturation as in here) and fitting a logarithmic curve and averaging was a useful step. Logarithmic curve doesn't fit well to the initial sections of the data, however this model has been a good predictor for me for estimating the LTV of customers.

By the way, the best parameters for the exponential curve is ($a$ and $b$ in order):

array([22.58256696,  0.0256301 ])
$\endgroup$
  • $\begingroup$ I find your answer very interesting, as I had not at all considered this approach. Based on your experience on similar curves, would you please consider also fitting the data using this approach and adding your results as an edit to your answer here for comparison to my answer? $\endgroup$ – James Phillips Mar 20 at 0:56
  • $\begingroup$ Sure, I've added the complete code! @JamesPhillips $\endgroup$ – gunes Mar 20 at 8:35
  • $\begingroup$ Please forgive the somewhat unprofessional nature of my reply, but that is some totally awesome curve fitting! The results are impressive, and your description of the technique is quite clearly stated. $\endgroup$ – James Phillips Mar 20 at 11:49
  • 1
    $\begingroup$ Here is the NIST page for the equation you mentioned: itl.nist.gov/div898/strd/nls/data/boxbod.shtml $\endgroup$ – James Phillips Mar 20 at 12:53
  • $\begingroup$ I remember searching these models and couldn't find anywhere back at the time. You at least demystified one for me.Thanks for nice reference! $\endgroup$ – gunes Mar 20 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.