How to estimate multiplicative model of spice harvesting?
Why do output coefficients not resemble true coefficients in a linear model?
This is a story about generating values by function, noising it and then trying to estimate the parameters of the function. The background of my story takes place at Dune desert.
I have been observing 80 workers harvesting spice on the Dune desert. I wrote down three characteristics of each worker and the volume of harvest each worker brought to granary throughout the whole season. These three characteristics were Home, Sex and army Rank.
All of a sudden, there was a sandstorm and I was captured by deity of spice. The deity took me to the middle of the desert and revealed to me the absolute knowledge about the load of harvest the worker brings to the granary. Relationship has the following form:
Harvest = Constant * Home * Sex * Rank * Noise
The deity revealed to me the true divine parameters of the phenomenon as well as the nature of Noise. The Noise is a random variable, normally distributed with mean 1, and standard deviation of 5%.
The deity warned me that if the workers are further oppressed by foremen to harvest more spice, it will send sandworms, which will put an end to spice harvesting.
Now the drama is that to stop worker exploitation I have to explain everything to the duke. But the duke does not believe in the existence the deity of spice. The duke thinks that the workers can harvest more output if oppressed harder. The only way to convince the duke is an econometric estimation which he values a lot.
- How can I estimate the parameters of this model?
- And how can I convince the duke that the relation is multiplicative and not additive?
Here is the track of the 80 workers data:
+------+-----------+--------+---------+
| Y | X1_Home | X2_Sex | X3_Rank |
+------+-----------+--------+---------+
| 2.82 | Ordos | M | Veteran |
| 1.02 | Atreides | F | Junior |
| 4.15 | Ordos | M | Elite |
| 2.78 | Horekonen | M | Elite |
| 2.07 | Ordos | F | Junior |
| 3.20 | Ordos | M | Veteran |
| 2.16 | Horekonen | M | Veteran |
| 2.25 | Ordos | F | Junior |
| 2.48 | Ordos | F | Veteran |
| 1.10 | Atreides | F | Veteran |
| 1.21 | Atreides | M | Junior |
| 1.61 | Horekonen | F | Junior |
| 1.07 | Atreides | F | Veteran |
| 1.06 | Atreides | F | Junior |
| 1.74 | Atreides | M | Elite |
| 1.14 | Atreides | F | Veteran |
| 3.41 | Ordos | M | Elite |
| 1.41 | Atreides | M | Veteran |
| 2.59 | Ordos | F | Veteran |
| 1.98 | Horekonen | F | Veteran |
| 2.01 | Horekonen | M | Junior |
| 2.98 | Ordos | M | Veteran |
| 4.18 | Ordos | M | Elite |
| 1.04 | Atreides | F | Veteran |
| 2.77 | Horekonen | M | Elite |
| 1.88 | Horekonen | M | Junior |
| 2.11 | Horekonen | M | Junior |
| 1.47 | Atreides | F | Elite |
| 1.15 | Atreides | M | Junior |
| 1.69 | Atreides | M | Elite |
| 1.47 | Horekonen | F | Junior |
| 2.15 | Horekonen | M | Veteran |
| 1.28 | Atreides | M | Veteran |
| 1.91 | Horekonen | F | Veteran |
| 2.23 | Ordos | F | Junior |
| 2.50 | Horekonen | M | Elite |
| 1.75 | Horekonen | F | Veteran |
| 2.22 | Horekonen | F | Elite |
| 2.88 | Ordos | M | Junior |
| 1.62 | Atreides | M | Elite |
| 1.67 | Horekonen | F | Junior |
| 2.43 | Ordos | F | Veteran |
| 0.92 | Atreides | F | Junior |
| 2.01 | Horekonen | M | Veteran |
| 1.09 | Atreides | F | Veteran |
| 2.12 | Ordos | F | Junior |
| 3.29 | Ordos | M | Veteran |
| 2.17 | Horekonen | M | Veteran |
| 3.17 | Ordos | F | Elite |
| 2.83 | Ordos | M | Junior |
| 1.81 | Atreides | M | Elite |
| 3.20 | Ordos | F | Elite |
| 1.91 | Horekonen | M | Junior |
| 0.92 | Atreides | F | Junior |
| 2.32 | Horekonen | F | Elite |
| 1.60 | Atreides | M | Elite |
| 1.52 | Atreides | F | Elite |
| 2.40 | Horekonen | F | Elite |
| 1.47 | Atreides | F | Elite |
| 1.51 | Horekonen | F | Junior |
| 2.58 | Ordos | M | Junior |
| 1.25 | Atreides | M | Veteran |
| 2.22 | Horekonen | M | Veteran |
| 1.22 | Atreides | M | Junior |
| 1.20 | Atreides | F | Veteran |
| 1.30 | Atreides | M | Veteran |
| 2.50 | Ordos | F | Junior |
| 2.23 | Ordos | F | Junior |
| 3.98 | Ordos | M | Elite |
| 2.26 | Horekonen | F | Elite |
| 3.16 | Ordos | F | Elite |
| 1.25 | Atreides | M | Junior |
| 2.20 | Ordos | F | Junior |
| 3.81 | Ordos | M | Elite |
| 1.24 | Atreides | M | Veteran |
| 1.66 | Horekonen | F | Junior |
| 2.28 | Ordos | F | Junior |
| 2.84 | Ordos | M | Veteran |
| 1.01 | Atreides | F | Junior |
| 1.23 | Atreides | M | Junior |
+------+-----------+--------+---------+
The true divine parameters revealed by deity:
+----------+-----------+-----------+
| variable | value | parameter |
+----------+-----------+-----------+
| constant | constant | 2.0 |
| X1_Home | Ordos | 1.4 |
| X1_Home | Horekonen | 1.0 |
| X1_Home | Atreides | 0.6 |
| X2_Sex | M | 1.1 |
| X2_Sex | F | 0.9 |
| X3_Rank | Elite | 1.3 |
| X3_Rank | Veteran | 1.0 |
| X3_Rank | Junior | 0.9 |
+----------+-----------+-----------+
The day two. After I received council and comments from the elders and sages of the statistician guild. I perused the path to estimate parameters with logarithms. I prepared the following initial matrix:
+-------+---------+-----------+----------+---------+---------+---------+--------+--------+
| | X1_Home | X1_Home | X1_Home | X1_Rank | X1_Rank | X1_Rank | X1_Sex | X1_Sex |
| ln(Y) | Ordos | Horekonen | Atreides | Elite | Veteran | Junior | M | F |
+-------+---------+-----------+----------+---------+---------+---------+--------+--------+
| 1.04 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.02 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.42 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 1.02 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.73 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.16 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.77 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.81 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.91 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.09 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.19 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.48 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.06 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.05 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.55 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.13 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 1.23 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.34 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.95 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.68 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.70 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 1.09 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 1.43 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.04 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 1.02 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.63 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.75 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.38 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.14 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.53 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.39 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.76 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.25 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.65 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.80 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.92 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.56 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.80 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 1.06 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.48 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.51 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.89 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| -0.08 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.70 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.08 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.75 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.19 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.78 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 1.15 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 1.04 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.59 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 1.16 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.65 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| -0.08 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.84 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.47 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.42 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.87 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.38 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.41 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.95 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.23 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.80 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.20 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.18 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 |
| 0.27 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.92 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.80 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.38 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.82 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 1.15 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 |
| 0.22 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
| 0.79 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.34 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 |
| 0.22 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.51 | 1.00 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.82 | 2.72 | 1.00 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 1.04 | 2.72 | 1.00 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 | 1.00 |
| 0.01 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 1.00 | 2.72 |
| 0.21 | 1.00 | 1.00 | 2.72 | 1.00 | 1.00 | 2.72 | 2.72 | 1.00 |
+-------+---------+-----------+----------+---------+---------+---------+--------+--------+
But the parameters calculated from such a matrix are not only worthless but misleading.
+----------+-----------+-----------+---+--------+---+
| constant | value | estimated | | divine | |
+----------+-----------+-----------+---+--------+---+
| constant | constant | 1.00 | | 2.00 | |
| X1_Home | Ordos | 1.46 | | 1.40 | * |
| X1_Home | Horekonen | 1.62 | * | 1.00 | |
| X1_Home | Atreides | 0.94 | | 0.60 | |
| X2_Sex | M | 0.82 | | 1.10 | * |
| X2_Sex | F | 1.00 | * | 0.90 | |
| X3_Rank | Elite | 1.00 | | 1.30 | * |
| X3_Rank | Veteran | 1.16 | * | 1.00 | |
| X3_Rank | Junior | 0.62 | | 0.90 | |
+----------+-----------+-----------+---+--------+---+
The Asterix sign shows the highest parameter within each variable. The highest values of divine parameters and estimated parameters do not match. For example Ordos has highest parameter within X1 variable. But estimated is Harkonen.
Dear elders and sages of the statistician guild, I turn to you for further council. Is the initial matrix crafted according to the art?
The day four
I submit manuscript in a file which reveals the divine process of harvest values creation. You can play with divine parameters and generate your own harvest values. The manuscript also contains my handcrafted procedure which automatically estimates the divine parameters. The procedure regains the parameters with great precision. There is also a sheet with my unsuccessful trial of estimating parameters through logarithms as advised by the elders of statistics guild.
Here are another two manuscripts demonstrating my procedure of regaining parameters with SQL code.