I'm looking to construct a maximum entropy model using general iterative scaling.
In the process, I've come across the SharpEntropy project for .NET and I'm using that as a reference for the implementation I'm building.
That said, the simple example uses the following observations (all nominal values):
Day Temp. Moisture Time of Day Take Umbrella (outcome)
--- ----- -------- ----------- -----------------------
1 Warm Dry No_Umbrella
2 Cold Dry No_Umbrella
3 Cold Rainy Umbrella
4 Cold Dry Umbrella
5 Warm Dry No_Umbrella
6 Cold Dry Early Umbrella
7 Cold Rainy Early Umbrella
8 Cold Dry Late No_Umbrella
9 Warm Rainy Late No_Umbrella
10 Warm Dry Late No_Umbrella
Looking through the code, it seems that in order to generate the binary functions (predicates) for the maximum entropy model, it's taking the labels across all the attributes, no matter where the value is found.
Meaning, if Warm was in the Moisture column, then that would be tallied when figuring out the entropy associated with the binary function observation of warm.
This seems very incorrect to me. Granted, for the data above, it happens to work because all of the labels are unique across all of the attributes.
Let's add another feature to the set above, the temperature of the prior day:
Day Prior Temp. Temp. Moisture Time of Day Take Umbrella (outcome)
--- ----------- ----- -------- ----------- -----------------------
1 Warm Warm Dry No_Umbrella
2 Cold Cold Dry No_Umbrella
3 Cold Cold Rainy Umbrella
4 Warm Cold Dry Umbrella
5 Warm Warm Dry No_Umbrella
6 Cold Cold Dry Early Umbrella
7 Cold Cold Rainy Early Umbrella
8 Warm Cold Dry Late No_Umbrella
9 Cold Warm Rainy Late No_Umbrella
10 Warm Warm Dry Late No_Umbrella
Now what SharpEntropy seems to do is generate the following binary function:
// I wanted to do this in LaTeX, but my LaTeX-fu failed me.
// This would be, f0, for example.
return (priorTemp == "Warm" || priorTemp == "Warm");
This seems wrong to me, and that you really want two binary functions, one for the prior day's temperature:
// f0
return priorTemp == "Warm";
And then one for the current day:
// f1
return temp == "Warm";
That said, is this the proper way to generate the binary functions, that each attribute on an observation is has it's own universe of values, even though they might semantically mean the same thing when considering a larger universe of values (say, temperature in general versus temperature on a given day)?