1
$\begingroup$

Let's say we have a dataset with these columns:

A | B | C | D | E | F | G

I want to predict [E,F,G] based on [A,B] with following rule: top 5 entries order by (SUM(D)/SUM(C)) desc

Pseudo-SQL query representation for above rule for a A-B combination:

SELECT 
E, F, G, (SUM(D)/SUM(C)) AS ctr
FROM dataset
WHERE A=... AND B=...
GROUP BY E, F, G
ORDER BY ctr DESC
LIMIT 5

I managed to partially solve problem using Decision Tree Classifier by pre-calculating results through a new dataset that becomes:

A | B | E | F | G

But DecisionTreeClassifier in Python can only predict ONE random E-F-G combination out of 5 possible variants in new dataset for a A-B pair:

left = df.drop(columns=["E", "F", "G"])
right = df.drop(columns=["A","B"])

model = DecisionTreeClassifier(random_state=1)
model.fit(left, right)
# no point splitting into train & test sets because model has pre-calculated results
predictions = model.predict([ [3,1]])
print(predictions)

What should be the right path to get the 5 variants for A-B pair I save in dataset (without LabelEncoder)?

Also periodically we add new entries to dataset and the more this "database" grows the more feature engineering will become more costly.

UPDATE

Issue solved by feature engineering and ranking:

A | B || C | D || E | F | G

Was converted to:

A | B | RANK || E | F | G

So DecisionTreeClassifier is able to get 5 suggestions expected:

trainedModel.predict([ [9,21,1], [9,21,2], [9,21,3], [9,21,4], [9,21,5] ]))

Steps to solve similar cases:

  1. (using pandas) create pandas dataframe from csv and groupby A-B-E-F-G then sum C,D
  2. (programmatically) loop through resulting dataframe, apply D/C sums, sort and generate ranking in dictionary applying limit 5 condition, making other cases empty
  3. (using pandas) import dictionary into a new dataframe, train and test model
Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ "But this is no longer true machine learning" What you've done is called feature engineering. It's standard practice and is often the first step to achieving better results. See "A Few Useful Things to Know about Machine Learning" by Pedro Domingos, specifically section 8 "Feature engineering is the key." $\endgroup$
    – Sycorax ♦
    Commented Feb 28, 2022 at 14:19
  • $\begingroup$ Your question is "Are statistics based predictions possible?" but between the confused description of what "machine learning" is and the lack of specificity of what a "statistics-based" prediction is, or how a "statistics-based" prediction is different from any other machine learning prediction, it's hard to know what you mean. Decision trees are perfectly valid machine learning models & feature engineering is the way to get there. $\endgroup$
    – Sycorax ♦
    Commented Feb 28, 2022 at 14:23
  • $\begingroup$ You'll need to explain what you want to do & where you're stuck, how a statistic-based prediction is different from any other prediction, and what the various features are supposed to mean, and what the purpose of sorting them is. Typically, sorting the data are irrelevant. But are you perhaps trying to rank the data in the table by some priority? That might be a ranking task. $\endgroup$
    – Sycorax ♦
    Commented Feb 28, 2022 at 14:25
  • $\begingroup$ Do you truly need predictions for E, F, G, or do you just need the ranking? Or do you need both ranking and predictions for E, F, G? The way the question is written, it seems like maybe you only need to predict the sorting induced by SUM(D)/SUM(C) efficiently. $\endgroup$
    – Sycorax ♦
    Commented Feb 28, 2022 at 15:02
  • $\begingroup$ I only need the ranking: output of predictions should be 5 E-F-G sorted by ranking! That's why I originally asked if this is a proper machine learning application: because system works exactly like a database (cannot generate its own predictions for A-B pairs it wasn't taught already). $\endgroup$
    – Lucian Gabriel Popescu
    Commented Feb 28, 2022 at 15:56

0

Reset to default