Choosing a model for machine learning problem

Question

Update: See bottom of question for insight into the real application which this small example is attempting to simplify.

---

Questin: In the different machine learning libraries, what kind of model suits the following problem? Here's a simplified version:

The algorithmic connection (which the model will hopefully learn) between X and y is as follows:

If X[i][0] != X[i][2] ----------------> y[i] is 0

If X[i][0] == X[i][2] ----------------> y[i] is 10

If X[i][0] == X[i][1] == X[i][2] -----> y[i] doubles (to 20 in this simplified example)

I've had some success with categorical outputs; in this example there are three output categories "0", "10", or "20".

In that case the model looked something like this:

inputs = keras.Input(shape=(3,), name="input")
x = keras.regularizers.l1_l2(0.01)
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
outputs = layers.Dense(3, activation="sigmoid", name="predictions")(x)
model = keras.Model(inputs=inputs, outputs=outputs)

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=.2)

y_train = keras.utils.to_categorical(y_train, 3)
y_test = keras.utils.to_categorical(y_test, 3)

model.compile(
    optimizer=keras.optimizers.Nadam(),
    loss=keras.losses.BinaryCrossentropy(),
    metrics=[keras.metrics.BinaryAccuracy()],
)

history = model.fit(
    x_train,
    y_train,
    batch_size=16,
    epochs=50)

But if the relationship between X and y is more complex e.g. there are 100 different outputs, would it still be advisable to use categorical output?

---

Update 1: More realistic example to illustrate the actual problem?

Putting together two users user1 (blue), user2 (pink) and a conversation starter question question1 (yellow): Assume the more similar the users are in profile, the higher the satisfaction-score of the conversation (y), and also if the question matches the users the score goes up (doubles in the simple example, y = y*2).

I posted a more realistic version of the problem here on Cross Validated some days ago. I hope it sheds light on the type of problem, and helps identify the right ML-approach given real data from users, questions and their conversations.

The goal of the real-world application is to understand which conversation starter questions fit to which user combinations. Real world conversation satisfaction could be the length of the conversation following a conversation starter question.

So we would like to predict: What conversation starter question should be asked given the two users who are about to talk.

Answer 1

I now have something that works for this simple example. The improvement lies in the preprocessing with preprocessing.StandardScaler().fit(y_rshaped). I was inspired by the preprocessing documentation which says:

Standardization of datasets is a common requirement for many machine learning estimators implemented in scikit-learn; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.

Here's the working code for the simple example in the question:

import random
import numpy as np

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

from sklearn import preprocessing
from sklearn.model_selection import train_test_split
import itertools

X_data = list((itertools.product([0, 1], repeat=3)))*100
random.shuffle(X_data)
print(X_data[0:8])
print("length of X_data = ", len(X_data))

[(1, 0, 0), (0, 0, 0), (1, 0, 0), (0, 0, 1), (0, 1, 0), (1, 1, 1), (0, 0, 0), (0, 1, 0)]

length of X_data = 800

scorelist = []
for tup in X_data:
    score = 0
    if tup[0] == tup[2]:
        score += 10
        if tup[0] == tup[1]:
            score = score*2
    scorelist.append(score)
print("y values are: ", scorelist[0:10])

y values are: [0, 20, 0, 0, 10, 20, 20, 10, 20, 0]

This is the preprocessing part that improved the results:

X = np.array(X_data)
y = np.array(scorelist)
y_reshaped = y.reshape(-1, 1)
scaler = preprocessing.StandardScaler().fit(y_reshaped)
y_transformed = scaler.transform(y_reshaped)

Here the model: It runs/learns in 30 seconds on my slow computer (anno 2007).

inputs = keras.Input(shape=(3,), name="digits")
x = keras.regularizers.l1_l2(0.01)
x = layers.Dense(8, activation="relu", name="dense_1")(inputs)
outputs = layers.Dense(1, activation="linear", name="predictions")(x)
model = keras.Model(inputs=inputs, outputs=outputs)

x_train, x_test, y_train, y_test = train_test_split(X, y_transformed, test_size=.2)

# Reserve 200 samples (out of 800) for validation
x_val = x_train[-200:]
y_val = y_train[-200:]
x_train = x_train[:-200]
y_train = y_train[:-200]

model.compile(
    optimizer=keras.optimizers.Adam(),
    loss=keras.losses.MeanSquaredError()
)
history = model.fit(
    x_train,
    y_train,
    batch_size=8,
    epochs=150
)

Epoch 1/150 [======] - loss: 1.4777

Epoch 2/150 [======] - loss: 1.1794

Epoch 3/150 [======] - loss: 1.0652

...

Epoch 150/150 [======] - loss: 8.6565e-14

Finally we predict X and compare to the real y values:

predictions = model.predict(X)

prediction_list = scaler.inverse_transform(predictions).astype(int).tolist()
prediction_deviation = [[(a[0]-b[0])] for a, b in zip(prediction_list, y_reshaped)]
print("predicted:", prediction_list[0:10])
print("real value:", y_reshaped.tolist()[0:10])
print("deviation:", prediction_deviation[0:10])

predicted: [[0], [20], [0], [0], [9], [19], [20], [9], [20], [0]]

real value: [[0], [20], [0], [0], [10], [20], [20], [10], [20], [0]]

deviation: [[0], [0], [0], [0], [-1], [-1], [0], [-1], [0], [0]]

The deviations are negligible for my purposes as it's clear which predictions are 0, 10 or 20.