Update: See bottom of question for insight into the real application which this small example is attempting to simplify.

Questin: In the different machine learning libraries, what kind of model suits the following problem? Here's a simplified version:

The algorithmic connection (which the model will hopefully learn) between X and y is as follows:

If X[i][0] != X[i][2] ----------------> y[i] is 0

If X[i][0] == X[i][2] ----------------> y[i] is 10

If X[i][0] == X[i][1] == X[i][2] -----> y[i] doubles (to 20 in this simplified example)

X[0] = [0,0,0] -> y[0] = 20
X[1] = [1,0,1] -> y[0] = 10
X[2] = [1,1,0] -> y[0] = 0

I've had some success with categorical outputs; in this example there are three output categories "0", "10", or "20".

In that case the model looked something like this:

inputs = keras.Input(shape=(3,), name="input")
x = keras.regularizers.l1_l2(0.01)
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
outputs = layers.Dense(3, activation="sigmoid", name="predictions")(x)
model = keras.Model(inputs=inputs, outputs=outputs)

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=.2)

y_train = keras.utils.to_categorical(y_train, 3)
y_test = keras.utils.to_categorical(y_test, 3)


history = model.fit(

But if the relationship between X and y is more complex e.g. there are 100 different outputs, would it still be advisable to use categorical output?

Update 1: More realistic example to illustrate the actual problem?

Putting together two users user1 (blue), user2 (pink) and a conversation starter question question1 (yellow): Assume the more similar the users are in profile, the higher the satisfaction-score of the conversation (y), and also if the question matches the users the score goes up (doubles in the simple example, y = y*2).

I posted a more realistic version of the problem here on Cross Validated some days ago. I hope it sheds light on the type of problem, and helps identify the right ML-approach given real data from users, questions and their conversations.

The goal of the real-world application is to understand which conversation starter questions fit to which user combinations. Real world conversation satisfaction could be the length of the conversation following a conversation starter question.

So we would like to predict: What conversation starter question should be asked given the two users who are about to talk.

  • 1
    $\begingroup$ Since this problem does not need any machine learning at all, maybe you could think of more realistic example to illustrate the actual problem? Otherwise the answers might diverge pretty far from what you'd expect. $\endgroup$
    – Tim
    Commented Feb 28, 2022 at 14:11
  • $\begingroup$ Thank you, Tim. That's a good point. I will update the question with a part giving insight to the problem. I posted the real problem here on Cross Validated some days ago, but I think it was too complex. This new question is an attempt to simplify the problem. $\endgroup$
    – thenarfer
    Commented Feb 28, 2022 at 14:59
  • $\begingroup$ I read that Cross Validated has many unanswered questions. Please don't hesitate to provide a helpful answers even though it might not be the full answer to how to approach this problem. Thanks everyone! $\endgroup$
    – thenarfer
    Commented Feb 28, 2022 at 22:28
  • $\begingroup$ @Tim , perhaps you would have a look and give an opinion on the usage of preprocessing.StandardScaler() from scikitlearn on this problem? I'm quite new to Machine Learning and perhaps I stumbled on a clumsy way of achieving results? $\endgroup$
    – thenarfer
    Commented Mar 2, 2022 at 0:37

1 Answer 1


I now have something that works for this simple example. The improvement lies in the preprocessing with preprocessing.StandardScaler().fit(y_rshaped). I was inspired by the preprocessing documentation which says:

Standardization of datasets is a common requirement for many machine learning estimators implemented in scikit-learn; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.

Here's the working code for the simple example in the question:

import random
import numpy as np

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

from sklearn import preprocessing
from sklearn.model_selection import train_test_split
import itertools
X_data = list((itertools.product([0, 1], repeat=3)))*100
print("length of X_data = ", len(X_data))

[(1, 0, 0), (0, 0, 0), (1, 0, 0), (0, 0, 1), (0, 1, 0), (1, 1, 1), (0, 0, 0), (0, 1, 0)]

length of X_data = 800

scorelist = []
for tup in X_data:
    score = 0
    if tup[0] == tup[2]:
        score += 10
        if tup[0] == tup[1]:
            score = score*2
print("y values are: ", scorelist[0:10])

y values are: [0, 20, 0, 0, 10, 20, 20, 10, 20, 0]

This is the preprocessing part that improved the results:

X = np.array(X_data)
y = np.array(scorelist)
y_reshaped = y.reshape(-1, 1)
scaler = preprocessing.StandardScaler().fit(y_reshaped)
y_transformed = scaler.transform(y_reshaped)

Here the model: It runs/learns in 30 seconds on my slow computer (anno 2007).

inputs = keras.Input(shape=(3,), name="digits")
x = keras.regularizers.l1_l2(0.01)
x = layers.Dense(8, activation="relu", name="dense_1")(inputs)
outputs = layers.Dense(1, activation="linear", name="predictions")(x)
model = keras.Model(inputs=inputs, outputs=outputs)

x_train, x_test, y_train, y_test = train_test_split(X, y_transformed, test_size=.2)

# Reserve 200 samples (out of 800) for validation
x_val = x_train[-200:]
y_val = y_train[-200:]
x_train = x_train[:-200]
y_train = y_train[:-200]

history = model.fit(

Epoch 1/150 [======] - loss: 1.4777

Epoch 2/150 [======] - loss: 1.1794

Epoch 3/150 [======] - loss: 1.0652


Epoch 150/150 [======] - loss: 8.6565e-14

Finally we predict X and compare to the real y values:

predictions = model.predict(X)

prediction_list = scaler.inverse_transform(predictions).astype(int).tolist()
prediction_deviation = [[(a[0]-b[0])] for a, b in zip(prediction_list, y_reshaped)]
print("predicted:", prediction_list[0:10])
print("real value:", y_reshaped.tolist()[0:10])
print("deviation:", prediction_deviation[0:10])

predicted: [[0], [20], [0], [0], [9], [19], [20], [9], [20], [0]]

real value: [[0], [20], [0], [0], [10], [20], [20], [10], [20], [0]]

deviation: [[0], [0], [0], [0], [-1], [-1], [0], [-1], [0], [0]]

The deviations are negligible for my purposes as it's clear which predictions are 0, 10 or 20.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.