First the question: I'm new to machine learning and have made a first attempt at predicting scores (see problem below). I'm now looking for the right way to set up the model (layers, activations, accuracy metrics, etc.) and to learn how to approach this problem better. I'm grateful for your insights and recommendation.
I've constructed a simulated dataset of [conversations]. Every conversation has [user1], [user2], [question]. The idea is that user1 and user2 have a conversation, and a conversation starter question is given for them to talk about.
A user is constructed from [age, gender, hobby]
A question is constructed from [question_age_type, question_hobby_type, 0]
The conversations in the dataset are scored by a function that gives higher points the more similar the users are and the more they are similar to the question profile.
Given a conversation simulation dataset with the following structure:
| "age1" | "gender1" | "hobby1" | "age2" | "gender2" | "hobby2" | "question_age" | "question_hobby" | "0" | score |
|---|---|---|---|---|---|---|---|---|---|
| 4 | 1 | 1 | 2 | 1 | 1 | 1 | 0 | 0 | 20 |
| 2 | 0 | 1 | 2 | 1 | 0 | 2 | 0 | 0 | 50 |
The data is split with X containing the conversations, and y containing the scores. Here's a sample:
print(X[0:3])
print("X.shape is: ", X.shape)
----------
output:
[[[4 1 1]
[2 1 1]
[1 0 0]]
[[2 0 1]
[2 1 0]
[2 0 0]]
[[1 1 0]
[3 1 2]
[2 3 0]]]
X.shape is: (3000, 3, 3)
The shape of X represents (conversations, 2users&1question, [age, gender, hobby]/[question_age_type, question_hobby_type, 0])
print(y[0:3])
print("y.shape is: ", y.shape)
----------
output:
[20 50 10]
y.shape is: (3000,)
I hope this explains the data that I wish to learn from. I hope to predict the score of two new users talking to each other and so pick the right question that gives the conversation the highest score. With the trained model I can create a dataset which has the highest scoring question(s) given all user combinations. I could stop here, but let me give some insight to my model set-up so far. It's not working too well and I'm sure I've set it up completely wrong given the structure/type of the problem:
inputs = keras.Input(shape=(9,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(1, activation="relu", name="predictions")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=.2)
x_train = x_train.reshape(int(len(conversations)*0.8), 9).astype("float32")
x_test = x_test.reshape(int(len(conversations)*0.2), 9).astype("float32")
y_train = y_train.astype("float32")
y_test = y_test.astype("float32")
# Reserve 300 samples for validation
x_val = x_train[-300:]
y_val = y_train[-300:]
x_train = x_train[:-300]
y_train = y_train[:-300]
model.compile(
optimizer=keras.optimizers.RMSprop(), # Optimizer
loss=keras.losses.MeanSquaredError(reduction="auto", name="mean_squared_error"),
metrics=[keras.metrics.CategoricalAccuracy(name="categorical_accuracy", dtype=None)],
)
history = model.fit(
x_train,
y_train,
batch_size=32,
epochs=200,
validation_data=(x_val, y_val),
)
print("Evaluate on test data")
results = model.evaluate(x_test, y_test, batch_size=64)
print("test loss, test acc:", results)
print("Generate predictions for 3 samples")
predictions = model.predict(x_test)
print("predictions shape:", predictions.shape)
