# What is the best way to get the most accurate results with this small dataset?

This is my first question here, I apologize if this is the wrong place or my formatting is not correct. My experience with machine learning and data science, in general, is a graduate-level survey course I took as an undergrad about a year ago. I decided to refamiliarize myself by starting to work on little projects. This is my first one, you may get the data here.

I downloaded some 1090 photos from Flickr that I might like as my wallpaper. Then I labeled them (with a score between 0-10, 10 being high), so now I have 1093 wallpapers with their respective scores. The final goal is to make a little program that downloads pictures off the internet and decides how much I will like them and if the score is above a certain threshold, it will set the picture as my wallpaper. For machine learning, I standardized them into 75 by 75 pixels. The pictures are RGB.

The file data.npy is 1093*(75*75*3) numpy matrix(array of arrays), meaning 1093 rows and 16875 columns. Each row is a photo and can be reshaped as (75, 75, 3) into a picture. The label.npy is the parallel array of scores.

This makes every RGB pixel value a feature so we have 16875 features (inspired by the features on MNSIT dataset). I thought of starting with Logistic Regression by sklearn and then Linear. I am using the usual numpy, sklearn. I am getting an accuracy of about 0.5 (50%). I am guessing this is because of having a very small dataset compared to the number of features. I have thought of feature extraction but either I did not do it right or something else but it did not work well.

UPDATE 0:

So by the feedback, I abandoned vanilla logistic/linear regression and tried to lower the number of features by resizing the file, data_50.npy now has the matrix of shape (1093, (50*50*3)) which makes my image of shape (50,50,3). I tried PCA feature extraction, revised neural networks, and built one on my own with an input, a hidden, and an output layer. Finally, I also implemented the Keras Mobilenet CNN. I placed the code for all of these in the same link with the data.

UPDATE 1:

As suggested, I added an output layer for classification into two classes and froze all other layers. I am also using ImageNet weights. I tried to follow the "Fine-tune InceptionV3 on a new set of classes" section at https://keras.io/applications. I am not sure if I set everything up right but here is what I have,

# !/usr/bin/env python3

from keras.applications.mobilenet import MobileNet
from keras.layers import Dense
from keras.applications.mobilenet import preprocess_input
from keras.models import Model
from keras.optimizers import SGD
import numpy as np

cut = 6
split_ratio = 0.7
resolution = 224

# getting data

# preparing data
matrix = preprocess_input(matrix)
N = matrix.shape[0]
label = label > cut
indicies = np.arange(N)
np.random.shuffle(indicies)

# testing and training split
train_x = matrix[indicies][:int(split_ratio * N)]
train_x = train_x.reshape((-1, resolution, resolution, 3))
train_y = label[indicies][:int(split_ratio * N)]
train_y = np.array([train_y, -(train_y - 1)]).T  # one hoting
test_x = matrix[indicies][int(split_ratio * N):]
test_x = test_x.reshape((-1, resolution, resolution, 3))
test_y = label[indicies][int(split_ratio * N):]
test_y = np.array([test_y, -(test_y - 1)]).T  # one hoting

base_model = MobileNet(weights='imagenet')

x = base_model.output
# Add logistic layer for 2 output classes
predictions = Dense(2, activation='softmax')(x)

# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)

# for i, layer in enumerate(model.layers):
#     print(i, layer.name)

for layer in model.layers[:len(model.layers) - 1]:
layer.trainable = False
model.layers[len(model.layers) - 1].trainable = True

# we need to compile the model for these modifications to take effect
# we use SGD with a low learning rate
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9),
loss='categorical_crossentropy',
metrics=['accuracy'])

model.fit(train_x, train_y)

score = model.evaluate(test_x, test_y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])


The accuracy stays at base-line.

I'd really appreciate it if someone had a look and I apologize since this has gotten a bit lengthy.

• I wouldn't recommend any generalised linear model for image classification. As pointed out by @jonnor, convolutional neural networks are king in this topic, but even if you preferred a more basic classifier, non-linear ones like KNN, SVM or regular neural networks will do better than logistic regresion. My advice would be that you try different types of model, compare the results, and stay with the simplest one among those that do OK. The two answers give you pretty good advice about most of the stuff that is not in this comment. – David Jun 2 '19 at 8:58
• You are training the MobileNet model from scratch. That will be incredibly data hungry. Instead specify the ImageNet weights, set all layers instead of the last one as trainable=False. You should also use the MobileNet preprocessing function to make sure your inputs are properly scaled – jonnor Jun 4 '19 at 8:12
• Okay, I think I did that or tried to? No gains unfortunately =(. – scribe Jun 6 '19 at 4:30

You're right that there may be an issue with trying to model so few observations with such a high number of variables. Read the Power and Sample Size section here. Essentially the problem you're running into is that there are so many variables that it's difficult to determine per instance of good or bad picture what is responsible for the good or bad rating, and what is just random noise.

So you've basically got to decide at this point if you want to increase your sample size, decrease your features, or more likely, do both!

One way to decrease features would be further resolution reduction, so instead of 75X75 you could do 25X25, for example, but this might be too little resolution for you to capture the things about images you care about.

Another strategy would be to give up on color and analyze images in black and white, getting rid of that third dimension on your variables.

Even with both of these strategies, you're looking at several hundred or even several thousand variables, so you'd still probably want to sit down and rate more pictures.

Finally, it's possible that your rating of pictures is somewhat arbitrary, in which case there is definitely an upper limit to the accuracy we can get with any of these models looking at the picture alone. For example, we'd need info on your mood, time of day, pictures you looked at earlier, to really precisely know what makes you think a picture is good or bad.

Nonetheless, when you've got an issue of limited data with many features, random forests can help you out! I was able to squeeze out a few more percentage points of accuracy using the following code, and I bet it could be optimized more, like defining a max_depth to stop over-fitting.

import sklearn.ensemble as ske

#RANDOM FORESTS
rfmodel = ske.RandomForestClassifier(n_estimators = 200,
bootstrap = True,
verbose = True)

#Run model to assess accuracy
rf_modelfit = rfmodel.fit(train_x, train_y)
accuracy = rf_modelfit.score(test_x, test_y)


It's also worth noting that you should compare your results to a naive baseline. In your case, your split makes it so that 45.288% of photos are good and 54.712% of photos are bad. In my tests with your logistic model, I get about 53.57% accuracy, which is worse accuracy than if we just classified every photo as bad, so the logistic model is basically a guess machine in its current state. The random forests model got 58.63% accuracy, so it's a slight improvement over the baseline!

• I understand that in binary classification 50% is random. A teacher of mine recommended PCA feature extraction. Do you think that is a good way to reduce dimensionality than just giving up on colour? In regards to an upper bound on accuracy, do you think 70% is a good goal? or what do you think that upper should be? – scribe Jun 2 '19 at 23:30
• 50% isn't a universal random in binary classification, it depends on the distribution of 0s and 1s. If you have 90% 1 and 10% 0, then random guessing would yield about 82% accuracy (0.9*0.9 + 0.1*0.1), but classifying everything as 1 would yield 90% accuracy, so a good naive baseline is to classify everything as your largest class. You could try PCA! At this point, like David said, you should try different things and see what works best, I just think that given your particular situation of low data with tons of variables, random forests would probably do relatively well. – yonderkens Jun 3 '19 at 10:02
• The upper bound is probably unobservable, since it depends on factors that we can't control for, so to get an idea for your upper bound you'd have to run more models and configurations of models, and if they all can't get above a certain level of accuracy, then you could argue that that level of accuracy is your upper bound. – yonderkens Jun 3 '19 at 10:06
• The ultimate solution though, as in the one that would have the biggest effect on accuracy, would be to increase sample size and use a neural network, like the other folks mentioned. – yonderkens Jun 3 '19 at 10:08
• I appreciate your efforts to help me! I tried the CNN recommended by @jonnor but it did not quite work well. Do you mind taking a look? I also tried other things, if you would like to have a look at them they are in the links above. – scribe Jun 4 '19 at 3:13

Convolutional Neural Networks (CNN) are the best performing models by far on image data. Use a pretrained model that you train the last layer of, and you might get OK results.

You may need to change the image size to fit one of the pretrained models. 128x128 and 96x96 are common small sizes. You can start with a small model such as MobileNet to see if the approach works. Here are some examples of pretrained networks in Python using Keras: https://keras.io/applications/

• I tried MobileNet, it did not improve my accuracy much, will you have a look? – scribe Jun 4 '19 at 3:11