What is the best way to get the most accurate results with this small dataset?

Question

This is my first question here, I apologise if this is the wrong place or my formatting is not correct. My experience with machine learning and data science, in general, is a graduate-level survey course I took as an undergrad about a year ago. I decided to familiarise myself by starting to work on little projects. This is my first one, you may get the data (and other referenced code) here.

I downloaded some 1090 photos from Flickr that I might like as my wallpaper. Then I labelled them (with a score between 0-10, 10 being high), so now I have 1093 wallpapers with their respective scores. The final goal is to make a little program that downloads pictures off the internet and decides how much I will like them and if the score is above a certain threshold, it will set the picture as my wallpaper. For machine learning, I standardised them into 75 by 75 pixels. The pictures are RGB.

The file data.npy is 1093×(75×75×3) NumPy matrix (array of arrays), meaning 1093 rows and 16875 columns. Each row is a photo and can be reshaped as (75, 75, 3) into a picture. The label.npy is the parallel array of scores.

This makes every RGB pixel value a feature, so we have 16875 features (inspired by the features on MNSIT dataset). I thought of starting with Logistic Regression by scikit-learn and then Linear. I am using the usual NumPy, scikit-learn. I am getting an accuracy of about 0.5 (50%). I am guessing this is because of having a very small dataset compared to the number of features. I have thought of feature extraction but either I did not do it right or something else, but it did not work well.

UPDATE 0:

So by the feedback, I abandoned vanilla logistic/linear regression and tried to lower the number of features by resizing the file, data_50.npy now has the matrix of shape (1093, (50503)) which makes my image of shape (50,50,3). I tried PCA feature extraction, revised neural networks, and built one on my own with an input, a hidden, and an output layer. Finally, I also implemented the Keras Mobilenet CNN.

UPDATE 1 (Final):

For now due to a lack of data and time, I have postponed this project indefinitely though since this question got some upvotes and discussion, I will not delete it.

Answer 1

You're right that there may be an issue with trying to model so few observations with such a high number of variables. Read the Power and Sample Size section here. Essentially the problem you're running into is that there are so many variables that it's difficult to determine per instance of good or bad picture what is responsible for the good or bad rating, and what is just random noise.

So you've basically got to decide at this point if you want to increase your sample size, decrease your features, or more likely, do both!

One way to decrease features would be further resolution reduction, so instead of 75×75 you could do 25×25, for example, but this might be too little resolution for you to capture the things about images you care about.

Another strategy would be to give up on color and analyze images in black and white, getting rid of that third dimension on your variables.

Even with both of these strategies, you're looking at several hundred or even several thousand variables, so you'd still probably want to sit down and rate more pictures.

Finally, it's possible that your rating of pictures is somewhat arbitrary, in which case there is definitely an upper limit to the accuracy we can get with any of these models looking at the picture alone. For example, we'd need info on your mood, time of day, pictures you looked at earlier, to really precisely know what makes you think a picture is good or bad.

Nonetheless, when you've got an issue of limited data with many features, random forests can help you out! I was able to squeeze out a few more percentage points of accuracy using the following code, and I bet it could be optimized more, like defining a max_depth to stop over-fitting.

import sklearn.ensemble as ske

#RANDOM FORESTS
rfmodel = ske.RandomForestClassifier(n_estimators = 200,  
                                     bootstrap = True,
                                     verbose = True)

#Run model to assess accuracy
rf_modelfit = rfmodel.fit(train_x, train_y)
accuracy = rf_modelfit.score(test_x, test_y)

It's also worth noting that you should compare your results to a naive baseline. In your case, your split makes it so that 45.288% of photos are good and 54.712% of photos are bad. In my tests with your logistic model, I get about 53.57% accuracy, which is worse accuracy than if we just classified every photo as bad, so the logistic model is basically a guess machine in its current state. The random forests model got 58.63% accuracy, so it's a slight improvement over the baseline!

Answer 2

Convolutional Neural Networks (CNN) are the best performing models by far on image data. Use a pretrained model that you train the last layer of, and you might get OK results.

You may need to change the image size to fit one of the pretrained models. 128x128 and 96x96 are common small sizes. You can start with a small model such as MobileNet to see if the approach works. Here are some examples of pretrained networks in Python using Keras: https://keras.io/applications/