I've been trying to teach myself machine learning using some books, kaggle and other resources so I'm still very new at this. I'm trying to do a competition on kaggle and I've obtained a dataset.

I'm a little confused about what exactly needs to be done with this data set next though. My assumption is that if I choose to use a univariate distribution to model this data, I need to pick the features(columns in my csv file) I want to use, somehow combine them into a single variable xi and pass each pair of values of and labels (xi,yi) into my model, adjusting the model's parameters to fit the data.

I'm assuming that if I chose a bivariate distribution to model this data, I would do something similar, except I would convert the features of my choice into two separate values x1,x2 and pass tuples (xi1,xi2,yi) into my model and adjust parameters to fit the data.

for an n-variate distribution I would do... the same thing and pass in a vector (xi1, ..., xin, yi) into my model.

Is this pattern of thinking correct? I know I'm not mentioning anything about model or feature selection but that's mainly because I feel like I need to understand this step between training data --> model before I can even try to implement a likelihood estimation algorithm or pick the features I want to use.

Also please correct any terminology I've misused as I still have no idea what I'm talking about.


1 Answer 1


First of all, probably a (competitive) Kaggle dataset is not the best dataset to start with. There are many university courses and textbooks online that have exercises specially designed for beginners.

I think you're confusing modelling a distribution (simplified: does this histogram look like a normal distribution or a uniform distribution) and modelling the relationship between the outcome $y$ and the predictors $x$ (given $x$, what is our estimate of $y$).

You have a bunch of training data, for which you know the mapping between $x$ (which might be a vector / tuple with $n$ elements) and $y$ (a scalar). Your task is to learn the relationship between $x$ and $y$, so that you can predict the $y$ value for other values of $x$. The first thing to consider is: Is your $y$ qantitative (can it have arbitrary numerical values, at least within some range) or is it categorical (only certain values, maybe not even numerical, i.e. 'yes'/'no', 'red'/'green'/'black')? For the first scenario, you want to perform a regression, for the latter a classification.

For your first problems, you don't have to perform feature selection or feature engineering. You also don't have to try to select the best kind of model -- just pick whatever you want to learn (or what is suggested by the exercise).

In this case, the step from training data to a model is quite easy: Your model will simply have as many inputs as your problem has predictors: For linear regression you use a model with $n+1$ parameters (one for each input plus the offset); for a neural network you select $n$ input nodes; $k$-nearest neighors will work in $n$-dimensional space. You train the model with the training data according to the training equations/algorithm (either you do some calculations by hand, or you use a compute library for it or you write your own program) and can then use the trained model to predict new instances (again: either calculate by hand or use a library or write some code).

Where to go from there? You should learn to do your modelling using a computer program (e.g. R, Weka, Numpy); try different types of models (linear regression, neural network, regression trees, SVM); learn metrics to compare different models (mean squared error; accuracy) and how to properly do this (training/test sets, cross validation, bootstrap); learn about overfitting (and bias and variance) and ways to prevent it (regularization); learn how to deal with the curse of dimensionality (normalization, feature engineering, feature selection). And tackle real-world problems.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.