How to do feature transformation on data, without knowing what the data mean? What are the usual transformations that are done? Is there a proper method to do this in a structured way?

  • $\begingroup$ It might help to (1) define "feature transformation" - sometimes it seems to be used narrowly to refer to principal component analysis & related data reduction techniques, & sometimes to mean feature engineering/creation/construction; (2) explain what kind of situations you're imagining - what about the data can be taken for granted?; & perhaps (3) suggest what kind of analysis the features will be used for. $\endgroup$ Oct 26, 2015 at 14:27
  • $\begingroup$ i am talking about in general so it includes what you have mentioned. And as I said you don't know what the data is about and the analysis of the features is used for regression in order to predict y for a new x. $\endgroup$ Oct 26, 2015 at 14:32
  • $\begingroup$ So, to make it concrete: you've been given a table with fields $y$ & $x_1 \ldots x_k$; you do know that each row represents observations on an individual unit, & that these are (more or less) independent & representative of a population for which you'll have to predict $y$ for some members given only $x_1 \ldots x_k$ for each; you don't know anything about these $x_1 \ldots x_k$ other than what you can guess from the contents of these fields. Something like that? Or are you imaging that $x_1 \ldots x_k$ are already more or less suitable for regression, each being an individual real ... $\endgroup$ Oct 26, 2015 at 14:53
  • $\begingroup$ Did you look at box-cox family of transformations ? $\endgroup$
    – user83346
    Oct 26, 2015 at 15:06
  • $\begingroup$ ... number representing the magnitude of some quantity? $\endgroup$ Oct 26, 2015 at 15:09

3 Answers 3


I'll try to keep it as generic as possible. You might have continuous, categorical or binary values. If you have categorical values, let's say x1 takes values such as (Monday, Tuesday, Wednesday, etc) then you need to process and encode them somehow. For the example above it's typical to encode monday as [1,0,0,0,0,0,0]. Tuesday would be encoded as [0,1,0,0,0,0,0] and so on. You need to convert everything to numbers.

This page from scikit-learn explains it: http://scikit-learn.org/stable/modules/preprocessing.html

Feature transformation could then be:

  1. Preprocessing -- This is typically: Scaling, Standardizing (z-score) or Normalization.

If you're not allowed to look at the data, I'd go with Normalization since it's a procedure that you can perform on streaming data immediately. This guarantees that none of your features will be greater than one or less than zero. You now have some bounds on each feature. This might be your answer, although I'll go further just in case.

  1. Dimensionality reduction -- http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition

These methods project the data onto lower dimensions -- helpful with high dimensional data (e.g. think of is as compression, getting rid of non-essential information). Again, for streaming data you might need something like the Incremental PCA. Another option (vs PCA) is the RBM http://scikit-learn.org/stable/modules/neural_networks.html#neural-network Either way, you have no guarantee that you will end up with an optimal solution since you're not taking the y's (regressor) into consideration.

  1. Feature selection -- http://scikit-learn.org/stable/modules/feature_selection.html#feature-selection

Several options are (but not limited to): L2 (Ridge regression) L1 (Lasso) and L1+L2 ElasticNet. If you use the L1 norm to penalize then your solutions will be sparse (e.g. you get rid of some features completely, some weights are zero). If you use L2, then the algorithm adjusts the importance (weight) of each feature. You don't have to use linear models (e.g. f(x) = x_1 + x_2 + ... x_n), you can choose whatever combination of features you want, e.g. (x_1^2 + x_2*x_1 + x_2 + etc). Then the regularization is performed on this model.

Or you could just use the random forests and you get this for free.

Generic answer without knowing what the data 'means' and not making any assumptions about it's distribution: encode non-numerical values to numbers, normalize (unit length) and use a regularized regression model.


One interesting answer is the No Free Lunch theorem. Basically it states that your ability to model data is inseparable from your understanding of it.

Features, though, suggest you are interested in obtaining the numerical inputs to model using a multivariate regression tool. If that is the case, it's a fairly well defined area that is documented by ML libraries such as Spark. Common examples are mapping words (or categories) into vectors, and normalizing a quantity's mean and stddev.

This topic gets way more complex if your input is text, or images and other multimedia formats. This is why zip files compress text well, but you need a specialized MPEG-4 video codec for that kind of data.


Most common statistical techniques are optimized for normally distributed data of roughly the same scale. The most common feature transformations are thus equating variance across variables and turning non-normally distributed data into normally distributed data.

A standard approach is to rescaling is to calculate the mean and standard deviation for every variable, throw away all variables whose standard deviation is 0, and then subtract off the mean and divide by the standard deviation. This has the virtue of preserving distances between measurements--if three points were at 4, 6, and 8 before, then afterwards we know the distance between the first point and second point will be equal to the distance between the second point and third point. This is unnecessary as pre-cleaning for most regression techniques, because this is part of the operation of the regression technique, but can be useful for feature selection techniques, including regularized regression.

I prefer explicit normalization instead. To explain it, I'll talk about units: we start off with the raw units (maybe this is a length measured in meters), convert it to fractiles (this is the amount of the data it is larger than, measured in fractions), and then convert to probits (this is what the z-score of a point with the same rank from a normally distributed variable would have been). This has the virtue of enforcing a multivariate Gaussian distribution on the data.

Which approach to prefer depends on what the underlying data means. If you have bimodal data, is the distance between the modes significant, or just the overall rank of the data? If you have skewed data, is the data 'centered' on the mean, or the median, and are movements to one side or the other equally meaningful?

Fractiles discard relative distances, focusing only on a point's position in the sample's order, and thus are a natural way to compare very different distributions. It easily handles outliers, which are always a concern with unfamiliar data. It's what I would start with if I had no special prior knowledge. But if we know the correct transformation, we should of course use that, as those distances may have actual meaning (and it will only sometimes be the fractile, or equivalently, the raw data if the data is normally distributed).

  • 1
    $\begingroup$ By converting to fractiles, keep in mind you are effectively modeling the single-variate distribution. I'd call this modeling, not normalization. The remaining multivariate data, having been normalized in this way, is called a copula. $\endgroup$ Oct 31, 2015 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.