# Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

1,994 questions
Filter by
Sorted by
Tagged with
11 views

### Trying to Unskew data with log transformation, unable to get a somewhat normal distribution

Wannabe Data scientist here trying to do some k-means clustering, please go easy on me if there is a really obvious answer :). I'm currently at the step where I'm trying to unskew my data using log ...
10 views

16 views

### Which data do I have to use for forecasting? The transformed one or the real one?

I have a data about museum visitor in 2011-2019. I divide this data into training set (2011-2018) and testing set (2019). The training set is nonstationary so I used Box-Cox transformation and ...
1k views

### How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below: ...
12 views

### Resample data from a histogram with higher resolution - assuming it follows a normal distribution

I have a problem which has me stumped on what to try next. I have some data from a farm, related to yield over a period of days, for a defined area. I have daily-resolution data, where the day's ...
40 views

### Transforming data or not

I have data that I will use as a feature to Elastic Net. I thought I should transform the data using either Box Cox or Yeo Johnson. The transformed data looks weird and I'm not sure if I should ...
15 views

### Modelling outcome variable with lognormal distribution

I have an outcome variable with a distribution as follows, and I need to do some regression modelling. I know that often such variables are transformed to get a normal distribution. However, it would ...
138 views

### Overfitting and data transformations

I've read a lot about overfitting here, and now I have a question about this subject. Ok, if we put together too much independent variables, primary looking for better fits (for exemple, higher RĀ², ...
17 views

### Whitening MNIST in R [closed]

I am analyzing the famous MNIST data set with R. I have never been analyzing handwritten digit data before, so I wonder wether it makes sense to whiten the 756-dimensional data in the sense that from ...
35 views

### Choosing a statistical method for customer preferences

I have received a data set about the ingredients of sweets of different brands, as well as information about prices in percent, sugar and profit in percent. The information on the ingredients are ...
124 views

### Is it appropriate to perform a t-test on data expressed as T-scores for individual participants?

I have data that comes as "fully corrected" T-scores (i.e., adjusted for age, sex, SES, etc., to a standardized score with a population mean 50 and SD 10) for 20 participants at time 1 and time 2. How ...
91 views

### Generate null distribution from pvalues

I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations. Some of the data are dummy experiments that model our control experiments ...
127 views

### How to get the actual mean absolute error in cross validation after transforming the target variable y?

For a target variable y, it is transformed using np.log1p. Then a random forest regression model is trained using the ...
14 views

### Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
30 views

### Log-transforming values on different scales

I'm trying to train a neural network to classify time series that represent the times between the sales of consumer products. I am training a single model across multiple products (spanning across ...
28 views

### Can I mix different data transformations for different variables in the same dataset?

Can I mix different data transformations for different variables in the same dataset? Target analysis: Principal Component Analysis.
22 views

### Boxcox Transformation Error - “contains observations that are not strictly positive”

I am trying to do a Box-Cox transformation for a data set in Stata, using the following code: boxcox Production_qty price technology gender, model(lambda) notrans(gender) It gave me an error saying &...
89 views

### Modelling the logarithm of a response

My response variable is positive and I decided to model the logarithm of the response. Some of the values are zero. For this reason I modelled $Z = \log(Y + 0.1)$. When I transform back, some of my ...
9 views

### Quantify data leakage train test set for feature scaling

Topic: First appling feature scaling (e.g. standardisation) to a data set and then splitting the data set into train and test set, can introduce data leakage. Question: Although this is clear to me ...
14 views

### How to Construct an Index

I'm sorry in advance that this question is a little bit broad. I'm trying to create on variable to measure the overall prestige of a university. I'm using data on federal obligation, tuition, as well ...
17 views

### power transformation with negative values?

Is there a name for this type of transformation: $$sgn(x) * |x|^p$$ where $p$ is an arbitrary number (e.g., 1/2 and 1/3 for square and cube roots, respectively) and $sgn$ ...
22 views

### When transforming data, why use resampling to estimate transformed values?

I am working through a machine learning book and am using the caret package in R. There is a function, preProcess, that uses ...
311 views

### Should the multivariate Box-Cox lambda value, of a variable against itself, be 1?

I have 10 variables, and am trying to determine which transformation between each variable provides the best linear relationship. To this end, I am using the Box-Cox method to determine a power ...
14 views

### Document AI : Using FUNSD dataset to train a GNN to classify 'Linked' entities

I have been using the FUNSD dataset to predict sequence labeling in unstructured documents per this paper: LayoutLM: Pre-training of Text and Layout for Document Image Understanding . The data after ...
114 views

### Matrix and vectors, why different notation for dimensions?

If we collect data and put it into a matrix of size (100,3), we tend to say we have three-dimensional data. We think of each column as a dimension. On the other side, if we have a vector of size (100,...
122 views

### R auto.arima with transformed covariates

I have a non-stationary output time-series (oil prices) that is to be forecasted with 20 different input time series. The series are all non-stationary. I am considering two approaches. Approach 1) ...
49 views

### Comparing linear models between categories after different transformations

Suppose I have a predictor variable $X$ split into two categories $X_a$ and $X_b$, and a response variable $Y$. I wish to perform linear regression but need to transform both $X_a$ and $X_b$ as to be ...
1k views

### Box-Cox, log or arcsine transformation? [closed]

Box-Cox, log and arcsine transformations have the aim of make the data more Normal. My question is: how can I choose between each one of these transformations? Which assumptions do I need to have to ...
137 views

### How to interpretation of the results of PCA [closed]

There is a larger matrix (1500 rows x 40 columns), 1500 observations x 40 variables. then I follow the procedures of PCA(Principle components analysis), 1. find correlation 2. find eigenvalues 3. ...
68 views

### Can I perform Z-score values on percentages?

Can I perform Z-score values on percentages? It is correct? Or should I perform Z-values starting from frequencies? I want to perform Z-score on percentages to 'declosed' my data, as, If I understood ...
9 views

### Joining semi-related tables and predicting counts based on generalised proportions

I am working with UK census data and have found it to be quite restrictive when it comes to combining multiple variables. I have managed to get 3 separate tables that are relevant for a project of ...
52 views

### Curvature of a pencil's stroke

I'm trying to evaluate the curvature of an image of a pencil stroke based on the image's pixels and their shade of gray. I'm trying to get the curvature of the line at every point of the stroke. The ...
47 views

### which transformations of dependent variable are allowed?

I was wondering which transformations of dependent variable (y) are "allowed"? My problem is that I am trying to compare three groups (=categorical variable), basically to find out if the groups have ...
5k views

### Extract data points from moving average?

Is it possible to extract data points from moving average data? In other words, if a set of data only has simple moving averages of the previous 30 points, is it possible to extract the original data ...
18 views

### How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
115 views

### How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
2k views

### Is Freeman ā Tukey's transformation the most powerful for percentages?

I've described a typical design for my experiments in this question. Well, 1-way RM ANOVA assumes a Gaussian distributed vector. I try $y=\arcsin{\sqrt{x}}$. But for some data it works, for some doesn'...
155 views

### Wiener transformation - K-Apriori Algorithm

Wiener transformation is used to transform a binary data to a real data? How it works? Is there a R package that implements this transformation? I have been searching about Market Basket Analysis and ...
147 views

### Transforming data with large negative and large positive values [duplicate]

The data I'm trying to analyze are the quadratic estimates from a quadratic fit to a curve. Most of the data vary between -.15 and .15. However, I have outliers in both directions up to things like -...
44 views

### transforming dependent bivariate data to independent data

Assume a bivariate data that has some sort of dependency between the two variables is generated by unknown distribution, but not bivariate normal. Is it possible to remove the dependency of the two ...
29 views

### Transforming a multivariate normal sample using the sinh-arcsinh transform

Let us say that we have sample from the multivariable normal distribution. I would like to understand how is possible to apply a transformation to this sample to produce sample that has the sinh-...
17 views

### Transformation of Time and Temperature into Aggregate Score

Is anyone familiar with a suitable transformation of time and temperature values into a single score? I am working with microbiological data and I have three continuous factors all measured at ...