Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Trying to Unskew data with log transformation, unable to get a somewhat normal distribution

Wannabe Data scientist here trying to do some k-means clustering, please go easy on me if there is a really obvious answer :). I'm currently at the step where I'm trying to unskew my data using log ...
1
vote
0answers
10 views

Time series and first order difference correlation coefficient to adjust for autocorrelation

I have come across an adjustment method that uses the correlation coefficient and a first difference to adjust for autocorrelation in a times series. All data points transformed using the formula: $ ...
0
votes
0answers
8 views

Transforming log-scaled splines regression outputs to an understandable scale

Please give me some advice. I am using brms package and mgcv package for two regression models: bernoulli lognormal The problem is that both of these models outputs are in lognormal scale. As much ...
0
votes
0answers
12 views

WOE of raw variable vs WOE of log of same variable | What can be the impact? [closed]

Should I expect different results if I transform "var 1" to WOE compared to "log(var 1)" to WOE? WOE transformation means "Weight of evidence" transformation
0
votes
1answer
117 views

Dealing with a dataset having target values on different scales?

I am currently working on a dataset having 10 features and one continuous target variable. One of the features is 'Country' , in which there are seven unique values [Argentina ,Denmark , France...etc]....
0
votes
0answers
17 views

How to understand what linear transformations are doing when estimating a regression?

I am trying to better understand, ideally in English, what exactly a linear transformation is doing when we compute something like OLS (but the process holds for basically every statistical model). We ...
0
votes
1answer
481 views

Using both raw and weight of evidence values in logistic regression

I am building a logistic regressin model for probability of take-up for a lending product. I have a number of continuous variables. In the past, I have always used EITHER weight-of-evidence ...
1
vote
1answer
114 views

Solving a double integral over transformations of joint bivariate standard normal values

My problem is about calculating the covariance between transformations of two test statistics based on the correlation between these test statistics. Let $X$ and $Y$ be two test statistics whose joint ...
0
votes
0answers
30 views

How to use the kernel trick on a XOR-like dataset

Let's say that I have the following data: I want to find a transformation of this dataset that will make it linearly separable. My thought was to bring the data around the origin and then multiply $...
1
vote
0answers
16 views

Which data do I have to use for forecasting? The transformed one or the real one?

I have a data about museum visitor in 2011-2019. I divide this data into training set (2011-2018) and testing set (2019). The training set is nonstationary so I used Box-Cox transformation and ...
0
votes
1answer
1k views

How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below: ...
0
votes
0answers
12 views

Resample data from a histogram with higher resolution - assuming it follows a normal distribution

I have a problem which has me stumped on what to try next. I have some data from a farm, related to yield over a period of days, for a defined area. I have daily-resolution data, where the day's ...
0
votes
0answers
40 views

Transforming data or not

I have data that I will use as a feature to Elastic Net. I thought I should transform the data using either Box Cox or Yeo Johnson. The transformed data looks weird and I'm not sure if I should ...
1
vote
0answers
15 views

Modelling outcome variable with lognormal distribution

I have an outcome variable with a distribution as follows, and I need to do some regression modelling. I know that often such variables are transformed to get a normal distribution. However, it would ...
1
vote
1answer
138 views

Overfitting and data transformations

I've read a lot about overfitting here, and now I have a question about this subject. Ok, if we put together too much independent variables, primary looking for better fits (for exemple, higher R², ...
0
votes
0answers
17 views

Whitening MNIST in R [closed]

I am analyzing the famous MNIST data set with R. I have never been analyzing handwritten digit data before, so I wonder wether it makes sense to whiten the 756-dimensional data in the sense that from ...
0
votes
0answers
35 views

Choosing a statistical method for customer preferences

I have received a data set about the ingredients of sweets of different brands, as well as information about prices in percent, sugar and profit in percent. The information on the ingredients are ...
1
vote
1answer
124 views

Is it appropriate to perform a t-test on data expressed as T-scores for individual participants?

I have data that comes as "fully corrected" T-scores (i.e., adjusted for age, sex, SES, etc., to a standardized score with a population mean 50 and SD 10) for 20 participants at time 1 and time 2. How ...
1
vote
1answer
91 views

Generate null distribution from pvalues

I have a set of experiments on which I apply the Fisher's exact test to statistically infer changes in cellular populations. Some of the data are dummy experiments that model our control experiments ...
1
vote
1answer
127 views

How to get the actual mean absolute error in cross validation after transforming the target variable y?

For a target variable y, it is transformed using np.log1p. Then a random forest regression model is trained using the ...
0
votes
0answers
14 views

Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
0
votes
0answers
30 views

Log-transforming values on different scales

I'm trying to train a neural network to classify time series that represent the times between the sales of consumer products. I am training a single model across multiple products (spanning across ...
0
votes
1answer
28 views

Can I mix different data transformations for different variables in the same dataset?

Can I mix different data transformations for different variables in the same dataset? Target analysis: Principal Component Analysis.
0
votes
0answers
22 views

Boxcox Transformation Error - “contains observations that are not strictly positive”

I am trying to do a Box-Cox transformation for a data set in Stata, using the following code: boxcox Production_qty price technology gender, model(lambda) notrans(gender) It gave me an error saying &...
0
votes
3answers
89 views

Modelling the logarithm of a response

My response variable is positive and I decided to model the logarithm of the response. Some of the values are zero. For this reason I modelled $Z = \log(Y + 0.1)$. When I transform back, some of my ...
0
votes
0answers
9 views

Quantify data leakage train test set for feature scaling

Topic: First appling feature scaling (e.g. standardisation) to a data set and then splitting the data set into train and test set, can introduce data leakage. Question: Although this is clear to me ...
0
votes
0answers
14 views

How to Construct an Index

I'm sorry in advance that this question is a little bit broad. I'm trying to create on variable to measure the overall prestige of a university. I'm using data on federal obligation, tuition, as well ...
0
votes
0answers
17 views

power transformation with negative values?

Is there a name for this type of transformation: \begin{equation} sgn(x) * |x|^p \end{equation} where $p$ is an arbitrary number (e.g., 1/2 and 1/3 for square and cube roots, respectively) and $sgn$ ...
0
votes
0answers
22 views

When transforming data, why use resampling to estimate transformed values?

I am working through a machine learning book and am using the caret package in R. There is a function, preProcess, that uses ...
0
votes
1answer
311 views

Should the multivariate Box-Cox lambda value, of a variable against itself, be 1?

I have 10 variables, and am trying to determine which transformation between each variable provides the best linear relationship. To this end, I am using the Box-Cox method to determine a power ...
1
vote
0answers
14 views

Document AI : Using FUNSD dataset to train a GNN to classify 'Linked' entities

I have been using the FUNSD dataset to predict sequence labeling in unstructured documents per this paper: LayoutLM: Pre-training of Text and Layout for Document Image Understanding . The data after ...
2
votes
2answers
114 views

Matrix and vectors, why different notation for dimensions?

If we collect data and put it into a matrix of size (100,3), we tend to say we have three-dimensional data. We think of each column as a dimension. On the other side, if we have a vector of size (100,...
0
votes
1answer
122 views

R auto.arima with transformed covariates

I have a non-stationary output time-series (oil prices) that is to be forecasted with 20 different input time series. The series are all non-stationary. I am considering two approaches. Approach 1) ...
1
vote
0answers
49 views

Comparing linear models between categories after different transformations

Suppose I have a predictor variable $X$ split into two categories $X_a$ and $X_b$, and a response variable $Y$. I wish to perform linear regression but need to transform both $X_a$ and $X_b$ as to be ...
3
votes
0answers
1k views

Box-Cox, log or arcsine transformation? [closed]

Box-Cox, log and arcsine transformations have the aim of make the data more Normal. My question is: how can I choose between each one of these transformations? Which assumptions do I need to have to ...
1
vote
0answers
137 views

How to interpretation of the results of PCA [closed]

There is a larger matrix (1500 rows x 40 columns), 1500 observations x 40 variables. then I follow the procedures of PCA(Principle components analysis), 1. find correlation 2. find eigenvalues 3. ...
0
votes
0answers
68 views

Can I perform Z-score values on percentages?

Can I perform Z-score values on percentages? It is correct? Or should I perform Z-values starting from frequencies? I want to perform Z-score on percentages to 'declosed' my data, as, If I understood ...
0
votes
0answers
9 views

Joining semi-related tables and predicting counts based on generalised proportions

I am working with UK census data and have found it to be quite restrictive when it comes to combining multiple variables. I have managed to get 3 separate tables that are relevant for a project of ...
2
votes
0answers
52 views

Curvature of a pencil's stroke

I'm trying to evaluate the curvature of an image of a pencil stroke based on the image's pixels and their shade of gray. I'm trying to get the curvature of the line at every point of the stroke. The ...
1
vote
0answers
47 views

which transformations of dependent variable are allowed?

I was wondering which transformations of dependent variable (y) are "allowed"? My problem is that I am trying to compare three groups (=categorical variable), basically to find out if the groups have ...
17
votes
3answers
5k views

Extract data points from moving average?

Is it possible to extract data points from moving average data? In other words, if a set of data only has simple moving averages of the previous 30 points, is it possible to extract the original data ...
0
votes
0answers
18 views

How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
1
vote
1answer
115 views

How to interpret a specific data transformation?

I came across this specific data transformation in the context of a physics application, which by itself is rather complex and hence out of the scope of this question. However since this ...
3
votes
0answers
2k views

Is Freeman — Tukey's transformation the most powerful for percentages?

I've described a typical design for my experiments in this question. Well, 1-way RM ANOVA assumes a Gaussian distributed vector. I try $y=\arcsin{\sqrt{x}}$. But for some data it works, for some doesn'...
0
votes
0answers
155 views

Wiener transformation - K-Apriori Algorithm

Wiener transformation is used to transform a binary data to a real data? How it works? Is there a R package that implements this transformation? I have been searching about Market Basket Analysis and ...
0
votes
0answers
147 views

Transforming data with large negative and large positive values [duplicate]

The data I'm trying to analyze are the quadratic estimates from a quadratic fit to a curve. Most of the data vary between -.15 and .15. However, I have outliers in both directions up to things like -...
1
vote
0answers
44 views

transforming dependent bivariate data to independent data

Assume a bivariate data that has some sort of dependency between the two variables is generated by unknown distribution, but not bivariate normal. Is it possible to remove the dependency of the two ...
1
vote
0answers
29 views

Transforming a multivariate normal sample using the sinh-arcsinh transform

Let us say that we have sample from the multivariable normal distribution. I would like to understand how is possible to apply a transformation to this sample to produce sample that has the sinh-...
0
votes
0answers
17 views

Transformation of Time and Temperature into Aggregate Score

Is anyone familiar with a suitable transformation of time and temperature values into a single score? I am working with microbiological data and I have three continuous factors all measured at ...
0
votes
0answers
44 views

Find intercept of almost flat lines

I have a set of lines (image below) which should meet in a number of points. As you can see, now the angular coefficient doesn't vary noticeably, making intercepts hard to find. What transformation do ...

1
2 3 4 5
40