Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

0
votes
0answers
15 views

Regression percentage versus absolute levels

I regressed interest rates using the percentage changes for each respective rate. The r^2 was significantly lower, .5, versus .85 for regression pure rate values--not transforming them into ...
2
votes
1answer
34 views

Transformation of MLE parameters standard deviations calculation

I have run an MLE estimation using R to estimate parameters $\theta_1, \theta_2, \theta_3$, which are: 0.0002022146 0.0222026625 -6.1910421067 the variance ...
0
votes
1answer
10 views

Box-Cox transformation in multiway ANOVA

I am trying to run a ANOVA with the model LOS = HR + RESP + TEMP + SpO2 All 5 variables are discrete numerical values(all integers). Initially, I got a Shapiro Wilks of p-value < 2.2e-16 with no ...
0
votes
1answer
22 views

Approximation of non linear function with multiple linear functions

How can a non-linear function be approximated by an appropriate amount of linear functions? In the picture below, it would be quite easy to draw 10-15 linear functions to describe all data points ...
1
vote
1answer
30 views

Does data normalization and transformation change the Pearson's correlation?

As we know that Pearson's correlation measures the linearity between two variables, I am wondering when applying normalization and transformation on the original dataset, does the normalization and ...
1
vote
1answer
62 views

Which of those methods is the best for transforming my data into a normal distribution (if any)?

I have problems getting my data to be normally distributed. I hope someone more experienced can help. I have the following data: ...
0
votes
0answers
10 views

Converting Annual Data into Biannual Frequency

I need to convert annual data into biannual, since most of my data have biannual frequency. However, some macroeconomic variables - like business environment quality or Herfindahl-Hirschman Index are ...
0
votes
0answers
16 views

How to align data series along x-Axis to achieve besst fit?

I have 4 datasets that I am trying to align to each other so I can so some further analysis. I asked a similar question earlier, that didn't get much attention, so I am trying again (this time with ...
0
votes
0answers
9 views

how to treat half normal, half non-normal data in spss [on hold]

I’m examining the effects of three types of interventions on 3 different groups and at 2 different time points (pre, post-intervention). The total sample size is 23 (two groups have 8 participants ...
0
votes
0answers
9 views

Using categories instead of WoE values

As for as I can understand, the Weight of Evidence strategy is the following: For continuous independent variables : First, create bins (categories / groups) for a continuous independent variable and ...
0
votes
0answers
7 views

Model evaluation based on WoE: test and train set

I am training Logistic Regression model. First, I prepare the variables. I am confused with the following: Should I transform variables to the ones with WoE (Weight of Evidence) values based on the ...
1
vote
1answer
15 views

Transforming categorical variables

Can someone explain me in which cases is a good idea to convert a categorical variable to numerical in order to used it in our model (either regression or classification). I have seen cases where even ...
0
votes
1answer
18 views

How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below: ...
0
votes
0answers
13 views

Interpret model estimates after log transformation [duplicate]

I asked this question before, but maybe on the wrong audience at math.stackexchange.com. So, sorry for the redundancy. I sat up a mixed-effects linear model with the dependent variable log-...
0
votes
0answers
24 views

Workflow in data preparation with Box-Cox transformation

I have a dataset with both missing values and outliers in continuous features. I would like to perform Box-Cox transformation on every continuous feature to reach the best distribution. Box-Cox works ...
0
votes
0answers
13 views
0
votes
1answer
73 views

How to pre-process audio recordings for training a machine learning model?

Task: Process audio data so that it can be used for training a machine learning model--which would be used for labeling unseen/unheard audio recording in future. Data: The audio recordings are ...
0
votes
0answers
5 views

Variable influence on Normalization process of a data set [closed]

I have a data set with approximately 40 variables (metabolites). 2 of these variables have a high variance and I'm worried that this will influence the normalization process causing not good results ...
1
vote
0answers
18 views

Mean and error bounds of log-transformed data using Gaussian process regression

To revive a past question and establish a definitive answer, how should the mean/mode and error intervals of log-transformed data be handled when applying Gaussian process regression? For example, I ...
0
votes
0answers
13 views

Dynamic transformations on an inferred time-scale [duplicate]

Sorry if this is a duplicate, I have limited knowledge on data science, so I don't know the correct terms to look for and don't know if there's already an answer out there. I have two datasets which ...
0
votes
0answers
11 views

Normal Scores (Blom) vs Simple Standardization

I am conducting survival anaylsis in SAS (PROC PHREG) and have come across an important consideration. Most of my clinical variables are continuous but very skewed. So I thought to use ...
0
votes
0answers
18 views

Prediction Challenge AI. Machine learning and data analysis.(Inconsistency data)

I am working on a personal rento in which based on some input data I have to predict some output data. The challenge is to predict the expenses in transactions, receipts and cards that users will have ...
0
votes
0answers
7 views

Why some missing values are recorded as -1? [duplicate]

I have a lot of missing values in a dataset in a velocity column. Some of the missing values are just blank cells, some are recorded as NaN, but for some column (velocity), the missing value was ...
0
votes
0answers
31 views

Application of Box-cox transformation consecutively

as far as I have searched even we can obtain optimal lambda value to transform data to normal distributed with constant variance in box cox transformation method we may have not proper normal ...
0
votes
0answers
23 views

Log transform with 'zero' values [duplicate]

I am doing some explorative work on two large datasets. One from 2001 and one from 2018. The dataset consists of measured soil-parameters and it contains lots of zero's. From the transformations ...
0
votes
0answers
6 views

DeepSets design when sets may have varying number of sample elements with some fixed value?

Trying to make use of a DeepSets machine learning design (as described here: https://arxiv.org/pdf/1703.06114.pdf) where in order to model data that is given as per-item samples when sets of the ...
0
votes
1answer
26 views

What is the best approach to transform scale from 1 to 10 into three categories?

I want discretize my attributes according to the class quality which is the output variable. quality ranges from 1 to 10 in my ...
0
votes
0answers
10 views

How do i select the attribute from the correlation and covariance metrics?

I have a values from heatmap which is contain correlation and covarian. for the correlation values, i got with this code : ...
0
votes
0answers
42 views

Transforming back after a log transformation with subtraction

I needed help with back transforming my data. My initial data was positively skewed so I had to log transform it, after which I did my statistical test. One of my regression test required for my ...
1
vote
2answers
86 views

Transformation of distribution before applying tests or models?

I have observations from an experiment, and each of the observations belongs to one of 2 groups (patients, controls). I would like to test for group differences in the data using a test. I would ...
2
votes
2answers
74 views

How to back transform a folded root?

I have some data where the response variable is a proportion, and I am experimenting with transformation using Tukey's family of folded powers, $f(p) = p^\lambda - (1 - p)^\lambda$, with values of $\...
1
vote
1answer
31 views

Which constant to add when applying 'Box-Cox transformation' to negative values?

Questions How big constant should we add to negative values when applying the 'Box-Cox transformation'? The data that I am handling is 'daily return of stocks' Shouldn't we subtract some amount after ...
0
votes
3answers
51 views

Does Categorical Variable need normalization/standardization?

since we do normalize as 10kg >>> 10 grams or 1000 >> 10. so incase of one hot encoding eg male=0 and female =1, are we giving more weight to female as 1>0 for training our models?
1
vote
0answers
25 views

Prediction interval of Y given AR model for log(Y) [duplicate]

I have been given an AR model with seasonal variation. \begin{equation} (1-\theta_1B)(1-\theta_2B^8)(log(Y_t)-\mu)=\epsilon_t \end{equation} Setting $X_t=log(Y_t)-\mu$ one gets the following \begin{...
3
votes
1answer
175 views

Does a log transform always bring a distribution closer to normal?

I have a highly right skewed data set with a large range of values (from 1 ~ 10^6) (can't share the actual data for work related reasons). When I plot the log of the data instead, the distribution ...
0
votes
0answers
24 views

Converting log transformed and differenced time series back into original in R

I have built a Garch model in R based on taking a log transformation and a one order difference on the original time series. I would like to know how develop a forecast based on the Garch model for ...
1
vote
1answer
36 views

How to deal with predictions if taking log of dependent variable

I have a very basic question about linear regression. I have a dataset where the response variable is largely skewed to the right -- if I take a log of it, the distribution becomes a lot closer to ...
0
votes
2answers
39 views

How to get PCA of the testing data? [duplicate]

I'd like to transform my data into pca (preprocessing data before I use data into classification model). I separate my data into data training and data testing. I used princomp in R to process pca ...
0
votes
0answers
16 views

Covariance matrix for a 2D state vector

I'm performing Optimal Interpolation (which in fact is a simplified Kalman filter with constant $\mathbf{K}$). My state variable is a 2D concentration field with a size of 370 x 400 on which I try to ...
0
votes
0answers
12 views

Ways to anonymize the data for logistic regression

We are testing a Proof of Concept with an external vendor to whom we need to provide the data in an anonymized way. Apart from removing the client details/identifiers, what are the best ways to ...
2
votes
1answer
60 views

Why feature transformation is needed in machine learning & statistics? Doesn't it affect the “interaction” between features?

Before feeding machine learning models, we can do data transformation and feature scaling depending on data distribution. For example, if a column is skewed, we can use Box-Cox transformation to ...
0
votes
1answer
33 views

Are trend-stationary series I(0)?

I have time-series of different interest rates. Graphs of all series show existence of trend. For some of these series ADF-test with constant rejects null hypothesis. For others, null hypothesis is ...
1
vote
0answers
10 views

Comparing my data to published reference range percentiles, do I consider MY percentile range or does distribution of my data not entirely matter?

I am comparing my data (left column) of Packed cell volume to published known reference ranges, to identify any students that are anaemic/polycythaemic (too low/high PCV). With the data in this image,...
0
votes
0answers
15 views

Fine-tuning on a differently normalized dataset

I have a classification model which has been trained on a simulated training set, with a given mean and std deviation which of course I apply also on the (simulated) testing data. Now I received new (...
1
vote
0answers
6 views

EFA with heterogeneously transformed variables

Can I perform an EFA (oblique rotation) with variables that have undergone different types of transformations? Can I still interpret the factor loadings and other output coherently? For ...
0
votes
1answer
43 views

Approximate the data to a single curve

The question might be simple, but I am not able to find the answer. Hence I am asking here. I did search google but didn't get an answer. I have a continuous stream of data coming from an API in the ...
0
votes
0answers
24 views

Cox Regression Specifics: Standardizing, Log Transforming

In the statistical analysis of this paper I have some questions regarding their approach. https://academic.oup.com/ndt/article/33/6/1001/3978817 “Variables with non-normal distributions were either ...
0
votes
0answers
43 views

PDF transformation for y=|x|

Suppose I have the random variable X with a pdf: $$f(x)=exp(-(x+1)) u(x+1)$$ where u is the unit step function; such that u = 0 for x<-1 and u=1 for x>-1 $$y= |x|$$ for $$-1<x<1$$ ...
1
vote
0answers
27 views

Can I log-transform realized volatility in a co-integration setting

I'm writing my master's thesis and looking to see if there exists fractional co-integration between the volatility of some large stock-indices. My estimates of realized volatility are based on the ...
1
vote
0answers
14 views

How can i make constant variance for ARIMA model?

I want to fit ARIMA model to univariate time series. For this i took log then difference twice but the variance of the series is not constant. i also try with box.cox transformation but variance is ...