Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

0
votes
1answer
50 views

How to normalize data of 0s and 1s?

the data consists of one 2, so it's not binomial. is there any way to transform it to fit the normal assumptions? I tried square root and log, both didn't work
1
vote
0answers
13 views

Selecting lag length for VAR Model. Differences or Levels?

I'm currently testing for optimal VAR lag length using the information criteria. I found that my variables are non-stationary (i.e. they have to be first differenced). When I identify the number of ...
0
votes
0answers
23 views

Transforming non-normal to normal distribution and back-transform

I would like to transform non-normal distribution to normal distribution, and back-transform to its original state (or at least close to the original state). From this article, I've read that you can ...
0
votes
0answers
24 views

Lower ARIMA accuracy with BOX-COX

I'm using ARIMA model for time series forecast. My data has increasing variance and I applied a BOX-COX transformation to stabilise it. Here are charts: After I run my app it turned out that ARIMA ...
0
votes
0answers
4 views

Is sketching a method for dimensionality reduction and its relation to random projection

I want to know if sketching can be categorized as a method of dimensionality reduction and more specifically feature extraction. Also, i want to understand if its related to random projection.
4
votes
2answers
55 views

Change variable to log transformed or keep original?

A log transformation of the dependent variable is sometimes recommended as a remedy for some cases of non-normal distribution of residuals after fitting a linear regression model. What is the proper ...
3
votes
0answers
21 views

Use of logs in a regression analysis [duplicate]

I'm doing a paper on the impact of FDI on GDP and unemployment in different countries and I'm doing an OLS regression analysis for my data. Would it make sense to use logs for only one variable? Or ...
0
votes
0answers
15 views

Shift data to use Gamma glm with “identity” link?

I am trying to fit a glm using the Gamma family and the "identity" link. As I am analyzing nutrient concentrations in coastal waters (that can not be negative and are unlikely to be zero) I assume the ...
0
votes
1answer
35 views

Should the multivariate Box-Cox lambda value, of a variable against itself, be 1?

I have 10 variables, and am trying to determine which transformation between each variable provides the best linear relationship. To this end, I am using the Box-Cox method to determine a power ...
0
votes
0answers
6 views

Evaluating association between variables when number of customer records differ

I have a data set containing customer info spanning a decade that, among other variables, includes contract size in dollars and contract length in years. All data are discrete. I am asked to determine ...
0
votes
1answer
27 views

How to deal with a poor quality data set? [closed]

I have come across a data set in which almost half of the data points are unusable (as in when further processed, they give negative results when the results should all be above 0). This is largely ...
3
votes
1answer
64 views

why has author divided by 1.5 in hands on machine learning with scikit learn

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow (76/718), and the author is talking about dividing the dataset into a test set which i follow, but then goes on to talk about ...
0
votes
1answer
12 views

Summation Bounds When Finding Transformation of 2 Poisson Random Variables

I am reviewing some material on functions of several random variables from Section 7.4 of John E. Freund's Mathematical Statistics, 6th Edition, and I'm stumped on how the author gets the upper bound ...
2
votes
1answer
35 views

Ancova: How to proceed when residuals are not normal distrubuted but transformation changes meaning of results

I am trying to analyze water nutrient (DIN) at three different sites along distance transects perpendicular to the coast. The transects were sampled in two years (year t1 and t2) in two seasons (1 ...
0
votes
0answers
14 views

Can I add a constant and then transform my explanatory variable? [duplicate]

My current model is of the form $$\log(\frac{y_i}{1-y_i}) = \beta_0 + \beta_1 x_1+\beta_2 x_2.$$ Is it okay if I consider the model $$\log(\frac{y_i}{1-y_i}) = \beta_0 + \beta_1 x_1+\beta_2 \log(x_2+1)...
0
votes
0answers
21 views

Transformation suggestions requested - multi-regression Q

I am thawing fish - big fish. I am looking to regress minutes(Hrs) of thaw against fish size, starting core temperatures, thaw water temperature, water speed, and species (dummy var). Using Minitab. ...
1
vote
1answer
39 views

Fitting a curve or raw transformation in a logistic regression

For my project, I am fitting a logistic regression to the some 'dummy' bank data. The head of the dataset looks like this ...
0
votes
0answers
16 views

How to determine the distribution a dataset follows without plotting or visualizing it?

I am given a data set of 100k instances and I am being told that it belongs to one of four statistical distributions (normal, truncated normal, poisson and uniform). I am wondering how I may go about ...
1
vote
1answer
32 views

Transforming data with positive, negative, and zero values

I have a multiple linear regression model with several dependent variables that have positive, negative, and zero values, and are not normally distributed. I can't do a natural log transformation ...
0
votes
0answers
42 views

XY Graph from two time signals

I have two measurement signals (let's say a voltage signal and a current signals), both with timestamps and corresponding values. The timestamps of the two signals doesn't match up and aren't evenly ...
1
vote
1answer
25 views

ARIMA with increasing variance

my time series data looks like that: I would like to use ARIMA model to forecast next steps in time series. Unfortunately because of increasing variance data is non-stationary. Is there any way to ...
2
votes
1answer
36 views

What to do in meta-analysis if confidence interval not symmetrical?

I'm about to make a meta analysis of a particular topic. I'm using Comprehensive Meta-Analysis (CMA) ver. 3.0 for that reason. However, after I input all the necessary data, the software refused to ...
3
votes
2answers
39 views

Example of dataset where the data collected at time-points $g(t_1), g(t_2), \ldots$

What would be some practical scenarios where we collect data at time-points $g(t_1), \ldots, g(t_n)$, where $g$ is an increasing function? For example, $g(t) = \exp(t)$ or $\ln t$. To be more clear, ...
1
vote
0answers
13 views

Am I doing the correct data transformation for Granger causality tests

I have seven sets of time series, below is my process flow, am I doing the correct thing here? especially step 4. Raw data transform and test stationary with unit root test (ADF), with level, first ...
0
votes
1answer
40 views

Should I apply log transformation to column with long-tail distribution before clustering? [closed]

I am doing clustering on a given data. When I plot the distributions of the individual features of this data, I found there are many columns that shows "long tail distribution". I am wondering ...
0
votes
0answers
10 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
1
vote
0answers
26 views

Linear unit-transformation (say, from liters to milliliters) of response variable in simple regression

The setting Consider the simple regression model: \begin{equation} y_i=\hat{\beta}_0 + \hat{\beta}_1 x_i + \hat{\epsilon}_i \end{equation} Now, suppose I want to apply a linear transformation to ...
1
vote
0answers
14 views

Different conditions on data measurement for ml

Is it possible to train a prediction model (on my case classifier with four classes) between data taken on different conditions? To be more specific I have two data sets and for my task I am allowed ...
0
votes
0answers
11 views

Longitudinal sequence of observations to predict a single target

So we observe a population every month and observe set of features $X_{month}$ number of features monthly $X_{Mar,2018}$, $X_{Apr,2018}$ and we want to predict the end-of year value $y_{2018}$, which ...
0
votes
0answers
10 views

What are some techniques to augment tabular data?

As we know we can perform data augmentation to "image dataset". We can apply random rotation, shifts, shear and flips over images. Are there techniques to augment tabular small dataset? I know the ...
0
votes
1answer
36 views

Difference between (log, square, root) transformation and Normalization

I am confused between the Transformation and Normalization/Standardization, The basic understanding I have is Transformation: will be used in situation when we have skewness in data and to distribute ...
0
votes
0answers
9 views

Product Yield in Capability Analysis

I couldn't find a 6 sigma tag so please let me know if this is posted in the wrong forum. I'm performing a capability analysis on a manufacturing process. However my process output is Pass/Fail. This ...
0
votes
1answer
22 views

R auto.arima with transformed covariates

I have a non-stationary output time-series (oil prices) that is to be forecasted with 20 different input time series. The series are all non-stationary. I am considering two approaches. Approach 1) ...
0
votes
0answers
25 views

Does BoxCox transformation work for logistic regression?

I'm working on a case study from this MIT course. I'm practicing classification problems. Here is the code for my model. (The dataset can be accessed from the link. I can add it to this post) ...
2
votes
0answers
35 views

Data obfuscation strategies that keep statistical properties intact

I am getting data from a provider that contains identifiable information from a number of different organizations. The data aren't particularly sensitive, but to be safe, the provider wants to perform ...
1
vote
1answer
38 views

How to weight observations to transform a distribution into normal?

Suppose X is a variable which follows some distribution (non normal) then how to define $f(X(k))$ (f is some functions of the variable X) such that $$f(X)X$$ is normally distributed and $0\leq f(X(k))...
1
vote
1answer
29 views

Is there a systematic method to do transformations on independent variables in logistic regression?

Is there a systematic method for Logistic Regression to do transformations on the independent variables, in order to conclude that the most optimal logistic regression model is fitted? Illustration ...
0
votes
1answer
46 views

Log Transformation in R

I need to transform my not normal distributed data to normal distributed variables. Therefore I need to log-transform them. Log10(x+1) has not worked to create a normal distribution. Therefore, I want ...
0
votes
0answers
19 views

Centralisation of a data

A pre-processing process called centralisation of a data set is often applied. What the process is and a benefit from doing so?
0
votes
1answer
71 views

How do I normalize a bimodal distribution?

I'm working with the Iris data.One of the variables,PetalWidth,Has a clear bimodal distribution My understanding is that Multivariate regression Assumes normality for each of the input variables Can ...
1
vote
0answers
17 views

Using complex number in non-negative matrix factorization (NMF)

In short, I wonder which kind of data can use complex number for NMF. And could an imaginary part possibly be a vector? For detail, as I saw some papers used complex number in NMF (1), I think it ...
0
votes
1answer
31 views

Characterizing a non -normal distribution of data

I have a data set with about 3000 members that I'd like to use as a feature for a binary classification algorithm. The variable seems to be skewed by nature because the histogram is tailed at pretty ...
0
votes
0answers
21 views

Correct numerical feature transformation for neural networks

Model: I am working on a "shallow" (3-layer) auto encoder neural network. The input layer receives a, say 25-dimensional, vector $x$ of numerical elements representing client purchases. Several ...
1
vote
0answers
33 views

Functional Forms of Independent Variables

If our objective is to ascertain the relationship (specifically, sign and significance of Beta coefficient) between independent variables and dependent variable in an OLS regression (cross sectional ...
0
votes
0answers
18 views

Transforming raw scores to scaled scores using a proprietary scale

I'm working with questionaire answers from the Child Behavior Checklist (CBCL) for my master's thesis. The 113 items on the questionaire are organized into several subscales for different areas of ...
1
vote
0answers
34 views

How to interpret goodness of fit after a log transformation

below I have 2 variables x and y. I have reason to believe they move together in percentages rather than absolute terms. ie if one goes up 10 % the other will go up 10%. For this reason, I have taken ...
1
vote
0answers
40 views

Comparing linear models between categories after different transformations

Suppose I have a predictor variable $X$ split into two categories $X_a$ and $X_b$, and a response variable $Y$. I wish to perform linear regression but need to transform both $X_a$ and $X_b$ as to be ...
2
votes
0answers
40 views

Dealing with non-linear variable in multiple linear regression model [duplicate]

I have a multiple linear regression model which should explain the variation store price elasticities using consumer characteristics of the market area surrounding a store. Therefore, my dependent ...
1
vote
0answers
27 views

How to recalculate Confidence interval after logarithmic and square transformations?

I'm performing test on one variance and in order to asses normality of X, I've transformed data with first natural logarithm transformation,then square transformation and calculated confidence ...
1
vote
1answer
26 views

What are the differences between these two methods of Semi-logarithmic plotting?

I have my measurement data in (x, y). I am trying to plot a semi-logarithmic plot of y versus log (x). It looks like that there are two ways to plot such a graph. Transform the values of x to log (x) ...