Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
1
vote
2answers
13 views

Interpreting table 1 in clinical research papers

Am new to clinical research and starting off by reading some clinical papers. In the paper, I came across a table like as shown below Am I right to understand that the value of ...
1
vote
1answer
10 views

Is (covariance) stationarity preserved under log or exponential transformation?

In this lecture note, it (proposition 2) says that strict stationarity is preserved under transformation. However, it doesn't give the proof of this statement. Second, what if the process is ...
1
vote
0answers
7 views

Rearranging a data set to create one with lags (Past obervations) [migrated]

Let's say the original data set has 100 observations X1, X2, X3, --- , X100 Now I want to rearrange the data as follows and create a new data set Current_State Lag1 Lag2 Lag3 Lag4 Lag5 X100 ...
0
votes
0answers
16 views

What are the limits of inverse hyperbolic sine for regression?

I know that if your data contains zeros, log transforming your variable can be problematic, and all the zeros become missing. It is often suggested to use the inverse hyperbolic sine transform, rather ...
1
vote
0answers
14 views

Would a log transformation on my features change the elastic net result

We built the elastic net model on a set of my features and control features. With that, we did various experiments to discuss the importance of the selected features. For example, we showed more of my ...
1
vote
0answers
22 views

How to interpret a transformed linear regression model

I'm playing with the trafo R-package and this small data. After using the assumptions function, I found the log shift opt transformation is the best for normalizing ...
0
votes
0answers
17 views

Bayesian networks [closed]

I want anyone who can help me developing a model of traffic safety evaluation using bayesian networks( this method is more advanced statistics)? I have a the 385th questionnaires which it will be ...
0
votes
0answers
12 views

Survey Data: What is tidy?

My question is similar to this one, but framed in the context of survey data. I'd like to format data from a survey in a tidy manor where some questions are yes/no answers and others are numerical ...
1
vote
1answer
18 views

Does this shape one cluster? and why angles change every time i run the code?

I have data and tried to do clustering on it. every time I run the code with the below statements it changes the angle of the shape but still the same below shape ...
0
votes
1answer
23 views

Is it ok if I log/square root transform my variables and then scale them to perform a PCA? [duplicate]

My goal is to carry out an hierarchical cluster analysis using the principal components that explain most of the variance. None of my variables is normal and therefore I think I should transform them (...
2
votes
0answers
26 views

The impact of Normalization when training MLP

I come across a problem where I trained two MLPs using the same dataset, but one was trained using the raw data and the second one was trained using the normalized version of the dataset. In this case,...
1
vote
1answer
19 views

Rescale column values to sum exactly 100

Let's say I have a dataset with 4 columns: country, election year, party name, vote share. country party name election year vote share UK CON 2017 42.4 UK LAB 2017 40.0 UK CON 2019 43.6 UK LAB ...
0
votes
0answers
14 views

Can I perform One-way ANOVA after transormations and division?

I'm wet scientinst and not that good at math so sorry in advance. I'm performing ELISA, and after that I'm getting absorbtion and my calibration curve in order to convert absorbtion to pg/ml. (Pic1) ...
0
votes
0answers
24 views

Data transformation with and without Min-max scaling

I am trying to test Tukey's Bulging Rule or Ladder of Powers (Ref Image). I discovered that I am able to reproduce the transformation (from right skewed to less right-skewed OR left skewed to less ...
0
votes
1answer
33 views

Making sense of an actual equation obtained after Box-Cox transformation

I have recently performed and analysed an experiment and I am currently stuck on making sense of the outcome. Any help would be much appreciated. The experiment consists of a simple DC voltage source ...
2
votes
1answer
14 views

Should we apply feature transformation for test data?

I am working on a regression problem. The data contains 13 features (after performing feature selection). to some of these features, I have applied log transformation and box-cox to fix the skewness. ...
1
vote
0answers
15 views

Explained variance modelling a diff (Δ𝑦)

I have a question that I'm struggling with, and it's related with the explained variance of model that uses a "diff" as independent variable ($Δy$=$y_t$-$y_{t-1}$) with the following form: $...
1
vote
0answers
24 views

Data Transformation for Non-linear Feature in Linear Regression

I am new to various types of non-linear data transformations. I am sorry if this question is too basic for experts. I read (https://stattrek.com/regression/linear-transformation.aspx) that there are ...
1
vote
0answers
13 views

When do I use correlation and entropy for Feature Selection?

I am wondering when to use correlation and entropy to select features from the dataset. I understand correlation and it is used to see how correlated two variables are and it is only used between two ...
0
votes
0answers
21 views

Finding the minimal sufficient statistics for this family

Let $X_i\big|_{i = 1...n}$ be random sample from the PMF: $P(X_i = 0) = \frac{1-\theta}2;\;P(X_i = 1) = \frac12 ; P(X_i = 2) = \frac\theta2$ where $\theta\in(0,1)$. Find the minimal sufficient ...
0
votes
0answers
5 views

How to set up data for Neural Network when individuals in study have different number of data points?

I have a situation where I have data on individuals. Let's just say I have data on each individual's running performance during a month. So for each run, I record things such as, distance, location, ...
0
votes
0answers
10 views

$T$ be minimal sufficient statistics. If $T = H(U)$ where $U$ has the same dimension as that of $T$, is $U$ minimal sufficient? [duplicate]

Suppose $T$ is minimal sufficient statistics for a family of distribution. If $T = H(U)$ where $U$ has the same dimension as that of $T$, then does it imply that $U$ is also minimal sufficient? I ...
1
vote
0answers
12 views

is log1p the “correct” way of doing log scale transformation of charts? [duplicate]

when transforming data to log scale for charting purposes, is it more "correct" in some way to always transform using log(x+1) (henceforth referred to as ...
0
votes
0answers
29 views

Name for generic quantile transform

Suppose I have two real-valued, continuous random variables, $A$ and $B$, with corresponding cumulative distribution functions $F_A, F_B$ with support $\text{supp}(A), \text{supp}(B)\subseteq \mathbb ...
0
votes
0answers
19 views

Dealing with skewed data but cannot use log-transformation

It seems that the popular solution to dealing with skewed data is to apply log-transformation. But in my case, the data is a rating score (range form 0-5). The distribution of the data looks like ...
0
votes
0answers
3 views

How to apply multiple transformation in the multilinear regression

I have one dependent variable (y) and two independent variables (X1 and X2). I noticed that the relationship between y and X1 is a linear relationship and doesn't need any transformation. On the other ...
0
votes
0answers
21 views

Invent first, find its use later

The typical pipeline in ML is Find a data-related problem that you want to solve Build a model or algorithm that feeds on data related to the problem to try to solve the problem Check if the solution ...
0
votes
0answers
14 views

How to calculate coefficient of variation on cube or square root transformed data?

I transformed data to be normal using a cube root transformation. When I try to calculate the coefficient of variation I have tried back transforming the mean and sd (i.e., raise it to 3 "x^3&...
1
vote
0answers
10 views

Why do Mosteller & Tukey use 1/6 to start their “flog” (logit) transformation

Mosteller & Tukey (Data Analysis and Regression) recommend transforming counted fractions with the flog transformation: $$ \frac{\log{(n)}}{\log{(m - n)}}. $$ ...
0
votes
1answer
21 views

Non normal residuals even after transforming data and generalised linear models

I have data where my x is a categorical variable that I have used dummy variables for (I have 4 categories as my dependent variable) and my y is a continuous variable (height). Edit: the independent ...
0
votes
0answers
13 views

Is Data Normalization absolutely required for training a neural network? [duplicate]

I am relatively new to Deep Learning and wanted to implement a Variational Autoencoder for images. For data preprocessing I only rescaled the pixel values from the range of [0..255] to [0...1]. The ...
0
votes
0answers
11 views

arima concept clarification on differencing term d

I have a simple question regarding the d term in the arima(p,d,q). I understand when d = 1, we are essentially differencing the time series by X(t)-X(t-1). however I am confused when d=2 please help ...
1
vote
0answers
24 views

Transforming y and x-axis

I have a model that predicts frequencies. True frequencies are close to 1 therefore the frequencies that predicted are mostly close to 1. Sample frequencies are: ...
0
votes
0answers
9 views

How should interpret the irf derived from a VAR in differences?

For example, I have two series: price and sales and they are non-stationary. I took the first difference of them to make them stationary. I then use the differenced sales and price to set up a VAR(p) ...
0
votes
0answers
16 views

How to Evaluate model's r-square after inverting from logarithmic

I stacked with question about inverting the r-square() model value after taking log1p(). My baseline LinearRegression model ...
0
votes
0answers
27 views

How do I interpret a simple linear regression model when both dependent and independent variables are square root transformed? [duplicate]

Overview I built a simple linear regression model to understand if Universal Healthcare Index predicts suicides. My independent variable is Universal Healthcare Index (scale from 1 to 100). The ...
1
vote
1answer
35 views

Does logit transformation of information entropy values make sense?

I have a vector of information entropy values that range between 0 and 1 which I want to explain with some explanatory variables. I realized that the distribution of the entropy values in my dataset ...
0
votes
0answers
16 views

Change in the coefficient of determination (R2) when multiplying dependent variables by the independent one in linear regression

I have three independent variables {x_1,x_2,x_3} that I use to fit to a dependent variable y using an OLS regression. The coefficients of determination (R2) of the variables in relation to y are: R2(...
0
votes
0answers
18 views

Label-encoding nominal variables

I am aware of the practice that label encoding is preferred for ordinal variables while ...
0
votes
0answers
34 views
0
votes
0answers
10 views

Box-Tidwell meaning of score-statistics

so I have I've tried to do a box-tidwell in R, which returns a lambda value of 0.34. Now I get that, this says I should transform my predictor variable with x^0.31. However, I'm not too sure how to ...
0
votes
0answers
47 views

Can't transform data into a normal distribution

We have data on the counts of a certain disease in our city. We're trying to see if there's a difference between the male and female instances of it. We have decided to t-test for this, but both of ...
1
vote
1answer
14 views

Un-standardising data with known minimum value

If you have a vector of standardised data and don't have access to the original vector. I know that generally if you don't have any information on the original data (mean and sd) you can't reverse the ...
0
votes
0answers
16 views

Interpretation of level-log when the outcome is binary [duplicate]

I know that textbooks state that when conducting level-log regression, I should generally divide the estimated coefficient by Beta / 100 for a percentage point interpretation. For example, a 1 % ...
3
votes
1answer
31 views

P-value does not change by transformation/standardization

In terms of regression model, I have read a statement that transforming/standardizing variables does not change its p-value as long as we keep the same model. I recently transformed a variable (took ...
0
votes
0answers
13 views

Independent variable as a proportion or percentage in ordinal logistic regression

I'm trying to run an Ordinal Logistic Regression model using the polr function from the MASS package in R. I have a DV with 4 levels and 3 IV's out of which 1 is in terms of proportions and the other ...
1
vote
0answers
23 views

How to pass an array of variable length to the input of the neural network?

I have a bunch of two-dimensional points that I want to feed into my neural network as an input. Those points are positions of the visible obstacles around my agent. The main challenge is that the ...
1
vote
0answers
33 views

How can I compare an expected value to log transformed data?

For a school project I need to show that the data I was given does not match the expected value found in literature. My data is the diametre of 150 particles and the distribution is pretty close to ...
0
votes
1answer
33 views

What is the l1-normalization of some data?

From this page and in this paper (first paragraph of chapter 2.1) there is the term of "$l_1$-normalization" or absolute normalization of a vector (i.e. some data). The scope is to turn the ...
0
votes
0answers
14 views

rationale and consequences of data transforming

I have been using statistics (traditional null-hypothesis testing, publishing p-values in our reports, etc) for years, but I admittedly mostly follow 'kitchen receipts'. I have problems understanding ...

1
2 3 4 5
42