Questions tagged [data-transformation]

Mathematical re-expression, often nonlinear, of data values. Data are often transformed either to meet the assumptions of a statistical model or to make the results of an analysis more interpretable.

Filter by
Sorted by
Tagged with
1 vote
0 answers
9 views

Efficient storage of functional data

I have access to a sample (size $N$) of functional data. Each observation corresponds to $C$ functions. Each function $f_{n,c}$ is represented by $T_n$ points for $1\geq n \geq N, 1\geq c \geq C$. All ...
user avatar
  • 2,049
0 votes
0 answers
13 views

Why variable representation plays a role in prediction?

I am working on binary classification using a random forest, where the data have 977 records and 6 columns. The class ratio is 77:23. I have two derived input variables. One variable is called ...
user avatar
  • 1,780
0 votes
0 answers
18 views

Can a transformed variable's SE be meaningfully interpreted?

Suppose, for simplicity, I have a simple linear regression model, and I have transformed the response variable by taking the square root. The pre-transformed standard error is equal to 17, and the ...
user avatar
1 vote
0 answers
10 views

Scaling a single continuous variable

I have a basic question. I know you have to scale your variables (fixed effects) when you have several continuous fixed effects, to be able to compare them. However, if I only have one continuous ...
user avatar
  • 11
0 votes
1 answer
30 views

Separating datasets vs one dataset with extra categorical feature

I have regression/classification problem. Dataset contains data from 4 sensors on 4 positions (1,2,3,4). Processes measured on all 4 positions are equivalent and same label and features describe all 4 ...
user avatar
  • 3
0 votes
0 answers
17 views

Mixed ANOVA - Can We Standardize After Box-Cox Transformed Values

in our mixed ANOVA (2 timepoints, 3 interventiongroups, so 2x3 factorial ANOVA), the homogeneity of the residual variance was violated as indicated by significant levene tests, for the post-...
user avatar
  • 61
0 votes
0 answers
13 views

Meta-analysis with ratio data

I am currently performing a meta-analysis where the outcome of interest is the ratio between the concentration of two sugars in urine/plasma (i.e. the results of dual-sugar test; see: https://doi.org/...
user avatar
0 votes
0 answers
13 views

Moving columns under other columns in R [migrated]

I have to work with an existing dataset and want to move the content of some columns (in this example the content of columns F to J) under the content of columns A to E so that the length of the ...
user avatar
0 votes
0 answers
13 views

Regression with independent variables as percentages [duplicate]

I'm trying to run a regression analysis where my dependent is number of deaths and some of my independent variables are things like: poverty rate (Ex - 0.16, 0.09), incarceration rates (Ex 0.0042, 0....
user avatar
  • 1
0 votes
0 answers
23 views

Which method of logarithmic transformation is correct to use in linear model?

I try to do a simple linear model with logarithmic transformation of y values. I found that depending on which method I use the results differ and I don't ...
user avatar
0 votes
1 answer
21 views

1 Stationary time series, 1 non stationary: do I need to transform BOTH, OR can I use VAR with 1 transformed and 1 stationary variable?

I am doing a time series forecast using VAR. I have 2 time series, "orders" and "calls" The orders time series is stationary The calls time series is non-stationary Let's say I use ...
user avatar
0 votes
0 answers
36 views

what data transformation function will reverse the order in the original data?

I want to apply a data transformation function that produces value between 0 and 1 and also the order is reversed. The max becomes min after transformation and the minimum becomes max value. Assuming ...
user avatar
0 votes
0 answers
5 views

Does combining train and test data introduces any potential bias?

Given an estimator $f$ and a dataset $D$ with $k>2$ classes where $S_{train}$ denotes the train set and equivalently $S_{test}$ the test set. Suppose we want to transform the problem and instead of ...
user avatar
1 vote
2 answers
120 views

transformation of a kernel density estimate to uniform distribution

I am interested in estimating the expected value of a function, $f(x)$ with respect to a probability density function, $P(x)$. I am exploring a method that requires I change variables from the ...
user avatar
2 votes
1 answer
49 views

uniformly sample from gaussian distributed data

I have data that is roughly gaussian distributed, bounded on a range of [x0, x1], w/ mean m and stadard deviation s. I want to ...
user avatar
3 votes
1 answer
65 views

E(log(x)) to E(x) [duplicate]

Sorry if this is a straightforward question, but I have tried digging into econometrics book and cannot find anything about it. I worked on a model with ...
user avatar
  • 41
0 votes
0 answers
30 views

How to use RFM along with Predictive model?

I am looking to segment my customers based on their transactional data with us. Based on google search, I came to know about RFM matrix, which records Recency, Frequency and Monetary value-based ...
user avatar
  • 1,780
0 votes
0 answers
21 views

Rule based label - For attrition risk

I have 3 domains of supplier data (Jan 2017 to Jan 2022) and they are as follows a) Purchase data - Contains all the purchase (of product) data made by the suppliers with us. It contains columns such ...
user avatar
  • 1,780
2 votes
1 answer
36 views

Why logit transform explanatory variable restricted between $0$ and $1$ only if it is close to $0$ or $1$?

In Bayesian Data Analysis 3rd edition chapter 14 on linear regression they write Since the explanatory variable is restricted to lie between 0 and 1 (recall that we have excluded uncontested ...
user avatar
0 votes
0 answers
13 views

fit transform so that variance is independent of mean as much as possible when only ranks are available

set.seed(0) n=10000 x=rnorm(n) m=10 y=replicate(m, x+rnorm(n)) scaledrank=function(x) { n=nrow(x) apply(x, 2, function(v) rank(v)/(n+1)) } X=scaledrank(y) ...
user avatar
0 votes
0 answers
25 views

Why does my data look the same after log, root, Box-Cox transformations?

I have to forecast the amount of cars sales for the next 12 months. The data I have gathered are from 2013-2021 (108 months). This is what the plot of my data looks like using Rstudio and its ...
user avatar
  • 21
0 votes
0 answers
15 views

How to optimize transform function to make the variance and mode of the variance roughly stable?

$ curl -s https://i.stack.imgur.com/rl1eT.gif | tail -c +43 | zcat x y x2 2030667 x2 2343967 ... I have data like the above. If you compute the mean and ...
user avatar
0 votes
0 answers
41 views

How to transform a set of negative values to obtain a normal distribution? [duplicate]

I am having quite a few problems with transforming a set of data with values between -1 and 0, as I need to normalise them. I tried to use the following formulae: [(𝑥−min(𝑥))/(max(𝑥)−min(𝑥))] AND ...
user avatar
0 votes
0 answers
12 views

Calculating the variance for a complex transformed non central t-distribution

I am currently stumped and wanted to know if somebody could advise me on how to calculate the variance if, assuming $T_{\delta,\nu}$ is a noncentral t-distribution with degrees of freedom $\nu$ and ...
user avatar
  • 63
2 votes
1 answer
12 views

Extract valuable information from a list of dates for Machine Learning

I have to create a model to predict if a patient has a disease or not based on pharmaceutical prescription data. I created a neural network and a random forest using as features the number of ...
user avatar
0 votes
0 answers
26 views

Overlapping data in time series model?

I have some time series data that I am trying to model - basically historical elasticity reads with daily sales data. I really am just interested in finding the 'mean' value - not necessarily predict ...
user avatar
  • 2,513
1 vote
1 answer
31 views

How to predict an outcome of the game (next row) based on all previous games (rows)?

I'm a data science student and I've come across a fairly unusual dataset (to me, which explains the vague title). It's of the following form: STAT_1 STAT_2 ... HOME AWAY NEXT_HOME NEXT_AWAY ...
user avatar
2 votes
0 answers
52 views

What is the distribution of a Cholesky transformed variable?

I have a situation where I have a vector $\textbf{x} \sim N(\mu_x, \sigma^2_x)$ and a vector $\textbf{y} \sim N(\mu_y, \sigma^2_y)$. I want to generate a new $\textbf{y}_2$ that transforms $\textbf{y}$...
user avatar
  • 219
0 votes
0 answers
18 views

Transforming to address normality assumption (data with 0s)

I have done a regression analysis which violates normality assumption (shapiro.test yields <0.05 and plotting also shows that non-normality). Since my y has values less than 1 greater than 0, I ...
user avatar
  • 1
1 vote
0 answers
27 views

How to construct a relative price index

I have prices of different food items for three time periods (2011, 2015 and 2018) at the household level. So I have a panel with 3 time periods and around 5000 households. I want to see the impact of ...
user avatar
  • 11
0 votes
0 answers
21 views

Interpreting Scaled Continuous Variables' and Unscaled Dummy Variables' Regression Co-efficient

I have regression coefficients, Logistic Regression to be precise, of 7 continuous and 2 binary (1 or 0) flag variables. The continuous variables are scaled using just the standard deviation of the ...
user avatar
1 vote
1 answer
62 views

Poisson distribution transformation

I'm quite new to biostatistics so I apologize if my question is too dumb. I'm studying data transformation in biostatistics to fit my data to the normal distribution. I started with the Poisson ...
user avatar
  • 11
0 votes
1 answer
32 views

Box-cox transformation on dependent variable before splitting the data

I'm working on a regression problem where the dependent variable is highly skewed (See below). My idea is to use a Box-Cox transformation to make the problem a bit more well-behaved. Is it good ...
user avatar
0 votes
1 answer
41 views

Transforming skewed distribution of dependent variable in linear regression?

As I understand it, the skewness of the response variable in a linear regression does not need to be normal (only the residuals need to be normally distributed). However, I was generally wondering if ...
user avatar
0 votes
0 answers
17 views

How to make quarterly population data stationary?

So, I am trying to build the Time Series model for the quarterly population estimate for Ontario provided by Stats Canada (https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1710000901). As seen ...
user avatar
  • 101
3 votes
4 answers
950 views

Why would you perform transformations over polynomial regression?

When performing linear regression, why might you choose to transform one of your predictor variables over using polynomial regression? Essentially what are the advantages (if any) of performing a ...
user avatar
0 votes
0 answers
14 views

Formal way to test what kind of differencing is necessary?

I'm working on a project that concerns time series data for South-Africa. My series has 34 explanatory variables and only (!) 30 yearly observations. The analysis is meant to be high-dimensional, ...
user avatar
  • 1
0 votes
0 answers
34 views

Transforming different Likert scales to one common scale

For a research paper in my masters degree, it is my task to compare patient satisfaction scores for telemedicine across several studies. To compare them I am asked to plot the results on a forest ...
user avatar
0 votes
0 answers
11 views

Comparing two uneven datasets, both of which were sampled in different ways

Dataset X is small (n=100). It was sampled based on a specific variable. Say, it took data based on 3 countries. Something like this: ...
user avatar
  • 101
0 votes
1 answer
320 views

How to inverse difference time series data?

I'm preparing a time series model with LSTM, I noticed that the time series data is not stationary so I used diff(period=1) in ...
user avatar
0 votes
0 answers
12 views

box-cox transformation formula

In this page: https://www.statisticshowto.com/box-cox-transformation/ The formula of box cox is: But the transformed data, e.g., lamdha=1 is ...
user avatar
1 vote
2 answers
56 views

Can I transform a parameter's posterior to a different parametrization?

I have a model with several parameters. I apply Bayesian inference with a uniform prior for all of the parameters. After the process is finished, I realize that I need one of those parameters $x$ ...
user avatar
  • 3,124
0 votes
0 answers
7 views

Survey analysis and data handling

I have survey responses from 67 raters. They responded to 7 clinical scenarios. Each scenario had a set of possible responses which are not numeric and not ordinal. A response can be one of 10 ...
user avatar
0 votes
0 answers
16 views

Different transformations for IV's in multiple linear regression

I am trying to fit a linear regression to predict a DV from 5 DV's. Checking the assumption of linearity between each DV and the IV, I found that each DV needs a different transformation: 2 are linear ...
user avatar
2 votes
0 answers
50 views

Multivariate Gaussian transformation that preserves dependencies?

I have a dataset consisting of 2 multidimensional (multi-column) arrays $X$ and $Y$. All columns in $X$ and $Y$ are drawn from continuous distribution and are correlated between each other (the ...
user avatar
10 votes
3 answers
1k views

If a data set appears to be normal after some transformation is applied, is it really normal?

Suppose you have a data set that doesn't appear to be normal when its distribution is first plotted (e.g., it's qqplot is curved). If after some kind of transformation is applied (e.g., log, square ...
user avatar
  • 609
0 votes
0 answers
30 views

What transformations are permitted before a t-test for significance?

I have been using t-tests to test for significant differences between the means of populations. Specifically, I am trying to determine if revenues have increased significantly before and after a ...
user avatar
1 vote
2 answers
133 views

Time Series classification problem (how to format data?)

I am working on a project where a physical test over time is conducted to decide whether an object is diagnosed as class $A$ or class $B$. Typically these tests can take around 2.5-3 hours and so each ...
user avatar
  • 47
1 vote
2 answers
66 views

Transformation of a skewed sample for estimating better the mean

Given a skewed sample whose distribution is not normal and was caused by various reasons. As a result the mean calculation is affected by the skewed distribution. Can the following steps assess the ...
user avatar
  • 11
0 votes
0 answers
40 views

Training a machine learning model on data that has several rows for each user

I have a dataset consisting of log files from a smartphone application. Currently, it creates a row each time a user clicks on something, i.e. a user clicks on the homepage, and a new row is created ...
user avatar

1
2 3 4 5
45