Structured data files in any format, collected together with the documentation that explains their production or use.

learn more… | top users | synonyms (1)

2
votes
0answers
17 views

Creating a copy/simulation of a dataset respecting statistical properties?

I often need to work with and analyse electronic medical record data which cannot be moved from the servers in the organisation where it resides, due to ethical reasons. Is anyone aware of a method or ...
0
votes
1answer
11 views

How to weight data set for training?

I'm trying to build a training set for a classifier. A vector evaluates to either conclusive 'C' or unconclusive 'U'. ...
0
votes
0answers
3 views

Help with multiple landmark and Non-Metric Data INPUT [on hold]

Thank you all in advance for being so kind as to put up with my extremely beginner level R-Studio (and statistics programming in general) skills. I am struggling with the most basic of tasks-combining ...
0
votes
0answers
16 views

Can you estimate data covariance using a Gaussian Mixture Model?

I have a 3D data set with spatially correlated noise. I read this introduction to handling correlations in data and it suggested that you can use a Gaussian Process (GP) to estimate the covariance ...
0
votes
1answer
38 views

Converting Annual GDP Growth Rate into Monthly GDP Growth Rate

I am researching the effects of Euroarea fiscal stimulus on inflation expectations during the crisis. The data for fiscal stimulus is given as an annual percentage of GDP, e.g. 1.5% of Euro Area GDP ...
1
vote
0answers
34 views

Dimension reduction for discrete qualitative and aggregated variables

I know about PCA for multiple dimensions of continuous features but here is a problem I have some trouble to find a method for. I don't have a list of individual countries but rather a discrete ...
0
votes
0answers
4 views

Merging data with text identifiers [migrated]

I am trying to merge a bunch of datasets at the company level. The problem is my datasets only identify company name with a text string (i.e. "Ham and Cheese LLP") and further sometimes misspell or ...
0
votes
1answer
14 views

Why are there testing factor levels in a pure training set?

To illustrate I use the segmentationOriginal dataset from the AppliedPredictiveModeling package: ...
0
votes
0answers
17 views

How to remedy seasonality in a multiple regression model

I am trying to build an economic model using multiple regression, and I am not sure how to remedy seasonal effects. I am collecting data across several different variables, and building three models ...
0
votes
0answers
6 views

Generate Correlated Outcomes Between Groups

For a simulation study, I want to generate an outcome (Y) for 3 groups (Control, Treatment1, and Treatment1). The outcomes for Control and Treatment1 are correlated by .50; the outcomes for Treatment1 ...
0
votes
0answers
20 views

Linear Regression Data Transformation

My regression model has pulse rate difference (continuous variable) as the DV and Body Mass Index (BMI, continuous variable) and Level of Exercise (low, moderate, high, categorical variable) as the 2 ...
0
votes
0answers
5 views

How to add a split data frame with different colums on each row back into a data frame? [migrated]

I have one column in a CSV of salary data with the following types of data: Nothing in cell "£2.5-£3.5 an hour" "£4.8 an hour" "£40,000-£60,000 a year" £60,000 a year My objective is to: Split ...
0
votes
1answer
46 views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
0
votes
0answers
14 views

Finding a Multiple Response Linear Regression Data Set

I'm looking for a dataset where one can apply multiple response linear regression. Ideally, this dataset would have a large number of responses and predictors relative to the sample size, and both ...
0
votes
0answers
4 views

Prepare data for scikit-learn [migrated]

I am working on a small NLP project of authorship attribution: I have some texts from two authors and I want to say who wrote them. I have some pre-processed text (tokenized, pos-tagged, ect.) and I ...
1
vote
1answer
20 views

How to create scaled scores using Likert Data with N/A options?

I have 7 survey questions that I asked students about regarding their online learning experience. Each question is on a 5 point Likert scale including -1 for not applicable. Now that I have the ...
5
votes
0answers
39 views

Problems with Outlier Detection

In a blog post Andrew Gelman writes: Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by ...
1
vote
1answer
83 views

Fixed effects or Random effects model?

I am trying to understand the difference between fixed and random effects modelling. The panel data I have is in the form of basic longitudinal panel time series. I know that I can use the Hauseman ...
0
votes
0answers
14 views

combining 4 sets of data, which are collected under different condition during time

I have four sets of event based data, which are collected during time. I did ANOVA test and realized that four sets are significantly different. I would like to find a relationship between ...
1
vote
1answer
29 views

Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few ...
4
votes
0answers
38 views

Best practices for documenting a data-science pipeline

Even though I try to keep it as simple as possible, the pipelines for some of my data science projects get rather complex. At some point it becomes necessary to document this pipeline so that someone ...
0
votes
1answer
25 views

IBM Synthetic Data generator

I am implementing some algorithms in association rule hiding. I need to apply these algorithms in different data sets. I searched for IBM synthetic data generator through Google but I could not find ...
0
votes
0answers
7 views

longitudinal data, functional data, high frequency data

I know what is longitudinal data and functional data. But I really do not know what's the exact definition for high frequency data (for example in finance). If high frequency just means the denseness ...
2
votes
2answers
31 views

Best way to visualise impact distance against launch speeds and angle

I am modelling the trajectory of a projectile, incorporating atmospheric drag. I am calculating the distance from the launch point for when the projectile impacts the ground. I have generated this ...
1
vote
1answer
29 views

K means and hierarchical

I have some insurance data - 5,00,000 records with around 50 variables :categorical,continuous and binary variables.Some of my categorical variables have 30+ categories.I am trying to build clusters ...
0
votes
1answer
28 views

Where do I get open data on open source projects? [closed]

I'm trying to find out how many open source projects related to hydrological/environmental modelling are out there, and derive some basic information (e.g. total number, deployments per year, etc.). ...
0
votes
0answers
18 views

applying statistics in “filling in” a data set?

Let me first say that my professional role is that of a mid level OLAP/OLTP developer (lesser experence with OLTP). Ive only had business statistics in school many years ago and I am currently taking ...
1
vote
0answers
12 views

Filtering noise in a data set

I'm trying to filter out noise from a Sonar. The idea is that the sonar in aimed upwards, after an object has come in it's range, I want to be able to tell if the object is moving away or coming ...
0
votes
0answers
10 views

Intrinsic topology and metrics… (looking for name of a method)

Suppose I have an n-dimensional dataset and its points are roughly in the shape of an n-dimensional horseshoe or something along those lines. Using euclidian distance might be a bad idea, since points ...
0
votes
0answers
68 views

R subsetting and factor levels

I was wondering why it is that when subsetting a dataframe, the new frame retains the factor levels of the parent dataframe. For example, consider an arcaheological data set that includes ...
2
votes
0answers
30 views

Data set structure

I'm having problem to understand how my dataset should be organized. I have unbalanced panel data with observations of products' prices as independent variable. Data is from a 3-year period. ...
0
votes
0answers
23 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...
0
votes
0answers
6 views

How to specify “unlimited” value

How to specify "unlimited" value of a variable in my dataset for regression purposes? I have a variable which can have both finite or unlimited amount of units.
0
votes
0answers
14 views

Can linear regression model variables be constantly looped in simulations to find the perfect model? [duplicate]

Say that I have a very large dataset and have a fairly large amount of variables, let's say 30. Since I don't know which variables matter and are good predictors for regression, I construct a for-loop ...
0
votes
0answers
18 views

Partial least squares structural equation modeling: simulate dataset based on a given model

Is there any R package or other software, with which one can generate a dataset based on a given pls structural equation model? For covariance based structural equation modeling I found the simsem ...
-1
votes
2answers
137 views

can a software engineer be good at data analytics using statistics and R [closed]

I am a software engineer and new to R. I started learning but at many steps I end up learning stats concepts. So I was wondering if it's a good idea for a software developer to move to data ...
1
vote
2answers
56 views

How to normalize data without an edge

Is there some calculation method which normalizes data (numbers) if I don't have minimum and maximum edge? Only one I know is: y = (x - min-x) / (max-x - min-x). But as I said I don't have an edge ...
3
votes
2answers
117 views

How can I measure fluctuation?

For a certain product I have a data series with its price for each day over a long period of time (20 years). I want to investigate how the fluctuation of the price changes over the whole time. ...
2
votes
2answers
76 views

Calculate data distribution formula from data points

I have 50.000 pieces of data in a list. 10 random samples: 4,877; 4,165; 6,152; 5,544; 6,942; 7,409; 6,415; 5,592; 6,947; 4,965; 4,688 I'm trying to do 2 ...
0
votes
0answers
34 views

Resource for practice data analytics projects

does anyone know of resources that offer data analytics/mining projects that I can tackle to help build my portfolio and present to interviewers? Thank you.
9
votes
0answers
95 views

Are data handling errors already 'priced in' to statistical analysis?

Ok, fair warning--this is a philosophical question that involves no numbers. I've been thinking a lot about how errors creep into data sets over time and how that should be treated by analysts--or if ...
0
votes
1answer
44 views

Data analysis in a comparative way

I'm trying to make data analysis to huge amount of data of psychophysical task. Well, participant should estimate if a heared sound belong to which frequency (we have 8 sounds). So for a played sound ...
3
votes
1answer
27 views

Is it better to pre-filter the entire data set or just the training sub-set?

I am currently working on a classifier for the qualitative spectral analysis of alloys. One of the problems that I faced is preparation of samples for the classifier training. Samples have to me ...
1
vote
0answers
38 views

Correlation and coefficient with overlapping data

I have a time series data, with 5-period rates sampled at 1 period intervals. Essentially $$r_i = x_{i+5} - x_i ; ~~ i=1,...,N-5$$ This creates an overlapping data problem for regression. As far ...
0
votes
1answer
106 views

Generating text data for training for doing named entity recognition and extraction

I'm trying to build an algorithm for doing named entity extraction. It goes like this. There is a large set of text documents [communications], from which specific information has to be extracted. The ...
0
votes
1answer
51 views

How to standardize text data for training Neural Networks?

I want to train neural network with text data(natural language) as input for classification purpose. One way for standardizing text data for neural network is to use N-GRAM/SKIP-GRAM representation ...
0
votes
0answers
32 views

Analysing multiple response question

Suppose I have a survey data set, where sample size is n=50. There is a variable on " What do you like about this product? " Answers: a. Less time consuming b. Low weight c. User ...
1
vote
0answers
61 views

Multiple imputation in SPSS: merge datasets?

I have two sets of data, measured from the same persons. The variables in the two sets are very different. There are some extreme values (occurring at random) in both sets I would like to handle using ...
4
votes
2answers
28 views

Do any statistical tests compare distributions when one has significantly fewer samples?

I've been researching different methods to compare two distributions for equality, or inequality rather. I want to compare actual user performance against projected performance, after a particular ...
2
votes
0answers
157 views

Linear mixed models or not?

We did a study with a course participants where we have one outcome variable and several metrics calculated for each message they posted on a discussion forum. Basically, the data are structure in the ...