Structured data files in any format, collected together with the documentation that explains their production or use.

learn more… | top users | synonyms (1)

-1
votes
1answer
22 views

convert categorical data into numerical data? [on hold]

I am doing project for post graduation....project is document clustering.in the project raise the problem is the text data convert into the numerical data?
3
votes
1answer
35 views

Getting an Excel file with many sheets into R [on hold]

I am importing data from an Excel file that contains 15 sheets. Each sheets represent a year in which there are observations on the same variables. Here is what I would like to do (elgantly) from R: ...
1
vote
0answers
15 views

Using single line chart with multiple, unrelated, data sets?

I would like to visualize two unrelated sets of data using a single line chart to check for any "correlation" (it's just for fun - nothing that claims to be statistically accurate). Just as an ...
1
vote
0answers
30 views

Slope analysis where one group is not normally distributed

I am currently doing analysis of five separate drug groups and their changes in levels of prescription over time. Having done linear regression on excel I am comparing the 'steepness' (m) of the ...
0
votes
0answers
5 views

trying to figure out how to make a suggestion given some input

I want to be able to express as a value (say 1-10 as an example), the strength of a suggestion that a person should do X tommorow. The rules are they should only do X in any 6 consecutive days out of ...
0
votes
0answers
31 views

Is my data ordinal or interval? [closed]

In the experiment, participants were given a score between one an ten for each tense (past, present, future) and mean total scores were then calculated for each tense (each tense had multiple ...
0
votes
0answers
19 views

Sparsity of the matrix

We got a matrix of 500 users and 30 tracks. This matrix is complete full (it means every user rated explicit all 30 tracks). Every row is a combination of user id, music id and rating. Every user + ...
0
votes
0answers
12 views

Is there any program that can simulate streams of hash-tags sets?

More specifically I need a stream of fictional hash-tag sets more like the one that twitter emits. I am aware that twitter lets you use 1% of its data. However, that won't be useful for testing ...
0
votes
0answers
15 views

Multiple time points, relation between quantified change in environment and sample population

For a current research, I'm trying to view a relation between a country-level variable index and the financial structure of companies in that country over time. I have annual data for four years, ...
0
votes
0answers
29 views

How to use a Likert-type scale in this case? [migrated]

I am very interested in a survey employing a Likert type scale in my research. I have come across the Rosenberg Self-Esteem Scale. I would like to use this scale to get a sense of how subjects feel ...
2
votes
1answer
40 views

Validation of machine learning algorithms implementation

What are some techniques by which we can verify that a current implementation of a machine learning algorithm is correct? Is using the results of a benchmark dataset is enough?
0
votes
0answers
9 views

combining samples from different gamma distributions normalized by means

I'm trying to figure out what to do with some experimental data from a psychoacoustic experiment. The data come from multiple subjects and are approximately gamma distributed; I am treating them as ...
0
votes
1answer
17 views

Designing an experiment for a marketing campaign using Incremental Response Modelling

I have the following hypothetical question, can anyone provide some clarification? I'm looking at designing an experiment or modelling what steps can be taken to maximise the Net Incremental Revenue ...
0
votes
0answers
16 views

Statistically adjust data

I am wondering if there are any existing techniques to support adjusting a data variable (e.g., price) using another data variable (e.g., market price). As we know that price for a product is usually ...
0
votes
0answers
12 views

scoring/predicting for new observations

I have two data sets of variables where one of them - the new observations - has no dependent variable. The data set without a dependent variable has around 20 times the number of records. ...
1
vote
2answers
47 views

Constructing a data set for visualization

I am enrolled in a data analysis course and part of a course, I need to find a time series data set and generate visualizations and predictions based on my understanding of the findings. Instead of ...
4
votes
1answer
41 views

How to make an effective sampling from a database of text documents?

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text ...
0
votes
0answers
31 views

Prove or disprove hypotheses [duplicate]

[removed, apparently its duplicated question..]
1
vote
0answers
28 views

Book reading list for Time Series data analysis [duplicate]

I'm looking for book recommendations for Time series Data Analysis. All my time series data is in the engineering domain, pressure, volume, voltage, images etc. I'm experienced in Matlab and mostly ...
1
vote
0answers
24 views

Quantifying data completeness in healthcare

In healthcare data, there are often situations when we don't know how complete or incomplete is the dataset for a given patient population. Let's say for example, there are 10,000 patients in the ...
1
vote
0answers
27 views

What is best practise for dimensionality reduction in rows of data

I was wondering what was best practise for dimensionality reduction in observations (as opposed to features) in a data-set? I often have data comprising of a multiple, random number of observations ...
2
votes
1answer
100 views

simulating data with a lot of predefined constraints in R

how to simulate data (in R) to generate , sample values 1) variables with specific correlation values for a particular model AND 2) with predefined regression coefficients? 3) Can we also set the ...
2
votes
0answers
28 views

Creating a copy/simulation of a dataset respecting statistical properties?

I often need to work with and analyse electronic medical record data which cannot be moved from the servers in the organisation where it resides, due to ethical reasons. Is anyone aware of a method or ...
0
votes
1answer
16 views

How to weight data set for training?

I'm trying to build a training set for a classifier. A vector evaluates to either conclusive 'C' or unconclusive 'U'. ...
0
votes
0answers
32 views

How to fit a Gaussian Mixture Model to data with correlated errors?

I'm restating this question in the hope of getting more interest. The usual function for scoring a Gaussian mixture model assumes independent measurements. But what if we have correlated measurements? ...
0
votes
1answer
54 views

Converting Annual GDP Growth Rate into Monthly GDP Growth Rate

I am researching the effects of Euroarea fiscal stimulus on inflation expectations during the crisis. The data for fiscal stimulus is given as an annual percentage of GDP, e.g. 1.5% of Euro Area GDP ...
1
vote
0answers
42 views

Dimension reduction for discrete qualitative and aggregated variables

I know about PCA for multiple dimensions of continuous features but here is a problem I have some trouble to find a method for. I don't have a list of individual countries but rather a discrete ...
0
votes
1answer
16 views

Why are there testing factor levels in a pure training set?

To illustrate I use the segmentationOriginal dataset from the AppliedPredictiveModeling package: ...
0
votes
0answers
19 views

How to remedy seasonality in a multiple regression model

I am trying to build an economic model using multiple regression, and I am not sure how to remedy seasonal effects. I am collecting data across several different variables, and building three models ...
0
votes
0answers
8 views

Generate Correlated Outcomes Between Groups

For a simulation study, I want to generate an outcome (Y) for 3 groups (Control, Treatment1, and Treatment1). The outcomes for Control and Treatment1 are correlated by .50; the outcomes for Treatment1 ...
0
votes
0answers
22 views

Linear Regression Data Transformation

My regression model has pulse rate difference (continuous variable) as the DV and Body Mass Index (BMI, continuous variable) and Level of Exercise (low, moderate, high, categorical variable) as the 2 ...
0
votes
1answer
56 views

How To Remove Outliers Properly From Simple Dataset

Say I have a simple data set that describes height of males in a classroom. I decide that whatever value falls 3.29 standard deviations away from the mean is actually an outlier. When removing these ...
0
votes
0answers
24 views

Finding a Multiple Response Linear Regression Data Set

I'm looking for a dataset where one can apply multiple response linear regression. Ideally, this dataset would have a large number of responses and predictors relative to the sample size, and both ...
1
vote
1answer
33 views

How to create scaled scores using Likert Data with N/A options?

I have 7 survey questions that I asked students about regarding their online learning experience. Each question is on a 5 point Likert scale including -1 for not applicable. Now that I have the ...
5
votes
0answers
46 views

Problems with Outlier Detection

In a blog post Andrew Gelman writes: Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by ...
1
vote
1answer
105 views

Fixed effects or Random effects model?

I am trying to understand the difference between fixed and random effects modelling. The panel data I have is in the form of basic longitudinal panel time series. I know that I can use the Hauseman ...
0
votes
0answers
15 views

combining 4 sets of data, which are collected under different condition during time

I have four sets of event based data, which are collected during time. I did ANOVA test and realized that four sets are significantly different. I would like to find a relationship between ...
1
vote
1answer
42 views

Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few ...
4
votes
0answers
40 views

Best practices for documenting a data-science pipeline

Even though I try to keep it as simple as possible, the pipelines for some of my data science projects get rather complex. At some point it becomes necessary to document this pipeline so that someone ...
0
votes
1answer
31 views

IBM Synthetic Data generator

I am implementing some algorithms in association rule hiding. I need to apply these algorithms in different data sets. I searched for IBM synthetic data generator through Google but I could not find ...
0
votes
0answers
10 views

longitudinal data, functional data, high frequency data

I know what is longitudinal data and functional data. But I really do not know what's the exact definition for high frequency data (for example in finance). If high frequency just means the denseness ...
2
votes
2answers
34 views

Best way to visualise impact distance against launch speeds and angle

I am modelling the trajectory of a projectile, incorporating atmospheric drag. I am calculating the distance from the launch point for when the projectile impacts the ground. I have generated this ...
1
vote
1answer
29 views

K means and hierarchical

I have some insurance data - 5,00,000 records with around 50 variables :categorical,continuous and binary variables.Some of my categorical variables have 30+ categories.I am trying to build clusters ...
0
votes
1answer
30 views

Where do I get open data on open source projects? [closed]

I'm trying to find out how many open source projects related to hydrological/environmental modelling are out there, and derive some basic information (e.g. total number, deployments per year, etc.). ...
0
votes
0answers
18 views

applying statistics in “filling in” a data set?

Let me first say that my professional role is that of a mid level OLAP/OLTP developer (lesser experence with OLTP). Ive only had business statistics in school many years ago and I am currently taking ...
1
vote
0answers
15 views

Filtering noise in a data set

I'm trying to filter out noise from a Sonar. The idea is that the sonar in aimed upwards, after an object has come in it's range, I want to be able to tell if the object is moving away or coming ...
0
votes
0answers
11 views

Intrinsic topology and metrics… (looking for name of a method)

Suppose I have an n-dimensional dataset and its points are roughly in the shape of an n-dimensional horseshoe or something along those lines. Using euclidian distance might be a bad idea, since points ...
0
votes
0answers
90 views

R subsetting and factor levels

I was wondering why it is that when subsetting a dataframe, the new frame retains the factor levels of the parent dataframe. For example, consider an arcaheological data set that includes ...
2
votes
0answers
30 views

Data set structure

I'm having problem to understand how my dataset should be organized. I have unbalanced panel data with observations of products' prices as independent variable. Data is from a 3-year period. ...
0
votes
0answers
26 views

How much can I restrict my data with outliers?

I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case. I have a dataset that I'm analyzing where in order to achieve "good" results, the data ...