All Questions
1,365 questions
1
vote
3
answers
3k
views
How to use LOF for outlier detection as I have training and test dataset?
I want to use the Local Outlier Factor (LOF) algorithm for outlier detection but it simply finds outliers on unlabed data as whole and you do not need to have a training and test set. However in my ...
- 1,677
2
votes
1
answer
3k
views
Any out source outlier (anomaly) detection package for Weka?
There is KDD'98 cup data waiting for me to run some anomaly detection algorithms but Weka does not include any of them, natively. Is there any 3rd part package that can be integrated to weka?
- 1,677
6
votes
3
answers
5k
views
Which data mining packages support anomaly detection?
I aim to have some anomaly detection process on my data but Weka, Rapidminer or Knime do not support anomaly detection algorithms. How would I take care of the process?
7
votes
1
answer
2k
views
How to test for outliers in an mlogit model in R
I am running a multinomial logistic regression using the mlogit package and mlogit function in R. Now I need to check for ...
- 689
5
votes
1
answer
2k
views
Robust outlier detection in curve fitting with correlated errors
Assume I have data originating from a model
$$
y_i=f(t_i)+e(t_i)
$$
with $f\in C_2(\mathbb{R})$.
The only thing I know about the errors is that they roughly happen to be of to different sources:
...
- 368
1
vote
0
answers
2k
views
Unsupervised anomaly detection with factor analysis (in R)
The basic idea i'm trying is to model the data with factor analysis, assuming a latent variable structure that underlies the observations. Labels for "real" anomalies are available and used for ...
- 273
2
votes
0
answers
2k
views
Standardising the removal of outliers from a small data set
I have been wading through the many discussions on outliers on this site but I am still unfortunately having difficulty determining what to do with my data set.
My study consists of a simple pre-post ...
1
vote
1
answer
295
views
Dataset and outlier question
I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule
targets datasets of continuous (not necessarily ...
- 23k
6
votes
1
answer
1k
views
Identifying fraudulent questionnaires
Questionaires are often used in social sciences.
Many people try to complete them very quickly and very often they only "guess" answers.
Is there any statistical technique or any research in this area,...
- 3,979
2
votes
1
answer
52
views
If I have good reason to expect a certain value as the sample mean and I obtain something very different, should I obtain another sample?
In my specific case, I am measuring the mean of a sample for different values of a factor (let's say the possibilities are integers from 1 to 100). I have very good reason to expect the mean of each ...
- 677
6
votes
3
answers
6k
views
How to add outliers to an existing data?
I want to test few similarity measures for outlier detection. I've got some data from UCI repository, for example: Breast-Cancer.
Is there a smart way to add artificial outliers to an existing data?
...
- 487
4
votes
1
answer
4k
views
RandomForests in Matlab and outliers detection
I am solving some regression problem with RandomForests in Matlab, using it's default TreeBagger class for this task. While I managed to get reasonable result already, there are few questions which I ...
- 501
3
votes
2
answers
2k
views
Detecting initial trend or outliers
In my test procedure I sequentially take 10 measurements of a recently perturbed physical system, and I often find the first few (between 0 and 4) measurements can be inaccurate because the system has ...
- 33
3
votes
2
answers
2k
views
Dixon test and normal distribution
I hope Michael Chernick will read this question.
I have applied Dixon test to 100k rows. Rows are like-
1 1.819691 2.565696 3.317881 1.491987 ...
- 227
28
votes
4
answers
196k
views
Detecting outliers using standard deviations
Following my question here, I am wondering if there are strong views for or against the use of standard deviation to detect outliers (e.g. any datapoint that is more than 2 standard deviation is an ...
- 1,165
15
votes
6
answers
34k
views
Is there a simple way of detecting outliers?
I am wondering if there is a simple way of detecting outliers.
For one of my projects, which was basically a correlation between the number of times respondents participate in physical activity in a ...
- 1,165
0
votes
1
answer
250
views
Difference between different methods of outlier determination?
how to calculate Cooks distance. How is this method of determining outlier different from the quartile method.
- 1
1
vote
0
answers
195
views
Spotting anomalies in time series [duplicate]
Possible Duplicate:
Simple algorithm for online outlier detection of a generic time series
I've got data on quantities sold and average price, by date, for a number of commodities and a number ...
- 111
3
votes
0
answers
245
views
Detecting outstanding events [duplicate]
Possible Duplicate:
Simple algorithm for online outlier detection of a generic time series
Observing the time series data I noticed there are some outstanding peaks (the picture below). I would ...
- 1,343
2
votes
0
answers
264
views
Data fitting with repeated measurements
we have an experiment to excite a system with some energy, then measure the decay as a function of time. We are measuring data at 4 times $t$ to fit to an exponential model: $y = a \exp(-t/p) + c$, ...
- 81
7
votes
2
answers
398
views
Algorithim to determine if point is "too far from the average"
Long story short I have a collection of about 30 scripts that manipulate data sets and place them in a database. These scripts report their running times as well as any errors that occur to a ...
- 191
1
vote
1
answer
137
views
p(x) in an anomaly detection system gives values greater 1 [duplicate]
Possible Duplicate:
Probability distribution value exceeding 1 is OK?
I followed the machine learning course by Andrew Ng and decided to implement an anomaly detection system for one of my ...
- 113
14
votes
3
answers
5k
views
STL on time series with missing values for anomaly detection
I am trying to detect anomalous values in a time series of climatic data with some missing observations. Searching the web I found many available approaches. Of those, stl decomposition seems ...
- 141
3
votes
1
answer
980
views
Is it ok to clean up data from different conditions independently?
I have a dataset, which contains measurments from many different conditions. Since my hypothesis suggested a large difference for the measurements in each condition, in order to clean the data, I ...
- 2,671
7
votes
1
answer
250
views
Is Ye Shiwen's 400m IM performance in Olympics "anomalous" statistically?
The 16-year-old Chinese swimmer Ye Shiwen swam more than 5 seconds faster than her personal best in women's 400m individual medley in London 2012 Olympics, winning her a gold. Her performance received ...
- 173
4
votes
2
answers
5k
views
Outlier detection for heavy-tailed data
Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range ...
- 233
6
votes
3
answers
486
views
How to tell how extreme an outlier is?
I am analyzing some data and want to look at one particular point and see how "extreme" it is.
Do I exclude this outlier from the data, calculate the dataset's standard deviation and average, then ...
- 213
5
votes
1
answer
4k
views
Data cleansing in regression analysis
I've got a dataset for Temperature & KwH and I'm currently performing the regression below. (further regression based on coeffs is performed within PHP)
...
3
votes
1
answer
2k
views
Examples of Lurking Variable and Influential Observation
I have read possible explanations for Lurking Variables and Influential Observations but I can't seem to construct a good example for myself.
A well-designed experiment includes design features that ...
user8968
2
votes
3
answers
683
views
Filtering techniques and noise
Suppose we have some house price data for 30 years (1970-1999). This is yearly data (30 data points). Suppose some major event $X$ happened on 1980. I want to see whether this event affected prices ...
- 773
1
vote
1
answer
487
views
Anomaly Detection Algorithm versus "just doing some statistics"
I've proposed using an anomaly detection algorithm in a project.
The algorithm would consist of choosing some features we think might be indicative of anomalous examples. Then using a training set ...
- 837
2
votes
3
answers
279
views
Removing outliers from newspaper content analysis
This has been driving me crazy, so I thought I'd come and ask for some help:
I'm doing a study involving content analysis of newspapers. The unit of analysis is each day, adding all stories for the ...
- 21
29
votes
2
answers
5k
views
In what order should you do linear regression diagnostics?
In linear regression analysis, we analyze outliers, investigate multicollinearity, test heteroscedasticty.
The question is: Is there any order to apply these? I mean, do we have to analyze outliers ...
- 295
10
votes
4
answers
11k
views
How to fit a model for a time series that contains outliers
I have fitted ARIMA(5,1,2) model using auto.arima() function in R and by looking order we can say this is not a best model to forecast. If outliers exist in the ...
- 375
2
votes
1
answer
746
views
Treatment of outliers in annual time series data
I have an annual time series of data of a growth-rate variable $X$ for 50 years. Most of the values for the variable $X$ are less than 10%. The exception are two values that are around 30%.
How do I ...
- 21
4
votes
1
answer
198
views
Estimate density from neighbor distances
Assuming I have a data set of known size, and there is one object that I want to test for being in an approximately uniform distributed region of the data set.
For the query object, I know the $k$ ...
- 42.9k
4
votes
3
answers
3k
views
Univariate clustering of time series
I just want to know if its possible to cluster an univariate time series, in order , say, to detect anomalies?
and do you have any online version for denstream code, in Matlab?
here is the time ...
- 57
8
votes
1
answer
3k
views
Using MAD as a way of defining a threshold for significance testing
If I have a set of terms each term having a particular frequency associated with it (the number of the times the term has appeared in fixed corpus of papers), then is the following method of ...
- 1,099
11
votes
6
answers
9k
views
Identifying outliers for non linear regression
I am doing research on the field of functional response of mites.
I would like to do a regression to estimate the parameters (attack rate and handling time) of the Rogers type II function.
I have a ...
- 111
4
votes
1
answer
3k
views
C++ library to play with statistics (detecting outliers in time series)
I am looking for a C++ library for statistics to play with outliers detection in time series (amongst other).
What I need:
Robust estimators, correlations, hypothesis tests, etc;
No dependencies ...
- 141
6
votes
2
answers
799
views
How to judge if a datapoint deviates substantially from the norm
This is Statistics 101, but I'm not a statistician and so can't seem to find the right technical jargon to google.
My company collects data at discrete points through time. Today's datapoint is ...
- 353
5
votes
1
answer
9k
views
Mahalanobis distance distribution of multivariate normally distributed points
I'm trying to understand the properties of Mahalanobis distance of multivariate random points (my final goal is to use Mahalanobis distance for outlier detection). My calculations are in python.
I ...
- 737
4
votes
2
answers
266
views
Detecting fishy data
Just a little thought I've been having. If we rolled a fair dice 60 times, and got 60 sixes in a row, we would (wrongly?) definitely assume that something fishy's going on. Is there any statistical ...
- 463
3
votes
2
answers
6k
views
Improving SVM performance on data with missing features and outliers?
I'm trying to learn R for ML purposes, and right now I'm building classifier for my data (10 dimensions, ~400 elements, 2 classes), which have some outliers in it, and a lot of missing values.
I'm ...
7
votes
2
answers
2k
views
Tests for univariate outliers: have Dixon's and Grubb's methods been discredited?
In contrast to the many threads on this site that recommend Dixon's and Grubb's tests, the author of one answer, at this thread, contends that "Really, these have been discredited long ago" and ...
- 12.9k
1
vote
0
answers
288
views
Dixon test for outlier but which one is an outlier?
Following my previous question, I used Dixon test for outliers with the help of Michael Chernick answer. So now I have pvalues for say 10 numbers (basically 10 patients). But I have around 50k pvalues ...
- 227
0
votes
1
answer
196
views
Determine whether a number "fits" with a group of numbers
I need to determine if a single number "fits" with a group of numbers. All the numbers will be percentages, so we'll use decimals.
For example:
The group is: ...
- 103
3
votes
1
answer
251
views
Statistical test for finding significant positions having deviated values
I have near about 50 files(each file corresponds to a patient) with 4 columns -
chromosome start.position stop.pos value
First 3 columns in all 50 files are same and fourth column is ...
- 227
5
votes
2
answers
167
views
Looking for aberrations in time based data
Looking at IO latency data for a storage array. When a drive is about to fail, one indication is an increase in the IO operations 'time to complete'. The array kindly provides this data in this format:...
- 51
0
votes
1
answer
2k
views
Identifying a number of significance from a histogram
I'm creating a program that identifies trending music artists that uses Twitter metrics. I have data in a histogram format that represents the frequency of twitter @mentions of an artist for the last ...
- 71