An outlier is an observation that appears to be unusual or not well described relative to a simple characterization of a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
23 views

anomaly detection with Markov chain

The paper uses a simple technique to detect intrusions in computer systems. I will briefly explain it and ask a question: The paper proposes a simple 1-order Markov chain modelling approach to detect ...
0
votes
2answers
41 views

How to downweigh outlier in a sum?

I have a simple problem. Assume following dataset: resids <- c(,9,8,7,12,14,8,9,15,4,9,10,200) n <- length(resids) p <- 2 Using this dataset I want to ...
3
votes
1answer
33 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
0
votes
0answers
17 views

In RAPIDMINER, how do I store the Outlier Score (LOF) for each process and eventually have them all in 1 result table? [closed]

I'm trying to create an iterative process in RapidMiner where I: Plot 2 attributes against each other (whilst keeping the ID for each tuple) Apply LOF operator (from Anomaly Detection Extension to ...
5
votes
1answer
78 views

Where must we use Bagging or Boosting?

I want to know when Bagging is better than Boosting? How I select appropriate method for my classification task? I think when we have many outliers in our data-set, Bagging must be better than ...
1
vote
1answer
31 views

Extreme outlier detection algorithm for erroneous latitudes/longitudes

I have a dataset with latitude/longitude of hotels of a "destination". A destination is a city neighbourhood, whole city, or small region, usually having between 3 and 50 hotels. About 1% of the ...
0
votes
0answers
12 views

Is there a consensus method for defining outliers in a data set? [duplicate]

I am working on a large data matrix and I would like to know if there is a consensus method for defining outliers in a data set? I can 'eye-ball' it on a density plot, but it would be nice not having ...
1
vote
1answer
79 views

Simple algorithm for online outlier detection of a generic time series II: Daily cycle within annual

I have several years of sensor data (temperature and relative humidity) that records every 1/2 hour. When the sensor dies, it often starts throwing bad data mixed in with good data before it dies ...
3
votes
1answer
98 views

Linear regression with violated assumptions

I am trying to find out the determinants of cognitive function. The outcome variable is the mini–mental state examination which is a 30 point questionnaire response that has score values from 0 to ...
1
vote
0answers
51 views

Should outliers in a time series be removed before or after detrending?

I am doing a classical time series analysis. When do I remove outliers in the data? After detrend or before detrend?
0
votes
0answers
11 views

Testing outlier influence on random effects in linear mixed effects models

I have been reading a little bit about diagnostics for linear mixed effects models and have started wondering about how outliers may influence random effects in addition to fixed effects. The paper on ...
2
votes
0answers
28 views

Anomaly Analysis (K-Means) - finding suspicious activities/operators

I am relativly new to the field of data mining and want to make a anomaly detection on transactional retail data. I want to use a simple anomaly detection (kmeans at the moment) for finding suspicious ...
0
votes
0answers
26 views

Outliers in Boxplots when calculating means

I have a dataset for which I am making boxplots. I do not want to include the outliers in the box plot so I give an argument outline=FALSE in the command. In the next step I want to put the mean ...
0
votes
0answers
16 views

Removing outliers in R plot function [migrated]

I'm trying to do a scatterplot in R and would love to remove an outlier I've already identified. My plot function: ...
0
votes
0answers
23 views

Identify outliers with median-absolute-deviation for timeseries data

I am having trouble understanding this particular method of detecting outliers in a time series. Below is the problem: I have a region-of-interest containing 15 voxels. Each voxel contains values ...
3
votes
4answers
219 views

Confused by location of fences in box-whisker plots

In one type of box-whisker plot, the fences at the ends of the whiskers are meant to indicate cutoff values beyond which any point would be considered an outlier. The standard definitions I've found ...
0
votes
1answer
23 views

Outliers and Influential observations in fixed effects regression

I am running a fixed effects regression with a very unbalanced panel data. There are a lot of residuals. Like for half of my observations I get large residuals. So I do not want to simply remove them ...
1
vote
0answers
47 views

Unbalanced Panel data using R - Removing outliers and heteroskedastcity

I am new in R and it’s my first time using it so I’ll appreciate the help. I am estimating income elasticity for electricity consumption using budget shares. I have data for 8 regions categorized into ...
0
votes
0answers
4 views

Removing MULTIPLE outliers in regression model in R [migrated]

this is in R ok so i've used cook distances to identify the points i would like to remove from a dataset of 506 variables that i have. i am able to remove ONE point (number 369) as follows: ...
1
vote
0answers
25 views

Identifying outliers from binned data

I have binned data (x-axis) that I've plotted against frequency (y-axis) to see the distribution of the bins and I got this scatterplot:- The bins are arranged from the lowest to the highest. As you ...
6
votes
2answers
104 views

Back-testing or cross-validating when the model-building process was interactive

I have some predictive models whose performance I would like to back-test (i.e., take my dataset, "rewind" it to a previous point in time, and see how the model would have performed prospectively). ...
3
votes
0answers
66 views

How are outliers dealt with in R after detected? [closed]

Once outliers in time series are detected in R how exactly are they dealt with before forecasting? I dont want commands to use i would like the method. Please do not give any answers to do with ...
2
votes
0answers
39 views

what to do with ridiculous but valid leverage points

So I'm having some difficulty fitting a linear model to the data (see other post here glm model fit - can't find a family/link combination that produces good fit). In particular, I'm worried ...
0
votes
0answers
16 views

remove factor level if factor level has outlier in any of other columns in R dataframe

Hello I have a dataframe with one column a factor of patient id's, other columns of continous variables. I want to remove patients from the dataframe if they have an outlier observation in any of the ...
2
votes
3answers
232 views

Putting less weight on certain data points in a series for forecasting

I have a data set that contains outliers (big orders) i need to forecast this series taking the outliers into consideration. I already know what the top 11 big orders are so i dont need to detect them ...
1
vote
0answers
32 views

Is E-Divisive with Medians (the Twitter BreakoutDetection algo) robust and efficient?

There are quite a few algorithms to detect changepoints, outliers, mean shifts, trend shifts etc. out there. Recently I've stumbled upon BreakoutDetection and while it's new and shiny I'd like to know ...
0
votes
2answers
42 views

How to show that a dataset does not contain significant outliers?

I have largish dataset: there is 200 variables and 100 samples. How could I show that the dataset does not contain any significant outliers? All variables have the same unit (millimeters) and have ...
1
vote
0answers
21 views

before clusterisation, should I remove observations with too few measurements?

I have a very unevenly distributed dataset of 462 twitter users. During the window of observation, some of these users have produced as many as 2000 tweets, while others as few as one. My end is to ...
0
votes
0answers
20 views

Fat-tailed data and SVM

Does SVM perform poorly when fat-tailed data with outliers is used? What are some things that could be done to improve learning with such data? Does the choice of kernel and/or kernel parameter ...
0
votes
0answers
25 views

Correlation analysis while detecting outliers

I have simple dataset here. Supposed I want to find out which customers who bought a certain item are more likely to come back after 10 months. I have 2 sets of data The repeat purchase % of users ...
3
votes
0answers
16 views

Regression: Should I use the prediction interval obtained given n=9 and an outlier (Cook's D= 0.558) present?

The data I'm working with has 9 observations. I'm using only one predictor variable. Using SAS, I fit the model and checked the residuals. The typical model assumptions appear to be met, but there ...
4
votes
1answer
78 views

Intelligently selecting outliers

I'm trying to remove what might be considered "unreasonable" data by evaluating the percent error in the mean and square root of the variance. Here's the setup: Let's say I have three bids on a ...
0
votes
1answer
31 views

Spike detection and removal in position data

Is there any good filter to remove big spikes in position data? I think lowpass filter should be good but is it possible to filter 2D position data with assumption its joint distributed? I mean, not ...
1
vote
0answers
10 views

how to determine if a test run in only one geographic area leads to activity different than in other geographic areas

I was recently given data from an experiment that I did not design. In this study, the people running it changed behavior in 1 designative market area (DMA) but not the other DMAs (209 other DMAs). ...
0
votes
0answers
27 views

What do you do with outliers when developing statistical models?

I am a beginner so I have an extremely tough time dealing with outliers. I wanted to ask the community to help me understand rule of thumbs or anythng that would help me deal with these questions ...
0
votes
0answers
23 views

Anomaly Detection with very small number of positives [closed]

I am trying to detect anomalies in a population comprising of 10 features and around 90,000 observations. Past investigations have revealed 18 positives. Given limited data for supervised learning, I ...
0
votes
0answers
49 views

Outside 1.5 times inter quartile range yet not outlier

I have a data set with the following density histogram and box plot. Summary info as given R function. ...
2
votes
1answer
186 views

how to determine skewness from histogram with outliers?

I have the following histogram created in Minitab. I am wondering whether this histogram is actually positively skewed, negatively skewed, or symmetric. By observing the graph itself, it seems that ...
2
votes
1answer
72 views

Elastic net: dealing with wide data with outliers

Recently I was working on a dataset with ~300 observations and 1500 predictors. I used the glmnet package in R to fit an elastic net model, which gave me a ...
6
votes
4answers
159 views

How to prepare/construct features for anomaly detection (network security data)

My goal is to analyse network logs (e.g., Apache, syslog, Active Directory security audit and so on) using clustering / anomaly detection for intrusion detection purposes. From the logs I have a lot ...
0
votes
1answer
38 views

How to predict in advance that a smart meter is failing?

I have an electricity consumption data set collected by smart meters over a year and a half for every hour. The objective is to predict whether the meter could fail earlier than it actually fails.I ...
6
votes
2answers
138 views

What is a good method to identify outliers in exam data?

I give my students an exam that has 8 questions on it. Each question is about a particular topic. The exam is made up on the fly by randomly selecting 1 question for each topic from a pool of ...
2
votes
0answers
60 views

Basic methods for detecting outliers

Let $X$ be a matrix of $n$ rows and $m$ columns. $n$ is the number of samples and $m$ is the number of gene expressions. Gene expressions are basically numerical continuous values. Assume we have a ...
1
vote
1answer
60 views

Detecting anomalies in a time series where new data points will be continuously added

I have a time series data and I will be adding more data points in a consistent manner. I want to figure out whether the new data point added is an outlier, in regards to the previously observed data ...
0
votes
0answers
14 views

Should outliers be remove first before identifying influential observations?

I have constructed a logistic regression model. I used half-normal probability plot and detected two outliers, which I removed. Then I want to identify influential observation, in order to improve the ...
0
votes
0answers
13 views

Adjusting for outlier in Fractional logit in R when dv is very small proportion

I used the code from this site: http://stackoverflow.com/questions/19893133/fractional-logit-model-r to estimate a fractional logit model. There are 90 observations in my dataMy dependent variable is ...
1
vote
1answer
51 views

Cook's D, testing for outliers

I am working on a multiple linear regression and I want to check for outliers using Cook's D. I have a problem interpreting it, as there are many points above the 4/N line, but only one is >1. How ...
0
votes
1answer
45 views

Eliminating observations with big residuals in regression

I busy with a regression model that seems to have heteroscedasticity. The model has 6 independent variables and one dependent variable. I did the regression and noticed heteroscedasticity. I then ...
1
vote
1answer
150 views

Choosing a k-value for Local Outlier Factor (LOF) detection analysis

I have a set of three-dimensional data, and I'm trying to use Local Outlier Factor analysis to identify the most unique or strange values. How does one decide the k-value to use in LOF analysis? I ...
1
vote
1answer
115 views

Issues in auto.arima algorithm when using external regressors and outlier correction

auto.arima is an automatic arima modeling function in forecast package in R that uses information criterion(example: AIC/BIC) to ...