Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms, but could also be used more generally in the sense of coarsening.

Filter by
Sorted by
Tagged with
0
votes
0answers
26 views

What is a histogram simplex? [closed]

Is a histogram simplex simply a vector containing all elements of a probability distribution, indexed by their probability bins? and what type of simplex is the histogram simplex: 0-simplex, 1-simplex,...
0
votes
0answers
14 views

PDF of a two state Markov chain with binned observation

I am trying to come up with a PDF to describe experimental data, which I can describe well with a simple Monte Carlo simulation. I have a two state Markov chain with equal transition probabilities ...
0
votes
0answers
18 views

How do I select number of bins to discretize the data?

So, I have been pondering on how I can select the number of bins in a dataset? I know we have different methods for selecting number of bins for histogram, but how do I select number of bins when ...
0
votes
0answers
52 views

How to find the optimal cut point of a categorical variable?

I have two categorical variables (x and z) as shown in the frequency plot below. Y-axis is the count of variable x. As evident in this plot, there is a clear relationship between x and z variables. I ...
1
vote
1answer
47 views

Using decision tree for unsupervised discretization?

I want to discretize a continuous variable $X$ into a given number of classes $k$ (assume for simplicity that $k$ is even). Decision trees (and related methods) are already used to discretize a ...
1
vote
1answer
40 views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
0
votes
0answers
13 views

frequency distribution of data [duplicate]

Let's say I have around 30k points of time series data with values ranging from 0 to 0.5. I have split this data into 5 buckets of 0.1 each which contains values within that range. I then plot these ...
0
votes
0answers
7 views

How could I implement data binning for differential privacy, similar to Apple's Count Mean Sketch or Hadamard Count Mean Sketch?

I'm looking at the Apple differential privacy document here and it has the paragraph: The noise injection step works as follows: After encoding the input as a vector using a hash function, each ...
0
votes
0answers
23 views

How inaccurate are empirical copulas when fit on real data?

Copula models are used widely to present the dependency structure among variables. However, they are often implemented by fitting well-known bivariate copulas like Gumbel and Clayton over the data. ...
0
votes
0answers
22 views

What is bin area during empirical copula estimation?

Two finite-sampled continuous random variables $X$ and $Y$ are transformed so that they are uniformly distributed, $U$ and $V$. With these as marginals, the empirical copula of $X$ and $Y$, denoted $C(...
0
votes
0answers
20 views

Margin of error in case of very small sample size

Say there are some sectors(Sector) and some counterparties(NumCpty). Each counterparty belongs to a unique sector. Some counterparties fail on a certain task(CptyFailed). I want to do binning of the ...
1
vote
0answers
14 views

Notation in “Fast Computation of Multivariate Kernel Estimators” by M. P. Wand

I'm new to kernel estimation methods and I've reading the paper "Fast Computation of Multivariate Kernel Estimators" by M. P. Wand. Particulary on page 434, it says "Let $(X_1, Y_1), ......
1
vote
0answers
32 views

What happens with the significance of binned variables?

For this project I was required to create a credit risk scorecard witht the 4 most relevant variables, so I binned all variables and selected them by chi2 and IV. I ran the logistic and linear ...
0
votes
0answers
32 views

Artifact in CDF with real data?

Looking for some help explaining why intuition is failing me in exploring this data. I've binned by dataset by a predictor variable to examine the response variable through CDF plots. I realize it may ...
0
votes
0answers
11 views

Experimental design to create and adjust cut offs to a continuous outcome

An internal tool is used to estimate similarity between two documents. The similarity measure is continuous, but user testing shows that discrete labels (e.g. low, medium, and high) are preferable to ...
1
vote
0answers
25 views

How to convert continuous variable dataset into binary discrete values using Chi-Square testing for decision making

I have a dataset that contains continuous values for an attribute ranging from 0 - 100. I want to convert these continuous variables into two discreet values (say Label L1 and Label L2), So that the ...
0
votes
0answers
22 views

What is the appropriate way to analyze data subsetted into bins and compare those bins across conditions?

I am wondering how to approach the analysis of a data set that I've obtained. I have animal trajectories moving toward a target under multiple experimental conditions. One of my analyses was to look ...
1
vote
0answers
17 views

Does Discretization improve Classifier Performance?

I am trying to understand the basics of how and when is it ok discretize a variable. Below are some papers that support Supervised Discretization: Improving Classification Performance with ...
0
votes
1answer
10 views

Store binnin models in R after fitting a dataset

I'm looking for algorithms to create bins of variables in order to reduce the noise. I have found several libraries for that, one if the chi2 library: https://www.rdocumentation.org/packages/...
3
votes
0answers
41 views

How can I prove the ill-effects of binning/discretization?

There is a binary classification model built where there is grouping of continuous variables into arbitrary ranges which I am told is to include a good number of outliers in the data set. How do I ...
-1
votes
2answers
94 views

Is it justified to discretize / bin a skewed variable in a classification problem?

How would a skewed variable impact a classification problem (logistic regression, tree model)? Is it justified to bin the skewed variable ? My data set comprises of younger demographic and fewer older ...
1
vote
0answers
27 views

Converting Continuous variable to Categorical [duplicate]

When should one consider converting continuous variable into categorical variable ? Are there guidelines ? Is it justified to bin skewed variable ? How should I determine the range / binning when I do ...
1
vote
1answer
42 views

Can you use clustering models such as k-mean or knn to do feature binning?

I currently working with a financial dataset in Python which contains a feature (among many others) called "interest rate", which represents the interest rate that a certain loan would have. ...
3
votes
2answers
123 views

Should discretized continous varibles be treated as numeric or ordinal (in a GLM)?

I am uncertain about how to treat a discretized / binned continuous variable in the glm() function in R. I see two possible ways of feeding it to the glm. Either I ...
0
votes
0answers
32 views

Increase in SVM Classifier performance after binning

I've been working on a classification problem, and I ran into something rather strange. The original problem has continuous features and three labels. I then mapped the continuous features to binary ...
0
votes
0answers
15 views

Inferring truncated distribution and mortality rates from age-binned population data

Ultimate goal: compare age-specific and age-standardized mortality rates between two populations with different age distributions. The population data are in age bins (slightly different for each ...
0
votes
0answers
10 views

Freedmans rule to find number of bins

I have a data set that is 30162 rows long. I am trying to split age into bins but I am struggling to understand the concept of the rule with my data set. As a quick examples this is what I did: <...
2
votes
0answers
24 views

Derive summary statistic Grouped Data & Frequency Distribution Table

I have the following data from the 2018 American Community Survey for a number of census block groups: ...
0
votes
1answer
21 views

Probability as a function of age from observations over several years

Hoping someone can help me correct the flaws in my logic. I have a number of water tanks, which may leak. I want to model the probability a water tank leaks given that it has never leaked before as a ...
0
votes
0answers
35 views

How to define a spearman correlation on a subset of the data

I have two ways to rank items (i.e. assign p-values to some data), and I want to quantify how similar these two lists are, ordering-wise. Since they are ranked by significance, the bottom part of both ...
0
votes
0answers
41 views

Dependence of target variable as a function of only one predictor

I have trained a classifier with target variable y (= 1 or 0) and predictors x1, x2, x3, x4, x5 (all discrete or continuous numerical variables, not normally distributed - x2 continuous with values ...
0
votes
1answer
98 views

Smoothing a binned averages

I am trying to smooth some binned data. I have a discrete variable X which might best be thought of as time and a continuous variable Y. I want to know the average Y value for each value of X and this ...
1
vote
0answers
30 views

Does aggregating ordinal data destroy the signal?

In an university assignment we are being asked to perform ordinal regression on wine data, predicting the quality of wine on a scale of 1 to 3, where 1 = inferior, 2 = average, 3 = superior. So far ...
0
votes
1answer
238 views

variable lengths differ (found for 'EMI') [closed]

I'm getting this error "variable lengths differ ". Please suggest me how to solve it . there are no Gas in data set ...
0
votes
1answer
27 views

Drawing histograms and how size of a bar influences probability?

This is a question concerning the width of the bar in a histogram. Let's say we have frequency distribution like this: As far as I've learned when you define size of the bar for example 0.5, for each ...
1
vote
1answer
44 views

Right way to bin the data-Fitting Voigt profiles to spectroscopy data

I have some measurements of the rate of a physical process versus energy. For each energy I have a number of counts and a measurement time associated to it. However, the step (in energy) at which the ...
0
votes
1answer
82 views

When to use equal frequency binning and when equal width binning?

When transforming numerical variables into categorical variables I'm not aware of when should I use equal frequency binning and when equal width binning. Seems that each of them has their own ...
3
votes
0answers
18 views

Predicting binary outcomes for observations given statistics on binned data

SAT Verbal scores range from 200 to 800 in increments of 10. MIT says that for the class of 2023, the acceptance rates were, for various score ranges 750-800 10% = 677/6504 700-740 06% = 312/5039 ...
0
votes
0answers
22 views

Using Binning before Mann-Whitney for Temperature Data

I have daily temperature for 2 cities and I am trying to see if we can conclude that one city is warmer than the other. I could use a Mann-Whitney for a whole year or I can bin the temperature into ...
0
votes
1answer
48 views

How to fill the bins?

Consider 1 million people earning money, sorted in increasing order. The kth decile, i.e. the kth 100,000 of them has an income share of $f(k)$ with $f(k)<f(k+1)$ and $\sum_{k=1}^{10} f(k)=1$. Let ...
0
votes
0answers
50 views

Is the regression coefficient the same for all categories of a categorical variable?

Let's imagine I build a scorecard with a single binned variable that can only take two values. In the weight of reference framework I would replace the two possible values by their weight of evidence ...
1
vote
0answers
34 views

How to treat low frequency continuous variable in machine leanring

Hello I am working on machine learning model for count data, and I have various features that are highly skewed. The frequency table for one of the feature is given below. ...
3
votes
2answers
217 views

A data-independant transformation to discretize a range of values non-uniformly

I am sure this is trivial, but I am looking for a transformation that nonuniformly discretizes all values of a range into several bins. The bins should be variant and I'd like them to be smaller ...
0
votes
0answers
82 views

How to calculate errors for cumulative distribution function

I have some data points of the form $(x_i,y_i,\delta y_i)$, where $y$ are counts and the error associated to each $y_i = N$ is $y_i = \sqrt{N}$. I want to create the cumulative distribution of these ...
0
votes
1answer
37 views

Logistic regression interpretation in SPSS statistics

I observed a very strange behavior while doing logistic regression, univariance analysis and correlation analysis. I have dependent binary variable and several independent variables that should be ...
1
vote
0answers
52 views

How to properly bin the data for a fit

I am working on a spectroscopy project in which we adjust the wavelength of a laser and get some counts on the detector from some laser-atom interactions. The data that we have is in the form: $(\...
1
vote
2answers
231 views

Is binning of continuous data always bad for statistical tests? [duplicate]

I was always thinking that binning of data if data is naturally continuous is bad. However, here is the case. The goal of study was to find if there is an association between a biomarker and disease ...
2
votes
1answer
189 views

Regression to Classification and back to Regression

Is it reasonable to transform regression problem into classification by binning target variable into classes and construct regression curve separately on each class?\ Precisely, if my goal is to ...
3
votes
2answers
1k views

good number of bins for logarithmic bin width

I was wondering how to estimate a good number of bins for my histogram. I know quite certainly that my data is well approximated by a LogNormal distribution. Previous studies have used logarithmic ...
2
votes
1answer
20 views

How do you test for bias in a circular reference plane?

I've been trying to get my head around how to do hypothesis testing for a circular scale in hypothesis testing, but I am having a lot of trouble. I know well how to test for linear scales, but when it ...

1
2 3 4 5