Questions tagged [binning]

Binning means grouping a continuous variable into discrete categories. It is particularly used in reference to histograms, but could also be used more generally in the sense of coarsening.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Estimating conditional mutual informations from 2D histograms

I have binned marginal and joint distributions of two event features X and Y, i.e. p(X), p(Y) and p(X,Y) where the marginal distributions in X and Y are obtained by summing p(X,Y) over the bins of the ...
user avatar
0 votes
1 answer
34 views

How can I find the start of a range in an automated way?

I am looking at real estate sales data for different regions. I bin the data by sold price range in 1000 increments (see below). I am looking to find the start of the price range or cluster in which ...
user avatar
  • 1
0 votes
1 answer
28 views

Converting a continuous output to risk score category and selecting the optimal number of bins

I am trying to convert a continuous measurement of a patient’s bone mineral density to a risk score which I will display to the user with the corresponding observed prevalence (observed probability of ...
user avatar
  • 35
1 vote
1 answer
88 views

How to setup the Panjer's recursion correctly?

I have a table of $k=(0,1,2,3,4,5,6)$ and $number=(40544,8082,1205,145,20,3,1)$ I need to fit data by a Compound Poisson-Gamma distribution and then make a discretization and compare results with ...
user avatar
  • 792
0 votes
0 answers
59 views

Is percentile a good method?

Hello I'm an undergraduate student doing research about prevalence of carpal tunnel syndrome among college students I want to follow the method of this research (prevalence of carpal tunnel syndrome ...
user avatar
1 vote
0 answers
13 views

Empirical distribution for feature binning

In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
user avatar
  • 179
13 votes
0 answers
146 views

Why is Binning, Weight of Evidence and Information Value so ubiquitous in the Credit Risk/Finance industry?

In the credit risk industry (and finance industry as a whole, at least here in the UK), there is a very common and accepted 'proper' way to build scorecards. The general framework seems to be: ...
user avatar
0 votes
0 answers
34 views

Choose best binning for binned maximum likelihood fit?

I am trying to find the strength of signal over a background using a continuous variable, whose distributions are known for the expected signal, the expected background, and the observed data, along ...
user avatar
  • 1
0 votes
2 answers
37 views

Combining quantile regression with binning

I'm trying to employ a framework where I uncover the marginal effects of the quantiles of one continuous variable on another continuous variable - something analogous to the Quantile-on-quantile (QQR) ...
user avatar
0 votes
0 answers
18 views

How do you estimate the mode of a histogram with logarithmic bin width?

For the purpose of estimating the median and other quantiles, I summarize samples in a histogram whose bins grow logarithmically in width. For example, to guarantee a 1% worst case absolute relative ...
user avatar
1 vote
0 answers
17 views

Convert a distribution of a varialbe to another one

Assume an experiment where the observable is t and is related to another variable E with the following formula: $$E(t)=A/(t-t_0)^...
user avatar
  • 121
0 votes
0 answers
18 views

Histogram where the bin heights take values into account

In a regular histogram the bin heights reflect the density of observations/data within them. Therefore, the areas of the bins on the chart reflect the quantity of observations they cover. They are for ...
user avatar
0 votes
0 answers
15 views

How to break up the following scale into equal segments?

In Excel, there is a conditional formatting feature like this: Let's say in order to figure out what color to use for a value, I split the gradient into 14 steps, such as: Red:6Parts,Yellow:0Parts ...
user avatar
  • 101
0 votes
0 answers
47 views

Is estimating histogram bin width of already binned dataset reasonable?

as a starting point I have a 1D dataset representing population counts along the spatial dimension x. The dataset is already binned, thus representing population counts per 100 meter intervals (=bins)....
user avatar
  • 1
0 votes
0 answers
63 views

Number of bins for discretization

How do I decide on the right number of bins to discretize my continuous data? Are there are tests/techniques to do the same? Could someone give me some idea into existing approaches?
user avatar
0 votes
0 answers
10 views

Proper statistical test for NGS dataset with two samples containing a library of 12,000 sequences each sorted into 4 bins with counts in each bin

Description of the experiment: A library containing 12,000 unique nucleotide sequences is integrated into two samples, a WT cell line and a knockout cell line. Each cell contains only one reporter ...
user avatar
  • 1
1 vote
2 answers
56 views

Binning Calibrated probability scores for business use

Context: We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to ...
user avatar
0 votes
0 answers
30 views

Which unsupervised clustering technique to identify potential high-value zones from volume profile?

Ideally, would like to implement an unsupervised approach (ultimately in C#) that is a starting point to define prospective high-value supply and demand zones from a multi-day composite volume profile....
user avatar
0 votes
1 answer
62 views

Variance of 2D binned uniform density

I will start off by saying that I am aware of this post, but applying Sheppard's correction in my case would seem to lead to a negative variance; so either I don't understand that post, or I have a ...
user avatar
0 votes
0 answers
16 views

How to you distinguish between the verbs "Roll up", "Bucket", "Bin", and "Cluster"?

I'm thinking that "rolling up" could be summarized as grouping plus some sort of aggregation function (I'm looking at https://www.tutorialspoint.com/dwh/dwh_olap.htm), and grouping is either ...
user avatar
0 votes
1 answer
48 views

Transform target/label variable into classes but classes are data-dependent: How to approach this correctly?

I was redirected from StackOverflow because my question is more about theory. I have a usual set-up with a pandas dataframe with some features and a numeric target variable (financial returns for ...
user avatar
  • 27
1 vote
0 answers
42 views

What can we learn from binning before correlation?

I've seen a lot of questions about binning data and how it produces wrong results. "wrong" here refers to arbitrary increases in correlation strength, or sometimes even achieving two ...
user avatar
  • 123
1 vote
2 answers
759 views

How to calculate the mean from bin endpoints and frequencies? [duplicate]

Sometimes data extracted from reports do not have individual values, like 4, 23, 43, but grouped together like this: income level people in this group 10k to 20k 44 20k to 40k 240 40k to 80k 400 ...
user avatar
  • 11
2 votes
0 answers
36 views

When should I use a dummy variable?

I have data for neighbourhoods with median income. I also have a standard low-income cutoff. (Just using regression analysis) I feel it would lead to a simpler and cleaner result if I just use the ...
user avatar
  • 21
1 vote
0 answers
37 views

Improve Multiclass Classification by Binning weak classes?

I have a imbalanced Dataset with 23 classes from Accounting Data. My goal is to provide a suggestion to the accountant, which Account a Transaction belongs to. Gradient Boosting and any other Ensemble ...
user avatar
0 votes
0 answers
35 views

How do I bin my PCA results i.e., 120 equal bins of my large data set?

I am a new researcher in Japan and working on my project. I am lost after reading one paper, the paper is from very good general he uses the technique of binning of PCA but I don't know how what does ...
user avatar
  • 101
1 vote
1 answer
32 views

Logistic regression using a predictor that's part of the outcome

Say I have 2 continuous variables measuring the same thing (e.g., at-home blood pressure monitor and in-office blood pressure cuff with in-office measurements being the gold standard). At a cut-off of ...
user avatar
1 vote
1 answer
38 views

Preferred way to sum different time series together (in software)

Is there a canonical/best approach to computationally summing different time series together? What I mean by that is the operation $$ \sum_i{s_i(t)} = S(t) $$ where $s_i(t)$ is the $i$-th time series, ...
user avatar
0 votes
0 answers
50 views

binning numerical variables?

I am dealing with a dataset composed of both numerical (discrete) and nominal variables and I have to classify a binary response. Since the dataset is imbalanced, I decided to oversample the minority ...
user avatar
2 votes
1 answer
3k views

Difference between equal frequency and quantile binning

Equal-frequency binning divides the data set into bins that all have the same number of samples. Quantile binning assigns the same number of observations to each bin. What is the difference between ...
user avatar
  • 21
3 votes
1 answer
65 views

One Hot Encoding of ranges of data vs. leaving data as is for Logistic Regression

Recently whilst doing an assignment using the PIMA Diabetes set I ran Logistic Regression using, amongst others: the age predictor as is segmented the age into ranges and applied OHE (with and ...
user avatar
0 votes
0 answers
26 views

Converting continuous predictor to category e.g. Age [duplicate]

I notice that on many examples one is keen to convert Age to a categorical age range. I am wondering if that is always necessary. The famous golf play decision tree example has ranges for temperatures ...
user avatar
1 vote
0 answers
188 views

Modeling covariates in multiple regression

My aim is to find the association between intake of chocolate (continuous predictor) and blood pressure (continuous outcome) in a multiple linear regression. I have to include many covariates in order ...
user avatar
  • 11
0 votes
2 answers
2k views

How can I determine the optimal binning system for a continuous variable in Python?

I've got two columns of data - a continuous variable that I'd like to treat as a categorical variable (i.e. bin it up), and a metric I want to measure by bin. Let's say the first column is income and ...
user avatar
0 votes
1 answer
38 views

Plotting average shows (log) linear trend but fitting line has 0 p value

I have some data and I am examining the relationship between two variables. When I form x-bins and take summary statistics of y in those bins, I see the plot below. The black line is the mean of each ...
user avatar
0 votes
0 answers
26 views

Which statistical techniques can be used to decide which definition of sets produces the most coherent grouping of data

To explain. I am a historian, and an almost complete statistical novice. I am interested in exploring the ways in which generational alignments might be identified, not via use of generational labels, ...
user avatar
0 votes
0 answers
95 views

Uniform distribution in a logarithmic/isolethargic binning

Assume a variable $x$ follows a uniform distribution i.e. $P(x)-=const$. In my case this is constant background as shown in the following figure with the green curve This is a distribution with a ...
user avatar
  • 121
1 vote
1 answer
37 views

In practice, how to discretize continuous regressor with minimal impact on coefficient (or minimal information loss)?

Suppose I have some continuous data that looks like this (this is a mini example, not my real data): ...
user avatar
  • 1,662
0 votes
0 answers
69 views

How do I select number of bins to discretize the data?

So, I have been pondering on how I can select the number of bins in a dataset? I know we have different methods for selecting number of bins for histogram, but how do I select number of bins when ...
user avatar
0 votes
0 answers
167 views

How to find the optimal cut point of a categorical variable?

I have two categorical variables (x and z) as shown in the frequency plot below. Y-axis is the count of variable x. As evident in this plot, there is a clear relationship between x and z variables. I ...
user avatar
  • 123
1 vote
1 answer
388 views

Using decision tree for unsupervised discretization?

I want to discretize a continuous variable $X$ into a given number of classes $k$ (assume for simplicity that $k$ is even). Decision trees (and related methods) are already used to discretize a ...
user avatar
1 vote
2 answers
1k views

Optimal binning methods for categorical variables

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the ...
user avatar
  • 233
0 votes
0 answers
15 views

frequency distribution of data [duplicate]

Let's say I have around 30k points of time series data with values ranging from 0 to 0.5. I have split this data into 5 buckets of 0.1 each which contains values within that range. I then plot these ...
user avatar
  • 1
1 vote
0 answers
37 views

How inaccurate are empirical copulas when fit on real data?

Copula models are used widely to present the dependency structure among variables. However, they are often implemented by fitting well-known bivariate copulas like Gumbel and Clayton over the data. ...
user avatar
  • 3,047
1 vote
0 answers
66 views

What is bin area during empirical copula estimation?

Two finite-sampled continuous random variables $X$ and $Y$ are transformed so that they are uniformly distributed, $U$ and $V$. With these as marginals, the empirical copula of $X$ and $Y$, denoted $C(...
user avatar
  • 3,047
0 votes
0 answers
30 views

Margin of error in case of very small sample size

Say there are some sectors(Sector) and some counterparties(NumCpty). Each counterparty belongs to a unique sector. Some counterparties fail on a certain task(CptyFailed). I want to do binning of the ...
user avatar
1 vote
0 answers
72 views

What happens with the significance of binned variables?

For this project I was required to create a credit risk scorecard witht the 4 most relevant variables, so I binned all variables and selected them by chi2 and IV. I ran the logistic and linear ...
user avatar
1 vote
0 answers
106 views

How to convert continuous variable dataset into binary discrete values using Chi-Square testing for decision making

I have a dataset that contains continuous values for an attribute ranging from 0 - 100. I want to convert these continuous variables into two discreet values (say Label L1 and Label L2), So that the ...
user avatar
0 votes
0 answers
28 views

What is the appropriate way to analyze data subsetted into bins and compare those bins across conditions?

I am wondering how to approach the analysis of a data set that I've obtained. I have animal trajectories moving toward a target under multiple experimental conditions. One of my analyses was to look ...
user avatar
1 vote
0 answers
57 views

Does Discretization improve Classifier Performance?

I am trying to understand the basics of how and when is it ok discretize a variable. Below are some papers that support Supervised Discretization: Improving Classification Performance with ...
user avatar
  • 547

1
2 3 4 5