Questions tagged [binary-data]

A binary variable takes one of two values, typically coded as "0" and "1".

Filter by
Sorted by
Tagged with
63
votes
4answers
46k views

Reduce Classification Probability Threshold

I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) > 0.5$, we will ...
60
votes
7answers
37k views

Binary classification with strongly unbalanced classes

I have a data set in the form of (features, binary output 0 or 1), but 1 happens pretty rarely, so just by always predicting 0, I get accuracy between 70% and 90% (depending on the particular data I ...
59
votes
10answers
18k views

Measuring entropy/ information/ patterns of a 2d binary matrix

I want to measure the entropy/ information density/ pattern-likeness of a two-dimensional binary matrix. Let me show some pictures for clarification: This display should have a rather high entropy: ...
58
votes
5answers
105k views

Is it meaningful to calculate Pearson or Spearman correlation between two Boolean vectors?

There are two Boolean vectors, which contain 0 and 1 only. If I calculate the Pearson or Spearman correlation, are they meaningful or reasonable?
45
votes
4answers
47k views

Would PCA work for boolean (binary) data types?

I want to reduce the dimensionality of higher order systems and capture most of the covariance on a preferably 2 dimensional or 1 dimensional field. I understand this can be done via principal ...
37
votes
1answer
49k views

Doing principal component analysis or factor analysis on binary data

I have a dataset with a large number of Yes/No responses. Can I use principal components (PCA) or any other data reduction analyses (such as factor analysis) for this type of data? Please advise how I ...
37
votes
5answers
33k views

Should you ever standardise binary variables?

I have a data set with a set of features. Some of them are binary $(1=$ active or fired, $0=$ inactive or dormant), and the rest are real valued, e.g. $4564.342$. I want to feed this data to a ...
34
votes
2answers
31k views

How to use both binary and continuous variables together in clustering?

I need to use binary variables (values 0 & 1) in k-means. But k-means only works with continuous variables. I know some people still use these binary variables in k-means ignoring the fact that k-...
33
votes
1answer
30k views

Is there Factor analysis or PCA for ordinal or binary data?

I have completed the principal component analysis (PCA), exploratory factor analysis (EFA), and confirmatory factor analysis (CFA), treating data with likert scale (5-level responses: none, a little, ...
30
votes
2answers
24k views

Clustering a binary matrix

I have a semi-small matrix of binary features of dimension 250k x 100. Each row is a user and the columns are binary "tags" of some user behavior e.g. "likes_cats". ...
27
votes
7answers
52k views

Why is gender typically coded 0/1 rather than 1/2, for example?

I understand the logic of coding for data analysis. My question below is on the use of a specific code. Is there a reason why gender is often coded as 0 for female and 1 for male? Why is this coding ...
25
votes
2answers
26k views

Similarity Coefficients for binary data: Why choose Jaccard over Russell and Rao?

From Encyclopedia of Statistical Sciences I understand that given $p$ dichotomous (binary: 1=present; 0=absent) attributes (variables), we can form a contingency table for any two objects i and j of a ...
23
votes
3answers
12k views

Generate random correlated data between a binary and a continuous variable

I want to generate two variables. One is binary outcome variable (say success / failure) and the other is age in years. I want age to be positively correlated with success. For example there should be ...
23
votes
3answers
6k views

Visualizing the calibration of predicted probability of a model

Suppose I have a predictive model that produces, for each instance, a probability for each class. Now I recognize that there are many ways to evaluate such a model if I want to use those ...
18
votes
5answers
4k views

How to do one-class text classification?

I have to deal with a text classification problem. A web crawler crawls webpages of a certain domain and for each webpage I want to find out whether it belongs to only one specific class or not. That ...
17
votes
2answers
10k views

optimizing auc vs logloss in binary classification problems

I am performing a binary classification task where the outcome probability is fair low (aroung 3%). I am trying to decide whether to optimize by AUC or log-loss. As much as I have understood, AUC ...
15
votes
3answers
18k views

What is the effect of dichotomising variables?

When dichotomising variables, what information is lost in the process? How does a dichotomisation help in the analyses?
15
votes
1answer
12k views

Looking for a step through an example of a factor analysis on dichotomous data (binary variables) using R

I have some dichotomous data, only binary variables, and my boss asked me to perform a factor analysis using the tetrachoric correlations matrix. I’ve previously been able to teach myself how to run ...
15
votes
1answer
2k views

Variance partitioning and longitudinal changes in correlation with binary data

I am analysing data on 300,000 pupils in 175 schools with a logistic linear mixed effects model (random intercepts). Each pupil occurs exactly once and the data spans 6 years. How do I partition ...
15
votes
3answers
424 views

Comparison of statistical tests exploring co-dependence of two binary variables

Suppose we observe data $(X_i,Y_i)_{i=1,...,n}$ on two binary variables: $X\in\{0,1\}$ and $Y\in\{0,1\}$. We would like to test if $X$ and $Y$ are co-dependent (related). Standard suggestions in ...
14
votes
3answers
24k views

Probit two-stage least squares (2SLS)

I was told that it's possible to run a two-stage IV regression where the first stage is a probit and the second stage is an OLS. Is it possible to use 2SLS if the first stage is a probit but the ...
13
votes
3answers
1k views

How would I bias my binary classifier to prefer false positive errors over false negatives?

I've put together a binary classifier using Keras' Sequential model. Of its errors, it predicts with false negatives more frequently than false positives. This tool is for medical application, where I'...
13
votes
2answers
31k views

How to plot binary (presence/absence - 1/0) data against continuous variables [duplicate]

I am trying to plot presence/absence (1/0) of a sample species against various environmental variables. I have put presence/absence on the y-axis and the environmental variable (in this case ...
13
votes
4answers
7k views

Forecasting binary time series

I have a binary time series with 1 when the car is not moving, and 0 when the car is moving. I want to make a forecast for a time horizon up to 36 hours ahead and for each hour. My first approach ...
13
votes
2answers
1k views

Is there an R package for continuous time longitudinal binary responses?

The bild package appears to be an excellent package for serial binary responses. But it is for discrete time. I would like to specify a smooth function of time ...
12
votes
3answers
7k views

How can I test whether my clustering of binary data is significant

I'm doing shopping cart analyses my dataset is set of transaction vectors, with the items the products being bought. When applying k-means on the transactions, I will always get some result. A random ...
12
votes
3answers
15k views

Distance Metrics For Binary Vectors

I have vectors of same length consisting of 1 and 0. I am trying to find out how similar they are. So far I am using hamming distance that I calculate sum of one vector then sum of second vector and ...
12
votes
4answers
13k views

What algorithm should I use to cluster a huge binary dataset into few categories?

I have a large (650K rows * 62 columns) matrix of binary data (0-1 entries only). The matrix is mostly sparse: about 8% is filled. I would like to cluster it into 5 groups - say named from 1 to 5. I ...
12
votes
2answers
2k views

Why we use log function for cross entropy?

I'm learning about a binary classifier. It uses the cross-entropy function as its loss function. $y_i \log p_i + (1-y_i) \log(1-p_i)$ But why does it use the log function? How about just use linear ...
12
votes
4answers
7k views

Logistic Regression and Inflection Point

We have data with a binary outcome and some covariates. I used logistic regression to model the data. Just a simple analysis, nothing extraordinary. The final output is supposed to be a dose-response ...
12
votes
2answers
2k views

How to choose optimal bin width while calibrating probability models?

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example Brier score, and its decomposition into resolution, ...
12
votes
1answer
10k views

How to interpret second-stage coefficient in instrumental variables regression with a binary instrument and a binary endogenous variable?

(fairly long post, sorry. It includes lots of background info, so feel free to skip to the question at the bottom.) Intro: I am working on a project where we are trying to identify the effect of a ...
11
votes
3answers
3k views

How to visualize Bayesian goodness of fit for logistic regression

For a Bayesian logistic regression problem, I have created a posterior predictive distribution. I sample from the predictive distribution and receive thousands of samples of (0,1) for each observation ...
11
votes
2answers
2k views

Modelling auto-correlated binary time series

What are the usual approach to modelling binary time series? Is there a paper or a text book where this is treated? I think of a binary process with strong auto-correlation. Something like the sign of ...
10
votes
3answers
16k views

logit - interpreting coefficients as probabilities

I seem to be missing some vital piece of information. I am aware that the coefficient of logistic regression are in log(odds), called the logit scale. Therefore to interpret them, ...
10
votes
3answers
946 views

Indicator variable for binary data: {-1,1} vs {0,1}

I am interested in treatment-covariate interactions in the context of experiments/randomized controlled trials, with a binary treatment assignment indicator $T$. Depending on the specific method/ ...
10
votes
3answers
8k views

For a classification problem if class variable has unequal distribution which technique we should use?

for eg. if I have a class variable credit scoring with two classes good and bad, where #(good) = 700 and #(bad)= 300. I do not want to shorten my data. which technique I should use? I was using SVM ...
10
votes
2answers
7k views

Recommended procedure for factor analysis on dichotomous data with R

I have to run a factor analysis on a dataset made up of dichotomous variables (0=yes, 1= no) and I don´t know if I'm on the right track. Using tetrachoric() I ...
10
votes
1answer
359 views

Is it ever a good idea to give “partial credit” (continuous outcome) in training a logistic regression?

I am training a logistic regression to predict which runners are most likely to finish a grueling endurance race. Very few runners complete this race, so I have severe class imbalance and a small ...
10
votes
1answer
2k views

What are dangers of calculating Pearson correlations (instead of tetrachoric ones) for binary variables in factor analysis?

I do research on educational games, and some of my current projects involve using data from BoardGameGeek (BGG) and VideoGameGeek (VGG) to examine relationships between design elements of games (i.e., ...
10
votes
1answer
225 views

Correlation between two binary variables within one categorical variable

The Problem: I have measured two binary variables within 1 categorical variable with 5 levels. Initially, I thought I'd be able to use Fisher's Exact test or some $N \times M \times K$ version of it. ...
9
votes
3answers
22k views

Calculating Jaccard or other association coefficient for binary data using matrix multiplication

I want to know if is there any possible way to calculate Jaccard coefficient using matrix multiplication. I used this code ...
9
votes
5answers
103k views

Is nominal, ordinal, & binary for quantitative data, qualitative data, or both?

I'm getting wrapped around data types and I need some help: If you look at the picture above (taken from here), it has the data types like this: Quantitative (Discrete, Continuous) Qualitative (...
9
votes
4answers
428 views

How should I approach this binary prediction problem?

I've got a dataset with the following format. There's a binary outcome cancer/no cancer. Every doctor in the dataset has seen every patient and given an independent judgment on whether the patient ...
9
votes
5answers
7k views

How to measure a classifier's performance when close to 100% of the class labels belong to one class?

In my data, I have a class variable, denoted as $C$. This class variable values are ${0, 1}$ (binary). Almost all observations of $C$ are 0 (close to 100%, more precisely, 97%). I would like a "...
9
votes
3answers
6k views

Binary time series

I have a binary time series: We have 2160 data (0=didn't happen, 1=happened) for one-hour period in 90 days. I want to forecast after these 90 days, where the next 1 will happen, and also Extend ...
9
votes
1answer
761 views

Monotone machine learning

I have a binary classification (supervised learning) problem, where all my features are boolean, with the following twist: I want to learn a classifier $f:\{0,1\}^n \to \{0,1\}$ that is monotone. In ...
8
votes
3answers
3k views

GLM with logit link and Gaussian family to predict a continuous DV between 0 and 1

Can you run a GLM using a logit link with a continuous DV (between 0 and 1)? Generally it's suggested to use a binomial family with a logit link, but I'm guessing that is because the model assumes a ...
8
votes
2answers
8k views

Prediction of a binary variable

I am establishing a model for prediction of a binary variable (Yes/No) depending on three continuous variables ($A$,$B$,$C$). I applied logistic regression analysis for a learning dataset vith the ...
8
votes
1answer
3k views

Visualizing longitudinal data with binary outcome

For longitudinal data with a numeric outcome, I can use spaghetti plots to visualize the data. For example something like this (taken from the UCLA Stats site): ...

1
2 3 4 5
24