Questions tagged [weighted-data]

Datasets where different pieces of data can have different "weights", i.e. different importance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
40 views

Best practice for subsampling training data and weights (in XGBoost)

I am trying to build an XGBoost model in pycharm and I have a general method question even though it relates to my model of choice (XGBoost). Any kind of general comments on the proper statistical ...
Magi's user avatar
  • 41
0 votes
0 answers
14 views

Using weights in prognostic models of Survival Analysis data

I have a dataset where I'm comparing Survival (overall and cancer-specific survival) between two treatment groups (Surgery vs. Radiation) for prostate cancer. As suggested by Noordzij et al (PMID ...
Uro_Star's user avatar
0 votes
0 answers
44 views

Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage

The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B. The formula for a weighted ...
nka5we's user avatar
  • 41
0 votes
0 answers
23 views

Complex survey design with multiple waves

The organization I work for has collected data from individuals in multiple waves. Their goal was to collect 333 individuals in 6 different groups (genderXgroup). If the first Wave did not reach 333 ...
StatsAreHard's user avatar
1 vote
0 answers
58 views

Power analysis for an IPTW (Inverse Probability of Treatment Weighting) model?

I have a sample of N=1,615, and 319 of those cases (19.75%) received the treatment. The prevalence of the outcome of interest is about 0.09 for the whole sample. Ultimately, I want to conduct IPTW to ...
Ray's user avatar
  • 11
0 votes
0 answers
68 views

Do I need to standardise variables when computing an index of interest?

Suppose I have to create an "index of interest" for the products of an e-commerce. By index of interest I mean a $parameter$ which, based on some variables, tells me which products are the ...
TF7's user avatar
  • 37
1 vote
0 answers
31 views

Weighted table probabilities without replacement

I'm trying to recall some statistical maths and could do with a little asisstance. I'm dealing with a weighted table, and I want to work out the probability of drawing certain values with no order, ...
Watch's user avatar
  • 11
0 votes
0 answers
29 views

Bootstrap specifics with weighted outcome estimation

I am using bootstrapping to estimate the standard error of a weighted estimate. The process to calculate the estimate is: Sample from the population Calculate a weight for each individual based on ...
fair21comic's user avatar
0 votes
1 answer
18 views

Determining comparable individual performance within a team sport

i have collated a significant body data for a local badminton league (> 20000 indvidual games) and while i have the means to extract data on a: per player per couple per team per club per season ...
Neil Cooper's user avatar
6 votes
3 answers
2k views

"Weighted Median": statistical properties and connection to the sample variance

Suppose we only have the following information: A set of sample means : $\bar{x_1}$, $\bar{x_2}$ ... $\bar{x_k}$ The sample size used to calculated each sample mean: $n_1$, $n_2$ ....$n_k$ The ...
Uk rain troll's user avatar
0 votes
0 answers
130 views

Do the weights of IPTW have to sum up to the population size?

I'm new to Inverse Probability of Treatment Weighting and trying to understand the mechanism. From several practical guides I understood, that the weights should add up to the population size for ...
TiTo's user avatar
  • 217
1 vote
1 answer
31 views

How does a machine learning model estimate unevenly correlated features?

Aside from additional data and feature engineering, how can I make sure my ML model picks up on patterns that only apply to certain thresholds or ranges? For example, assume I use data about users and ...
Joachim Rives's user avatar
0 votes
0 answers
58 views

How to calculate confidence intervals in a subgroup of a weighted sample?

I have a weighted data sample (size : 20000) and I want to calculate a 95% confidence interval for the mean of a variable in the dataset, but only in a specific subgroup within this dataset (size of ...
mtr's user avatar
  • 1
0 votes
1 answer
266 views

Formula for standardized mean difference in Cobalt package for categorical variables

I am having troubles in understanding the formula in cobalt package used for standardized mean difference calculation in BINARY variables ...
user19745561's user avatar
0 votes
1 answer
301 views

Discrepancy between standardized mean difference in cobalt and smd packages

Let's suppose I have weighted my population using WeightIt ...
user19745561's user avatar
0 votes
0 answers
64 views

Distribution of Weighted Sample Variance

Say we have $N$ independent Gaussian variables $X_i$, each drawn from a mean-zero Gaussian with variance $\sigma_i^2$, where in this case I assume I know the $\sigma_i^2$. Define the weighted average ...
StevenMurray's user avatar
1 vote
0 answers
296 views

How to fit data with asymmetric error bars?

General question: How do I fit a model to data when the data points have asymmetric error bars? What is the cost function I use to calculate residuals, and from that, how do I calculate confidence ...
Jagerber48's user avatar
0 votes
1 answer
263 views

Calculate risk ratio in weighted population

I have a propensity score weighted population (using IPTW) and I want to compute risk ratios on my weighted population. For this, I am using a weighted Poisson regression. Let's suppose that "...
user19745561's user avatar
3 votes
1 answer
847 views

Unbalanced variables after IPTW - entropy balancing?

After using inverse probability of treatment weighting (IPTW) on the variables of my dataset, there is still an imbalance in one covariate between the two groups. My outcome is binary (yes/no) and it ...
user19745561's user avatar
2 votes
1 answer
428 views

Recognize extreme weights in inverse probability treatment weighting

I have some doubts about how to recognize if there are extreme weights after balancing my population with inverse probability treatment weighting. For instance, let's look at these results [code at ...
user19745561's user avatar
1 vote
1 answer
318 views

Calculating outcomes in IPTW

I have a few questions concerning what to do next after balancing your population with IPTW. This is a code just to have an example: ...
user19745561's user avatar
3 votes
1 answer
951 views

Reduced effective sample size in balanced population after inverse probability treatment weighting

I am new to inverse probability treatment weighting (IPTW) and I am trying to understand it. However, I am not a statistician and I am having some troubles with advanced statistics concepts. As far as ...
user19745561's user avatar
2 votes
1 answer
523 views

Create table before and after IPTW with frequencies in R

I am doing propensity score weighting with the package "Weightit" and I want to have a Table 1 "Before and After Weighting" like this one: As you can see, data are displayed ...
user19745561's user avatar
0 votes
0 answers
13 views

How to use weights in the right way?

I have a dataset and in the description is written <<In order to improve the representativeness of some segments of the population, the variable pesofit (...
io_boh's user avatar
  • 101
1 vote
0 answers
25 views

Amplification effect of retweets on uncertainty

Consider you are scoring tweets for tone based on some sentiment analysis implementation. Each tweet has hypothetically a 90% chance of being correctly scored, while 10% get it wrong for whatever ...
geotheory's user avatar
  • 647
0 votes
0 answers
213 views

Estimate weights given a weighted geometric median

Problem 1 Given a set of points and their weighted geometric median, find the weights associated with each point. For instance, given the points ...
npetrov937's user avatar
0 votes
0 answers
57 views

Combining standard errors of two separate variables

I have a dependent variable associated with a SE and an independent variable associated with a SE. I would like to weight my regression such that both of these SEs are taken into account. What is the ...
M BG's user avatar
  • 61
2 votes
1 answer
71 views

Conducting multiple weighted t.tests to count over what fraction of a population a model performs significantly better or worse than another

I have two different model families $A_i$ and $B_i$ each of which predicts a continuous value for an item $i$ at various points in time throughout a given day. For a given day, and item $i$, I can ...
Palace Chan's user avatar
  • 1,003
0 votes
0 answers
37 views

How to get weighted mean with two weighting sources

I have some data on a cyclists distance traveled and hours it took. ...
radio23's user avatar
  • 23
1 vote
0 answers
32k views

Calculate probability of dying in school shooting using spatial statistics

Some public commentators say that the chance of dying in a school shooting is very low, so they conclude that Americans over-react to them. https://www.washingtonpost.com/outlook/school-shootings-are-...
Iterator516's user avatar
0 votes
0 answers
35 views

Smoothing time series with Adjusted R2-weighted averages

I have two parameters (a,b) resulting from an exponential estimation of a curve. I have estimated this curve every hour for one month. In other words, I have a total of 720 parameters a and b, and I ...
angelavtc's user avatar
2 votes
1 answer
62 views

weighted median result is integer

I am computing the weighted median for a dataset using the R reldist package. The input data are integers but my result is an integer number too. Should't my result ...
aaaaa's user avatar
  • 314
0 votes
0 answers
133 views

How to bias-correct proportions for a prop.test in R?

Let's say I ask 200 participants in a study what experience brought the most joy to their lives: A, B, C, or D. ...
format_0's user avatar
1 vote
0 answers
27 views

How reliable is conclusion drawn out from a known data?

Most of time conducting experiment is expensive. Suppose I and my collaborator decided to use a known databank(NHANE, NCI's data) to do some digging with some untested hypothesis given. Most of time ...
user45765's user avatar
  • 1,406
0 votes
0 answers
135 views

How to assign weights to a set of ordered objects?

I have been trying to assign weights to set of objects. The problem that I have is as following. I have a set of objects ordered either ascending or decending. . Each object shall be provided a weight ...
sveer's user avatar
  • 103
3 votes
1 answer
197 views

Can I use oversampling to increase sample weights?

I am working on a balanced binary classification problem. I don't need to adjust the proportion of either of the classes, as it's already 50/50 . However, some of the samples are more valuable than ...
Vladimir Belik's user avatar
0 votes
1 answer
34 views

Age adjustment or Age standardization state of the art?

I want to run some comparisons of a quantity between different populations. Such quantity comes from a model that includes, among other variables, the variable age. So, one of the variables affecting ...
Compare's user avatar
3 votes
0 answers
90 views

How do I calculate a finite-population correction for a weighted sample?

I have a weighted sample from a population. These are probability weights (as samples are taken with unequal probability). What's the variance in the estimated total? All samples are exchangeable (...
Closed Limelike Curves's user avatar
0 votes
0 answers
60 views

How to weight my data in SAS - categorical and numerical variables

I've collected data from 1000 animals (categorical and numerical variables) from 2 different farms. My problems are: one farm corresponds to 68% of my data, in some of my variables I've observed more ...
cristina's user avatar
0 votes
0 answers
38 views

Weigh the percentages based on the number of inhabitants

I am analyzing the diffusion of a service in some regions. Is there a legitimate criterion for weighing percentages by the number of inhabitants? I would like to "improve" the results of the ...
LJG's user avatar
  • 131
1 vote
0 answers
845 views

How do we calculate statistics on data weighted by relative frequencies?

I have the following data with some values and their corresponding weights. groups values weights a 1 0.25 a 3 0.75 b 8 1.00 The weights represent relative frequencies of each value in the group ...
Troll's user avatar
  • 113
0 votes
0 answers
45 views

weight variable based on opposite direction variable

I have two variables. From these two variables, I want to produce one weight variable. The first (A) is a percentage, 0.0 - 1.0. The second variable (B) is a count, 0 to infinity. I want to weight the ...
adin's user avatar
  • 235
0 votes
1 answer
138 views

Q: Aggregating audience and critic movie ratings, and creating a composite rating of the two

I’m building a movie review aggregation site that combines user ratings and critical reviews for a given movie. The objective is to create a list of the “best of the best” movies which were rated ...
GaussianBlur's user avatar
2 votes
0 answers
130 views

Variance in variance-weighted variance estimate?

Apologies for the confusing title, but I couldn't resist. Much can and has been said about computing the unbiased variance using a sample of points, weighting by the variances of each point (for ...
astrobenji's user avatar
3 votes
2 answers
3k views

What does the "weights" argument do when fitting Kaplan-Meier curveswith the survival package?

In R's survival package, there is an optional weights argument you can supply when you fit a Kaplan-Meier curve. I can't find any documentation about what this does or what exactly a weighted Kaplan ...
Eli's user avatar
  • 2,642
6 votes
1 answer
2k views

Best way to construct a QQ-plot

I want to assess the normality of a dataset (which is log-normally distributed data transformed back to normal) using a Q-Q plot. I stumbled on the fact that there are many ways to build such a plot, ...
Aubergine's user avatar
  • 183
0 votes
0 answers
47 views

Weighting the loss function based on previous seen true positive rates

Similiar to class imbalance there is always something I would call "learnability imbalance" in multi-class classification. What I mean by that: Even when the classes are evenly distributed ...
Theo P's user avatar
  • 59
0 votes
1 answer
215 views

Calculating Weighted Quantiles causing NA error by coercion in R

I'm new to R and I'm trying through two different packages and formulas to calculate quantiles from a weighted dataset, as you can see below: ...
Luis López's user avatar
15 votes
2 answers
2k views

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

Say I commit the following sins while building a predictive model: I take my dataset and split it into four subsets: Three for training (Train_A, Train_B, and Train_C) and one for validation. I ...
Jdclark's user avatar
  • 155
0 votes
1 answer
415 views

NHANES Weighting

I am in the process of analyzing some data derived from 8 cycles of NHANES data. Due to the nature of my research, my sample size is rather small (n=98). Since NHANES surveys utilize a complex ...
PhD2B's user avatar
  • 21

1
2 3 4 5