Questions tagged [weighted-data]

Datasets where different pieces of data can have different "weights", i.e. different importance.

206 questions
Filter by
Sorted by
Tagged with
1 vote
25 views

Weighted averages (average customer-product interactions)

I'm trying to wrap my head around a seemingly simple weighted average calculation. I have a table joined from different sources to include customer, product and product_interaction_id : customer ...
326 views

Sandwich variance estimator or bootstrap-based variance for stabilized inverse probability weighting (IPW)

Multiple published papers describe IPW as akin to having population with multiply copies of the same individuals. Hence, the correlation should be accounted and corrected using sandwich variance ...
• 895
1 vote
26 views

Subjective confidence as weights in regression models

I have data where subjects rate a quantity on a certain scale ($y$) but also add their subjective confidence as to how sure they are in their choice ($w$). My initial thought was to add $w$ as weights ...
• 3,560
160 views

Question on the effect of test reliability on weighting of a test battery

Originally, a test battery had 4 components: two 100 item multiple-choice tests, one oral test, and one essay test. Each component measured different topics. Each of the 4 components was weighted 25%. ...
• 3,407
28 views

Generate weighted median from activity per timespan

I have a set of data where I have a number of observations over the course of a year per individual. Generally speaking, I want to know the average activeness of the individuals that participated in ...
• 97
25 views

Weighing Data Issue

I am looking at e-cig prevalence within a city. I used surveys to collect data from residents, and I have a query around weighing data. I have made the assumption, due to over and underrepresentation ...
1 vote
128 views

Best practice for subsampling training data and weights (in XGBoost)

I am trying to build an XGBoost model in pycharm and I have a general method question even though it relates to my model of choice (XGBoost). Any kind of general comments on the proper statistical ...
• 41
25 views

Using weights in prognostic models of Survival Analysis data

I have a dataset where I'm comparing Survival (overall and cancer-specific survival) between two treatment groups (Surgery vs. Radiation) for prostate cancer. As suggested by Noordzij et al (PMID ...
136 views

Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage

The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B. The formula for a weighted ...
• 49
31 views

Complex survey design with multiple waves

The organization I work for has collected data from individuals in multiple waves. Their goal was to collect 333 individuals in 6 different groups (genderXgroup). If the first Wave did not reach 333 ...
1 vote
88 views

Power analysis for an IPTW (Inverse Probability of Treatment Weighting) model?

I have a sample of N=1,615, and 319 of those cases (19.75%) received the treatment. The prevalence of the outcome of interest is about 0.09 for the whole sample. Ultimately, I want to conduct IPTW to ...
• 11
75 views

Do I need to standardise variables when computing an index of interest?

Suppose I have to create an "index of interest" for the products of an e-commerce. By index of interest I mean a $parameter$ which, based on some variables, tells me which products are the ...
• 37
1 vote
31 views

Weighted table probabilities without replacement

I'm trying to recall some statistical maths and could do with a little asisstance. I'm dealing with a weighted table, and I want to work out the probability of drawing certain values with no order, ...
• 11
1 vote
68 views

Bootstrap specifics with weighted outcome estimation

I am using bootstrapping to estimate the standard error of a weighted estimate. The process to calculate the estimate is: Sample from the population Calculate a weight for each individual based on ...
22 views

Determining comparable individual performance within a team sport

i have collated a significant body data for a local badminton league (> 20000 indvidual games) and while i have the means to extract data on a: per player per couple per team per club per season ...
2k views

"Weighted Median": statistical properties and connection to the sample variance

Suppose we only have the following information: A set of sample means : $\bar{x_1}$, $\bar{x_2}$ ... $\bar{x_k}$ The sample size used to calculated each sample mean: $n_1$, $n_2$ ....$n_k$ The ...
245 views

Do the weights of IPTW have to sum up to the population size?

I'm new to Inverse Probability of Treatment Weighting and trying to understand the mechanism. From several practical guides I understood, that the weights should add up to the population size for ...
• 249
34 views

How does a machine learning model estimate unevenly correlated features?

Aside from additional data and feature engineering, how can I make sure my ML model picks up on patterns that only apply to certain thresholds or ranges? For example, assume I use data about users and ...
72 views

How to calculate confidence intervals in a subgroup of a weighted sample?

I have a weighted data sample (size : 20000) and I want to calculate a 95% confidence interval for the mean of a variable in the dataset, but only in a specific subgroup within this dataset (size of ...
• 1
645 views

Formula for standardized mean difference in Cobalt package for categorical variables

I am having troubles in understanding the formula in cobalt package used for standardized mean difference calculation in BINARY variables ...
• 159
519 views

Discrepancy between standardized mean difference in cobalt and smd packages

Let's suppose I have weighted my population using WeightIt ...
• 159
94 views

Distribution of Weighted Sample Variance

Say we have $N$ independent Gaussian variables $X_i$, each drawn from a mean-zero Gaussian with variance $\sigma_i^2$, where in this case I assume I know the $\sigma_i^2$. Define the weighted average ...
• 203
1 vote
443 views

How to fit data with asymmetric error bars?

General question: How do I fit a model to data when the data points have asymmetric error bars? What is the cost function I use to calculate residuals, and from that, how do I calculate confidence ...
• 136
405 views

Calculate risk ratio in weighted population

I have a propensity score weighted population (using IPTW) and I want to compute risk ratios on my weighted population. For this, I am using a weighted Poisson regression. Let's suppose that "...
• 159
1k views

Unbalanced variables after IPTW - entropy balancing?

After using inverse probability of treatment weighting (IPTW) on the variables of my dataset, there is still an imbalance in one covariate between the two groups. My outcome is binary (yes/no) and it ...
• 159
800 views

Recognize extreme weights in inverse probability treatment weighting

I have some doubts about how to recognize if there are extreme weights after balancing my population with inverse probability treatment weighting. For instance, let's look at these results [code at ...
• 159
1 vote
518 views

Calculating outcomes in IPTW

I have a few questions concerning what to do next after balancing your population with IPTW. This is a code just to have an example: ...
• 159
2k views

Reduced effective sample size in balanced population after inverse probability treatment weighting

I am new to inverse probability treatment weighting (IPTW) and I am trying to understand it. However, I am not a statistician and I am having some troubles with advanced statistics concepts. As far as ...
• 159
914 views

Create table before and after IPTW with frequencies in R

I am doing propensity score weighting with the package "Weightit" and I want to have a Table 1 "Before and After Weighting" like this one: As you can see, data are displayed ...
• 159
1 vote
25 views

Amplification effect of retweets on uncertainty

Consider you are scoring tweets for tone based on some sentiment analysis implementation. Each tweet has hypothetically a 90% chance of being correctly scored, while 10% get it wrong for whatever ...
• 637
230 views

Estimate weights given a weighted geometric median

Problem 1 Given a set of points and their weighted geometric median, find the weights associated with each point. For instance, given the points ...
58 views

Combining standard errors of two separate variables

I have a dependent variable associated with a SE and an independent variable associated with a SE. I would like to weight my regression such that both of these SEs are taken into account. What is the ...
• 61
73 views

Conducting multiple weighted t.tests to count over what fraction of a population a model performs significantly better or worse than another

I have two different model families $A_i$ and $B_i$ each of which predicts a continuous value for an item $i$ at various points in time throughout a given day. For a given day, and item $i$, I can ...
• 1,003
41 views

How to get weighted mean with two weighting sources

I have some data on a cyclists distance traveled and hours it took. ...
• 23
1 vote
34k views

Calculate probability of dying in school shooting using spatial statistics

Some public commentators say that the chance of dying in a school shooting is very low, so they conclude that Americans over-react to them. https://www.washingtonpost.com/outlook/school-shootings-are-...
• 163
40 views

Smoothing time series with Adjusted R2-weighted averages

I have two parameters (a,b) resulting from an exponential estimation of a curve. I have estimated this curve every hour for one month. In other words, I have a total of 720 parameters a and b, and I ...
69 views

weighted median result is integer

I am computing the weighted median for a dataset using the R reldist package. The input data are integers but my result is an integer number too. Should't my result ...
• 344
151 views

How to bias-correct proportions for a prop.test in R?

Let's say I ask 200 participants in a study what experience brought the most joy to their lives: A, B, C, or D. ...
1 vote
27 views

How reliable is conclusion drawn out from a known data?

Most of time conducting experiment is expensive. Suppose I and my collaborator decided to use a known databank(NHANE, NCI's data) to do some digging with some untested hypothesis given. Most of time ...
• 1,445
161 views

How to assign weights to a set of ordered objects?

I have been trying to assign weights to set of objects. The problem that I have is as following. I have a set of objects ordered either ascending or decending. . Each object shall be provided a weight ...
• 103
232 views

Can I use oversampling to increase sample weights?

I am working on a balanced binary classification problem. I don't need to adjust the proportion of either of the classes, as it's already 50/50 . However, some of the samples are more valuable than ...
• 1,561
37 views

Age adjustment or Age standardization state of the art?

I want to run some comparisons of a quantity between different populations. Such quantity comes from a model that includes, among other variables, the variable age. So, one of the variables affecting ...
104 views

How do I calculate a finite-population correction for a weighted sample?

I have a weighted sample from a population. These are probability weights (as samples are taken with unequal probability). What's the variance in the estimated total? All samples are exchangeable (...
71 views

How to weight my data in SAS - categorical and numerical variables

I've collected data from 1000 animals (categorical and numerical variables) from 2 different farms. My problems are: one farm corresponds to 68% of my data, in some of my variables I've observed more ...
38 views

Weigh the percentages based on the number of inhabitants

I am analyzing the diffusion of a service in some regions. Is there a legitimate criterion for weighing percentages by the number of inhabitants? I would like to "improve" the results of the ...
• 131
1 vote
968 views

How do we calculate statistics on data weighted by relative frequencies?

I have the following data with some values and their corresponding weights. groups values weights a 1 0.25 a 3 0.75 b 8 1.00 The weights represent relative frequencies of each value in the group ...
• 113
50 views

weight variable based on opposite direction variable

I have two variables. From these two variables, I want to produce one weight variable. The first (A) is a percentage, 0.0 - 1.0. The second variable (B) is a count, 0 to infinity. I want to weight the ...
• 235
154 views

Q: Aggregating audience and critic movie ratings, and creating a composite rating of the two

I’m building a movie review aggregation site that combines user ratings and critical reviews for a given movie. The objective is to create a list of the “best of the best” movies which were rated ...