Questions tagged [weighted-data]
Datasets where different pieces of data can have different "weights", i.e. different importance.
210 questions
1
vote
0
answers
22
views
Incidence influenced by test frequency
I’m puzzled with a study design. Say I want to study a disease incidence, but this disease is asymptomatic mostly and its detection merely relies on regular testing. Some people test more frequently ...
- 11
0
votes
1
answer
25
views
Primer on Entropy Balancing
I've scoured the internet, but I can't for the life of me, find a comprehensive primer on entropy balancing.
I am currently in the process of cleaning data in order to create weights for the part of ...
- 1
0
votes
1
answer
22
views
What are the degrees of freedom for performing a 1-sample weighted t-test
I have 7 years of observations relating to mortality experience on a pool of lives. I assume each year's experience is independent of the other years. For each year, we have an expected number of ...
- 3
4
votes
0
answers
53
views
How should I weight points for which the uncertainties are unknown in a quantile regression?
Background
I am trying to analyse a compilation of data from different experimental studies and would appreciate advice on how to think through weighting points in a quantile regression.
I am not ...
- 20.4k
1
vote
1
answer
29
views
Weighted averages (average customer-product interactions)
I'm trying to wrap my head around a seemingly simple weighted average calculation.
I have a table joined from different sources to include customer, product and product_interaction_id :
customer
...
4
votes
2
answers
493
views
Sandwich variance estimator or bootstrap-based variance for stabilized inverse probability weighting (IPW)
Multiple published papers describe IPW as akin to having population with multiply copies of the same individuals. Hence, the correlation should be accounted and corrected using sandwich variance ...
- 905
1
vote
1
answer
33
views
Subjective confidence as weights in regression models
I have data where subjects rate a quantity on a certain scale ($y$) but also add their subjective confidence as to how sure they are in their choice ($w$). My initial thought was to add $w$ as weights ...
- 3,670
2
votes
2
answers
162
views
Question on the effect of test reliability on weighting of a test battery
Originally, a test battery had 4 components: two 100 item multiple-choice tests, one oral test, and one essay test. Each component measured different topics. Each of the 4 components was weighted 25%.
...
- 3,427
0
votes
0
answers
35
views
Generate weighted median from activity per timespan
I have a set of data where I have a number of observations over the course of a year per individual. Generally speaking, I want to know the average activeness of the individuals that participated in ...
- 97
0
votes
0
answers
28
views
Weighing Data Issue
I am looking at e-cig prevalence within a city. I used surveys to collect data from residents, and I have a query around weighing data.
I have made the assumption, due to over and underrepresentation ...
- 1
1
vote
0
answers
163
views
Best practice for subsampling training data and weights (in XGBoost)
I am trying to build an XGBoost model in pycharm and I have a general method question even though it relates to my model of choice (XGBoost). Any kind of general comments on the proper statistical ...
- 41
0
votes
0
answers
27
views
Using weights in prognostic models of Survival Analysis data
I have a dataset where I'm comparing Survival (overall and cancer-specific survival) between two treatment groups (Surgery vs. Radiation) for prostate cancer. As suggested by Noordzij et al (PMID
...
- 1
0
votes
0
answers
165
views
Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage
The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B.
The formula for a weighted ...
- 49
0
votes
0
answers
37
views
Complex survey design with multiple waves
The organization I work for has collected data from individuals in multiple waves. Their goal was to collect 333 individuals in 6 different groups (genderXgroup). If the first Wave did not reach 333 ...
1
vote
0
answers
110
views
Power analysis for an IPTW (Inverse Probability of Treatment Weighting) model?
I have a sample of N=1,615, and 319 of those cases (19.75%) received the treatment. The prevalence of the outcome of interest is about 0.09 for the whole sample.
Ultimately, I want to conduct IPTW to ...
- 11
0
votes
0
answers
76
views
Do I need to standardise variables when computing an index of interest?
Suppose I have to create an "index of interest" for the products of an e-commerce. By index of interest I mean a $parameter$ which, based on some variables, tells me which products are the ...
- 37
1
vote
0
answers
31
views
Weighted table probabilities without replacement
I'm trying to recall some statistical maths and could do with a little asisstance.
I'm dealing with a weighted table, and I want to work out the probability of drawing certain values with no order, ...
- 11
1
vote
1
answer
88
views
Bootstrap specifics with weighted outcome estimation
I am using bootstrapping to estimate the standard error of a weighted estimate. The process to calculate the estimate is:
Sample from the population
Calculate a weight for each individual based on ...
- 63
0
votes
1
answer
23
views
Determining comparable individual performance within a team sport
i have collated a significant body data for a local badminton league (> 20000 indvidual games)
and while i have the means to extract data on a:
per player
per couple
per team
per club
per season
...
6
votes
3
answers
2k
views
"Weighted Median": statistical properties and connection to the sample variance
Suppose we only have the following information:
A set of sample means : $\bar{x_1}$, $\bar{x_2}$ ... $\bar{x_k}$
The sample size used to calculated each sample mean: $n_1$, $n_2$ ....$n_k$
The ...
1
vote
0
answers
302
views
Do the weights of IPTW have to sum up to the population size?
I'm new to Inverse Probability of Treatment Weighting and trying to understand the mechanism. From several practical guides I understood, that the weights should add up to the population size for ...
- 281
2
votes
1
answer
37
views
How does a machine learning model estimate unevenly correlated features?
Aside from additional data and feature engineering, how can I make sure my ML model picks up on patterns that only apply to certain thresholds or ranges? For example, assume I use data about users and ...
- 143
0
votes
0
answers
78
views
How to calculate confidence intervals in a subgroup of a weighted sample?
I have a weighted data sample (size : 20000) and I want to calculate a 95% confidence interval for the mean of a variable in the dataset, but only in a specific subgroup within this dataset (size of ...
- 1
0
votes
1
answer
756
views
Formula for standardized mean difference in Cobalt package for categorical variables
I am having troubles in understanding the formula in cobalt package used for standardized mean difference calculation in BINARY variables
...
- 159
0
votes
1
answer
594
views
Discrepancy between standardized mean difference in cobalt and smd packages
Let's suppose I have weighted my population using WeightIt
...
- 159
0
votes
0
answers
108
views
Distribution of Weighted Sample Variance
Say we have $N$ independent Gaussian variables $X_i$, each drawn from a mean-zero Gaussian with variance $\sigma_i^2$, where in this case I assume I know the $\sigma_i^2$. Define the weighted average ...
- 203
1
vote
1
answer
565
views
How to fit data with asymmetric error bars?
General question: How do I fit a model to data when the data points have asymmetric error bars? What is the cost function I use to calculate residuals, and from that, how do I calculate confidence ...
- 146
0
votes
1
answer
466
views
Calculate risk ratio in weighted population
I have a propensity score weighted population (using IPTW) and I want to compute risk ratios on my weighted population. For this, I am using a weighted Poisson regression.
Let's suppose that "...
- 159
3
votes
1
answer
1k
views
Unbalanced variables after IPTW - entropy balancing?
After using inverse probability of treatment weighting (IPTW) on the variables of my dataset, there is still an imbalance in one covariate between the two groups. My outcome is binary (yes/no) and it ...
- 159
3
votes
1
answer
909
views
Recognize extreme weights in inverse probability treatment weighting
I have some doubts about how to recognize if there are extreme weights after balancing my population with inverse probability treatment weighting.
For instance, let's look at these results [code at ...
- 159
1
vote
1
answer
592
views
Calculating outcomes in IPTW
I have a few questions concerning what to do next after balancing your population with IPTW.
This is a code just to have an example:
...
- 159
3
votes
1
answer
2k
views
Reduced effective sample size in balanced population after inverse probability treatment weighting
I am new to inverse probability treatment weighting (IPTW) and I am trying to understand it. However, I am not a statistician and I am having some troubles with advanced statistics concepts.
As far as ...
- 159
2
votes
1
answer
1k
views
Create table before and after IPTW with frequencies in R
I am doing propensity score weighting with the package "Weightit" and I want to have a Table 1 "Before and After Weighting" like this one:
As you can see, data are displayed ...
- 159
1
vote
0
answers
25
views
Amplification effect of retweets on uncertainty
Consider you are scoring tweets for tone based on some sentiment analysis implementation. Each tweet has hypothetically a 90% chance of being correctly scored, while 10% get it wrong for whatever ...
- 647
0
votes
0
answers
236
views
Estimate weights given a weighted geometric median
Problem 1
Given a set of points and their weighted geometric median, find the weights associated with each point.
For instance, given the points ...
- 45
0
votes
0
answers
60
views
Combining standard errors of two separate variables
I have a dependent variable associated with a SE and an independent variable associated with a SE. I would like to weight my regression such that both of these SEs are taken into account. What is the ...
- 61
2
votes
1
answer
74
views
Conducting multiple weighted t.tests to count over what fraction of a population a model performs significantly better or worse than another
I have two different model families $A_i$ and $B_i$ each of which predicts a continuous value for an item $i$ at various points in time throughout a given day. For a given day, and item $i$, I can ...
- 1,003
0
votes
0
answers
41
views
How to get weighted mean with two weighting sources
I have some data on a cyclists distance traveled and hours it took.
...
- 23
1
vote
0
answers
34k
views
Calculate probability of dying in school shooting using spatial statistics
Some public commentators say that the chance of dying in a school shooting is very low, so they conclude that Americans over-react to them.
https://www.washingtonpost.com/outlook/school-shootings-are-...
- 359
0
votes
0
answers
40
views
Smoothing time series with Adjusted R2-weighted averages
I have two parameters (a,b) resulting from an exponential estimation of a curve. I have estimated this curve every hour for one month. In other words, I have a total of 720 parameters a and b, and I ...
- 11
2
votes
1
answer
71
views
weighted median result is integer
I am computing the weighted median for a dataset using the R reldist package.
The input data are integers but my result is an integer number too.
Should't my result ...
- 344
0
votes
0
answers
154
views
How to bias-correct proportions for a prop.test in R?
Let's say I ask 200 participants in a study what experience brought the most joy to their lives:
A, B, C, or D.
...
- 1
1
vote
0
answers
28
views
How reliable is conclusion drawn out from a known data?
Most of time conducting experiment is expensive. Suppose I and my collaborator decided to use a known databank(NHANE, NCI's data) to do some digging with some untested hypothesis given. Most of time ...
- 1,465
0
votes
0
answers
169
views
How to assign weights to a set of ordered objects?
I have been trying to assign weights to set of objects. The problem that I have is as following.
I have a set of objects ordered either ascending or decending.
. Each object shall be provided a weight ...
- 103
3
votes
1
answer
242
views
Can I use oversampling to increase sample weights?
I am working on a balanced binary classification problem. I don't need to adjust the proportion of either of the classes, as it's already 50/50 . However, some of the samples are more valuable than ...
- 1,571
0
votes
1
answer
39
views
Age adjustment or Age standardization state of the art?
I want to run some comparisons of a quantity between different populations. Such quantity comes from a model that includes, among other variables, the variable age. So, one of the variables affecting ...
- 1
3
votes
0
answers
111
views
How do I calculate a finite-population correction for a weighted sample?
I have a weighted sample from a population. These are probability weights (as samples are taken with unequal probability). What's the variance in the estimated total? All samples are exchangeable (...
- 1,165
0
votes
0
answers
73
views
How to weight my data in SAS - categorical and numerical variables
I've collected data from 1000 animals (categorical and numerical variables) from 2 different farms. My problems are: one farm corresponds to 68% of my data, in some of my variables I've observed more ...
- 1
0
votes
0
answers
40
views
Weigh the percentages based on the number of inhabitants
I am analyzing the diffusion of a service in some regions. Is there a legitimate criterion for weighing percentages by the number of inhabitants? I would like to "improve" the results of the ...
- 131
1
vote
0
answers
1k
views
How do we calculate statistics on data weighted by relative frequencies?
I have the following data with some values and their corresponding weights.
groups
values
weights
a
1
0.25
a
3
0.75
b
8
1.00
The weights represent relative frequencies of each value in the group ...
- 113