Measure how representative a feature sample is of the feature space, and the effect on statistical significance

Question

I may not be asking this question correctly, but my curiousity concerns separating the statistical significance of sampling about features (as opposed to whole samples), how to measure this, and how to integrate this into a model. Because of this, I will pose my question as a word problem:

Say you have been taking data for 15 years on giving out free hats to members of a group across many locations, as you are trying to model how many hats you should actually buy so that as few as possible are wasted (like a supply chain), and you plan on the hats being customized for each location. You don't have enough samples of handing out hats at each individual location (say you've done this 15 times for each location), but you have 100 locations, so given the assumption that behavior across locations is a function of location-agnostic features, you can use behavior from all locations to predict the number of hats you need for each individual locations.

So for this example, every sample is one instance of handing out hats, and each has a handful of these "potentially agnostic" features, like

• how long you were handing out hats for
• where the hats were handed out (city, region, time zone, etc)
• day of the year
• the expected size of the group at the location at that time
• number of hats you had
• the weather the day you gave them out
• whether it was a holiday locally or not
• the type of hats
• most recent GDP growth was that quarter (idk maybe people like free t-shirts less in an up-economy), and
• (obviously) the number of shirts you actually handed out.

...but then you throw in a feature like "number of comet appearances that year". Now, you still have 1500 samples that all have a "comets" feature, but because this feature varies so slowly, and the max number of "comets" you can have in your dataset could be but is likely not a representative sample (because sampling was over a 15 year period), it is highly improbable that behavior based on "comets" will be accurate over many future sample. Moreover, it could be the case that this poorly-sampled feature seems like it correlates well to the hats problem, but logically we know it would be a mistake to trust it. From my perspective, this makes it seem like the dataset has statistically significant amount of data, but some subset of the features are not representative.

My question is this: does this phenomenon have a name or formal definition in statistics, how does one measure if a feature sample in a dataset is representative of the range of that feature space, and what can we do to mitigate models from correlating falsely to such poorly sampled features?

Answer 1

does this phenomenon have a name or formal definition in statistics,

Not that I know of. "Overfitting" sort of describes this, but not exactly. There's also the term "spurious feature".

how does one measure if a feature sample in a dataset is representative of the range of that feature space

You can do statistical tests on a particular feature versus the response variable. For instance, you can get the p-value of their correlation.

what can we do to mitigate models from correlating falsely to such poorly sampled features?

Regularization is designed to address overfitting, but it isn't generally customized to different features. You could come up with a more complicated variant of regularization, such as giving different features different $\lambda$'s based on their statistical significance. You could also scale the features based on their significance, which would in many cases be equivalent.