yoav_aaa
  • Member for 5 years, 6 months
  • Last seen this week
Data scientist interview question: Linear regression low $R^2$ and what would you do
5 votes

I'm not sure what the interviewer was after but when facing a poorly preforming model these are the things I consider and an answer I would love hearing as an interviewer (been interviewing for a ...

View answer
A visual explanation regarding a generalised Pearson's correlation for two variables
Accepted answer
4 votes

This is one option: There is zero correlation for groups within vertical lines. But overall correlation is positive.

View answer
Is regression possible for unstructured text?
Accepted answer
4 votes

Yes. The second format fits a straight-forward classification problem. We have multiple classes(Red/Yellow/Green/Blue) and text based features. There are two main components needed in order to creat ...

View answer
Does poor clustering results entail poor classification results?
Accepted answer
2 votes

No, similar distribution of Y's does not entail poor classification results. See this for example, two features X1, X2 and binary response variable(colored green and red). Clustering results are ...

View answer
Estimate size of each group with uncertainties from a KDE plot
2 votes

Look into Gaussian mixture models. You basically assume an existence of 2 sub populations in your data. Fit a model with relevant parameters per each of the populations. Then using the parameters ...

View answer
data set 'bootstrapping'. Is there a name for this?
2 votes

I can think of a formal technique fitting this situation. semi-supervised learning. In the heart of this technique lays the assumption that all data(labeled and not labeled) have a common structure ...

View answer
Why do actuaries roll high values of some variables like seats (categorical variable) into a lower value?
2 votes

This is a part of the feature engineering stage in modeling. Two key concepts in features engineering are: 1) Features should represent reality(or our assumption about reality). 2) Features different ...

View answer
PCA explained variance low
2 votes

PCA is a decomposition process. Meaning it takes an existing vector space and transforms it into another vector space. If i understand correctly, you chose the first N components of the transformed ...

View answer
DQN - agent doesn't improve policy
1 votes

One possible explanation might be a result of the game setting. Regardless of the cell the player is in - if the probability($P$) of it winning equals(or less) the probability of it meeting an enemy(...

View answer
How to calculate tf-idf for a single term
Accepted answer
1 votes

The modeling strategy suggested in the paper refers to temporal representation(both frequency and context) of words. From what I understand, they attempt to learn the changes in these representations ...

View answer
Uncertainty in measurement error
1 votes

I'm not sure if this is what you aiming for but this is one approach that might be suitable. Start with modeling the over 6 error as a probabilistic event. I.E, lets say that for each measure event $...

View answer
Using log-log graph to find equation of power law relationship?
Accepted answer
1 votes

I did the following calculation and got a different y value for x=0.25. Intercept and slope are similar to OP's question. x = (0.25, 1, 2, 2.75, 4.25, 6.5, 8, 13.25, 16.25, 19.25, 19.75, 26.5, 31, 37....

View answer
T-test : Average Time Spent on Website
Accepted answer
1 votes

T-Test assumes samples are independent of each other. In the first case where each sample is identified by a user-date tuple, you might be violating this assumption. For example imagine the hypotheses ...

View answer
Dimensionality reduction of events data
1 votes

As I understand it you are facing a problem of ordering(and then ranking) an unsupervised set of users. These type of problems can be tackled using a clustering algorithm. There are many such ...

View answer
Clustering of time series and their transformations
Accepted answer
1 votes

A clear cut answer would depend on the time series, actual transformations and clustering algorithm you have in mind. Reason is most clustering algorithms are based on different similarity function ...

View answer
Expected value of linear regression coefficient over different probability distributions
1 votes

There is a concept named Asymptotic Normality. In our case means that over high enough number of estimations, the estimators distribution is normal. Technically this derives: $\hat{\beta} \sim \...

View answer
Detecting outliers in contextual time-series data
Accepted answer
1 votes

I know of three possible problem settings: 1. Detecting contextual anomalies in the time series - the anomalies are individual instances of the time series which are anomalous in a specific context, ...

View answer
Machine Learning dealing with NaN values
Accepted answer
1 votes

Handling NaN values belongs to the feature engineering part of developing machine learning models. Different types of models make different assumption about the underline features distributions and ...

View answer
What techniques can be used for predictive modeling which incorporate sector specific information
1 votes

I would attempt modeling this type of revenue data as a time-series problem. Since you assume some sectors will go nil, you effectively assume the data is not stationary(average changes across time). ...

View answer
Why and when would one begin with PCA on $X$ when predicting $y$?
1 votes

PCA can be used as a step for pre-processing the model features. From my experience: Pros: Reducing dimensionality. Reducing noise and possibly improving model performance. Cons: Sensitivity to ...

View answer
Estimating the error in the standard deviation
Accepted answer
1 votes

Its common to assume that the distribution-variance of the sample and that of the entire population are similar, so no surprise you didn't find an answer quickly. None the less you can use this ...

View answer
How to build feature vectors from profile data
1 votes

As for the number of common friend i suggest using the Jacard index. Its basically the ratio between shared friends both friends.

View answer
How to apply feature engineering for classification efficiently using polynomials
1 votes

Some things to consider: Saying your sample size is constant, then each additional feature n will reduce you generalization by log(n) factor -- you can find a good explanation for this here: http://...

View answer
Which statistical test to use for binned data?
0 votes

One potential direction is using contingency table. It is a good way of 'modeling' the relations between frequency distributions of more than one variable.

View answer
How to set up a machine learning model with no data
0 votes

One way addressing problems where you have little(or no data) is 'replacing' it with priors. Bayesian modeling has the priors concept at its core. If your using python take a look at pymc(and start ...

View answer
What kind of statistical model I should use?
0 votes

My two cents: I would frame this as a classification problem where target label is 'died/survived'. Features are demographics, physical and transportation method. Assuming I have enough good data, ...

View answer
what's the use of lag? when using it do I have to lag my dependent variable or is it only the independent variable?
0 votes

Using log transformations and lagged features are two separate techniques that could be combined together. In linear models, when regressors ${x_1, x_2..x_n}$ have multicollinearity one possible ...

View answer
How to determine a sample size
0 votes

As I understand it, your question is consisted of two smaller questions: What is the needed sample size to evaluate my classification model(algorithm)? How do I select my evaluation(labeled) data ...

View answer
Compare same time serie from two different sources
Accepted answer
0 votes

There are several algorithms/methods for measuring similarity between two time series. A first possible step in deciding which one will work best is comparing the two time series meta features: Length,...

View answer
Cluster analysis with boosting models for better predictions?
0 votes

For me it makes sense using clustering as a method of feature engineering. Assuming your intuition on the relation between predictors clusters and target variable is correct, helping models finding ...

View answer