New answers tagged python
0
I was going through same problem, After some research here is my solution:
If you are using tensorflow :
Multi label loss:
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=tf.cast(targets,tf.float32))
loss = tf.reduce_mean(tf.reduce_sum(cross_entropy, axis=1))
prediction = tf.sigmoid(logits)
output =...
1
The intercept estimate of 15.724 is the global intercept, around which the (72) random intercepts vary. The random intercepts are estimated as samples from a normal distribution with a variance of 40.384 and mean of zero - hence the need for a global (fixed) intercept.
0
I have done the same thing before, you should be able to find the approach from bioinformatics books. But here I will give some simple solution, which might not the approach you should take but it's just a hint of where you should look into.
The non-conditional probability of certain amino acid, you would think it should be 1/20, which is not the case. ...
-1
Since I posted this question over a year ago, I've taken a class on generalized linear models and learned a lot. Since this post is viewed somewhat frequently, I thought I would add some guidance I wish I had at the time.
Link functions to consider for the Gamma model:
Log - Forces the predictions/output to "respect the domain" by ensuring all predicted ...
0
Here is a somewhat tangential answer that might help you think about the issue (it doesn't address the discrepancies between packages though). Too long to add as a comment so I'm noting it here.
Confidence intervals for the survival function of the Kaplan-Meier curves are calculated at the time of each event (so you get the same type of step function that ...
0
Multiply your AR(2) polynomial by each of your input series polynomials . It is straight forward in this case but it can get complicated if you have moving average structure in your error term or dynamic structure in one or more of your exogenous input AND if you have a constant in your model.
I don't know if your software has this feature (extract pure ...
0
At the risk of overreaching, I think you're putting the cart before the horse. Your startup has objectives but no data. The question you really need to answer is: "If my company had data, would it even make a difference?" Unless your companies business model is 100% ML driven, most likely its growth objectives are not initially dependent on strong ML ...
0
One way addressing problems where you have little(or no data) is 'replacing' it with priors.
Bayesian modeling has the priors concept at its core.
If your using python take a look at pymc(and start off as simple as you can).
3
Just write unit tests.
The scenario you're describing may not be that uncommon. For example, when a team works on putting machine learning algorithm into production, they may divide the tasks between several developers, e.g. someone prepares data preprocessing pipeline, someone else the machine learning model prototype, etc. In such scenario you may need to ...
0
I would say simply initialize the model's weights at random. Which type the different features of your data are doesn't really make a difference here, as in any case you'd have to convert all of them to the same type (typically float) before feeding them to your model.
0
After discussing with Ben Letham, one of the Facebook Prophet library creators, we've found a suitable solution to the standardisation problem.
Relative comparisons remain valid: If we subtract C from each weekly(t), then the difference between weekly(t) and weekly(t+1) (e.g., Monday vs. Sunday) will be the same no matter what C is. So suppose we have ...
2
What I do in these occasions is that I compute the dataset mean and std beforehand. So I do a single pass over the whole dataset just to compute the mean and std. Then I use these two numbers to normalize each image before I feed them to the model.
Another option would be to use a running mean and std, which would converge to the true mean and std after the ...
0
A lot depends on how you prepare the data. It would be very valuable if you use some external data with respect to postal code as suggested by @Dan. For example you can try to build a simple regression model that tries to predict number of damage claims. For this, inputs may be population, average income, month of the year, number of insurance holders in ...
3
it's not true that deep learning models outperforms the traditional machine learning or even statistical models. It all depends on the problem, what kind of data you have, how many, what's the dimensional etc.
In general, DL might outperform other models in situations where you have a lot of data (10k, 100k, 1m) and when the model is very nonlinear such as ...
1
You have to transform your data so that if two data points are close to each other they are more likely to be related.
Your times you can for instance convert to seconds from 1. Januar 1970, 00:00 (Unix time)
The postal code is most likely already in a way that two postal codes that are close to each other numerically are also close to each other ...
0
You must use k-modes(an extension of k-means) for categorical problems. I had a similar business problem and I used K-modes clustering algorithm and solved my issue. Below is the URL that will help you understand the algorithm.
https://www.youtube.com/watch?v=b39_vipRkUo
2
The ACF is the ratio between the covariance and the variance ..If the variance is zero the ACF is inestimable as the covariance is also zero.
If the variance is zero the ar(1) coefficient is zero AND is redundant to the estimated mean yielding a standard error of 0. .
1
Linear regression fits the model
$$(1) \ \ y = \theta_1 z_1 + ... \theta_K z_K + \epsilon$$
where $z_1,...,z_K$ are data - you can call them "features" if you want - and $\theta_1,...,\theta_K$ are parameters to be estimated, $\epsilon$ is unobserved error and $y$ is observed output variable.
In polynomial regression the model is
$$(2) \ \ y = \theta_1 +...
2
1) almost so.
The proportions can be considered as conditional probability of finding a white person, when looking for someone randomnly in each state. You may then define a joint probability function for the varibles "county" and "skin color". This means that, if you want to turn your list of proportions in a unique distribution, you have to include ...
1
The t-test is a test of the difference between two means and KDE plots are not always a good way to look for that. Plus your sample size is pretty big, which makes small difference significant. Plus, although it's hard to tell, it looks like there is an outlier around -1 but only for y.
answered Oct 26 at 21:36
0
Are you dealing with simple or complex seasonality in your data? For example, monthly data may show monthly seasonality whereas hourly data may show daily and weekly seasonality.
I think your decomposition would need to reflect whether the seasonality in your data is simple or complex. Simple seasonality would produce a single seasonal component when ...
2
on repeated trials it shows that daily seasonality is pretty much zero (though my algorithms suggest that there is one.)
I am not sure about Prophet, but STL will fit a seasonal component whether or not one is present. This may account for your observation.
I personally would fit seasonal and non-seasonal models to your data and assess forecast accuracy on ...
answered Oct 24 at 13:09
S. Kolassa - Reinstate Monica
58.2k10 gold badges113 silver badges216 bronze badges
1
It is not a problem to have several observations on the same day. The prophet library allows it.
The reason why it is not a problem is because prophet is a regression method and it is no problem to give two observations with the same covariate to a regression method.
(The prophet paper: https://doi.org/10.7287/peerj.preprints.3190v2)
1
There are a few problems here some of which are just terminology. You don't have a distribution, you have data. You want to estimate a distribution from your data. An MLE isn't a distribution, it's a type of estimation--the two distributions you are estimating are a KDE and a Gaussian (which you are estimating with MLE).
Using KS hypothesis testing, you ...
0
My first question would be, why would you use deep learning for tokenization (performance and accuracy-wise)? Apart from some corner cases it's a pretty straightforward task. Existing statistics-based systems and rule-based methods are fast and quite accurate, unlike deep learning approaches which are computationally expensive and require large datasets to ...
1
Graphical comment: Here is a session in R, which takes random samples of size 500 from
uniform, gamma, exponential, and normal distributions.
Descriptive statistics are shown for each sample, with very brief
comments about the sample statistics. Histograms of all four samples
at the end. (Although $n = 500$ is a relatively large sample size, not
even a ...
0
Conv1D has a parameter called data_format which by default is set to "channels_last". So, by default it expects inputs to be of the form (batch_size,steps,channels). To quote from the Documentation:
data_format: A string, one of "channels_last" (default) or "channels_first". The ordering of the dimensions in the inputs. "channels_last" corresponds to ...
0
Following is the link for the Python wrapper for CMU tweet tagger,
https://github.com/ianozsvald/ark-tweet-nlp-python
Its a Simple Python wrapper around runTagger.sh of ark-tweet-nlp. It passes a list of tweets to runTagger.sh and parses the result into a list of lists of tuples, each tuple represents the (token, type, confidence). Download the ark-tweet-...
1
No.
If you read the code, it shows that the argument to regularizers.l2() matches your definition of $\lambda$. regularizers.l2() is just an alias that calls L1L2.
class L1L2(Regularizer):
"""Regularizer for L1 and L2 regularization.
# Arguments
l1: Float; L1 regularization factor.
l2: Float; L2 regularization factor.
"""
...
1
Are Python classes in Spark notebooks beneficial or common in the data scientist community? [closed]
In general you are writing pipelines and you relay on fact that "you run n+1 cell only if previous n completed with success" and you don't have to rely on super structurized code. On the other hand you want to have some functionalities closed in classes/functions and then run them with different parameters, but don't stick too much to classes since 90% of ...
1
Sample size for ANOVA. If you know the number of groups, the common variance of the group populations, the size of the difference in means you want to detect, and the power with which you want to detect that difference, then you can find the number of replications needed in each group. Most statistical software packages have a 'power and sample size' ...
3
At first, I thought a solution might require summing over all possible sequences of draws, with a resulting combinatorial explosion. But, thinking about the problem from a slightly different angle, there's actually an efficient way to approximate the solution. It only requires solving a simple system of linear equations.
Assuming the target probabilities ...
0
Initially, and even after some of the exchanges in Comments, I'm not sure I understand the question. Is this anything like what you're looking for? If not, maybe there are ideas here you can use. Everything below is in R:
pct = c(99,97,95,82, 68,57,48,33, 11,9,1,0)
prob = pct/sum(pct); prob
[1] 0.165000000 0.161666667 0.158333333 0.136666667 0.113333333 0....
Top 50 recent answers are included
