Skip to main content
Cross Validated

Return to Question

added 12 characters in body
Source Link
zoned post meridiem
zoned post meridiem
  • 163
  • 2
  • 6

I'm relatively new to stats, and I have a large dataset of strings (n=114541) for which I'm interested in a binary event: whether X occurs or not (specifically, whether a specific language construct is present in thea given string or not). X occurs rarely (443/n). X is poisson-distributed.

The strings in the dataset are arranged in a time series, and I'm trying to determine whether there's a significant change in the occurrence of X before and after a given moment in time. Because X follows a poisson distribution, I can easily calculate lambda before and after that moment in time (let's say that lambda1 = 127/35465, and lambda2 = 316/79076).

lambda1/lambda2 (or lambda1-lambda2) gives me an indication of whether the probability of X changed. What I need to know is if that change is significant (i.e., in the example above, if the difference between 0.00399 before the switch point and 0.0035 after is significant).

I've read that the C-test can test for significance between two lambda values in a poisson distribution, but I'm unsure if that's the right test. And if it is: is there a way of running it in Scipy (I've looked through the documentation, and couldn't find any mention of it.)?

I'm relatively new to stats, and I have a large dataset of strings (n=114541) for which I'm interested in a binary event: whether X occurs or not (specifically, whether a specific language construct is present in the string or not). X occurs rarely (443/n). X is poisson-distributed.

The strings are arranged in a time series, and I'm trying to determine whether there's a significant change in the occurrence of X before and after a given moment in time. Because X follows a poisson distribution, I can easily calculate lambda before and after that moment in time (let's say that lambda1 = 127/35465, and lambda2 = 316/79076).

lambda1/lambda2 (or lambda1-lambda2) gives me an indication of whether the probability of X changed. What I need to know is if that change is significant (i.e., in the example above, if the difference between 0.00399 before the switch point and 0.0035 after is significant).

I've read that the C-test can test for significance between two lambda values in a poisson distribution, but I'm unsure if that's the right test. And if it is: is there a way of running it in Scipy (I've looked through the documentation, and couldn't find any mention of it.)

I'm relatively new to stats, and I have a large dataset of strings (n=114541) for which I'm interested in a binary event: whether X occurs or not (specifically, whether a specific language construct is present in a given string). X occurs rarely (443/n). X is poisson-distributed.

The strings in the dataset are arranged in a time series, and I'm trying to determine whether there's a significant change in the occurrence of X before and after a given moment in time. Because X follows a poisson distribution, I can easily calculate lambda before and after that moment in time (let's say that lambda1 = 127/35465, and lambda2 = 316/79076).

lambda1/lambda2 (or lambda1-lambda2) gives me an indication of whether the probability of X changed. What I need to know is if that change is significant (i.e., in the example above, if the difference between 0.00399 before the switch point and 0.0035 after is significant).

I've read that the C-test can test for significance between two lambda values in a poisson distribution, but I'm unsure if that's the right test. And if it is: is there a way of running it in Scipy (I've looked through the documentation, and couldn't find any mention of it)?

Source Link
zoned post meridiem
zoned post meridiem
  • 163
  • 2
  • 6

C-Test for comparing poisson means in Scipy

I'm relatively new to stats, and I have a large dataset of strings (n=114541) for which I'm interested in a binary event: whether X occurs or not (specifically, whether a specific language construct is present in the string or not). X occurs rarely (443/n). X is poisson-distributed.

The strings are arranged in a time series, and I'm trying to determine whether there's a significant change in the occurrence of X before and after a given moment in time. Because X follows a poisson distribution, I can easily calculate lambda before and after that moment in time (let's say that lambda1 = 127/35465, and lambda2 = 316/79076).

lambda1/lambda2 (or lambda1-lambda2) gives me an indication of whether the probability of X changed. What I need to know is if that change is significant (i.e., in the example above, if the difference between 0.00399 before the switch point and 0.0035 after is significant).

I've read that the C-test can test for significance between two lambda values in a poisson distribution, but I'm unsure if that's the right test. And if it is: is there a way of running it in Scipy (I've looked through the documentation, and couldn't find any mention of it.)