How to statistically discover significant process change effectiveness?

Question

I am currently working on a project where I need to assess the effectiveness of changes made in a production process. Our initial success rate was 50%, and after making some alterations, we've observed varying success rates. I want to accurately determine at what point these changes can be considered statistically significant.

To achieve this, I plan to use two statistical approaches:

• Traditional Null Hypothesis Testing: To assess if the improvements are statistically significant.
• Bayesian Analysis: To update our understanding of the process's effectiveness based on new data.

Additionally, my goal is to identify specific change points - success rates at which the p-value becomes significant in the null hypothesis testing and where the Bayesian probability exceeds 50%.

I intend to:

• Generate data reflecting a 50% initial success rate.
• Apply null hypothesis testing and Bayesian analysis to this data.
• Evaluate different success rates (e.g., 70%, 80%, 90%) to understand their impact on p-values and Bayesian probabilities.
• Identify and plot the specific success rates that mark significant changes as per both methods.

Could someone guide me on how to implement this approach correctly, particularly in identifying these change points? Also, are there any suggestions for improving this methodology or alternative methods I should consider?

Answer 1

Self-Answer:

Since my question about determining significant change points in process effectiveness using Null Hypothesis Testing and Bayesian Analysis didn't receive any responses, I went ahead and worked out a solution myself. I'm sharing my findings here in case it helps anyone else facing a similar challenge.

Approach Summary:

I focused on two main statistical methods:

• Traditional Null Hypothesis Testing for assessing statistically significant improvements.
• Bayesian Analysis to update our belief about the effectiveness based on the new data.

The key was to identify specific change points - those success rates where the p-value in the null hypothesis testing becomes significant, and the Bayesian probability exceeds 50%. Detailed Implementation: Initial Setup and Data Generation:

I created a baseline scenario with a 50% success rate in our production process, using 32 total items and 16 successes.

Applying Two Statistical Methods:

• Null Hypothesis Testing (T-Test): I compared success rates before and after implementing process changes. The objective was to determine if the differences were statistically significant, marked by lower p-values.
• Bayesian Analysis: I used this to update our understanding of process effectiveness based on the observed data. I applied a Beta distribution to estimate the probability of effectiveness post-changes.

Evaluating Different Success Rates:

I tested success rates of 70%, 80%, and 90% to observe the impact on the p-values and Bayesian probabilities. Identifying Change Points:

• For the T-Test, I interpolated to find the success rate at which the p-value is 0.05.
• In Bayesian analysis, I similarly interpolated to find the success rate at which the probability of effectiveness exceeds 50%.

Plotting the Results:

I plotted the results to visually represent the change points as per both statistical methods, with red dashed lines marking these points.

Python Code:

import numpy as np
from scipy.stats import ttest_ind, beta
from scipy.interpolate import interp1d
import matplotlib.pyplot as plt

# Sample data and initial success rate
total_items = 32
success_before = 16  # 50% of 32
data_before = np.concatenate([np.ones(success_before), np.zeros(total_items - success_before)])

# Null Hypothesis Testing and Bayesian Analysis for varying success rates
success_rates_to_test = [0.7, 0.8, 0.9]
p_values = []
probabilities = []

for rate in success_rates_to_test:
    success_after = int(rate * total_items)
    data_after = np.concatenate([np.ones(success_after), np.zeros(total_items - success_after)])

    # T-Test
    _, p_val = ttest_ind(data_before, data_after)
    p_values.append(p_val)

    # Bayesian Analysis
    alpha_prior, beta_prior = 1, 1
    alpha_posterior = alpha_prior + success_after
    beta_posterior = beta_prior + (total_items - success_after)
    probability = beta.cdf(rate + 0.05, alpha_posterior, beta_posterior) - \
                  beta.cdf(rate - 0.05, alpha_posterior, beta_posterior)
    probabilities.append(probability)

# Interpolation for Null Hypothesis Testing to find the success rate at which p-value is 0.05
interp_func_p_value = interp1d(p_values, success_rates_to_test, kind='linear', bounds_error=False, fill_value="extrapolate")
success_rate_at_p_05 = interp_func_p_value(0.05)

# Creating the interpolation function for Bayesian probabilities
interp_func_bayesian = interp1d(probabilities, success_rates_to_test, kind='linear', bounds_error=False, fill_value="extrapolate")

# Identifying the change point for Bayesian analysis
threshold_probability = 0.5
bayesian_change_point = interp_func_bayesian(threshold_probability)

# Plotting results with both T-Test and Bayesian change points
plt.figure(figsize=(12, 6))

# T-Test Plot with Interpolated Change Point
plt.subplot(1, 2, 1)
plt.plot(success_rates_to_test, p_values, marker='o', linestyle='-', color='blue', label='Original Data')
plt.axvline(x=success_rate_at_p_05, color='r', linestyle='--', label='Interpolated Change Point at {:.2f}'.format(success_rate_at_p_05))
plt.xlabel('Success Rate')
plt.ylabel('P-Value')
plt.title('T-Test P-Values with Interpolated Change Point')
plt.legend()
plt.grid(True)

# Bayesian Analysis Plot with Change Point
plt.subplot(1, 2, 2)
plt.plot(success_rates_to_test, probabilities, marker='o', linestyle='-', color='green', label='Original Data')
plt.axvline(x=bayesian_change_point, color='r', linestyle='--', label='Change Point at {:.2f}'.format(bayesian_change_point))
plt.xlabel('Success Rate')
plt.ylabel('Probability')
plt.title('Bayesian Probabilities for Different Success Rates')
plt.legend()
plt.grid(True)

plt.tight_layout()  # Adjust layout to ensure visibility of all elements
plt.show()

Outcome:

I believe the plots demonstrate the points of significant change in process effectiveness as indicated by both the T-Test and Bayesian Analysis.

Conclusion:

This exercise was quite insightful. I think it affirms that the combination of Null Hypothesis Testing and Bayesian Analysis is likely effective for identifying significant change points in process effectiveness. I hope my approach and the solution I've shared here will be helpful to others facing similar analytical challenges.