I was initially dealing with huge sets of couple of values. I used a custom heuristic to compute a score from each couple of values and turn the set into an array of values. I sorted it and assigned each value to x and its normalized rank (between 0 and 1, both excluded) to y.

Here is an example :

[![enter image description here][1]][1]

I tried to fit a sigmoid function but I think it is misled by the huge amount of centered dat and doesn't account for extreme values, whereas I actually plan on using this new function to assign as score between 0 and 1 to any new value. Right now with that solution it would rank to roughly 0.93 any value past a threshold of around 31.

How can I lower the impact of the centered points? I thought about simply removing some but I don't know the right way to do that, if it is even the right way.

Here is my code :

    import json
    import numpy as np
    from scipy.optimize import curve_fit
    import matplotlib.pyplot as plt
    
    filename = "scores.json"
    
    with open(filename, 'r') as f:
        data = json.load(f)
    
    x_values = [point['x'] for point in data]
    y_values = [point['y'] for point in data]
    
    x_data = np.array(x_values)
    y_data = np.array(y_values)
    
    def sigmoid(x, L, x0, k):
        return L / (1 + np.exp(-k * (x - x0)))
    
    initial_guess = [1, np.median(x_data), 1]
    
    params, covariance = curve_fit(sigmoid, x_data, y_data, p0=initial_guess, maxfev=10000)
    
    L, x0, k = params
    
    print(f"Optimized parameters: L = {L}, x0 = {x0}, k = {k}")
    
    plt.scatter(x_data, y_data, marker='+', label='Data')
    
    x_fit = np.linspace(min(x_data), max(x_data), 400)
    y_fit = sigmoid(x_fit, *params)
    plt.plot(x_fit, y_fit, label='Fitted Sigmoid', color='red')
    
    plt.legend()
    plt.show()

And here are the found parameters :

    # Optimized parameters: L = 0.9305200252602871, x0 = 2.107303517527327, k = 0.24761667539895446

  [1]: https://i.sstatic.net/mdM8LWxD.png