I was initially dealing with huge sets of couple of values. I used a custom heuristic to compute a score from each couple of values and turn the set into an array of values. I sorted it and assigned each value to x and its normalized rank (between 0 and 1, both excluded) to y.

Here is an example :

I tried to fit a sigmoid function but I think it is misled by the huge amount of centered dat and doesn't account for extreme values, whereas I actually plan on using this new function to assign as score between 0 and 1 to any new value. Right now with that solution it would rank to roughly 0.93 any value past a threshold of around 31.

How can I lower the impact of the centered points? I thought about simply removing some but I don't know the right way to do that, if it is even the right way.

Here is my code :

import json
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

filename = "wukong_wScore.json"

with open(filename, 'r') as f:
    data = json.load(f)

x_values = [point['x'] for point in data]
y_values = [point['y'] for point in data]

x_data = np.array(x_values)
y_data = np.array(y_values)

def sigmoid(x, L, x0, k):
    return L / (1 + np.exp(-k * (x - x0)))

initial_guess = [1, np.median(x_data), 1]

params, covariance = curve_fit(sigmoid, x_data, y_data, p0=initial_guess, maxfev=10000)

L, x0, k = params

print(f"Optimized parameters: L = {L}, x0 = {x0}, k = {k}")

plt.scatter(x_data, y_data, marker='+', label='Data')

x_fit = np.linspace(min(x_data), max(x_data), 400)
y_fit = sigmoid(x_fit, *params)
plt.plot(x_fit, y_fit, label='Fitted Sigmoid', color='red')

plt.legend()
plt.show()

And here are the found parameters :

# Optimized parameters: L = 0.9305200252602871, x0 = 2.107303517527327, k = 0.24761667539895446

I was initially dealing with huge sets of couple of values. I used a custom heuristic to compute a score from each couple of values and turn the set into an array of values. I sorted it and assigned each value to x and its normalized rank (between 0 and 1, both excluded) to y.

Here is an example :

I tried to fit a sigmoid function but I think it is misled by the huge amount of centered dat and doesn't account for extreme values, whereas I actually plan on using this new function to assign as score between 0 and 1 to any new value. Right now with that solution it would rank to roughly 0.93 any value past a threshold of around 31.

How can I lower the impact of the centered points? I thought about simply removing some but I don't know the right way to do that, if it is even the right way.

Here is my code :

import json
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

filename = "scores.json"

with open(filename, 'r') as f:
    data = json.load(f)

x_values = [point['x'] for point in data]
y_values = [point['y'] for point in data]

x_data = np.array(x_values)
y_data = np.array(y_values)

def sigmoid(x, L, x0, k):
    return L / (1 + np.exp(-k * (x - x0)))

initial_guess = [1, np.median(x_data), 1]

params, covariance = curve_fit(sigmoid, x_data, y_data, p0=initial_guess, maxfev=10000)

L, x0, k = params

print(f"Optimized parameters: L = {L}, x0 = {x0}, k = {k}")

plt.scatter(x_data, y_data, marker='+', label='Data')

x_fit = np.linspace(min(x_data), max(x_data), 400)
y_fit = sigmoid(x_fit, *params)
plt.plot(x_fit, y_fit, label='Fitted Sigmoid', color='red')

plt.legend()
plt.show()

And here are the found parameters :

# Optimized parameters: L = 0.9305200252602871, x0 = 2.107303517527327, k = 0.24761667539895446