4 added 380 characters in body
source | link

Can you provide[d] an R based solution?

No, both on principle (I can help you understand better the approach, not solve the problem) and because your description is too vague to write an R code.

given a non-uniform sampling of the parameter space

You should provide a description of this non-uniform sampling on $p$ or $P$. From the available description, I can only guess you are using a certain proposal distribution, say $h(p)$, instead of the correct prior, say $g(p)$. In which case, each simulation should be [importance] weighted by $g(p)/h(p)$.

Can you provide[d] an R based solution?

No, both on principle (I can help you understand better the approach, not solve the problem) and because your description is too vague to write an R code.

given a non-uniform sampling of the parameter space

You should provide a description of this non-uniform sampling on $p$ or $P$. From the available description, I can only guess you are using a certain proposal distribution, say $h(p)$, instead of the correct prior, say $g(p)$. In which case, each simulation should be [importance] weighted by $g(p)/h(p)$.

3 added 380 characters in body
source | link

Possible issues with the approach:

[Warning: the OP completely rewrote the question after I made those points, so most quotes relate to an earlier version!]

my model is stochastic. This means that a given set of parameters will result in different simulated summary statistics.

This is the standard setting for statistical inference, so I do not understand why this is stressed in the question. In an ABC algorithm, the simulated summary statistics are random variables and as such make the method valid (asymptotically).

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data. When you mention 30 times, is $n=30$?

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

the distance between the observed and simulated summary statistics could > be very similar between different parameters.

This is natural given the stochastic nature of the data. If one dataset could uniquely identify the parameter behind it, there would be no need for statistics.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

Possible issues with the approach:

my model is stochastic. This means that a given set of parameters will result in different simulated summary statistics.

This is the standard setting for statistical inference, so I do not understand why this is stressed in the question. In an ABC algorithm, the simulated summary statistics are random variables and as such make the method valid (asymptotically).

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data.

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

Possible issues with the approach:

[Warning: the OP completely rewrote the question after I made those points, so most quotes relate to an earlier version!]

my model is stochastic. This means that a given set of parameters will result in different simulated summary statistics.

This is the standard setting for statistical inference, so I do not understand why this is stressed in the question. In an ABC algorithm, the simulated summary statistics are random variables and as such make the method valid (asymptotically).

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data. When you mention 30 times, is $n=30$?

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

the distance between the observed and simulated summary statistics could > be very similar between different parameters.

This is natural given the stochastic nature of the data. If one dataset could uniquely identify the parameter behind it, there would be no need for statistics.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

2 added 380 characters in body
source | link

Possible issues with the approach:

my model is stochastic. This means that a given set of parameters will result in different simulated summary statistics.

This is the standard setting for statistical inference, so I do not understand why this is stressed in the question. In an ABC algorithm, the simulated summary statistics are random variables and as such make the method valid (asymptotically).

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data.

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

Possible issues with the approach:

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data.

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

Possible issues with the approach:

my model is stochastic. This means that a given set of parameters will result in different simulated summary statistics.

This is the standard setting for statistical inference, so I do not understand why this is stressed in the question. In an ABC algorithm, the simulated summary statistics are random variables and as such make the method valid (asymptotically).

My strategy was to simulate my model several times to each parameter set and use the average as the representation of the result for that parameter set.

This construction of a summary statistics must mimic the one for the original data, meaning you must have the same "several times" for the original data.

I didn't use a uniform prior to the parameters, but instead keep updating it to search is the areas with best fit.

Using an arbitrary distribution for the parameters is fine, provided one corrects for this change by an importance sampling weight. However, adapting the simulation of the parameters based on earlier ABC steps offers no convergence guarantee.

because I wanted to impose some constrains in the parameters I use transformed versions of then (like log or others).

At this level of description, there is nothing wrong with using another parameterisation, provided it is one-to-one and that the prior is transformed by the Jacobian formula.

This, plus something intrinsic to the model, lead to a correlation between some parameters (one could get similar results with different combinations of some parameters).

This is hard to understand. Correlation between the parameters is either a consequence of correlation in the prior or of dependences in the likelihood. Your parenthesised explanation is however off-key: finding two different values of the parameter with the same posterior value or with similar simulations has nothing to do with correlation.

Now, how do I compute the posterior distribution for my parameters?

ABC produces a sample from the ABC posterior. This posterior has a density that cannot be computed in closed form.

My simplistic strategy was the following (a) Accept the best 5% simulations (b) Fit a multivariate normal distribution to the parameters

The first bit is regular ABC, the second bit is a further approximation that does not seem necessary, as one could instead use a kernel approximation. The prior need be incorporated via further importance weighting.

As an option, I redid the previous step using as weighting the score

What is the meaning of score in this context? The distance? In which case the post-processing solution of Beaumont et al. (2003) should be considered.

1
source | link