1

I am using curve_fit to fit a curve to some set of data points (x,y) in the 2D space. curve_fit has this p0 parameter as we know.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

The second thing returned by curve_fit is pcov and when I take the diagonal of pcov and square root it, I get a vector v of values.

Then I sum all these values (from this vector v) and I get a number S: something which I interpret (correct or not, I am not sure?!) as an overall std.dev. (or say sum of std.deviations).

I am noticing that when I vary p0 I get different curves and they have different S values. But also, sometimes I think the curves look visually not much different but their S values differ a lot.

I don't fully understand this pcov matrix, hence my confusion. It is variance-covariance matrix of what?!

My question is this: this S value

1) is it a measure of how well my curve fits to the data?

or

2) is it more like a measure of how fast the optimization process (which happens inside curve_fit) converges (given the particular p0 value which I used)?

I hope it is 1) and thus I can use this number S as a quality measure for the curve fitting process.

Is that so or not?

Also, any explanations related to the above mentioned doubts would be much appreciated.

peter.petrov
  • 38,363
  • 16
  • 94
  • 159
  • Someone suggested this question maybe I should post to another `SE` site? To which one? Please let me know. – peter.petrov Jan 14 '21 at 13:23
  • 1
    see https://stackoverflow.com/questions/14854339/in-scipy-how-and-why-does-curve-fit-calculate-the-covariance-of-the-parameter-es – Stef Jan 14 '21 at 14:00
  • As an addition to my question see also https://stackoverflow.com/questions/25234996/getting-standard-error-associated-with-parameter-estimates-from-scipy-optimize-c – peter.petrov Jan 15 '21 at 01:17

2 Answers2

1

Actually neither 1) nor 2). The covariance matrix gives the errors and correlations of fit parameters. You can have large errors while the fit is "looking good". Large errors mean that the prediction of the model is not very good. When two parameters are basically doing the same thing correlation (off diagonal elements) can be very large, while the model is still good. The goodness of a fit is better checked, in a first approach, be a reduced chi-square. Most fitting is done by minimizing chi-square. The speed of convergence depends on many things including the complexity of the high dimensional ( dimensions as number of parameters) chi-square hyper-surface, while the error is only estimated from the curvature of an approximately parabolic (local) minimum.

When going into detail it becomes complicated very fast, but as a rough idea, that's it, somehow.

Adendum

The following question came up: Is it that a small S usually indicates a good fit, while a good looking fit not necessarily has a small S?

In the fitting process the best parameters are eventually a function of the input parameters. The errors of a parameter reflects how a parameter would change if the input data would change slightly. Say we have a + b x_i = y_i (this is linear but one can generalize it. we assume no error in x). Then we will have a = f( x, y ). The error s_a is related to d/d y_i f(x, y) Actually, it is error propagation -> how do errors in y_i affect a etc. So most of the time I'd say that a small error means a good fit, in a linear system for sure. In strange non-linear cases I am quite certain that one can construct a case (steep local minimum) where the variations of parameters is small with respect to variations in input values, while the fit itself is quite bad. In this case one would have small errors and a bad fit at the same time. One would probably see it in the chi-square value, though.

On the other hand, and as mentioned before, you could have a good looking fit with a very small chi-square but large errors. This is not a problem per se. But if you use the model with the fitted parameters to predict other values, you need to perform error propagation. As a consequence the prediction might be very good in reality, but math tells you to announce a low confidence.

Some math is given here

mikuszefski
  • 3,943
  • 1
  • 25
  • 38
  • Thanks. What are these fit parameters? Also, if the fit is visually looking good should I trust this, or should I trust the large error? I guess I should trust the visual look. – peter.petrov Jan 14 '21 at 23:01
  • Also... are you certain what the other answer suggests is not correct? From the hundreds of experiments which I did, it seems to me that when S is small the model also looks good. But if the model (the fitted curve) looks good, then S is not necessarily small (especially when I have just a few data points to fit to, but a complicated model function which I am trying to fit). Does that sound OK to you? – peter.petrov Jan 14 '21 at 23:04
  • https://stackoverflow.com/questions/25234996/getting-standard-error-associated-with-parameter-estimates-from-scipy-optimize-c – peter.petrov Jan 15 '21 at 01:17
  • @peter.petrov somewhat ok for me. coming originally from experimental physics I agree that experience is a good hint for what is going on. I'll make an edit. Please tell me if this then answers your question. – mikuszefski Jan 15 '21 at 06:33
  • Thank you. I will try to reread and try to understand what you're saying. – peter.petrov Jan 15 '21 at 10:41
0

In my understanding it is more like your 1st point.

From the docs:

The estimated covariance of popt. The diagonals provide the variance of the parameter estimate. To compute one standard deviation errors on the parameters use perr = np.sqrt(np.diag(pcov)).

So the smaller pcov the smaller the error on the parameter estimate.

-- edit ---

pcov indicate how certain the solver is that the provided parameters, popt, are optimum and unique. This doesn't necessarily means that they provide a good fit to the data.

Raf
  • 1,628
  • 3
  • 21
  • 40
  • Hm... OK, thanks, but you sound a bit uncertain :) I am uncertain too :) that's why I'm asking this question. I don't understand `errors`... Errors/variances compared to what? To the data points `(x,y)`, or to how much the parameters themselves varied until they converged to the "almost optimal" curve. – peter.petrov Jan 14 '21 at 13:16
  • If you really want to dig it and understand, I suggest to do some exercise. I'd get a set of points that perfectly fit a curve, and see what happens with pcov. Then start changing things one by one and see what happens again =) I've just done with the example provided in the docs, and remove the `y_noise`... fit is perfect, `pcov` is zero. That's a clue ;) – Raf Jan 14 '21 at 14:00
  • OK... so you mean such tests show it is indeed 1) and not 2)? The number S - it is indeed a measure of how close/well we are approximating the data? – peter.petrov Jan 14 '21 at 14:53
  • @peter.petrov changed my opinion after reading @mikuszefski answer and I played a bit with high order polynomials. Try to fit a set of well behaved points (say a straight line) into a high order polynomial function. The polynomial is stuck and cannot fit the data, although the optimum solution is quite straightforward - so you get horrible fit with a really small `pvoc` – Raf Jan 18 '21 at 08:22