As noted, the weak point of Gauss-Newton minimization comes from correlation of parameters. There is another type of minimization, called steepest descent, where we use the only the gradient of the model function to determine the \(\delta _k\) shifts to apply to the \(p_k\) values. This does not require matrix inversion, so high correlation does not degrade the accuracy. However, since interactions between parameters are not taken into account, steepest descent tends to converge more slowly and there are some edge cases where the algorithm completely fails to find a minimum.
Steepest descent minimization can be implemented using the same math as used for Gauss-Newton, by modifying the Hessian to remove the cross-derivatives. In other words, where \(H_{kk} = \sum _j w_j [\frac {\partial M(\textbf {p},x_j)}{\partial p_k}]^2\) and \(H_{jk} = 0 \) for \(j\neq k\). Thus, the diagonal Hessian terms remain unchanged, but the off-diagonal terms are all set to zero. A diagonal matrix will never be singular provided that all diagonal terms are non-zero.