LAUTOSCALE=.TRUE. problem in LBFGS?

[ehermes]

Hello,

I've had an issue with the LBFGS optimizer for a while now, and I may have even mentioned it in a previous thread, but now I've found a good test case that shows that the problem is related to the inverse curvature calculation. Basically, during a geometry minimization the energy will occasionally spike due to atoms moving far too close to one another. I noticed that these spikes were preceded by the printed value of invcurv in the OUTCAR file becoming large (over 0.1, occasionally reaching 0.5). I found that by setting LAUTOSCALE=.FALSE. in the INCAR, the geometry relaxation becomes monotonic decreasing after an initial spike, and converges with 30% fewer ionic steps. I imagine that lowering the value of MAXMOVE may also solve this issue, but I have not tested that yet.

My question is, does this look like a bug? Or perhaps is it simply a poor default setting for MAXMOVE? Or are my systems just pathological? In any case, I'd be interested in hearing your opinions on these examples.

(Note: If you run my calculation as they are presented here you will not converge to the same value, because I am using a custom-patched version of VASP that implements the three-body contributions to Grimme's D3 dispersion correction).

[chill]

Our L-BFGS implementation will never guarantee a monotonic decrease in energy. It does not use a "proper" line search. This is so that the same code can be easily used with NEB optimization, which does not have an energy function to optimize (instead the force on the images is optimized). The LAUTOSCALE=.TRUE. parameter attempts to set an optimal INVCURV based upon the curvature along the previous optimization step (except in the first step, when it uses forward finite difference to estimate INVCURV). This scheme will not always work. If the previous step was along a stiff direction and the current direction is soft, then INVCURV will be too large. This happens mostly in heterogeneous systems (e.g. molecule on a surface). In more homogeneous systems such as relaxation of a bulk crystal the LAUTOSCALE settings should be much more efficient.

Interestingly, even with the "spikes" in energy LAUTOSCALE will typically converge. This is because the error that is made during the bad step is only along one degree of freedom, which gets corrected in the next step since the LBFGS algorithm now knows the curvature in that direction. But, as you saw, explicitly setting INVCURV may be faster in some systems.

Ideally, for minimizations, we should use a back tracking line search that could reject moves that caused the energy to increase. Then it would always be safe to use LAUTOSCALE, which should generally lead to a faster convergence. The issue with having to explicitly set INVCURV is that there is no ideal value. It basically needs to be smaller than the stiffest mode in the system to guarantee convergence. LAUTOSCALE is often an improvement because it determines a good value for INVCURV with no additional DFT calculations (except for the first step as explained above).

LAUTOSCALE attempts to strike a balance between having to do a fancy line search at each step (something that would involve extra DFT calculations to determine the step length) and having to explicitly set INVCURV ahead of time.