The rms calculation for small molecule docking needs a reference structure with the same ligand (atom-for-atom) that it’s being compared to. If you’re using an apo structure as the native, or are using a reference structure which contains a ligand different from your experimental ligand structure, then any rms value you get (assuming the program doesn’t exit with an error) is nonsense.
The rms values are really only useful for benchmarking purposes, anyway. If you’re doing a docking run with a new ligand with an unknown binding pose, you can just ignore the value (or omit the -in:file:native, which should turn off that calculation.)
.
