Usually when you get the “NAN occurred in H-bonding calculations!” it’s because you have a structure that has a major deficiency in it. Most commonly you have overlapping atoms. How are you preping you strucutures before you do your docking run? Do you have a number of residues which all have, say, zero coordinates?