The GDTMM is simply a different metric for structure-structure difference. It’s a variant of the Global Distance Test metric that is used in evaluating CASP entries. The reason for using it instead of rmsd is that rmsd has very poor behavior with respect to small outliers. For example, if you get the majority of the protein very nearly correct, but completely mess up a single loop, you could have a much worse rmsd than if you’re off for the entire protein. GDTMM attempts to correct those sorts of issues.
The -cluster:sort_groups_by_energy sorts the clusters based on the energy of the lowest energy member – the cluster with the lowest energy for the lowest energy member becomes cluster zero, etc. (Okay, that’s not *quite* true. It looks like the sorting metric is actually a weighted combination of energy and cluster size. You can control how much the cluster size contributes with the -cluster:population_weight flag. It defaults to 0.09, which means the ranking is based 91% on score, and 9% on cluster size.)