Not familiar enough with Calibur to make a direct comparison, but in Rosetta clustering, the superposition and rmsd is indeed computed pairwise (although there are optimizations such that the full matrix isn’t calculated for large numbers of structures – the first 400 or so are taken a representative of the whole set and used to form seeding clusters, and then afterward only enough rmsd calculations as needed to assign into existing clusters is used.) I’m not familiar enough with the concept of “threshold” in Calibur to make a comparison with Rosetta’s cluster radius.
Rosetta uses all Calpha atoms by default, so it would include both the peptide and the protein when doing the alignment and rmsd calculation. (Though there are flags which allow a limited amount of control on this.)