How exactly did you get the rmsds in the two cases, and how far off are they?
If I read things correctly, the rmsdCA that’s being put out by FlexPepDock is the rmsd for the Calpha atoms of the peptide only, without any sort of superposition. If you calculate the rmsd of the whole complex, or try to do a superposition of the peptide (or the whole complex) first, you’ll likely not get equivalent values.