Make sure there’s no name collision between the three-letter code that you’ve assigned for your noncanonical and any of the existing noncanonicals in the database. If you named your noncanonical XYZ, try:
grep “NAME XYZ” Rosetta/main/database/chemical/residue_type_sets/fa_standard/residue_types/* -r
grep “IO_STRING XYZ” Rosetta/main/database/chemical/residue_type_sets/fa_standard/residue_types/* -r
To mutate, use the MutateResidueMover, which takes a full residue type name. Make sure that your noncanonical has the properties PROTEIN ALPHA_AA, as well as one of L_AA or ACHIRAL_BACKBONE. (Note that Rosetta automatically generates D-amino acid equivalents of all L-amino acids, so if you want the D-amino acid, prepend “D” to the name when using the MutateResidueMover. So “DXYZ” instead of “XYZ”.)