After asking around, there doesn’t seem to be a way to tell the part of Rosetta doing design to limit itself to a certain number of mutations from the wild-type sequence. There’s a facility that could possibly be made to work, and there sounds like interest (independent of your question) in adding that functionality in, but nothing that exists currently.
What you could do now, though, is to use the FavorNativeResidue mover to add an adjustable bias toward the native residues. By dialing that value up and down you can cause Rosetta to change the likelihood of putting in mutations. At the right level, you should be able to get around a certain number of mutations.
That’s not going to be exact, though. There’s always a possibility of gettting more or fewer mutations than you’ve attempted to “dial in”. If you really do have a hard limit at 5 mutations, you can use the MutationsFilter to throw out those structures with too many mutations.
.
