Symmetry definition for coiled-coil dimer and trimer

For two anti-parallel helices, the symmetry is also C2, just like the two parallel helices case. The only difference is where the axis of symmetry is pointing. For the parallel case, the axis of symmetry is aligned along the helices. For the anti-parallel case, the axis is perpendicular to the helices.

I’m not sure if there’s a good way in the symmetry code itself to enforce the parallel versus anti-parallel arrangement. My guess is that what you’d get is dependent on the exact protocol you’re using and what the energy function thinks of the two cases. You may be able to apply constraints (e.g. distance constraints between the ends of the two helices) to favor one over the other.

Regarding the trimer, if you have a three helix bundle with two parallel and one anti-parallel, then that’s not really “symmetric”. There’s no line you can draw through that conformation which gives a rotation less than 360 degrees resulting in an identical conformation – it’s just not geometrically feasible. That said, I think the Rosetta symmetry mechanism may be flexible enough to allow it. None of the automated tools are likely to generate such a symmetry definition file for you, but you may be able to hand edit one. Generate a number of different symmetry files of differing complexities and try to get an understanding of what each of the parts mean. You may be able to adjust one of the parameters for the third member of the trimer such that it flips it end over end.

Worst case scenario, you have to model it without symmetry. That may actually be the better course of action, anyway. Using the symmetry machinery implies things about sidechain positions and backbone conformations which might not actually be the case for this trimer.