Symmetry assumes perfect structural/chemical identity between all subunits. So for this to work the individual repeats must be assumed to be identical. They must also be repeated in a perfectly symmetrical manner. Given these approximations the repeat system could potentially be modeled as a helix. Because this is a repeat protein there is supposed to be a covalent link between the repeat units. This cannot be modeled symmetrically as of now. But as you mention this can be avoided by trimming of residues at the end. Or adding a hard constraint between the covalently attached termini in repeats.
The symmetry definition file can be generated from a preexisting helical structure using the make_symmdef_file.pl with the -m HELIX flag. If you want to predict a denovo structure then you have to make sure to edit the symmetry definition file to make sure to completely randomize the starting position of the subunits as to remove the memory of the preexisting helix. The alternative is to make a symmetry definition file denovo. The syntax for this is found at http://www.rosettacommons.org/manuals/archive/rosetta3.2_user_guide/app_symmetry.html. You would need two types of statements: rot Rz_angle VALUE and trans 0,0,VALUE. I haven’t done any denovo stuff with helix symmetry in Rosetta 3 but I did get that to work in Rosetta 2. So unfortunately I don’t have a helical symmetry denovo symmetry definition file lying around. It should not be that difficult to make if you get the hang of it.
As far as protocol goes, for abinitio use the fold-and-dock app. For docking the SymDock app. In both cases a symmetry definition file needs to be generated.
If you have problems with generating the symmetry definition file then post it here and I can give you feedback.