Rosetta won’t rotate methyl hydrogen the way real physics does. Once they’re placed they’re static. It is certainly a reasonable idea to rely on the nearby heavyatom instead, as you suggest. (The only hydrogens I ever use in constraints are backbone hydrogens, which of course aren’t degenerate.)
Ambiguous constraints are used when you want the model to satisfy any of a group of possibilities. For example, you have five positive charges on one side of an interface, and 5 negative charges on the other side, and you want at least one of the many possible pairs to form.
Here, ambiguousness to account for degeneracy would be used like so:
AtomPair HG11 3 H 11 BOUNDED 1.5 5 3
AtomPair HG12 3 H 11 BOUNDED 1.5 5 3
AtomPair HG13 3 H 11 BOUNDED 1.5 5 3
This says, “whichever of HG11/2/3 has the best distance, score only that one”. (Since 1.5 to 5 is a large range, probably at least two would return 0 score at any one time anyway).
Now that I understand better what you’re doing, I may be able to point you to something inside AmbiguousNMRDistanceConstraint. If you load up the file rosetta_source/src/core/scoring/constraints/AmbiguousNMRDistanceConstraint.cc and go to line 95 or so, you will see a huge block of code translating atom names. Translated into English, it appears this code is doing things like “if the name is HG, then convert it to mean 1HG2, 2HG2, 3HG2, 1HG1, 2HG1”. So maybe the AmbigousNMRDistanceConstraint is internally handling the degeneracy. Unfortunately there doesn’t appear to be a well documented key to what it’s doing – you’ll want to look at that code directly.
w/r/t BOUNDED versus HARMONIC functional forms: I suggest you fire up a plotting program and input the functional forms for those to see what they look like at different distances (or do it by hand, whatever). The constraint and the functional form are not interdependent, you can mix and match. I use BOUNDED for just about everything because I can have a wide zero-score basin. I also like to combine all my constraints (via ambiguous) with CONSTANT constraints set at a score of a few units; this prevents highly-unsatisfied constraints from blowing up the protein during minimization (if atoms are 100 angstroms apart, I want the score to stay equal if it moves to 99 angstroms apart, and I want the absolute value to be small).