512 MB might be pushing it, but there are certainly ways of cutting down on Rosetta memory usage.
One of the biggest users of memory in Rosetta is the ResidueType set – if you restrict the number of residues which Rosetta knows about, you can greatly reduce memory usage. In Rosetta/database/chemical/residue_type_sets/fa_standard/ there are files “residue_types.txt.slim” and “patches.txt.slim” if you copy them to the non-slim versions of the files (make a backup of the originals first), this should reduce memory usage. That will only allow you to model simple proteins, though. Depending on your input system or modeling requirements, you may need to add residues or patches from the full set to your slimmed down set.
Another memory saving option is the “-delete_old_poses” option – this can reduce memory usage in some cases, but causing Rosetta to aggressively get rid of old structural information it (probably) doesn’t need anymore.