memory issue rosetta3.4 scripts

[Anonymous]

Hi,
I’am using a simple rosetta scripts design where I want to do three subsequent steps of mover and relax (see script at the end).
When I am using a .res file (232aa) with only NATAA it works fine, but when I am using a .res file with some positions ALLAA he throws an errorcode (6) and breaks with “bad alloc” after the first cycle mover+relax. Accordingly, if I only use one mover and one designs step he is happy. I only get him to work with all 3 design+relax steps and some ALLAA if I allocate 32G of memory instead of my normal 16G for the job. Why is that? Are the mover steps in PROTOCOLS section not independent of each other? Is there a more memory saving way to approach this?

Thanks a lot!

Here is the script:
{ROSETTASCRIPTS>
{SCOREFXNS>
{scorefcn weights=”score12″/>
{/SCOREFXNS>
{TASKOPERATIONS>
{InitializeFromCommandline name=init/>
{ReadResfile name=”resfile” filename=”bbarrel.res”/>
{/TASKOPERATIONS>
{FILTERS>
{/FILTERS>
{MOVERS>
{PackRotamersMover name=”MovPack” scorefxn=”scorefcn” task_operations=resfile,init/>
{FastRelax name=”FaRelax” scorefxn=”scorefcn” repeats=5 task_operations=init/>
{/MOVERS>
{APPLY_TO_POSE>
{FavorNativeResidue bonus=-1.25/>
{/APPLY_TO_POSE>
{PROTOCOLS>
{Add mover_name=”MovPack”/>
{Add mover_name=”FaRelax”/>
{Add mover_name=”MovPack”/>
{Add mover_name=”FaRelax”/>
{Add mover_name=”MovPack”/>
{Add mover_name=”FaRelax”/>
{/PROTOCOLS>
{/ROSETTASCRIPTS>

[Anonymous]

A) Scripts work better if you attach them – otherwise the forum software eats the brackets and we can’t see it. (I can see it using administrative mode, but folks in general can’t. I replaced < with { for your post, but that still ruins tabs. Wow, 16/32 GB is a TON of memory for this. Is that 32 GB per process, or 32 GB shared over some large number of processors? You should be allowing at least 1GB/processor for Rosetta, and more for very large design jobs.

C) How many positions get ALLAA? What ex flags are you using? Also useful, from the logfile, would be the statement of how many rotamers were used and the size of the interaction graph, lines like:
core.pack.interaction_graph.interaction_graph_factory: Instantiating PDInteractionGraph
core.pack.pack_rotamers: built 2808 rotamers at 20 positions.
core.pack.pack_rotamers: IG: 8537520 bytes

D) There are two offhand tricks to reduce memory use. The big one is -linmem_ig 20. linmem_ig hugely reduces memory use in packing; for some applications this comes at a time cost, for some it’s basically free, and sometimes it speeds up design. Here’s a citation:

Bioinformatics Research and Applications
Lecture Notes in Computer Science Volume 4983, 2008, pp 343-354
On-the-Fly Rotamer Pair Energy Evaluation in Protein Design
Andrew Leaver-Fay,
Jack Snoeyink,
Brian Kuhlman

D.2) Your other option is to reduce static memory use by getting rid of unnecessary ResidueTypes. Try:
-chemical:exclude_patches LowerDNA UpperDNA Cterm_amidation SpecialRotamer protein_cutpoint_upper protein_cutpoint_lower VirtualBB ShoveBB VirtualDNAPhosphate VirtualNTerm CTermConnect sc_orbitals pro_hydroxylated_case1 pro_hydroxylated_case2 ser_phosphorylated thr_phosphorylated tyr_phosphorylated tyr_sulfated lys_dimethylated lys_monomethylated lys_trimethylated lys_acetylated glu_carboxylated cys_acetylated tyr_diiodinated N_acetylated C_methylamidated MethylatedProteinCterm

This prevents creation of a lot of special-case residue types, and reduces memory waste in highly parallel environments. (This might not work for 3.4, or your particular case, but we can iron out the details as needed.)

[Anonymous]

thanks for the help.
Let me give you some more details on the questions you guys asked:
1. The designs is on a 232 AA protein: 184 NATAA, 43 ALLAA, 5 PIKAA
2. Ex level: 3
3. the 16G or for the working solution 32G was for 1 process that is one core
4. rotamers used, as for 2.: 103107 rotamers at 232 positions
5. the full error line is:
terminate called after throwing an instance of ‘std::bad_alloc’
what(): std::bad_alloc

I attached the script as well as the log output

Another question.
Is this way of just calling the Mover and Fast Relax several times in the PROTOCOL section the proper way? Is there a way with a loop I could do it where he would also purge data from the previous steps. And why does he save data anyway… I mean the independent movers work fine, only the series makes a problem.
Thanks!!

[Anonymous]

Great, thanks a lot!
Here some final updates:
-linmem_ig with 16G works with ex 3
-exclude patch doesnt help. Still crashes
-ex 2 also works with 16G and even with 8
… and I will do both I think. ex 2 and linmem_ig 20.
Thanks again!

[Anonymous]

Whenever I’ve had too large of memory consumption, the reason has usually been that I’ve messed up my settings and accidentally did design when I wasn’t intending to. The info for all rotamers for all 20 aa really eats up memory, especially for big proteins.

I’d first recommend checking your resfile, and making sure it isn’t designing residues you don’t want it to design.

I’d also second the recommendation for the -linmem_ig flag, and not being too aggressive with your command line flags. -ex1 and -ex2 are usually sufficient for the extra rotamer flags. And they’re not even needed for certain movers, like your FastRelax mover (basically anything that incorporates minimization). Though it’s probably the PackRotamers mover which is giving you memory issues.

[Anonymous]

You should definitely try -linmem_ig…your interaction graph is 4 GB here, and your rotamer pair energies (RPE) will easily be a lot more than that. I concur with Rocco that you should drop to -ex1 -ex2 and not use -ex3 for now. It is possible to use -ex3 only on certain residues you care the most about via the resfile.

I’m a little vague on if that’s the “right” amount of memory use for your settings – to some degree it depends on the physical arrangement of the design positions (design positions close together will combinatorially generate a lot more RPEs) – but it seems reasonable given your settings.

I don’t think you have a memory accumulation/leak problem. You’ll notice your two packing iterations:

core.pack.pack_rotamers: built 100032 rotamers at 232 positions.
core.pack.pack_rotamers: IG: 4072260304 bytes

core.pack.pack_rotamers: built 103107 rotamers at 232 positions.
Got some signal… It is:6

You are building more rotamers in the second case, which means more RPEs as well; this explains why you run out of memory in the second case. (I think RPEs are not accounted for in the IG size, and Andrew isn’t around to ask).

The reason you are building more in the later case is that you’ve relaxed to a better conformation (after your first relax run), and fewer rotamers are getting thrown out by bump_check, or some parts of the protein are closer and the rotamers interact, or something similar.

I don’t think there’s anything wrong with your pseudoloop and I don’t think you have a memory leak.

[Anonymous]

First off, I think you might have the standard confusion as to how -ex flags work. The numbers after the ex don’t refer to the *level* of the extra rotamers, it refers to the *chi* on which it operates. -ex1 means to do “standard”-level (+/- 1 standard deviation) extra sampling on the first chi rotamer. -ex2 means do standard-level on the second chi. -ex3 means standard extra sampling on the third chi angle, for those residues which have it. If you had some bizarre set of circumstances, you could specify -ex3 without -ex1 or -ex2, and you’d do standard extra sampling on the third chi, but no extra sampling on chi 1 or chi 2. (If you want to affect the level of sampling, you need to use flags like “-ex1:level”, and keep in mind the designations aren’t strictly bigger=more.)

—

From your interaction graph, it looks like you’re running ~440 rotamers per position. This seems really high for mostly repacking. On a test protein, I ran 330 rotamers/position for full redesign (-ex1 & -ex2), and 20 rotamers/position for repack only.

My guess is that you’re still accidentally designing residues you don’t want to be redesigning. What does your resfile look like? Particularly check that you have the default section (the part prior to “start”) set appropriately – keep in mind that the default is to do full redesign.

[Anonymous]

As for the resfile. I explicitly set every position. So to my understanding the default part is not necessary?! With the ex level, I set: “-ex1 -ex2 -ex3” … to my understanding the number of ex levels, by addressing more chi angles, changes from coarse (only movements at the “stem”) to more fine (even the “branches” move). So with “-ex1 -ex2 -ex3 -ex4” I should get the most fine sampled rotamer set, no?! Sorry, if me stating only “ex3” was misunderstood, I meant “ex1 to ex3” with that. But maybe you could explain me another aspect. With the first mover RPEs are precalculated as to the level they “fit” with the memory. Are they purged in between movers, or are they passed on with the pose to the next mover? And why doesn’t he max out the memory in the second mover step. I mean, that he died before the second mover with the additional rotamers is because he wanted to add the new RPEs to the pose and it does not fit? Is there a step to explicitly limit the memory of stored RPEs… than he shouldn’t have a problem, even if he has to calculate some anew, no? Sorry for asking such basic questions, I’m not familiar with the architecture of these things….
Thanks

[Anonymous]

“I explicitly set every position. So to my understanding the default part is not necessary?!”

The default is unused if you specify something for each position.

EX flags:
The flags / resfile commands EX 1 or -ex1, EX 2 or ex2, etc, turn on “extra sampling” for the 1st, 2nd, 3rd, and 4th chi angles. All rotamers always sample all chi angles for all residue types; this turns on “extra” sampling, so that more rotamers are made available, by adding +- 1 standard deviation about the existing rotamer wells. This is done on a per-chi basis. The “levels” of extra chi sampling do things like adding +- 0.5 SD instead of +- 1 SD, or +- (0.5, 1, 1.5) instead of just 1. For 99% of purposes, -ex1 -ex2 is sufficient. (I might have the default wrong; it might be 0.5 instead of 1; we can look the detail up if it matters).

For RPEs:
A) I was wrong, the RPEs are included in that 4 GB number that the interaction graph reports. This makes it strange that you are running out of memory when your IG size is not likely far greater than 4 GB. I think it fails in the later step simply because the earlier steps have moved you to a conformation that takes more memory to compute, for whatever reason (more interacting rotamers).

None of the RPE data should be shared between calls to PackRotamersMover, so you should not have a memory accumulation. The data should be purged.

-linmem_ig IS the flag for capping the amount of memory used, as you request – that’s why it works here. linmem means linear memory – the amount of memory grows at most linearly (and slowly) with task size, rather than combinatorially.

[Author]

Posts