trRosetta protocol Segmentation Fault

[Anonymous]

Hello,

So I’ve been trying to implement the new trRosetta protocol and I keep getting a segmentation fault error. Is there any idea what I might be doing incorrectly? The Rosetta Crash Log is below and is followed by the output logs (both raw text files are also included as attachments).

My guess is it has something to do with the: “File does not exist: $Rosetta/main/database/protocol_data/tensorflow_graphs/tensorflow_graph_repo_submodule/trRosetta/model_v1/variables/variables.index”. When I checked in the $Rosetta/main/database/protocol_data/tensorflow_graphs/tensorflow_graph_repo_submodule/trRosetta there is just model_v1/saved_model.pb, which I think is the trained tensorflow model? But it doesn’t seem to get expanded/unpacked (is that even a thing for tensorflow models? Way out of my depth here) so Rosetta fails to find the variables and crashes. But that is entirely a guess.

Thanks in advance for any help/assistance! 

##############################################################################################################

#

# Rosetta crash log. Please submit the contents of this file to http://crash.rosettacommons.org/

# or use the script Rosetta/main/source/scripts/python/public/report_crashes.py

#

[START_CRASH_REPORT]

[ROSETTA_VERSION]: 2021.16+release.8ee4f02ac57

[COMMIT_DATE]: 2021-04-20T11:45:16.822269

[APPLICATION]: /Users/ncohen/Desktop/rosetta/main/source/bin//trRosetta.tensorflow.macosclangrelease

[MODE]: Release

[EXTRAS]: tensorflow 

[OS]: Apple Mac OS X

[COMPILER]: Clang version “12.0.0 (clang-1200.0.32.29)”

[STDLIB]: libc++ version 10000

[START_OPTIONS]

 -in:file:fasta=r10_1.fasta -out:mute=all -out:unmute=apps basic.citation_manager protocols.relax protocols.trRosetta protocols.trRosetta_protocols basic.tensorflow_manager core.simple_metrics -trRosetta:msa_file=r10.a3m -trRosetta:backbone_randomization_mode=ramachandran -trRosetta:mutate_gly_to_ala=false

[END_OPTIONS]

[START_BACKTRACE]: RAW_LIBC

1   libutility.dylib                    0x0000000115b5017e utility::signal_handler(int) + 318

2   libsystem_platform.dylib            0x00007fff7037d5fd _sigtramp + 29

3   ???                                 0x0000000000000000 0x0 + 0

4   libprotocols_a.2.dylib              0x000000010f9f4464 protocols::trRosetta::trRosettaMultipleSequenceAlignment::trRosettaMultipleSequenceAlignment(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 68

5   libprotocols_a.2.dylib              0x000000010f9f6de2 protocols::trRosetta::trRosettaProtocolBase::set_input_msa_file(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 242

6   libprotocols_b.6.dylib              0x000000010db3d776 protocols::trRosetta_protocols::constraint_generators::trRosettaConstraintGenerator::load_msa_and_run_neural_network() const + 406

7   libprotocols_b.6.dylib              0x000000010db3d5c2 protocols::trRosetta_protocols::constraint_generators::trRosettaConstraintGenerator::apply(core::pose::Pose const&) const + 34

8   libprotocols_b.6.dylib              0x000000010de59ad9 protocols::trRosetta_protocols::movers::trRosettaProtocolMover::generate_trRosetta_constraints(core::pose::Pose const&, double) const + 1353

9   libprotocols_b.6.dylib              0x000000010de56d30 protocols::trRosetta_protocols::movers::trRosettaProtocolMover::apply(core::pose::Pose&) + 2448

10  libprotocols.1.dylib                0x0000000111ef2fc0 protocols::jd2::JobDistributor::run_one_job(std::__1::shared_ptr<protocols::moves::Mover>&, long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, unsigned long&, unsigned long&, bool) + 7648

11  libprotocols.1.dylib                0x0000000111ef05c2 protocols::jd2::JobDistributor::go_main(std::__1::shared_ptr<protocols::moves::Mover>) + 194

12  libprotocols.1.dylib                0x0000000111eeb772 protocols::jd2::FileSystemJobDistributor::go(std::__1::shared_ptr<protocols::moves::Mover>) + 66

13  trRosetta.tensorflow.macosclangrele 0x000000010c13b59b main + 635

14  libdyld.dylib                       0x00007fff70184cc9 start + 1

15  ???                                 0x0000000000000002 0x0 + 2

[END_BACKTRACE]

[FILE]: SIGSEGV

[LINE]: 11

[START_MESSAGE]

Segmentation Fault

[END_MESSAGE]

[END_CRASH_REPORT]

#################################################################################################################

apps.public.trRosetta.trRosetta: Starting trRosetta application.

apps.public.trRosetta.trRosetta: Application created 21 January 2021 by Vikram K. Mulligan, Flatiron Institute (vmulligan@flatironinstitute.org).

apps.public.trRosetta.trRosetta: Original trRosetta neural network developed by Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. (https://doi.org/10.1073/pnas.1914677117).

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: No native structure specified.  Not setting native structure in trRosettaProtocol mover.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Set scorefunction for centroid stage 0 to trRosetta_cen0.wts.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Set scorefunction for centroid stage 1 to trRosetta_cen1.wts.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Set scorefunction for centroid stage 2 to trRosetta_cen2.wts.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Set scorefunction for centroid stage 3 to trRosetta_cart.wts.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Set scorefunction for full-atom refinement to ref2015.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Setting full-atom refinement scorefunction atom_pair_constraint weight to 5.0 (was 0.0).

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Setting full-atom refinement scorefunction dihedral_constraint weight to 1.0 (was 0.0).

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Setting full-atom refinement scorefunction angle_constraint weight to 1.0 (was 0.0).

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Starting trRosetta structure prediction protocol.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Building pose from sequence.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Randomizing backbone dihedrals using protocol “ramachandran”.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Removing clashes.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Initializing constraint generator from multiple sequence alignment file “r10.a3m”.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  distance constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  omega dihedral constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  theta dihedral constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  phi angle constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  distance constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  omega dihedral constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  theta dihedral constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Configuring trRosettaConstraintGenerator  to set  phi angle constraints.

protocols.trRosetta_protocols.movers.trRosettaProtocolMover: Generating trRosetta constraints.

protocols.trRosetta_protocols.constraint_generators.trRosettaConstraintGenerator: Loading trRosetta model version 1.

2021-04-27 11:06:08.916050: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /Users/ncohen/Desktop/rosetta/main/database/protocol_data/tensorflow_graphs/tensorflow_graph_repo_submodule/trRosetta/model_v1/

2021-04-27 11:06:09.037685: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }

2021-04-27 11:06:09.209187: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

2021-04-27 11:06:09.224621: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9466c06c90 initialized for platform Host (this does not guarantee that XLA will be used). Devices:

2021-04-27 11:06:09.224667: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

2021-04-27 11:06:09.760118: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.

2021-04-27 11:06:09.760227: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /Users/ncohen/Desktop/rosetta/main/database/protocol_data/tensorflow_graphs/tensorflow_graph_repo_submodule/trRosetta/model_v1/variables/variables.index

2021-04-27 11:06:09.760252: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 844210 microseconds.

AN INTERNAL ERROR HAS OCCURED. PLEASE SEE THE CONTENTS OF ROSETTA_CRASH.log FOR DETAILS.

 

 

[Anonymous]

The warning about the lack of variables.index can be ignored; that’s not the issue here.  It looks like it’s crashing when trying to load your multiple sequence alignment file.  Is the file in A3M format?  Can you share the MSA?

[Anonymous]

Hi vmulligan, 

Thanks for the help! Sure thing, I’ve attached it here (had to put it in .txt format to upload). It was generated with the hhblits server so I assumed it was good-to-go but looking at it, it starts with a line

#A3M#

That I don’t recognize. Would removing that work/help?

[Anonymous]

The #A3M# line is always added by the HHBLITS server.  Rosetta should be ignoring it, but you can try removing it.

One other thing to try: could you compile with mode=debug (instead of mode=release) and post the output from the crash log?  It will likely be more informative.

[Anonymous]

 So removal of the #A3M# file allowed the app to run and I got an output structure. 

Sorry haven’t compiled in debug mode yet, will try to do tonight/this weekend and update with the crash log results. 

[Anonymous]

Sorry for the delay, here is the result of the run in debug mode. Both log file and crash log attached. 

[Author]

Posts