Member Site Forums PyRosetta PyRosetta – General Parallel processing? Reply To: Parallel processing?


    I was one of the authors on the underlying C++ job distributors, but I’ll freely admit I know nothing about the python layer. I can tell you this:

    A) The underlying C++ needs to be recompiled with extras=mpi to get MPI style parallel processing. I’m sure you don’t have this if you got binaries, and I’m not sure how to generate it for the python bindings.

    B) The job distributor you are likely to be using supports an option called -run::multiple_processes_writing_to_one_directory. In this mode, non-communicating Rosetta processes will use the filesystem to signal by making temporary files myjob_0001.in_progress, etc. If the number of processors is small and the jobs are long, then this method allows use of multiple processors on one big job in one directory with minimal overwriting. This method has NO GUARANTEES against overwriting, duplication of effort, or screwing up the scorefile with simultaneous writes. It should be sufficient for your use.

    C) You can always run 8 jobs in 8 different directories. Just make sure you start them with independent random number seeds, and you’ll get trajectories as effectively as one job eight times as long. This is functionally equivalent to (and the same speed as) MPI parallelization.

    If you have more questions about the underlying c++, you may want to repost on the rosetta3.0 board, I don’t check this one often.