How to use MPI with PyRosetta?

[Anonymous]

Hi,

I am trying to parallelize a protocol using MPI in PyRosetta. I see that PyRosetta does provide interfaces for MPI based job distributors. But, I also saw a documentation stating that PyRosetta doesn’t support MPI (https://www.rosettacommons.org/docs/latest/build_documentation/TACC) and recommends using MPI framework provided by Python using mpi4py library. 

Is MPI supported within PyRosetta? How does one typically parallelize PyRosetta protocols?

[Anonymous]

Most use of MPI in Rosetta is at a trivial level – that is, to parallelize out each output structure, each of which is generated completely independently of each other. (Generally speaking – there are some protocols which need more communication.) So for the most part you don’t really *need* MPI to run Rosetta protocols, so long as you can handle the output file parallelization yourself.

The tricky thing when doing MPI with PyRosetta is that the C++ Rosetta code needs to be specially compiled to support MPI, which when you’re also specially compiling it to interact with Python gets a little tricky.

So for most things it’s best just to use the Python-level MPI parallelization, and not worry about (re)compiling PyRosetta for MPI. Done correctly, this should work even for more complex protocol (not just the batching out each output structure), so long as the Python-level MPI appropriately handles the distribution/collection of work/results.

If all you’re concerned about is the output-structure level parallelism, recent versions of PyRosetta include an MPI job distributor (which uses mpi4py) with pyrosetta.mpi.MPIJobDistributor

[Author]

Posts