rosetta compile with mpi options

[Anonymous]

Hello

I am trying to compile rosetta with mpi options on our linux based university cluster but the compilation fails with the following error,

Before compiling, I copied the site.settings.tacc.lonestar.intel into site.settings in /tools/build/, since in one of the posts it was mentioned that this is suitable for compiling on intel based clusters.

$ ./scons.py bin mode=release extras=mpi
scons: Reading SConscript files …
Running versioning script … fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
Done. (0.0 seconds)
scons: done reading SConscript files.
scons: Building targets …
mpicxx -o build/src/release/linux/2.6/64/x86/gcc/4.4/mpi/apps/public/AbinitioRelax.o -c -std=c++98 -isystem external/boost_1_46_1/ -pipe -ffor-scope -Wall -Wextra -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -s -Wno-unused-variable -finline-limit=487 -DNDEBUG -DUSEMPI -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc/4.4 -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/opt/apps/intel/2011.0.013/mkl/include -I/opt/apps/intel/2011.0.013/include -Iexternal/boost_1_46_1 -Iexternal/dbio -I/usr/include -I/usr/local/include src/apps/public/AbinitioRelax.cc
mpicxx: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory
scons: *** [build/src/release/linux/2.6/64/x86/gcc/4.4/mpi/apps/public/AbinitioRelax.o] Error 127
scons: building terminated because of errors.

Here is the result of echo $PATH,
/opt/apps/openmpi/1.6.5-intel/bin:/opt/apps/intel/2011.0.013/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin:/opt/apps/moab/current/bin:/home/aa20/rosetta_2013wk36_bundle/main/source/bin:/home/aa20/bin

I have also tried to compile using only gcc and not intel by setting the relevant paths and copying the site.settings.topsail to site.setting. In this case the compilation proceeds without any overt errors but when I execute my script with mpiexe ./script, it throws up errors indicating problem with loading shared libraries. The IT people who maintain our cluster strongly recommend using intel compilers. What should I do?

I would greatly appreciate any help!

[Anonymous]

Interesting. There’s only two differences between the site.settings.tacc.lonestar.intel and the site.settings.topsail files. This first is that the “include_path” line is commented out for lonestar, and the second is that lonestar has a ` “cxx” : “mpicxx”, ` line in the overrides section. I don’t think the issue you’re running into is with include_path line (although it might be worth a try uncommenting it to see.) The overrides is what’s selecting the intel compiler versus the gcc compiler.

Best I can tell, you may not have your environment variables set up appropriately for intel mpi compiles. Have you been able to successfully compile other (non-Rosetta) programs with intel/mpi? For the clusters I’ve used mpi on, there’s been specific incantations that have to be in the shell startup files in order to get things to work. (Unfortunately, they’re specific to the cluster you’re using.) If you don’t have them, or if you didn’t do them properly, MPI builds will die miserably. You may need to talk to the cluster admins to see what these should be.

If that’s not an issue, it may be that your cluster is set up idiosyncratically. The $PATH variable is probably not an issue – that’s for executables, but not for finding libraries. What are your $LD_LIBRARY_PATH and $LIBRARY_PATH environment variables? (while we’re at it, what’s the $INCLUDE variable?) Basically, the libimf.so file should be in a directory listed in the $LD_LIBRARY_PATH variable for the Rosetta compile to work. You may want to talk with the cluster admins as to where that file is located, or you might be able to find it yourself by digging around the /opt/apps/intel/ directory (try the command: find /opt/apps/intel/ -name “libimf.so” ).

[Anonymous]

Hello, thank you for the prompt response. The libimf.so *is* in the library path.
$ echo $LD_LIBRARY_PATH
/opt/apps/openmpi/1.6.3-intel/lib:/opt/apps/intel/2011.0.013/mkl/lib/
intel64:/opt/apps/intel/2011.0.013/lib/intel64

$ ls -l /opt/apps/intel/2011.0.013/lib/intel64/libimf.so
-rwxr-xr-x 1 root root 3264667 Apr 28 2011 /opt/apps/intel/2011.0.013/
lib/intel64/libimf.so

echo $INCLUDE
/opt/apps/intel/2011.0.013/mkl/include:/opt/apps/intel/2011.0.013/include

echo $LIBRARY_PATH
/opt/apps/intel/2011.0.013/mkl/lib/intel64:/opt/apps/intel/2011.0.013/lib/intel64

It is also in the following locations:

find /opt/apps/intel/ -name libimf.so
/opt/apps/intel/2013.0.028/itac/8.1.0.024/intel64/bin/rtlib/libimf.so
/opt/apps/intel/2013.0.028/composer_xe_2013.1.117/compiler/lib/ia32/libimf.so
/opt/apps/intel/2013.0.028/composer_xe_2013.1.117/compiler/lib/intel64/libimf.so
/opt/apps/intel/2013.0.028/composer_xe_2013.1.117/compiler/lib/mic/libimf.so
/opt/apps/intel/2011.0.013/compiler/lib/ia32/libimf.so
/opt/apps/intel/2011.0.013/compiler/lib/intel64/libimf.so
/opt/apps/intel/2011.0.013/composerxe-2011.4.191/compiler/lib/intel64/libimf.so

The cluster admin suggested that “python build environment is messing up the
library path”. I am fairly new to scripts and command line and have asked for more details from him but
meanwhile, I thought I would post it here.

thanks

[Anonymous]

So, the cluster admin suggested i use openmpi/1.4.4-gcc to compile mpi.

It appears to be compiling but is also throwing up following errors

/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:644: warning: unused parameter ‘oldcomm’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:644: warning: unused parameter ‘comm_keyval’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:644: warning: unused parameter ‘extra_state’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:644: warning: unused parameter ‘attribute_val_in’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:644: warning: unused parameter ‘attribute_val_out’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:671: warning: unused parameter ‘comm’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:671: warning: unused parameter ‘comm_keyval’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:671: warning: unused parameter ‘attribute_val’
/opt/apps/openmpi/1.4.4-gcc/include/openmpi/ompi/mpi/cxx/comm_inln.h:671: warning: unused parameter ‘extra_state’

Not sure what these mean. Any ideas?

[Anonymous]

Thanks J. Sorry, I am very new to command line and scripting. Still learning the lingo.
Warning is what I meant

[Anonymous]

Big difference between warnings and errors, but you shouldn’t have to worry about these warnings. They say that some variable in that file was defined but not used.

-J

[Anonymous]

The issue you run into here is that compiler *errors* mean something went wrong with the compile, and needs to be fixed before you get a working program. Compiler *warnings* just mean that something is sub-optimal. Your compile still worked and still gave you a working program (unless you also get errors), it’s just that certain things happened that might indicate that developers need to look into things.

For you as an end user, “warnings” can be safely ignored (they’re there for developers) – it’s only “errors” that you need to be concerned with.

[Author]

Posts