unable to propagate location of libraries to scons for Rosetta 3.1 [FIXED…SORT OF…]

[Anonymous]

I am unable to propagate our cluster environment to the Rosetta scons scripts (most likely the library environment)

Here is my situation:

On our cluster, compilers are loaded using “module” (e.g. “module load compilers/gcc/4.4.2” or “module load mpi/openmpi/1.4.3_gcc”).
-> Consequently, compilers (gcc, g++, …) are not in “normal” PATH, and the same goes for some required libraries (and for INCLUDE paths also)
-> Since Scons does not automatically propagate the external environment used to execute scons to the commands used to build target files, commands paths and library paths need to be defined in one of the Rosetta .settings file.

For the compiler executables, I am able to prepend the necessary paths using either user.settings or site.settings
I am also able to propagate INCLUDE paths to scons (I see -I/mypaths in the g++ command when I execute scons)

For the libraries, I however have not succeeded. Despite adding the libraries to site.settings (see attached file), library paths do not seem to be propagated.

Here is what I typically get :

$ ./scons.py bin
scons: Reading SConscript files …
svn: ‘.’ is not a working copy
scons: done reading SConscript files.
scons: Building targets …
g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
/software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
scons: *** [build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o] Error 1
scons: building terminated because of errors.

$ ls -l /software/mpfr-2.4.1_gcc/lib
-rw-r–r– 1 root clumeq 968598 Jan 18 2010 libmpfr.a
-rwxr-xr-x 1 root clumeq 999 Jan 18 2010 libmpfr.la
lrwxrwxrwx 1 root clumeq 16 Jan 18 2010 libmpfr.so -> libmpfr.so.1.2.0
lrwxrwxrwx 1 root clumeq 16 Jan 18 2010 libmpfr.so.1 -> libmpfr.so.1.2.0
-rwxr-xr-x 1 root clumeq 458672 Jan 18 2010 libmpfr.so.1.2.0

Note that executing the above g++ command within my environment (outside of scons) works fine.

Any help with propagation of library paths to the Rosetta scons scripts would be GREAT!

[UPDATE]
using a g++ wrapper-script, I got things to compile fine. Just to be on the safe side, I also used wrapper scripts for gcc, ar, as, ld
$ cat ~/bin-wrapper/g++
#!/bin/bash
export PATH=/home/smg3d/bin-wrapper:/software/gcc-4.4.2/bin:/software/base-gcc/bin:/clumeq/bin:/usr/local/ge6.2u5/bin/lx24-amd64:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/clumeq/bin:/home/smg3d/bin
export LD_LIBRARY_PATH=/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib:/software/mpfr-2.4.1_gcc/lib
export C_INCLUDE_PATH=/software/base-gcc/include:/software/gmp-4.3.1_gcc//include:/software/mpfr-2.4.1_gcc/include
export CXX_INCLUDE_PATH=/software/base-gcc/include
export CPP_INCLUDE_PATH=/software/base-gcc/include
export CPLUS_INCLUDE_PATH=/software/base-gcc/include
/software/gcc-4.4.2/bin/g++ “$@”

[Anonymous]

My supercomputer cluster also uses module. The changes we use for it are in site.settings.topsail (neighboring site.settings). Have you tried those changes?

[Anonymous]

Thanks a lot for the quick response.

Yes, I had tried it on a fresh rosetta_source, but it gave the same result as above. I copied the site.settings.topsail to site.settings (also replaced “INCLUDE” by “C_INCLUDE_PATH”, since I do not have a INCLUDE environment variable, but my C_INCLUDE_PATH has all the INCLUDE paths defined by the modules). See the attached site.settings

The result is the same :
$ ./scons.py bin
scons: Reading SConscript files …
Copy(“user.options”, “user.options.template”)
Copy(“user.settings”, “user.settings.template”)
Copy(“/rap/eer-775-aa/rosetta-3.1/rosetta_source/src/pilot_apps.src.settings”, “/rap/eer-775-aa/rosetta-3.1/rosetta_source/src/pilot_apps.src.settings.template”)
svn: ‘.’ is not a working copy
scons: done reading SConscript files.
scons: Building targets …
g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
/software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
scons: *** [build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o] Error 1
scons: building terminated because of errors.

As additional information regarding my cluster module settings:

$ module list
No Modulefiles Currently Loaded.
$ module avail

---

/clumeq/Modules/versions

---

3.2.7

---

/clumeq/Modules/modulefiles

---

apps/abaqus-6.9-1 blas-libs/lapack/3.2.1_gcc misc-libs/libpng/1.2.42
apps/ansys-12.1-cfx blas-libs/mkl/10.2.2.025 misc-libs/libpng/1.4.0
apps/ansys-12.1-fluent compilers/gcc/4.4.2 misc-libs/matplotlib/1.0.0_intel
apps/cp2k/2_1-branch compilers/gcc/4.4.2+g77 misc-libs/mpfr/4.3.1_gcc
apps/espresso-4.1.3 compilers/intel/11.1.059 misc-libs/netcdf/3.6.3_intel
apps/flex-2.5.35 compilers/sunstudio/12.1 misc-libs/netcdf/4.1.1_intel
apps/geant-4.9.3 java/jdk1.6.0 misc-libs/netcdf-4.0.1
apps/gpaw/0.7.2 lapack misc-libs/numpy/1.4.1_intel
apps/grass-gis-6.4.0 misc-libs/ParMetis/3.1.1_gcc misc-libs/petsc/2.3.3-p15
apps/gromacs/4.5.1_intel misc-libs/SuperLU/2.3.a_gcc+atlas misc-libs/petsc/3.1-p4
apps/gromacs-4.0.7 misc-libs/SuperLU/4.0_gcc misc-libs/scipy/0.8.0_intel
apps/lammps/3Dec10 misc-libs/SuperLU/DIST-2.3.a_gcc misc-libs/szip-2.1
apps/lammps-30Mar10 misc-libs/ase/3.4.1 mono-2.6.1
apps/namd/2.7b4 misc-libs/boost/1.44.0_intel mpi/openmpi/1.3.4_gcc
apps/octave-3.2.4 misc-libs/charm/6.2.2_intel mpi/openmpi/1.3.4_intel
apps/ox-console/5.10 misc-libs/clhep-2.0.4.3 mpi/openmpi/1.3.4_sunstudio
apps/powerflow-4.1c misc-libs/clhep-2.0.4.5 mpi/openmpi/1.4.1_gcc
apps/powerflow-4.2b misc-libs/fftw/2.1.5 mpi/openmpi/1.4.3_gcc
apps/python-2.6.4 misc-libs/fftw/2.1.5_intel mpi/openmpi/1.4.3_intel
apps/r-2.10.1 misc-libs/fftw/3.2.2 tools/gdb/7.1
apps/structure-2.3.3 misc-libs/gmp/4.3.1_gcc tools/pdtoolkit/3.15_gcc
apps/wine-1.2-rc2 misc-libs/gmp/4.3.1_intel tools/tau/2.19_gcc
blas-libs/GotoBlas2 misc-libs/gsl/1.14_intel tools/tau/2.19_gcc-ompi
blas-libs/atlas/3.9.20 misc-libs/hdf/hdf4.2r4
$ module load compilers/gcc/4.4.2
$ module list
Currently Loaded Modulefiles:
1) misc-libs/mpfr/4.3.1_gcc 2) misc-libs/gmp/4.3.1_gcc 3) compilers/gcc/4.4.2
$ printenv PATH
/software/gcc-4.4.2/bin:/software/base-gcc/bin:/usr/local/bin:/bin:/usr/bin
$ printenv LD_LIBRARY_PATH
/software/mpfr-2.4.1_gcc/lib:/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib
$ printenv C_INCLUDE_PATH
/software/base-gcc/include:/software/gmp-4.3.1_gcc//include:/software/mpfr-2.4.1_gcc/include

[Anonymous]

I’ve asked someone else to take a look at this, but in the meantime – are you compiling on the cluster head node or the compute nodes? I’ve had issues with being unable to compile on compute nodes due to missing software (in my case, the ICC license was only installed on the head node, so you couldn’t compile it on the compute nodes).

[Anonymous]

I am compiling on the head node.

As mentionned, when I type directly the g++ command that scons spits out (and fails on), it compiles fine (both executed in the same environment).

Also note that I can reproduce the scons error within my user environment by unsetting LD_LIBRARY_PATH:

$ ./scons.py bin
…
g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
/software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory

$ g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc

$ unset LD_LIBRARY_PATH

$ g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
/software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory

Puzzling why paths and includes get propagated, but not the libraries…

[Anonymous]

Hi smg3d,

If I understand your problem correctly, your compilation environment requires g++ to be dynamically linked against libmpfr.so.1 which is located in /software/mpfr-2.4.1_gcc/lib. Since this linking error is a problem with how g++ is linking and not how Rosetta is linking, changing the library_path in scons won’t help because that will only be used when linking Rosetta!

I suspect that somehow the scons script is running under a process that is not owned by you, where the environment is not set up correctly. To test this, please try comparing the environment you get when you execute the command from the command line with the one that scons is run under. Execute the following on the command line and report back what you see:


export LD_LIBRARY_PATH_BACKUP=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:foobar
whoami
printenv


Now in the file SConscript located in the base mini directory, locate the line


for kind in build.options.cat:


and following that line insert the following debug statements:


print "### DEBUG ###"
import os
print "whoami: %s " % os.popen('whoami').read()
print "os.eviron: %s " % os.environ
print "### END DEBUG ###"


now execute


./scons.py


and it should print some debug information at the top. Please report report that as well.

Don’t forget to reset the LD_LIBRARY_PATH.

If it looks like that is the problem. There are a couple of things to try.

One: Create a wrapper script for g++ that takes adds the correct paths to the LD_LIBRARY_PATH, echos a debug message and then calls g++. If you call this script “g++” and put it somewhere on the path before the standard g++, it will get called each time g++ is called. The debug message should help you verify that it is actually getting called.

Two: I’ve attached a script called scons_to_makefile.pl.txt (change the extension and execution permissions) that scrapes the scons compilation commands and puts them in a makefile. You’ll want to run this on the output of a full rebuild of Rosetta on a system where it builds correctly. You can then try the running the make script and see if g++ is liking correctly. If that doesn’t immediately work, you might have luck just trying to execute all the statements somehow–ie pasting them into a terminal.

Please write back if this doesn’t give you enough help to get it built!

-matt

[Anonymous]

Hi matt. Thanks for your comments. The help provided by yourself and smlewis is greatly appreciated.

I have tried your suggestion, but as you will see the scons script is run by me, and my environment is accessible within scons (at least at the stage where you suggest the insertion of environment check).

I have put the report requested in an attachment.

Regarding suggestion #2 above, I tried something like that a few days ago. I ran the output of “scons -n” (Don’t build; just print commands.). It went fine for a while, but than crashed… probably missing something done in the scons script that is not listed by “scons -n”. I did not explore this avenue further however…

If I understand well, within scons, it is the LIBPATH that stores the library paths… correct? There is also LIBS… Would it be a good idea to check for LIBPATH / LIBS at various stage of the scons script execution? I tried something like that yesterday, but could not get the syntax right to print LIBPATH. In any case, it is weird that my paths and includes get propagated, but not my libraries.

-stéphane

[Anonymous]

Hi Stéphane,

As for the LIBPATH/LIBS scons variables you can try printing them out with


print env.Dictionary()


where env is what scons thinks of as the “environment”, but I don’t think this is the issue–

Ok here are more things to try. From your command line run,


ldd `which g++`


This will print out which libraries are dynamically linked into g++. Now make a wrapper script for g++ like I described in the last message (let me know if you can’t figure out how.) that also echo in the script ldd `which g++` as well as ldd `cc1plus` for good measure. This should verify the error message that you’re seeing–namely that the location of the library is not found.

The next step is to reduce the test case and contact your system administrator to see if they know what’s going on with the module loading service. One way to try to reduce the test case is to make your own simpler scons build system for a small program and see if it is still failing. If you make it a really clean example, your sys admin will be much more likely to help you figure out what’s going on!

Thanks for sticking with this.

-matt

[Anonymous]

Dear Matt,

I could not do the proposed test earlier since cluster has been down for maintenance since yesterday 17h00.

So, first, here are the shared library dependencies for /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus :
$ ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b90a1def000)
libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b90a2055000)
libc.so.6 => /lib64/libc.so.6 (0x0000003b25e00000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b25a00000)

Here is my wrapper script for g++ :
$ cat ~/bin-wrapper/g++
#!/bin/bash
ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
/software/gcc-4.4.2/bin/g++ “$@”

and finally, as expected, here is the result from a build attempt:

./scons.py bin mode=release
scons: Reading SConscript files …
cat: write error: Broken pipe
svn: ‘.’ is not a working copy
scons: done reading SConscript files.
scons: Building targets …
g++ -o build/src/release/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
libmpfr.so.1 => not found
libgmp.so.3 => /usr/lib64/libgmp.so.3 (0x0000003e43000000)
libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)
/software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory

So, libmpfr.so.1 => not found is as expected from my initial problem. The other thing is that scons pulled libgmp.so.3 => /usr/lib64/libgmp.so.3 ( I think it should be libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 ).

I will keep on digging…

[Anonymous]

I just noticed the availability of the option log=environment from looking into tools/build/options.settings

The output of ” ./scons.py log=environment ” is in attachment. It definitely shows that LIBPATH is fine within scons…

[Anonymous]

Matt,

I just tried something (and as I was doing it, I started thinking that maybe you already had proposed it…, which of course you did in message #6). I guess my brain went dead at the same time as my cluster went down…

So this time I included a statement for LD_LIBRARY_PATH in my g++ wrapper-script.

$ cat ~/bin-wrapper/g++
#!/bin/bash
export LD_LIBRARY_PATH=/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib:/software/mpfr-2.4.1_gcc/lib
ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
/software/gcc-4.4.2/bin/g++ “$@”

And Voilà! Finally compiling! I will later report if all is working fine with this somewhat ugly workaround.

Once again, thanks to you and smlewis for all the help.

Here is the output using the g++ wrapper script :

$ ./scons.py -j8 bin mode=release
scons: Reading SConscript files …
svn: ‘.’ is not a working copy
scons: done reading SConscript files.
scons: Building targets …
g++ -o build/src/release/linux/2.6/64/x86/gcc/protocols/loops/looprelax_protocols.os -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -fPIC -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/protocols/loops/looprelax_protocols.cc
libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b2a1b5e9000)
libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b2a1b84f000)
libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)
g++ -o build/src/release/linux/2.6/64/x86/gcc/protocols/loops/LoopRelaxMover.os -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -fPIC -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/protocols/loops/LoopRelaxMover.cc
libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b524d55a000)
libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b524d7c0000)
libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)

[Anonymous]

It looks like you have at least identified the problem and hopefully it will work.

The ./scons.py log=environment is new to me. I’ll have to remember that

Best,
Matt

[Author]

Posts