unable to propagate location of libraries to scons for Rosetta 3.1 [FIXED…SORT OF…]

This topic has 11 replies, 3 voices, and was last updated 13 years, 11 months ago by Anonymous.

Viewing 7 reply threads

Author

Posts
- January 27, 2011 at 3:14 pm #751
  Anonymous
  I am unable to propagate our cluster environment to the Rosetta scons scripts (most likely the library environment)
  
  Here is my situation:
  
  On our cluster, compilers are loaded using “module” (e.g. “module load compilers/gcc/4.4.2” or “module load mpi/openmpi/1.4.3_gcc”).
  -> Consequently, compilers (gcc, g++, …) are not in “normal” PATH, and the same goes for some required libraries (and for INCLUDE paths also)
  -> Since Scons does not automatically propagate the external environment used to execute scons to the commands used to build target files, commands paths and library paths need to be defined in one of the Rosetta .settings file.
  
  For the compiler executables, I am able to prepend the necessary paths using either user.settings or site.settings
  I am also able to propagate INCLUDE paths to scons (I see -I/mypaths in the g++ command when I execute scons)
  
  For the libraries, I however have not succeeded. Despite adding the libraries to site.settings (see attached file), library paths do not seem to be propagated.
  
  Here is what I typically get :
  
  $ ./scons.py bin
  scons: Reading SConscript files …
  svn: ‘.’ is not a working copy
  scons: done reading SConscript files.
  scons: Building targets …
  g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
  /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
  scons: *** [build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o] Error 1
  scons: building terminated because of errors.
  
  $ ls -l /software/mpfr-2.4.1_gcc/lib
  -rw-r–r– 1 root clumeq 968598 Jan 18 2010 libmpfr.a
  -rwxr-xr-x 1 root clumeq 999 Jan 18 2010 libmpfr.la
  lrwxrwxrwx 1 root clumeq 16 Jan 18 2010 libmpfr.so -> libmpfr.so.1.2.0
  lrwxrwxrwx 1 root clumeq 16 Jan 18 2010 libmpfr.so.1 -> libmpfr.so.1.2.0
  -rwxr-xr-x 1 root clumeq 458672 Jan 18 2010 libmpfr.so.1.2.0
  
  Note that executing the above g++ command within my environment (outside of scons) works fine.
  
  Any help with propagation of library paths to the Rosetta scons scripts would be GREAT!
  
  [UPDATE]
  using a g++ wrapper-script, I got things to compile fine. Just to be on the safe side, I also used wrapper scripts for gcc, ar, as, ld
  $ cat ~/bin-wrapper/g++
  #!/bin/bash
  export PATH=/home/smg3d/bin-wrapper:/software/gcc-4.4.2/bin:/software/base-gcc/bin:/clumeq/bin:/usr/local/ge6.2u5/bin/lx24-amd64:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/clumeq/bin:/home/smg3d/bin
  export LD_LIBRARY_PATH=/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib:/software/mpfr-2.4.1_gcc/lib
  export C_INCLUDE_PATH=/software/base-gcc/include:/software/gmp-4.3.1_gcc//include:/software/mpfr-2.4.1_gcc/include
  export CXX_INCLUDE_PATH=/software/base-gcc/include
  export CPP_INCLUDE_PATH=/software/base-gcc/include
  export CPLUS_INCLUDE_PATH=/software/base-gcc/include
  /software/gcc-4.4.2/bin/g++ “$@”
- January 27, 2011 at 3:18 pm #4874
  Anonymous
  My supercomputer cluster also uses module. The changes we use for it are in site.settings.topsail (neighboring site.settings). Have you tried those changes?
  - January 27, 2011 at 3:42 pm #4876
    Anonymous
    Thanks a lot for the quick response.
    
    Yes, I had tried it on a fresh rosetta_source, but it gave the same result as above. I copied the site.settings.topsail to site.settings (also replaced “INCLUDE” by “C_INCLUDE_PATH”, since I do not have a INCLUDE environment variable, but my C_INCLUDE_PATH has all the INCLUDE paths defined by the modules). See the attached site.settings
    
    The result is the same :
    $ ./scons.py bin
    scons: Reading SConscript files …
    Copy(“user.options”, “user.options.template”)
    Copy(“user.settings”, “user.settings.template”)
    Copy(“/rap/eer-775-aa/rosetta-3.1/rosetta_source/src/pilot_apps.src.settings”, “/rap/eer-775-aa/rosetta-3.1/rosetta_source/src/pilot_apps.src.settings.template”)
    svn: ‘.’ is not a working copy
    scons: done reading SConscript files.
    scons: Building targets …
    g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
    /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
    scons: *** [build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o] Error 1
    scons: building terminated because of errors.
    
    As additional information regarding my cluster module settings:
    
    $ module list
    No Modulefiles Currently Loaded.
    $ module avail
    
    /clumeq/Modules/versions
    
    3.2.7
    
    /clumeq/Modules/modulefiles
    
    apps/abaqus-6.9-1 blas-libs/lapack/3.2.1_gcc misc-libs/libpng/1.2.42
    apps/ansys-12.1-cfx blas-libs/mkl/10.2.2.025 misc-libs/libpng/1.4.0
    apps/ansys-12.1-fluent compilers/gcc/4.4.2 misc-libs/matplotlib/1.0.0_intel
    apps/cp2k/2_1-branch compilers/gcc/4.4.2+g77 misc-libs/mpfr/4.3.1_gcc
    apps/espresso-4.1.3 compilers/intel/11.1.059 misc-libs/netcdf/3.6.3_intel
    apps/flex-2.5.35 compilers/sunstudio/12.1 misc-libs/netcdf/4.1.1_intel
    apps/geant-4.9.3 java/jdk1.6.0 misc-libs/netcdf-4.0.1
    apps/gpaw/0.7.2 lapack misc-libs/numpy/1.4.1_intel
    apps/grass-gis-6.4.0 misc-libs/ParMetis/3.1.1_gcc misc-libs/petsc/2.3.3-p15
    apps/gromacs/4.5.1_intel misc-libs/SuperLU/2.3.a_gcc+atlas misc-libs/petsc/3.1-p4
    apps/gromacs-4.0.7 misc-libs/SuperLU/4.0_gcc misc-libs/scipy/0.8.0_intel
    apps/lammps/3Dec10 misc-libs/SuperLU/DIST-2.3.a_gcc misc-libs/szip-2.1
    apps/lammps-30Mar10 misc-libs/ase/3.4.1 mono-2.6.1
    apps/namd/2.7b4 misc-libs/boost/1.44.0_intel mpi/openmpi/1.3.4_gcc
    apps/octave-3.2.4 misc-libs/charm/6.2.2_intel mpi/openmpi/1.3.4_intel
    apps/ox-console/5.10 misc-libs/clhep-2.0.4.3 mpi/openmpi/1.3.4_sunstudio
    apps/powerflow-4.1c misc-libs/clhep-2.0.4.5 mpi/openmpi/1.4.1_gcc
    apps/powerflow-4.2b misc-libs/fftw/2.1.5 mpi/openmpi/1.4.3_gcc
    apps/python-2.6.4 misc-libs/fftw/2.1.5_intel mpi/openmpi/1.4.3_intel
    apps/r-2.10.1 misc-libs/fftw/3.2.2 tools/gdb/7.1
    apps/structure-2.3.3 misc-libs/gmp/4.3.1_gcc tools/pdtoolkit/3.15_gcc
    apps/wine-1.2-rc2 misc-libs/gmp/4.3.1_intel tools/tau/2.19_gcc
    blas-libs/GotoBlas2 misc-libs/gsl/1.14_intel tools/tau/2.19_gcc-ompi
    blas-libs/atlas/3.9.20 misc-libs/hdf/hdf4.2r4
    $ module load compilers/gcc/4.4.2
    $ module list
    Currently Loaded Modulefiles:
    1) misc-libs/mpfr/4.3.1_gcc 2) misc-libs/gmp/4.3.1_gcc 3) compilers/gcc/4.4.2
    $ printenv PATH
    /software/gcc-4.4.2/bin:/software/base-gcc/bin:/usr/local/bin:/bin:/usr/bin
    $ printenv LD_LIBRARY_PATH
    /software/mpfr-2.4.1_gcc/lib:/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib
    $ printenv C_INCLUDE_PATH
    /software/base-gcc/include:/software/gmp-4.3.1_gcc//include:/software/mpfr-2.4.1_gcc/include
- January 27, 2011 at 4:28 pm #4877
  Anonymous
  I’ve asked someone else to take a look at this, but in the meantime – are you compiling on the cluster head node or the compute nodes? I’ve had issues with being unable to compile on compute nodes due to missing software (in my case, the ICC license was only installed on the head node, so you couldn’t compile it on the compute nodes).
  - January 27, 2011 at 4:47 pm #4879
    Anonymous
    I am compiling on the head node.
    
    As mentionned, when I type directly the g++ command that scons spits out (and fails on), it compiles fine (both executed in the same environment).
    
    Also note that I can reproduce the scons error within my user environment by unsetting LD_LIBRARY_PATH:
    
    $ ./scons.py bin
    …
    g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
    /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
    
    $ g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
    
    $ unset LD_LIBRARY_PATH
    
    $ g++ -o build/src/debug/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O0 -g -ggdb -ffloat-store -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
    /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
    
    Puzzling why paths and includes get propagated, but not the libraries…
- January 27, 2011 at 5:21 pm #4880
  Anonymous
  Hi smg3d,
  
  If I understand your problem correctly, your compilation environment requires g++ to be dynamically linked against libmpfr.so.1 which is located in /software/mpfr-2.4.1_gcc/lib. Since this linking error is a problem with how g++ is linking and not how Rosetta is linking, changing the library_path in scons won’t help because that will only be used when linking Rosetta!
  
  I suspect that somehow the scons script is running under a process that is not owned by you, where the environment is not set up correctly. To test this, please try comparing the environment you get when you execute the command from the command line with the one that scons is run under. Execute the following on the command line and report back what you see:
  
  export LD_LIBRARY_PATH_BACKUP=$LD_LIBRARY_PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:foobar whoami printenv
  
  Now in the file SConscript located in the base mini directory, locate the line
  
  for kind in build.options.cat:
  
  and following that line insert the following debug statements:
  
  print "### DEBUG ###" import os print "whoami: %s " % os.popen('whoami').read() print "os.eviron: %s " % os.environ print "### END DEBUG ###"
  
  now execute
  
  ./scons.py
  
  and it should print some debug information at the top. Please report report that as well.
  
  Don’t forget to reset the LD_LIBRARY_PATH.
  
  If it looks like that is the problem. There are a couple of things to try.
  
  One: Create a wrapper script for g++ that takes adds the correct paths to the LD_LIBRARY_PATH, echos a debug message and then calls g++. If you call this script “g++” and put it somewhere on the path before the standard g++, it will get called each time g++ is called. The debug message should help you verify that it is actually getting called.
  
  Two: I’ve attached a script called scons_to_makefile.pl.txt (change the extension and execution permissions) that scrapes the scons compilation commands and puts them in a makefile. You’ll want to run this on the output of a full rebuild of Rosetta on a system where it builds correctly. You can then try the running the make script and see if g++ is liking correctly. If that doesn’t immediately work, you might have luck just trying to execute all the statements somehow–ie pasting them into a terminal.
  
  Please write back if this doesn’t give you enough help to get it built!
  
  -matt
  - January 27, 2011 at 6:24 pm #4881
    Anonymous
    Hi matt. Thanks for your comments. The help provided by yourself and smlewis is greatly appreciated.
    
    I have tried your suggestion, but as you will see the scons script is run by me, and my environment is accessible within scons (at least at the stage where you suggest the insertion of environment check).
    
    I have put the report requested in an attachment.
    
    Regarding suggestion #2 above, I tried something like that a few days ago. I ran the output of “scons -n” (Don’t build; just print commands.). It went fine for a while, but than crashed… probably missing something done in the scons script that is not listed by “scons -n”. I did not explore this avenue further however…
    
    If I understand well, within scons, it is the LIBPATH that stores the library paths… correct? There is also LIBS… Would it be a good idea to check for LIBPATH / LIBS at various stage of the scons script execution? I tried something like that yesterday, but could not get the syntax right to print LIBPATH. In any case, it is weird that my paths and includes get propagated, but not my libraries.
    
    -stéphane
- January 27, 2011 at 10:15 pm #4885
  Anonymous
  Hi Stéphane,
  
  As for the LIBPATH/LIBS scons variables you can try printing them out with
  
  print env.Dictionary()
  
  where env is what scons thinks of as the “environment”, but I don’t think this is the issue–
  
  Ok here are more things to try. From your command line run,
  
  ldd `which g++`
  
  This will print out which libraries are dynamically linked into g++. Now make a wrapper script for g++ like I described in the last message (let me know if you can’t figure out how.) that also echo in the script ldd `which g++` as well as ldd `cc1plus` for good measure. This should verify the error message that you’re seeing–namely that the location of the library is not found.
  
  The next step is to reduce the test case and contact your system administrator to see if they know what’s going on with the module loading service. One way to try to reduce the test case is to make your own simpler scons build system for a small program and see if it is still failing. If you make it a really clean example, your sys admin will be much more likely to help you figure out what’s going on!
  
  Thanks for sticking with this.
  
  -matt
  - January 28, 2011 at 9:12 pm #4896
    Anonymous
    Dear Matt,
    
    I could not do the proposed test earlier since cluster has been down for maintenance since yesterday 17h00.
    
    So, first, here are the shared library dependencies for /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus :
    $ ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
    libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b90a1def000)
    libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b90a2055000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003b25e00000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003b25a00000)
    
    Here is my wrapper script for g++ :
    $ cat ~/bin-wrapper/g++
    #!/bin/bash
    ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
    /software/gcc-4.4.2/bin/g++ “$@”
    
    and finally, as expected, here is the result from a build attempt:
    
    ./scons.py bin mode=release
    scons: Reading SConscript files …
    cat: write error: Broken pipe
    svn: ‘.’ is not a working copy
    scons: done reading SConscript files.
    scons: Building targets …
    g++ -o build/src/release/linux/2.6/64/x86/gcc/apps/public/AbinitioRelax.o -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/apps/public/AbinitioRelax.cc
    libmpfr.so.1 => not found
    libgmp.so.3 => /usr/lib64/libgmp.so.3 (0x0000003e43000000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)
    /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus: error while loading shared libraries: libmpfr.so.1: cannot open shared object file: No such file or directory
    
    So, libmpfr.so.1 => not found is as expected from my initial problem. The other thing is that scons pulled libgmp.so.3 => /usr/lib64/libgmp.so.3 ( I think it should be libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 ).
    
    I will keep on digging…
- January 28, 2011 at 9:31 pm #4897
  Anonymous
  I just noticed the availability of the option log=environment from looking into tools/build/options.settings
  
  The output of ” ./scons.py log=environment ” is in attachment. It definitely shows that LIBPATH is fine within scons…
- January 28, 2011 at 10:34 pm #4899
  Anonymous
  Matt,
  
  I just tried something (and as I was doing it, I started thinking that maybe you already had proposed it…, which of course you did in message #6). I guess my brain went dead at the same time as my cluster went down…
  
  So this time I included a statement for LD_LIBRARY_PATH in my g++ wrapper-script.
  
  $ cat ~/bin-wrapper/g++
  #!/bin/bash
  export LD_LIBRARY_PATH=/software/gcc-4.4.2/lib64:/software/gcc-4.4.2/lib:/software/base-gcc/lib:/software/base-gcc/lib64:/software/gmp-4.3.1_gcc//lib:/software/mpfr-2.4.1_gcc/lib
  ldd /software/gcc-4.4.2/libexec/gcc/x86_64-unknown-linux-gnu/4.4.2/cc1plus
  /software/gcc-4.4.2/bin/g++ “$@”
  
  And Voilà! Finally compiling! I will later report if all is working fine with this somewhat ugly workaround.
  
  Once again, thanks to you and smlewis for all the help.
  
  Here is the output using the g++ wrapper script :
  
  $ ./scons.py -j8 bin mode=release
  scons: Reading SConscript files …
  svn: ‘.’ is not a working copy
  scons: done reading SConscript files.
  scons: Building targets …
  g++ -o build/src/release/linux/2.6/64/x86/gcc/protocols/loops/looprelax_protocols.os -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -fPIC -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/protocols/loops/looprelax_protocols.cc
  libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b2a1b5e9000)
  libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b2a1b84f000)
  libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
  /lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)
  g++ -o build/src/release/linux/2.6/64/x86/gcc/protocols/loops/LoopRelaxMover.os -c -std=c++98 -pipe -ffor-scope -W -Wall -pedantic -Wno-long-long -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -fPIC -DNDEBUG -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -I/software/base-gcc/include -I/software/gmp-4.3.1_gcc/include -I/software/mpfr-2.4.1_gcc/include -Iexternal/boost_1_38_0 -I/usr/local/include -I/usr/include src/protocols/loops/LoopRelaxMover.cc
  libmpfr.so.1 => /software/mpfr-2.4.1_gcc/lib/libmpfr.so.1 (0x00002b524d55a000)
  libgmp.so.3 => /software/gmp-4.3.1_gcc//lib/libgmp.so.3 (0x00002b524d7c0000)
  libc.so.6 => /lib64/libc.so.6 (0x0000003e41400000)
  /lib64/ld-linux-x86-64.so.2 (0x0000003e41000000)
- January 29, 2011 at 5:07 am #4900
  Anonymous
  It looks like you have at least identified the problem and hopefully it will work.
  
  The ./scons.py log=environment is new to me. I’ll have to remember that
  
  Best,
  Matt
Author

Posts

Viewing 7 reply threads

You must be logged in to reply to this topic.