‘MPI’ has not been declared

[Anonymous]

Hello,

I have built the MPI binaries many times without issue… until now.  I have two machines with the same Ubuntu build, openmpi, and site.settings file on them.  The build goes fine on one, but fails on the other (I did the openmpi installation tests and they succeed).

In the build output I see: “‘MPI’ has not been declared”.   Below is more of the output… thank you very much in advance for your time!

src/protocols/mpi_refinement/MPI_Refine_Master.cc: In member function ‘virtual void protocols::mpi_refinement::MPI_Refine_Master::init()’:

src/protocols/mpi_refinement/MPI_Refine_Master.cc:204:15: error: ‘MPI’ has not been declared

  int ncores = MPI::COMM_WORLD.Get_size();

               ^

…..

 

scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os] Error 1

scons: building terminated because of errors.”

 

 

[Anonymous]

What does “which mpicc” return?  What do the scons lines look like (the ones that probably start with mpicc)?  The individual scons lines should show paths to libraries – are the mpi libraries showing up like they should?  (You can determine “should” by comparison to the machine that works).

I assume the problem is that whatever the system is finding as MPI libraries aren’t right, if they don’t define MPI.  (I assume if it wasn’t finding the libraries at all, it would complain that the library files weren’t found).

It compiles fine without MPI, right?

[Anonymous]

Update, fixed:

I started looking through threads and found the following are needed… the machine I had the error on was a clean install of the OS so they were missing:

sudo apt-get install libopenmpi-dev

I also needed:

sudo apt-get install zlib1g-dev

Are these things always necessary?

[Anonymous]

Great!  Thanks for the info and the update to the docs.

I actually thought I was in good shape because I saw no errors, but I also don’t  see any .mpi executables… : (  I’ll have to look into that tonight and tomorrow.

 

[Anonymous]

Actually the standard binaries built (not sure what I messed up)… the MPI versions did not.  The error is the same “MPI has not been declared”.

I thought maybe something went wrong with the libdev install so I ran it again but it appears to be fine unless I’m misinterpreting the following… so where else am I missing something?

“sudo apt-get install libopenmpi-dev

Reading package lists… Done

Building dependency tree       

Reading state information… Done

libopenmpi-dev is already the newest version (1.10.2-8ubuntu1).

The following packages were automatically installed and are no longer required:

  libpango1.0-0 libpangox-1.0-0 linux-headers-4.4.0-21 linux-headers-4.4.0-21-generic linux-headers-4.4.0-24 linux-headers-4.4.0-24-generic

  linux-image-4.4.0-21-generic linux-image-4.4.0-24-generic linux-image-extra-4.4.0-21-generic linux-image-extra-4.4.0-24-generic

Use ‘sudo apt autoremove’ to remove them.

0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded.”

Thanks!

 

[Anonymous]

On my VM here on my laptop, it builds fine… I noticed this difference between the build statements, but I don’t know why the path is missing from the build statement on my workstation… ($LD_LIBRARY_PATH and $PATH are set the same way on both machines)

 

Laptop:

 

“mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/AnchoredDesign.mpi.linuxgccrelease -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=$ORIGIN -Wl,-rpath=$ORIGIN/../lib build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/interface_design/anchored_design/AnchoredDesign.o -Lexternal/lib -Lbuild/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Lsrc -Lbuild/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Lexternal -L/home/starone/openmpi/lib -L/home/starone/Public/PyRosetta.Namespace -L/home/starone/Public/PyRosetta.Namespace/rosetta -L/usr/lib -L/usr/local/lib -ldevel -lprotocols.7 -lprotocols.6 -lprotocols_e.5 -lprotocols_d.5 -lprotocols_c.5 -lprotocols_b.5 -lprotocols_a.5 -lprotocols_h.4 -lprotocols_g.4 -lprotocols_f.4 -lprotocols_e.4 -lprotocols_d.4 -lprotocols_c.4 -lprotocols_b.4 -lprotocols_a.4 -lprotocols.3 -lprotocols_b.2 -lprotocols_a.2 -lprotocols.1 -lcore.5 -lcore.4 -lcore.3 -lcore.2 -lcore.1 -lbasic -lnumeric -lutility -lObjexxFCL -lz -lcppdb -lsqlite3 -lcifparse….”

 

Workstation:  (-L/home/starone/openmpi/lib is missing)

 

“mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os -c -std=c++98 -ffor-scope -isystem external/boost_1_55_0/ -isystem external/ -isystem external/include/ -isystem external/dbio/ -pipe -Wall -Wextra -pedantic -Werror -Wno-long-long -Wno-strict-aliasing -march=core2 -mtune=generic -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -Wno-unused-parameter -fPIC -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DPTR_BOOST -DNDEBUG -DUSEMPI -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc/5.4 -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -Iexternal/boost_1_55_0 -Iexternal/libxml2/include -Iexternal -Iexternal/dbio -I/usr/include -I/usr/local/include src/protocols/mpi_refinement/MPI_Refine_Master.cc”

 

 

[Anonymous]

The first thing to try when there are apparent pathing issues is to `ln -s site.settings.killdevil site.settings`.  This adds some of your enviroment variables into SCons in a different way.  (Those files are in source/tools/build).  

[Anonymous]

I tried the step you suggested (thank you!), but nothing changed.  One thing that is striking is the difference in the parameters… on my laptop I see “-L” parameters indicated which include all the elements of the LD LIBRARY PATH…  they aren’t present on the workstation complie option list.

Why would this be?

[Anonymous]

I’m not a Linux person so I just accept whatever is installed when I installed the OS on the machines, the only changes I made were in the .bashrc files regarding the paths.  They are both Intel machines.

This is what I added to that file:  (sorry for the strange characters… hopefully you can figure it out.  I must be inadvertantly hitting on some escape characters or something)

 

source /home/starone/Public/PyRosetta.Namespace/SetPyRosettaEnvironment.sh

export PATH=/opt/ncbi-blast-2.3.0+/bin:$PATH

export BLASTDB=/opt/ncbi-blast-2.3.0+/db

export PATH=/opt/blast-2.2.26/bin:$PATH

export PATH=$HOME/openmpi/bin:$PATH

export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH

 

These are the contents of the site.settings file I have been using on both machines:

 

import os

settings = {

“site” : {

“prepends” : {

“program_path” : os.environ[“PATH”].split(“:”),

“library_path” : os.environ[“LD_LIBRARY_PATH”].split(“:”),

},

“appends” : {

},

“overrides” : {

},

“removes” : {

},

},

}

 

[Anonymous]

Ok… I think I see what you are getting at.  I will try it in the morning.

(sorry if there is and I just don’t know about it):  I haven’t been an engineer for many years, but when I have to build something in Linux I always wonder why there isn’t something slick like MS’s Visual Studio.  I did Microsoft development for a long time before moving into management and the ability to visualize things from within a development studio is so much more helpful to me than using command line interactions… feel like I’m back in the MS-DOS era : )

Thanks again for your help!

[Anonymous]

Progress!  Well, the compilation looks like its succeeding now, but the linking is failing… pretty much as expected I suppose.  I think you’re right that I used different methods for installing openmpi (I installed on the laptop a while ago), but it wasn’t by choice.  There is a good deal of conflicting info on the web on this stuff as I’m sure you know.  On the workstation I downloaded the package from the OpenMPI website and followed the steps in the YouTube video linked there. 

I noticed that there are about 80 items in openmpi/lib directory on my laptop, but only about 20 on the workstation.

I documented the steps I followed… appearently they are incorrect, can you tell me the proper method.

 

1. Go to: https://www.open-mpi.org/software/ompi/v2.0/ 

2. Download the latest stable build tar.gz 

3. Make directory Public/openmpi 

4. Copy the compressed file to the directory 

5. Open a terminal window 

6. Change to the openmpi directory 

7. Execute: ./configure –prefix=$HOME/openmpi 

8. Execute: make all 

9. Execute: make install 

10. Change to your home directory 

11. Execute: sudo gedit .bashrc 

12. Add the following lines to the end of the file: 
    
    export PATH=$HOME/openmpi/bin:$PATH 
    
    export LD_LIBRARY_PATH=$HOME/openmpi/lib:$LD_LIBRARY_PATH 

13. Save and close the editor 

14. Execute: mpirun -np n hostname   (where ‘n‘ is the number of processors in your machine) 

 

These are the errors I’m getting now:

mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/per_residue_energies.mpi.linuxgccrelease -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=/home/starone/Public/Rosetta/main/source/build/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Wl,-rpath=$ORIGIN -Wl,-rpath=$ORIGIN/../lib build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/analysis/per_residue_energies.o -Lexternal/lib -Lbuild/src/release/linux/4.4/64/x86/gcc/5.4/mpi -Lsrc -Lbuild/external/release/linux/4.4/64/x86/gcc/5.4/mpi -Lexternal -L/home/starone/Public/PyRosetta.Namespace -L/home/starone/Public/PyRosetta.Namespace/rosetta -L/home/starone/openmpi/lib -L/usr/lib -L/usr/local/lib -ldevel -lprotocols.7 -lprotocols.6 -lprotocols_e.5 -lprotocols_d.5 -lprotocols_c.5 -lprotocols_b.5 -lprotocols_a.5 -lprotocols_h.4 -lprotocols_g.4 -lprotocols_f.4 -lprotocols_e.4 -lprotocols_d.4 -lprotocols_c.4 -lprotocols_b.4 -lprotocols_a.4 -lprotocols.3 -lprotocols_b.2 -lprotocols_a.2 -lprotocols.1 -lcore.5 -lcore.4 -lcore.3 -lcore.2 -lcore.1 -lbasic -lnumeric -lutility -lObjexxFCL -lz -lcppdb -lsqlite3 -lcifparse -lxml2

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to `ompi_mpi_cxx_op_intercept’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to `MPI::Win::Free()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to `MPI::Datatype::Free()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.6.so: undefined reference to `MPI::COMM_WORLD’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.3.so: undefined reference to `MPI::Comm::Comm()’

collect2: error: ld returned 1 exit status

scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/helix_from_sequence.mpi.linuxgccrelease] Error 1

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)’:

pmut_scan_parallel.cc:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x17): undefined reference to `ompi_mpi_cxx_op_intercept’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function `MPI::Intracomm::Clone() const’:

pmut_scan_parallel.cc:(.text._ZNK3MPI9Intracomm5CloneEv[_ZNK3MPI9Intracomm5CloneEv]+0x3a): undefined reference to `MPI::Comm::Comm()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function `MPI::Graphcomm::Clone() const’:

pmut_scan_parallel.cc:(.text._ZNK3MPI9Graphcomm5CloneEv[_ZNK3MPI9Graphcomm5CloneEv]+0x35): undefined reference to `MPI::Comm::Comm()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/apps/public/design/pmut_scan_parallel.o: In function `MPI::Cartcomm::Sub(bool const*) const’:

pmut_scan_parallel.cc:(.text._ZNK3MPI8Cartcomm3SubEPKb[_ZNK3MPI8Cartcomm3SubEPKb]+0x194): undefined reference to `MPI::Comm::Comm()’

 

[Anonymous]

Wait, why are we talking about manually installed openmpi?  I thought you installed it via apt?  “sudo apt-get install openmpi openmpi-dev” or something similar?

[Anonymous]

Thanks!  I’ll give it a try

[Anonymous]

I tried using sudo apt-get… it just reports that the packages are already installed. 

These errors are different than I was getting before… now I’m seeing alot of this:

scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/beta_peptide_modeling.mpi.linuxgccrelease] Error 1

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to `ompi_mpi_cxx_op_intercept’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to `MPI::Datatype::Free()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to `MPI::Comm::Comm()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libutility.so: undefined reference to `MPI::Win::Free()’

collect2: error: ld returned 1 exit status

scons: *** [build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/antibody.mpi.linuxgccrelease] Error 1

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to `ompi_mpi_cxx_op_intercept’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to `MPI::Win::Free()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to `MPI::Datatype::Free()’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.6.so: undefined reference to `MPI::COMM_WORLD’

build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/libprotocols.1.so: undefined reference to `MPI::Comm::Comm()’

collect2: error: ld returned 1 exit status

 

[Anonymous]

On the workstation (problem machine) I first did a manual install using the steps listed in #16 above that I found in a video linked from the openmpi website.  The generated fewer files.  On my laptop I did the apt-get method there are more files and everything works fine.

Sorry I didn’t get to try your suggestion as I woke up this morning feeling horrible and went back to bed after the kids were off to school.  Hopefully I’ll be feeling better tomorrow so I can have at it.

[Anonymous]

Thanks, but no changes…

Ok, as I see it I have one machine where the build process can find, for example, “MPI::COMM_WORLD” and another on which it can’t.  Both machines have the same OpenMPI package installed and both have the same library paths defined. 

So, what I’d like to do is find out from which library file, on the machine where the build works, exports “MPI::COMM_WORLD”.  This way I can see if it exists on the machine where it wont build.

 

[Anonymous]

Maybe this explains something? 

Both machines are running Ubuntu 16.04, but my laptop (where everthing builds fine) was an upgrade, while I did a clean install on my workstation (where the build fails).  Everything else between the machines in terms of software, shells, settings, etc, are the same.

[Anonymous]

Does anyone know about installing OpenMPI?  I ran ‘ompi_info’ on both machines and the installation options and version info are different.  I attached both files for review.

CatalayseU:  Laptop, build successful

SynthaseU: Workstation, build fails

Thank you!!

[Anonymous]

Could this “C++ bindings: no” be the problem?

If it is, does anyone know how to change it and rebuild?

Thanks

[Anonymous]

Ok, looks like this is what happened…  The C++ bindings are off by default when building OpenMPI (though I can’t imagine why!).  So I redid the build with this added to the ./configure command:  “–enable-mpi-cxx”.

Then I redid the remaining steps as I outlined above.  I then did the MPI build of Rosetta and it was successful.

How do I mark this as answered?

Thanks for your help guys.

[Anonymous]

Thank you!

“which mpicc” returns: “/home/starone/openmpi/bin/mpicc”

Here is an individual line:

“mpiCC -o build/src/release/linux/4.4/64/x86/gcc/5.4/mpi/protocols/mpi_refinement/MPI_Refine_Master.os -c -std=c++98 -ffor-scope -isystem external/boost_1_55_0/ -isystem external/ -isystem external/include/ -isystem external/dbio/ -pipe -Wall -Wextra -pedantic -Werror -Wno-long-long -Wno-strict-aliasing -march=core2 -mtune=generic -O3 -ffast-math -funroll-loops -finline-functions -finline-limit=20000 -s -Wno-unused-variable -Wno-unused-parameter -fPIC -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS -DPTR_BOOST -DNDEBUG -DUSEMPI -Isrc -Iexternal/include -Isrc/platform/linux/64/gcc/5.4 -Isrc/platform/linux/64/gcc -Isrc/platform/linux/64 -Isrc/platform/linux -Iexternal/boost_1_55_0 -Iexternal/libxml2/include -Iexternal -Iexternal/dbio -I/usr/include -I/usr/local/include src/protocols/mpi_refinement/MPI_Refine_Master.cc”

Yes, it builds fine without MPI.

[Anonymous]

You will always need to install MPI to compile in MPI.  libopenmpi-dev is the stuff that gets #included, which is why you got those MPI not defiend errors.  I am updating the build documentation to clarify you do need the -dev package here.

You will always need zlib support for Rosetta.  zlib1g-dev is the usual ubuntu package that’s necessary (and it’s not usually present on ubuntu; it varies by system).  That’s the only thing we consider an external dependency other than “a compiler”.  (The feature this buys you is that Rosetta can basically always interpret gzipped inputs natively, and can optionally output many things as .gz as well).  Usually a missing zlib is exposed by “cannot find -lz”, explained in the build documentation (https://www.rosettacommons.org/docs/latest/build_documentation/Build-Documentation#setting-up-rosetta-3_dependencies-troubleshooting)

[Anonymous]

I’m surprised that the site.settings change did nothing – usually it forces a full recompile.  (I guess if it wasn’t compiling before that’s not noticeable).

Does the workstation have something unusual set up w/r/t the shell you are calling scons from versus the default shell for your user account?  Perhaps the environment variables don’t exist in the scons environment if they aren’t being set in your .zshrc or whatever?  (I’m guessing here.)

[Anonymous]

The SetPyRosettaEnvironment.sh shouldn’t do anything with respect to an MPI compile – that just sets things up for PyRosetta, and the compilation of commandline Rosetta is completely separate from it.

Just to confirm, the error message you’re getting is still the same as the one at the top of this thread, right? The “error: ‘MPI’ has not been declared” in MPI_Refine_Master.cc (or similar), right?

If that’s the case, that’s a header issue, and not a library issue — the -L settings and the LD_LIBRARY_PATH shouldn’t affect that. I’d focus more on the “-I” settings for the include path. (One complication here is that you showed a linker commandline for the laptop compile, but a compile commandline for the workstation – that’s why you see the -L’s in one but not in the other.)

My initial guess would be that the path to the MPI includes is missing. That’d be a little odd, though, as you would expect that you’d get a different error in that case. But it might be an issue of *where* the MPI libraries you’re using are located.

If you’re using the Ubuntu system libopenmpi, then it looks like the headers should be (for Xenial) in /usr/lib/openmpi/include/ It doesn’t look like you have that enabled. I’m not sure where it’s pulling  the mpi.hh from, though: probably not from where you expect it. Try adding 

"include_path" : ["/usr/lib/openmpi/include/", ],

To the “prepends” section of your site.settings file and recompiling.

I’m not sure why it’s working on your laptop, though, if you’re using the same packages. Perhaps because you did a manual install of the library in /home/starone/openmpi/ and it’s pulling from that. (If you do want to use the manually installed version rather than the system version, you’ll need to change the paths.)

[Anonymous]

yes, same error

[Anonymous]

Not on the workstation… I followed those steps because when I tried sudo apt-get install openmpi I think I remember it saying it wasn’t found.  I found a page today that indicates that I have to use ‘sudo apt-get install openmpi-bin’… is that right? 

[Anonymous]

I can’t tell you what’s right; I haven’t installed MPI on linux in years.  What you just said is similar to what I suggested so it’s worth trying.  You’ll want the -dev package too, remember.  You should be able to do something like “apt list | grep mpi” on the machine that works to see which mpi packages you installed, and just do the same on the other machine…?

[Anonymous]

I’d definitely recommend using the system versions (the one from apt-get) if you can. MPI is a little bit finicky at times, and it’s a lot easier if you leave setting things up properly to people who know the operating system well, rather than trying to get things set up appropriately yourself.  In fact, if you continue to have issues, I’d recommend removing/uninstalling the manually installed version, to make sure that having two versions of the MPI libraries aren’t confusing things. (You can get issues if Rosetta is picking up the headers of one and the libraries from the other.)

[Anonymous]

Okay, *now* you’re getting issues with your -L settings. The killdevil settings should have fixed this, but you might have to supplement this with the openmpi location

"library_path" : os.environ["LD_LIBRARY_PATH"].split(":") + ["/usr/lib/openmpi/lib/",],

If that works for compilation, in order to run things successfully, you may need to actually add /usr/lib/openmpi/lib/ to your LD_LIBRARY path

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib/

 You can do this either any time you need to run MPI, or you could add it to your .bashrc to have it on permanently.

[Anonymous]

Thank you!  I will give that a try in the morning.

One odd thing I noticed is that on my laptop there are about 79 items in the openmpi bin folder but on the workstation there are less than half that number.  I looked at the build lines and openmpi is one of the -L options. 

I will try what you suggested but I was hoping for your thoughts on the differening file counts… maybe the path is correct but the libraries are missing somehow?

[Anonymous]

Are these both apt-get installed packages? Or are you comparing a manual install to the apt-get install?

I would not be surprised if the apt-get install is smaller (or larger). The people who do the Ubuntu packaging probably know which files are required and which are extra, and remove anything that is not really necessary for general Ubuntu usage. (Or alternatively, there may be multiple ways to do the build, and the Ubuntu packagers do all of them, as they don’t know which ones would really be needed for your particular use case.)

[Author]

Posts