You should be able to link against Rosetta as a library. I’m aware that some groups have worked on using Rosetta as a library linked into some other program, but I’m not sure how much success they had. FoldIT and PyRosetta both use the C++ libraries in this fashion, but those are both in-house, and put together with a lot of inside knowledge. I don’t know how to go about doing the linking.
Phenix uses Rosetta but I’m pretty sure that’s by making system calls to Rosetta executables.
I will point out that if you just want the scorefunctions, you don’t have to build the protocols (the largest) library to get it. The scorefunction and all of the classic terms are in the core library.