We have experimented with linking directly to Rosetta in Phenix using Boost.Python – pretty similar to what PyRosetta does, but custom-made. It was relatively easy to link to the libraries; just look for .so files in the build/ subdirectory, and add them to the standard linker call (for our purposes, I added “-lbasic -lnumeric -lutility -lObjexxFCL -lcppdb -lsqlite3 -devel -lcore.1 -lcore.2 -lcore.3 -lcore.4 -lcore.5”, which may be overkill but definitely works). If you have linker issues, you can move the .so objects to a more central location, or build them as static libraries (.a files) instead – look at the build documentation for details.
Re-implementing the energy function is a huge undertaking and could easily take years. It is not simply a matter of coding the published equations and reading in the database files; there is a considerable amount of empirical knowledge embedded in the source code which is not documented elsewhere as far as I can tell. It should be relatively straightforward to call Rosetta methods to calculate residue energies, judging by the PyRosetta manual. I’m not sure about pairwise energies, however.