how to transfer the rosetta score to physical unit?

I think the Rosetta Energy Score has no unit. Well, in their publications they use the unit named REU (Rosetta Energy Unit). I don’t think one can convert it to physical unit. But the scores are still comparable for the same protein, or protein of similar size.

I do not suggest doing this, but because the scores are from a KBP, you should be able to use the reverse boltzmann equation to obtain the actual energy. See equation 1 of:

Knowledge Based Potentials: the Reverse Boltzmann
Methodology, Virtual Screening and Molecular Weight
Dependence

Chrysi Konstantinou Kirtay*a, John B. O. Mitchella and James A. Lumley
http://onlinelibrary.wiley.com/doi/10.1002/qsar.200430926/pdf

For the ddG application, there was a calibration set with experimentally known values that was compared to the ddG prediction in REUs, and from the line of best fit a conversion of prediction to kcal/mol was obtained. (See Kellogg et al. (2011) Proteins 79:830-838 for details)

Most of the other protocols have not be calibrated against experimental results. While people sometimes pull the calibration from one protocol to another, that’s not necessarily the most accurate thing to do. The absolute energies (and to a lesser extent the relative energies) you get are dependent on the protocol you use and how extensive the sampling is. And while there might be theoretical reasons to hope that there is a universal correlation between REU and kcal/mol for all situation, there hasn’t been the experimental validation done to confirm that.

So unless you have a test set which you can calibrate a particular protocol against, the general recommendation is to set aside the notion that you can directly convert from REU to kcal/mol, and instead just treat the output as a general “lower=better” metric. (Also keep in mind that even if you have a calibration curve, in all likelihood it will be incredibly noisy, netting you an order of magnitude estimation at best.)