Rosetta reports energies in Rosetta Energy Units.
I know it’s a trite answer but it’s true. Some components of the scorefunction (derived from molecular mechanics) produce scores in kcal/mol. Other components produce scores weighted to be in the same range, but not tied to particular units. If you want to call them a physical unit, then kcal/mol is the closest, but the truest answer is just Rosetta Energy Units.