We report the direct determination of nonradiative lifetimes in Si/SiGe asymmetric quantum well structures designed to access spatially indirect (diagonal) interwell transitions between heavy-hole ground states, at photon energies below the optical phonon energy. We show both experimentally and theoretically, using a six-band k center dot p model and a time-domain rate equation scheme, that, for the interface quality currently achievable experimentally (with an average step height >= 1 A), interface roughness will dominate all other scattering processes up to about 200 K. By comparing our results obtained for two different structures we deduce that in this regime both barrier and well widths play an important role in the determination of the carrier lifetime. Comparison with recently published experimental and theoretical data obtained for mid-infrared GaAs/AlxGa1-xAs multiple quantum well systems leads us to the conclusion that the dominant role of interface roughness scattering at low temperature is a general feature of a wide range of semiconductor heterostructures not limited to IV-IV materials.