Deriving an arrow of time from time-reversal symmetric microscopic dynamics is a fundamental open problem in many areas of physics, ranging from cosmology, to particle physics, to thermodynamics and statistical mechanics. Here we focus on the derivation of the arrow of time in open quantum systems and study precisely how time-reversal symmetry is broken. This derivation involves the Markov approximation applied to a system interacting with an infinite heat bath. We find that the Markov approximation does not imply a violation of time-reversal symmetry. Our results show instead that the time-reversal symmetry is maintained in the derived equations of motion. This imposes a time-symmetric formulation of quantum Brownian motion, Lindblad and Pauli master equations, which hence describe thermalisation that may occur into two opposing time directions. As a consequence, we argue that these dynamics are better described by a time-symmetric definition of Markovianity. Our results may reflect on the formulations of the arrow of time in thermodynamics, cosmology, and quantum mechanics.