This study investigates flow variability at different scales and its effects on the dispersion of a passive scalar in a regular street network by means of direct numerical simulations (DNS), and compared to wind tunnel (WT) measurements. Specific scientific questions addressed include: (i) sources of variability in the flow at street-network scale, (ii) the effects of such variability on both puff and continuous localised releases, (iii) additional sources of uncertainty related to experimental setups and their consequences. The street network modelled here consists of an array of rectangular buildings arranged uniformly and with periodic horizontal boundary conditions. The flow is driven by a body force at an angle of 45 degrees relative to the streets in the network. Sources of passive scalars were located near ground level at three different types of locations: a short street, an intersection between streets and a long street. Flow variability is documented at different scales: small-scale intra-street variations linked with local flow topology; inter-street flow structure differences; street-network scale variability; and larger-scale spatial variations associated with above-canopy structures. Flow statistics and the dispersion behaviour of both continuous and short-duration (puff) releases of a passive scalar in the street network are analysed and compared with the results of wind-tunnel measurements. Results agree well with the experimental data for a source location in an intersection, especially for flow statistics and mean concentration profiles for continuous releases. Larger differences arise in the comparisons of puff releases. These differences are quantified by computing several puff parameters including time of arrival, travel time, rise and decay times. Reasons for the differences are discussed in relation to the underlying flow variability identified, differences between the DNS and WT setup and uncertainties in the experimental setup. Implications for the propagation of short-duration releases in real urban areas are discussed in the light of our findings. In particular, it is highlighted that in modelling singular events such as accidental releases, characterising uncertainties is more meaningful and useful than computing ensemble averages.