Evidence for the arrow of time is quantified by the Kullback-Leibler divergence (DKL) between the distribution of forward and backward trajectories of a system. Most efforts to apply this idea rely on models, e.g. Markov models for transitions among discrete states. But these models can be wrong, and this leads to uncontrolled errors in our inferences about irreversibility. We present a method to infer the arrow of time from trajectories without relying on a predefined model of the system's dynamics, instead using trajectory data directly. In practice such estimates can be made only for short duration trajectories, so we face the problem of extrapolating to longer times. More fundamentally random errors due to finite sample size become systematic errors in DKL. This is the same problem that arises in other contexts, such as entropy estimation. Following earlier work we give a theory for the systematic errors as a function of sample size and show that we can identify this behavior and extrapolate to infinite data, getting correct answers within errors for model systems. Importantly we recover DKL = 0 in systems that obey detailed balance, where naive estimates from finite data would give nonzero answers. We apply these methods to the trajectories of neural activity in several systems and find evidence of irreversibility even in single neurons. This opens a path to exploring the brain’s representation of the arrow of time.