Single atoms form a model system for understanding the limits of single-photon detection. Here, we develop a non-Markovian theory of single-photon absorption by a two-level atom to place limits on the absorption (transduction) time. We show the existence of a finite rise time in the probability of excitation of the atom during the absorption event which is infinitely fast in previous Markov theories. This rise time is governed by the bandwidth of the atom-field interaction spectrum and leads to a fundamental jitter in time stamping the absorption event. Our theoretical framework captures both the weak and strong atom-field coupling regimes and sheds light on the spectral matching between the interaction bandwidth and single-photon Fock state pulse spectrum. Our work opens questions whether such jitter in the absorption event can be observed in a multimode realistic single-photon detector. Finally, we also shed light on the fundamental differences between linear and nonlinear detector outputs for single-photon Fock-state vs coherent-state pulses.