We study the orthogonal perturbation of various coherent function systems (Gabor systems, Wilson bases, and wavelets) under convolution operators. This problem is of key relevance in the design of modulation signal sets for digital communication over time-invariant channels. Upper and lower bounds on the orthogonal perturbation are formulated in terms of spectral spread and temporal support of the prototype, and by the approximate design of worst-case convolution kernels. Among the considered bases, the Weyl–Heisenberg structure which generates Gabor systems turns out to be optimal whenever the class of convolution operators satisfies typical practical constraints.
