FFT is one of the 20th Century's greatest inventions, running as $O(n \log(n))$ rather than as $O(n^2)$ as a simple implementation of a discrete Fourier transform would. But what about half-order Fourier transforms, or arbitrary real order iterated Fourier transforms?
Literature on this topic is quite sparse.
I don't think the Mehler kernel has a FFT equivalent. An easy way to create a fractional FFT is to project $x$ onto the eigenspaces through $$P_k(x) = \frac14\sum_{l=0}^3 i^{-lk} F^l(x),\qquad P_k^2=P_k,P_mP_k=0$$ where $F^l$ is the FFT iterated $l$ times, then $S(x) = \sum_{k=0}^3 e^{2i\pi k/8} P_k(x)$ satisfies $$S^2(x)=\sum_{k=0}^3 i^k P_k(x)=F(x)$$
After a little Google searching. I found a webpage The Fractional Fourier Transform about a book which seems to be very comprehensive and useful. The website states:
Little need be said of the importance and ubiquity of the ordinary Fourier transform in many areas of science and engineering. As a generalization of the ordinary Fourier transform, the fractional Fourier transform is only richer in theory and more flexible in applications--but not more costly in implementation. This book provides a comprehensive and widely accessible account of the transform covering both theory and applications.
Also on the webpage is a link to Matlab code for fast computation of the fractional Fourier transform which may be what you are looking for.
