Fourier analysis of the shape of a signal's log-power spectrum. Because this transform applies to magnitudes and not the phases (i.e. it ignores half of the data), the cepstral Buffer is half the size of the FFT Buffer.
Very loosely speaking the cepstrum is an "FFT of an FFT". There are various forms of cepstral analysis - this UGen applies what is sometimes called the "power cepstrum": a Fourier analysis of the spectral log-magnitudes. (The log is important because that's part of what helps separate source-and-filter components in this procedure.) The reverse, ICepstrum, performs IFFT and exponentiates the result, to derive new magnitudes to be stored back into the fftbuf (phases are left untouched; they are essentially meaningless).
a Buffer of half the size as the original FFT Buffer.
a standard FFT chain as produced by the FFT UGen or various PV_ UGens.
a cepstral "chain" which can be treated in many regards like an FFT chain: you can apply PV_ transforms, for example.
In the above example, the cepstrum plot itself may be difficult to interpret (although you can clearly see what the PV_BrickWall unit has done), but the liftered spectrum should look reasonably smooth and show the main peaks and troughs of the spectrum (try running the example without ICepstrum to see the unsmoothed version).
Note: the cepstral analysis here includes the DC bin and ignores the Nyquist bin, so as to ensure that the resulting number of bins is a power of two.