In search of "better-than-DCT" unitary transforms for encoding of residual signals

It is well known that the discrete cosine transform (DCT) closely approximates the optimal KarhunenLove transform (KLT) under the first-order stationary Markov condition with a strong inter-pixel correlation. However, if the inter-pixel correlation is weak or becomes negative, transforms other than the DCT would possibly become better. In this letter, we present a design framework for finding new unitary transforms according to two principles: 1) they are indeed more efficient than the DCT in encoding of signals with a weak or negative inter-pixel correlation (e.g., the residual signals after motion-compensation or intra- prediction) and 2) each has a fixed transform matrix (i.e., signal-independent) and can be implemented as efficiently as the DCT. Some of the new transforms will be used to demonstrate an improved RD performance in the video coding scenario.