Fisher Linear Discriminant Analysis

Fisher LDA is a supervised dimensionality reduction technique

• Define mean feature vector $\overline{\mathbf{x}}\triangleq \frac{1}{N}\sum _{i=1}^N{\mathbf{x}}_i$
• Define mean class $k$ feature vector ${\mathit{\mu }}_k\triangleq \frac{1}{N_k}\sum _{i=1}^N{\mathbf{x}}_i\mathbf{1}\{y_i=k\}$.

Fisher LDA attempts to find dimensions that best discriminate the labels by maximizing the following objective $$J(\mathbf{w})=\frac{{\mathbf{w}}^{⊺}{S}_B\mathbf{w}}{{\mathbf{w}}^{⊺}{S}_W\mathbf{w}}$$ with $$S_B\triangleq \sum _{k=1}^K({\mathit{\mu }}_k-\overline{x})({\mathit{\mu }}_k-\overline{x}{)}^{⊺}\mathsf{\text{ and }}{S}_W\triangleq \sum _{k=1}^K\sum _{i=1}^N\mathbf{1}\{y_i=k\}({\mathbf{x}}_i-{\mathit{\mu }}_k)({\mathbf{x}}_i-{\mathit{\mu }}_k{)}^{⊺}$$

$S_B$ is called the between scattering matrix

$S_W$ is called the within scattering matrix