Principal Components

A quick summary, all is detailed below:

• $X$ centered $1_n \, X=0$,
• SVD best approximation to $x$,
• Cols of $V\rightarrow$ new variables,
• Principal Components $C=US\quad C^\prime C=S^2\quad C=XV$,
• Principal axes $Z=U^\prime X=SV^\prime$,
• Supplementary points $x_{k\cdot}V = x_{k\cdot}Z^\prime S^{-1}$,
• Distance between two points,
  
  $$\left( x_{k\cdot}-x_{\ell\cdot} \right)^\prime \left( x_{k\c... ...ot} \right) = \sum^T_{j=1} \left( c_{kj}-c_{\ell j} \right)^2,$$
  
  

• Transition Formulae $Z=S^{-1}C^\prime X\quad C=XZ^\prime S^{-1}$,
• Scale problem solution: $Q$ changes the metric,
• Generalized PCA 2nd generalized svd,
• $B=D^{\frac{1}{2}}X\,Q^{\frac{1}{2}}$, $U = D^{-\frac{1}{2}}E$, $V = D^{-\frac{1}{2}}A$,
• Special case: correspondence analysis,
• $N \; V=\sum\sum$, $r=\frac{1}{2}\, N\cdot 1_p$, $c=\frac{1}{2}\, N^\prime\cdot 1_n$, $X=rc^\prime$,
• $\left( X,D^{-1}_c,D^{-1}_r \right)\:(-)\: \left( X^*,D^{-1}_c,D^{-1}_r \right)$,
• Number of Components. PRESS(M) = $\frac{1}{np} \, \sum^n\sum^p \left( x^m_{ij} - x_{ij} \right)^2$.

Maximization of $u^\prime Au$ with $u^\prime Mu=1$

• for $u$ eigenvector of $M^{-1}A$,
• ${\cal F}$.

1) $X$ centered, all points (observations) same weight.

$$1 \, X=0\quad\mbox{and}\quad x_{ij} = \sum^r_{t=1}x_{it}s_t v_{jt},$$

$p$ variables can be replaced by the $r$ columns of $v$.

$$x^{(k)}_{ij} = \sum^k_{t=1}u_{it}s_t v_{jt}$$

is the best (optimal) approximation of rank $k$ for $X$.

The columns of $V$ are the directions along which the variances are maximal.

Definition: Principal components are the coordinates of the observations on the basis of the new variables (namely the columns of $V$) and they are the rows of $G=XV=US$. The components are orthogonal and their lengths are the singular values $C^\prime C=SU^\prime US=S^2$.

In the same way the principal axes are defined as the rows of the matrix $Z=U^\prime X=SV^\prime$. Coordinates of the variables on the basis of columns of $U$.

$$Z=S^{-1}C^\prime X \quad\mbox{and}\quad G=XZ^\prime S^{-1}.$$

However, this decomposition will be highly dependent upon the unity of measurement (scale) on which the variables are measured. It is only used, in fact, when the $X_{\cdot k}$ are all of the same ``order''. Usually what is done is that a weight is assigned to each variable that takes into account its overall scale. This weight is very often inversely proportional to the variable's variance. So we define a different metric between observations instead of

$$\begin{eqnarray*} d\,\left( x_{i\cdot}, x_{j\cdot}\right) &=& \left( x_{i\cdot}-... ...2_1}\\ & \ddots\\ & & \frac{1}{\sigma^2_p} \end{array}\right). \end{eqnarray*}$$

The same can be said of the observations; some may be more ``important'' than others (resulting from a mean of a group which is larger).

2) General PCA (X,Q,D)

$X$ centered with respect to $D$ :$X D\,1_n=0$.

Generalized singular value decomposition.

$$\begin{array}{lcrclcr} X & = & USV^\prime & \mbox{with} & V^\prime QV & = & I\\ &&&& U^\prime DU & = & I. \end{array}$$

Practical Computation:

$$\begin{eqnarray*} B &=& D^{\frac{1}{2}}X\, Q^{\frac{1}{2}} = ESA^\prime\quad A^\... ...\ V &=& D^{-\frac{1}{2}}A\\ [2ex] X^{(K)} &=& US^{(K)}V^\prime. \end{eqnarray*}$$

Principal Components are the columns of: $C=US=XQV$.

$V$ eigenvectors of $X^\prime DXQ$.

$$Q= \left( \begin{array}{ccc} \frac{1}{\sigma^2_1}\\ & \ddots\\ & & \frac{1}{\sigma^2_p} \end{array}\right).$$

$V$ eigenvectors of correlation matrix.