Tag Archives: covariance

Plot covariance estimates in a GaussianMixture cluster

covariance

Moving covariance matrix functions from GMM to GaussianMixture in sklearn

So in sklearn 0.18 the GMM model is deprecated and looks to be removed in 0.20. The replacement is GaussianMixture available from sklearn.mixture. I have some code that currently uses GMM and needed to port it to GaussianMixture. One of the major convienence features to GMM are accessors to mean, covars, etc. The Covariance matrix itslef is critical for many of the clustering applications that would motivate the original use of GMM. This post shows how to port that covariance accessor to the promoted GaussianMixture implementation.

Covariance Matrix

Remember that the covariance matrix is a matrix representing the covariances between all of the elements between two vectors, X and Y:

Cov[X,Y]=E[(X − E[X])(Y − E[Y])]=E[XY] − E[X]E[Y]

An alternative form, perhaps more expressive when considering the matrix forms that we are dealing with here and using Σ as the covariance matrix, and μ is the mean of any random vector X:

Σ = E[(X − μ)(X − μ)T ] = E[XXT ] − μμT

Python Intelligent Algorithms

Working with a GMM clustering over the iris dataset, there is a dependency on the make_ellipses method published by Ron Weiss ronweiss@gmail.com.
The most non-trivial move required to shift from GMM to GaussianMixture involves:

v, w = np.linalg.eigh(gmm._get_covars()[n][row_idx[:, None], col_idx])

which needs to shift to

v, w = np.linalg.eigh(gmm.covariances_[n][row_idx[:, None], col_idx])

in order to work with the newer model package.
The full listing of the function as I have ported it is then:

def make_ellipses(gmm, ax, x, y):
"""
Extracts a covariance matrix in 2D from a higher dimensional feature space.
Calculates an ellipse along maximal variance in a GMM object in 2D,
both direction and the respective magnitude, i.e., the eigenvector and eigenvalue
of the covariance matrix. It writes the resulting ellipse onto an existing pyplot plot.
:param gmm: sklearn GaussianMixture object
:param ax: plot axis - the 2D subset of the full feature space
:param x: the first dimension of the 2D plot axis
:param y: the second dimension of the 2D plot axis
:return:
"""
for n, color in enumerate('rgb'):
    row_idx = np.array([x, y])
    col_idx = np.array([x, y])
    # FIXME GMM has method _get_covars not present in GaussianMixture
    #v, w = np.linalg.eigh(gmm._get_covars()[n][row_idx[:, None], col_idx])
    v, w = np.linalg.eigh(gmm.covariances_[n][row_idx[:, None], col_idx])
    u = w[0] / np.linalg.norm(w[0])
    angle = np.arctan2(u[1], u[0])
    angle = 180 * angle / np.pi  # convert rads to degrees
    v *= 9
    ell = mpl.patches.Ellipse(gmm.means_[n, [x, y]], v[0], v[1], 180 + angle, color=color)
    ell.set_clip_box(ax.bbox)
    ell.set_alpha(0.5)
    ax.add_artist(ell)

For an excellent discussion on the use of the ellipses for plotting the covariances of your GaussianMixture see GMM covariances