Machine learning basics (part 10): Probabilistic models

Some useful statistical formulas

Expectation (expected values)

In practice we calculate — for random variable X — the mean taking into account probabilities of different values. For instance, for a dice with probabilities pj the expected value is calculated:

Variance

It measures how spread out the values are, with its equation as:

The square root of variance is known as standard deviation, σj .

Data matrix

Covariance

The variance looks at the variation in one variable compared to its mean. By generalizing to two variables we get covariance which is a measure of how dependent the two variables are in statistical sense:

If two variables X_j and X_l are independent, then covariance is 0 (they are uncorrelated), while if both increase and decrease at the same time, covariance is positive, and if one goes up while the other goes down, covariance is negative.

The covariance matrix presents how the data vary along with each dimension (variable).