Joint Marginal and Conditional

Joint, Marginal, and Conditional Distributions

We engineers often ignore the distinctions between joint, marginal, and conditional probabilities - to our detriment.

Joint probability is the probability of two or more things happening together. f(x, y | q) where f is the probability of x and y together as a pair, given the distribution parameters, q. Often these events are not independent, and sadly this is often ignored. Furthermore, the correlation coefficient itself does NOT adequately describe these interrelationships.

Consider first the idea of a probability density or distribution:
f(x | q) where f is the probability density of x, given the distribution parameters, q. For a normal distribution, q = (m, s²)^T where m is the mean, and s is the standard deviation. This is sometimes called a pdf, probability density function. The integral of a pdf, the area under the curve (corresponding to the probability) between specified values of x, is a cdf, cumulative distribution function, F(x | q). For discrete f, F is the corresponding summation.

A joint probability density two or more variables is called a multivariate distribution. It is often summarized by a vector of parameters, which may or may not be sufficient to characterize the distribution completely. Example, the normal is summarized (sufficiently) by a mean vector and covariance matrix.

marginal probability: f(x | q) where f is the probability density of x, for all possible values of y, given the distribution parameters, q. The marginal probability is determined from the joint distribution of x and y by integrating over all values of y, called "integrating out" the variable y. In applications of Bayes's Theorem, y is often a matrix of possible parameter values. The figure illustrates Joint, marginal, and conditional probability.

Schematic showing joint, marginal, and conditional densities

conditional probability: f(x | y; q) where f is the probability of x by itself, given specific value of variable y, and the distribution parameters, q. (See Figure) If x and y represent events A and B, then P(A|B) = n_AB/n_B , where n_AB is the number of times both A and B occur, and n_B is the number of times B occurs. P(A|B) = P(AB)/P(B), since P(AB) = n_AB/N and P(B) = n_B/N so that

conditional.gif (2047 bytes)

Note that in general the conditional probability of A given B is not the same as B given A. The probability of both A and B together is P(AB), and P(A|B) X P(B) = P(AB) = P(B|A) X P(A), if both P(A) and P(B) are non-zero. This leads to a statement of Bayes's Theorem:

P(B|A) = P(A|B) X P(B)/P(A). Conditional probability is also the basis for statistical dependence and independence.

independence: Two variables, A and B, are independent if their conditional probability is equal to their unconditional probability. In other words, A and B are independent if, and only if, P(A|B)=P(A), and P(B|A)=P(B). In engineering terms, A and B are independent if knowing something about one tells nothing about the other. This is the origin of the familiar, but often misused, formula P(AB) = P(A) X P(B), which is true only when A and B are independent.

conditional independence: A and B are conditionally independent, given C, if
Prob(A=a, B=b | C=c) = Prob(A=a | C=c) X Prob(B=b | C=c) whenever Prob(C=c) > 0. So the joint probability of ABC, when A and B are conditionally independent, given C, is then Prob(C) X Prob(A | C) X Prob(B | C) A directed graph illustrating this conditional independence is A ? C ? B.