Bayesian Networks

Louis Oliphant

oliphant@cs.wisc.edu

CS540 section 2

Remember

chain rule – any joint probability can be represented as a sum of conditional probabilities

conditional independence – A is independent of B given C

Motivation for Bayesian Networks

Size of full joint probability table

N variables that can take on k possible values

size of full joint is kN

Would be nice to keep all information, but in a compacted form

Bayesian networks

A simple, graphical notation for conditional independence assertions and hence for compact specification of full joint distributions

Syntax:

a set of nodes, one per variable

a directed, acyclic graph (link ≈ "directly influences")

a conditional distribution for each node given its parents:

P (Xi | Parents (Xi))

In the simplest case, conditional distribution represented as a conditional probability table (CPT) giving the distribution over Xi for each combination of parent values

Example

Topology of network encodes conditional independence assertions:

Weather is independent of the other variables

Toothache and Catch are dependent upon Cavity and they are conditionally independent from each other given Cavity

Example

I'm at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn't call. Sometimes it's set off by minor earthquakes. Is there a burglar?

Variables: Burglary, Earthquake, Alarm, JohnCalls, MaryCalls

Network topology reflects "causal" knowledge:

A burglar can set the alarm off

An earthquake can set the alarm off

The alarm can cause Mary to call

The alarm can cause John to call

Example contd.

Compactness

A CPT for Boolean Xi with k Boolean parents has 2k rows for the combinations of parent values

Each row requires one number p for Xi = true
(the number for Xi = false is just 1-p)

If each variable has no more than k parents, the complete network requires O(n · 2k) numbers

I.e., grows linearly with n, vs. O(2n) for the full joint distribution

For burglary net, 1 + 1 + 4 + 2 + 2 = 10 numbers (vs. 25-1 = 31)

Semantics

The full joint distribution is defined as the product of the local conditional distributions:

P (X1, … ,Xn) = Πi = 1 P (Xi | Parents(Xi))

e.g., P(j  m  a  b  e)

= P (j | a) P (m | a) P (a | b, e) P (b) P (e)

n

Two Parts to every Model

Inference (classifying)

Inference by Enumeration (exact method)

Variable Elimination (exact method)

Sampling methods (approximation method) (next time)

Induction (learning) (next time)

parameter learning

Given a Bayes Net graph, fill in the values in the CPTs using some training set of data

structure learning

Given some training set construct the Bayes Net topography

Inference in Bayes Networks

Three types of nodes during inference

Query Nodes

These are the ones you want to know the probability distribution about

Evidence Nodes

These are the ones you know what their values are

Hidden Nodes

These are the ones you don't know anything about

More powerful than simply having a fixed classification feature like with decision trees, neural nets, etc.

You can query about any node, or any set of nodes

Another Example

Inference by Enumeration

Inference by Enumeration

P(X|e)=P(X,e)= y P(X,e,y)

Query Variables

Evidence Variables

Hidden Variables

Normalization Constant

(computed after the fact)

Inference by Enumeration

Lets figure out what the probability is of a burglary given the fact that both John and Mary have called.

P(B|j,m) = P(B,j,m) = eaP(B,e,a,j,m)

Remember the Chain Rule. You can rewrite any joint probability as a product of conditional probabilities

So figuring out P(b|j,m) (half of the problem)

P(b|j,m)=eaP(b) P(e) P(a|b,e) P(j|a) P(m|a)

=P(b) eP(e) aP(a|b,e) P(j|a) P(m|a)

Example

Inference by Enumeration

+

+

+

P(b)

.001

p(e)

.002

p(a|b,e)

.95

p(j|a)

.90

p(m|a)

.70

p(j|a)

.90

p(m|a)

.70

p(j|¬a)

.05

p(m|¬a)

.01

p(j|¬a)

.05

p(m|¬a)

.01

p(¬a|b,e)

.05

p(a|b,¬e)

.94

p(¬a|b,¬e)

.06

p(¬e)

.998

Notice the repeated sub-structures. It would be nice to do them once and

store the result. This would speed up computation.

Inference by Enumeration

You try it:

P(JohnCalls=true | Burglary=true)

setup the equation for finding the probability.

Variable Elimination

Variable Elimination

Point-wise product between two factors

Summing out variable A

Variable Elimination

Summary

Bayesian networks provide a natural representation for (causally induced) conditional independence

Topology + CPTs = compact representation of joint distribution

Query, Evidence, and Hidden nodes

Inference by Enumeration

Variable Elimination

2005-11-17T04:33:12