Prev: W10, Next: W12
Zoom: Link, TopHat: Link (341925), GoogleForm: Link, Piazza: Link, Feedback: Link.
Slide:
# Adversarial Search
📗 The search problem for games (one or more opponents are searching at the same time) is adversarial search: Link, Wikipedia.
➩ The initial state is the beginning of the game.
➩ Each successor represent an action or a move in the game.
➩ The goal states are called terminal states, the search problem tries to find the terminal state with the lowest cost (highest reward).
📗 The opponents search at the same time and tries to minimize their costs (or maximize their rewards).
➩ For zero-sum games, the sum of the rewards and costs for the two players is 0 at every terminal state. The opponent minimizes the reward (or maximizes the cost) of the first agent: Wikipedia.
In-class Discussion
📗 [1 points] pirate got gold coins. Each pirate takes a turn to propose how to divide the coins, and all pirates who are still alive will vote whether to (1) accept the proposal or (2) reject the proposal, kill the pirate who is making the proposal, and continue to the next round. Use strict majority rule for the vote, and use the assumption that if a pirate is indifferent, they will vote reject with probability 50 percent. How will the first pirate propose? Enter a vector of length , all integers, sum up to .
📗 Answer (comma separated vector): .
[L11Q]
Other students' answers:
In-class Discussion
📗 [1 points] \(n\) lions are hiding behind a row of trees, a wild bunny appears, lion 1 can choose to eat the bunny or stay hidden, if lion 1 stays hidden, the game ends, if lion 1 chooses to eat the bunny, lion 2 can choose to eat lion 1 or stay hidden ... If \(n\) is a (uniform) random integer between and (include both numbers), what is the probability that the bunny is eaten?
📗 Answer: .
[L11Q]
Other students' answers:
In-class Discussion
📗 [1 points] Two players, A and B, objects, each player can pick 1 or 2 each time. Pick the last object to win. If you pick first, how many should you pick?
📗 In the diagram, the node has the player name (or G for end of game) and the subscript means how many objects are left. The terminal nodes where player A wins are marked green, and where player B wins are marked blue.
📗 Answer: .
# Minimax Algorithm
➩ The game is solved backward starting from the terminal states.
➩ Each agent (player) chooses the best action given the optimal actions of all players in the subtrees (subgames).
📗 For zero-sum games, the optimal value at an internal state for the MAX player is called \(\alpha\left(s\right)\) and for the MIN player is called \(\beta\left(s\right)\).
📗 The time and space complexity is the same as DFS with \(D = d\).
In-class Quiz
📗 [4 points] Highlight the solution of the game (drag from one node to another to highlight the edge connecting them).
Algorithm
# Non-Determinstic Games
📗 For non-deterministic games, some internal states represent moves by Chance (can be viewed as another player), for example, dice roll or coin flip. For those states, expected cost or reward are used instead of max or min.
📗 DFS on games with Chance is called expectiminimax: Wikipedia.
In-class Quiz
ID:📗 [4 points] Consider a zero-sum sequential move game with Chance. player moves first, then Chance, then . The values of the terminal states are shown in the diagram (they are the values for the Max player). What is the (expected) value of the game (for the Max player)?
📗 Note: in case the diagram is not clear, the probabilities from left to right is: , and the rewards are .
📗 Answer: .
[L11Q]
# Pruning
📗 When a branch will not lead the current agent (player) to win, the branch can be pruned (both players will not need to search the subtree). DFS with pruning is called Alpha-Beta Pruning: Wikipedia.
➩ During DFS, keep track of both \(\alpha\left(s\right)\) and \(\beta\left(s\right)\) for each state.
➩ Here, the \(\alpha\left(s\right)\) and \(\beta\left(s\right)\) values are the current best value of an internal state so far (based only on the successor states that are expanded), which are not necessarily the final optimal values.
➩ Prune the subtree after \(s\) if \(\alpha\left(s\right) \geq \beta\left(s\right)\).
In-class Quiz
📗 [4 points] Highlight the nodes that will be alpha-beta pruned in the following game.
[L11Q]
Algorithm
# Static Evaluation Function
📗 The heuristic to estimate the value at an internal state for games is called a static (board) evaluation (SBE) function: Wikipedia.
➩ For zero-sum games, SBE for one player should be the negative of the SBE for the other player.
➩ At terminal states, the SBE should agree with the cost or reward at that state.
📗 For Chess, the SBE can be computed by a neural network based some features such as: material, mobility, king safety, center control; or a convolutional neural network treating the board as an image.
📗 IDS can be used with SBE.
➩ In iteration \(d\), the depth is limited to \(d\) and the SBE of the internal states at depth \(d\) are used as their cost or reward.
In-class Quiz
📗 [4 points] The subscripts are heuristics (static evaluation, or estimated alpha and beta values) at the internal nodes.
In-class Discussion
Name:📗 [1 points] Find the optimal strategy against a min player that uses a random strategy with probability \(p\):
Probability \(p\): 1
Heuristic: 0
Winner: -
1 slider 0
# Monte Carlo Tree Search
📗 Random subgames can be simulated by selecting random moves for both players: Wikipedia.
📗 The move corresponding to the highest expected reward (win rates) can be picked.
➩ The move corresponding to the highest optimistic estimate of the reward (win rates) can be also picked.
Example
📗 Alpha GO uses Monte Carlo Tree Search with more than \(10^{5}\) play-outs: Wikipedia.
Math Notes
📗 The optimistic estimate of the reward is called upper confidence bound of the rewards (or win rates here): \(\dfrac{w_{s}}{n_{s}} + c \sqrt{\dfrac{\log T}{n_{s}}}\), where \(w\) is the number of wins after state \(s\), \(n\) is the number of simulations after \(s\), and \(T\) is the total number of simulations.
➩ More details will be discussed in the reinforcement learning lectures.
# Rationalizability
📗 Unlike sequential games, for simultaneous move games, one player (agent) does not know the action taken by the other player.
📗 Given the actions of the other players, the optimal action is called the best response.
📗 An action is dominated if it is worse than another action given all actions of the other players.
➩ For finite games (finite number players and finite number of actions), an action is dominated if and only if it is never a best response.
➩ An action is strictly dominated if it is strictly worse than another action given all actions of the other players. A dominated action is weakly dominated if it is not strictly dominated.
📗 Rationalizability (IESDS, Iterative Elimination of Strictly Dominated Strategies): iteratively remove the actions are that dominated (or never best responses for finite games): Wikipedia.
In-class Discussion
📗 [1 points] Write down an integer between and that is the closest to two thirds \(\dfrac{2}{3}\) of the average of everyone's (including yours) integers.
📗 Answer: .
[L11Q]
Other students' answers:
In-class Discussion
📗 [1 points] Two identical antiques are lost. The airline only knows that its value is at most dollars, so the airline asks their owners (travelers) to report its value (non-negative integers larger than or equal to . The airline tells the travelers that they will be paid the minimum of the two reported values, and the traveler who reported a strictly lower value will receive dollars in reward from the other traveler. If you are one of the travelers, what will you report?
📗 Answer: .
[L11Q]
Other students' answers:
In-class Quiz
ID:📗 [4 points] Perform iterated elimination of strictly dominated strategies (i.e. find rationalizable actions). Player A's strategies are the rows. The two numbers are (A, B)'s payoffs, respectively. Recall each player wants to maximize their own payoff. Enter the payoff pair that survives the process. If there are more than one rationalizable action, enter the pair that leads to the largest payoff for player A.
| A \ B | I | II | III |
| I | |||
| II | |||
| III | |||
| IV |
📗 Answer (comma separated vector): .
[L11Q]
# Nash Equilibrium
📗 Another solution concept of a simultaneous move game is called a Nash equilibrium: if the actions are mutual best responses, the actions form a Nash equilibrium: Wikipedia.
In-class Quiz
ID:📗 [4 points] What is the row player's value in a Nash equilibrium of the following zero-sum normal form game? A (row) is the max player, B (col) is the min player. Use the checkboxes in the table to highlight the best responses: the first checkbox for the row player and the second checkbox for the column player.
| A \ B | I | II | III | IV |
| I | ||||
| II | ||||
| III | ||||
| IV |
📗 Answer: .
[L11Q]
# Prisoner's Dilemma
📗 A symmetric simultaneous move game is a prisoner's dilemma game if the Nash equilibrium (using strictly dominant actions) is strictly worse for all players than another outcome: Link, Wikipedia.
➩ For two players, the game can be represented by a game matrix: \(\begin{bmatrix} - & C & D \\ C & \left(x, x\right) & \left(0, y\right) \\ D & \left(y, 0\right) & \left(1, 1\right) \end{bmatrix}\), where C stands for Cooperate (or Deny) and D stands for Defect (or Confess), and \(y > x > 1\). Here, \(\left(D, D\right)\) is the only Nash equilibrium (using strictly dominant actions) but \(\left(C, C\right)\) is strictly preferred by both players.
In-class Discussion
📗 Guess the outcome? [Player on the left, Player on the right]
➩ A: Split, Split
➩ B: Split, Steal
➩ C: Steal, Split
➩ D: Steal, Steal
[L11Q] Other students' answers:
📗 Again,
➩ A: Split, Split
➩ B: Split, Steal
➩ C: Steal, Split
➩ D: Steal, Steal
[L11Q] Other students' answers:
📗 Again,
➩ A: Split, Split
➩ B: Split, Steal
➩ C: Steal, Split
➩ D: Steal, Steal
[L11Q] Other students' answers:
📗 Again,
➩ A: Split, Split
➩ B: Split, Steal
➩ C: Steal, Split
➩ D: Steal, Steal
[L11Q] Other students' answers:
📗 Last time,
➩ A: Split, Split
➩ B: Split, Steal
➩ C: Steal, Split
➩ D: Steal, Steal
[L11Q] Other students' answers:
# Normal-form of Sequential Games
📗 Sequential games can be converted into normal-form games where the actions are sequences of actions in the sequential games.
➩ The solution to the sequential game must be a Nash equilibrium of the normal-form game.
➩ Some Nash equilibria of the normal-form game may not be a solution (technically called a Subgame Perfect Equilibrium (SPE)) of the sequential game.
📗 The non-SPE Nash equilibria involve non-credible threats where the second player will use a non-optimal action in a subgame that causes the first player to use another action (so that the subgame will not be reached).
In-class Discussion
📗 [1 points] Country A can choose between Attack (A) and Not attack (N). If Country A chooses Not attack, the game ends, otherwise, Country B can choose to Fight (F) and Escape (E). Country A prefers (A, E) over (N) over (A, F). Country B prefers (N) over (A, E) over (A, F).
➩ How many Nash equilibria are there?
➩ What if Country B has the option to Burn the bridge (B) or Keep the bridge (K) at the beginning, and if B is chosen, the action E cannot be used?
📗 Answer (comma separated vector): .
[L11Q]
# Coordination Games
📗 Symmetric (simultaneous-move) games can have asymmetric Nash equilibria, but since the players are symmetric (identical), they might not know how to coordinate.
➩ Agreement before the game.
➩ Focal points (default actions).
➩ Randomized (mixed strategy).
In-class Discussion
📗 [1 points] There are three parking lots A, B, C with capacities \(15, 10, 5\). If a lot is below capacity, everyone in that lot gets \(+10\), and if a lot is above capacity, everyone in that lot gets \(-5\). Which parking lot would you choose?
A B
C
[L11Q]
Other students' answers:
In-class Discussion
📗 [1 points] There are three parking lots A, B, C with capacities \(15, 10, 5\). If a lot is below capacity, everyone in that lot gets \(+10\), and if a lot is above capacity, everyone in that lot gets \(-5\). Which parking lot would you choose?
A B
C
[L11Q]
Other students' answers:
In-class Discussion
📗 [1 points] There are three parking lots A, B, C with capacities \(15, 10, 5\). If a lot is below capacity, everyone in that lot gets \(+10\), and if a lot is above capacity, everyone in that lot gets \(-5\). Which parking lot would you choose?
A B
C
[L11Q]
Other students' answers:
# Mixed Strategy
📗 A mixed strategy is when a player randomizes between multiple actions: Wikipedia.
📗 A pure strategy is when a player uses only one action with probability 1.
📗 A mixed strategy Nash equilibrium is a Nash equilibrium for the game in which mixed strategies are allowed (called mixed extension of the original game).
➩ If the mixed strategies are mutual best responses, then they form a mixed strategy Nash equilibrium (see Math Note for the mathematical definition).
Math Note
📗 For a two-player general-sum game with rewards \(r\) (both players are maximizing), an action profile \(\left(a_{1}, a_{2}\right)\) is a pure strategy Nash equilibrium if \(r_{1}\left(a_{1}, a_{2}\right) \geq r_{1}\left(a', a_{2}\right)\) for every \(a'\) valid action of player \(1\); and \(r_{2}\left(a_{1}, a_{2}\right) \geq r_{2}\left(a_{1}, a'\right)\) for every \(a'\) valid action of player \(2\).
📗 An mixed strategy profile \(\left(\pi_{1}, \pi_{2}\right)\), where \(\pi_{i}\left(a\right)\) is the probability that player \(i\) chooses action \(a\), is a mixed strategy Nash equilibrium if \(\mathbb{E}\left[r_{1}\left(\pi_{1}, \pi_{2}\right)\right] \geq \mathbb{E}\left[r_{1}\left(\pi', \pi_{2}\right)\right]\) for every \(\pi'\) mixed strategy of player \(1\); and if \(\mathbb{E}\left[r_{2}\left(\pi_{1}, \pi_{2}\right)\right] \geq \mathbb{E}\left[r_{2}\left(\pi_{1}, \pi'\right)\right]\) for every \(\pi'\) mixed strategy of player \(2\).
➩ Here, the expectation is over the randomness of the mixed strategies, or \(\mathbb{E}\left[r_{i}\left(\pi_{1}, \pi_{2}\right)\right] = \displaystyle\sum_{a_{1}, a_{2}} r_{i}\left(a_{1}, a_{2}\right) \pi_{1}\left(a_{1}\right) \pi_{2}\left(a_{2}\right)\), where the sum is over all valid \(\left(a_{1}, a_{2}\right)\) actions of players \(1\) and \(2\).
➩ This condition is usually difficult to check: another equivalent condition is: \(\left(\pi_{1}, \pi_{2}\right)\) is a mixed strategy Nash equilibrium if \(\mathbb{E}\left[r_{1}\left(\pi_{1}, \pi_{2}\right)\right] \geq \mathbb{E}\left[r_{1}\left(a', \pi_{2}\right)\right]\) for every \(a'\) valid action of player \(1\); and if \(\mathbb{E}\left[r_{2}\left(\pi_{1}, \pi_{2}\right)\right] \geq \mathbb{E}\left[r_{2}\left(\pi_{1}, a'\right)\right]\) for every \(a'\) valid action of player \(2\).
Example
📗 For rock paper scissor game, there is no pure Nash equilibrium, but there is one mixed Nash equilibrium where every player uses each action with probability \(\dfrac{1}{3}\): Link.
📗 The game matrix is:
| \(a_{1} \setminus a_{2}\) | Rock | Paper | Scissors |
| Rock | \(0\) | \(-1\) | \(1\) |
| Paper | \(1\) | \(0\) | \(-1\) |
| Scissors | \(-1\) | \(1\) | \(0\) |
Nash Equilibrium:
[Note] Use the space to explain the steps or just take notes:
[L11Q]
In-class Quiz
ID:📗 [4 points] Given the following game payoff table, suppose the row player uses a pure strategy, and column player uses a mixed strategy playing L with probability \(q\). What is the smallest and largest value of \(q\) in a mixed strategy Nash equilibrium?
| Row \ Col | L | R |
| U | ||
| D |
Note: the following is a diagram of the best responses (make sure you understand what they are and how to draw them). The red curve is the best response for the column player and the blue curve is the best response for the row player.
📗 Answer (comma separated vector): .
[L11Q]
In-class Quiz
ID:📗 [4 points] Given the following BoS (Battle of Sexes) game, what is the row (Romeo) player's (expected) value (i.e. payoff) in the mixed strategy Nash equilibrium?
| Romeo \ Juliet | Bach | Stravinsky |
| Bach | ||
| Stravinsky |
📗 Answer: .
[L11Q]
Example
📗 Run vs Pass game: YouTube.
# Best Response Dynamics vs Fictitious Play
📗 Best response dynamics starts with random action, and one player at a time use the best response to the other player's actions from the previous iteration.
➩ Time-averages converge to a mixed strategy Nash for zero-sum games.
➩ No result on convergence for general-sum games except for special classes of games such as potential games.
📗 Fictitious play algorithm uses best response to the mixed strategy to the other players' actions in the whole history.
In-class Quiz
ID:📗 [1 points] Given the game , in step \(t\) = , the row player uses action , column player uses action , and fraction of times (including \(t\)) the row player uses each action is , and column player uses is , compute the next step of best response dynamics (row player first) and fictitious play in step \(t + 1\). The actions are \(\left\{0, 1, 2\right\}\) for both players.
📗 Answer (matrix with multiple lines, each line is a comma separated vector): .
[L11Q]
test prt,lio,nim,mnx,chn,abp,abs,tee,tt,tdl,ie,zs,btb,pkg,pkt,pkr,cm,bos,brd q
📗 Notes and code adapted from the course taught by Professors Blerina Gkotse, Jerry Zhu, Yudong Chen, Yingyu Liang, Charles Dyer.
📗 If there is an issue with TopHat during the lectures, please submit your answers on paper (include your Wisc ID and answers) or this Google form Link at the end of the lecture.
📗 Anonymous feedback can be submitted to: Form. Non-anonymous feedback and questions can be posted on Piazza: Link
Prev: W10, Next: W12
Last Updated: November 09, 2026 at 12:45 AM