\documentclass[11pt]{article}
\include{lecture}
%\usepackage{alg}

\begin{document}
\lecture{13}{3/02/2010}{Expanders}{Amanda Hittson, Nathan Collins and 
Matthew Anderson}


In the last lecture we introduced randomized computation in terms of
machines that have access to a source of random bits and that return
correct answers more than $\frac{1}{2}$ of the time.  We showed that
$\BPP$ has polynomial-size circuits and the conjecture in the community
is that $\BPP = \P$.

Today we'll introduce expanders, a type of graph that is useful in
improving (amplifying) randomized algorithms with little or no
additional random bit overhead.  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%% Original notes included the following theorem 
%%%%% that is now in lecture 12 (ie not this lecture, the previous one)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% But first we'll prove a theorem
% relating $\BPP$ to the polynomial time hierarchy.

% \section{Hierarchy Results for $\BPP$}

% Generally, whenever we introduce a new computational
% model, we look for hierarchy results: Does the class of problems we
% can solve using the model in question depend on the resources we are
% allowed?  Most hierarchy results use a computable enumeration of all
% machines of the type in question, but the following can be shown.
% % I left this as an exercise because he said ``I'm going to leave it
% % as a exercise that''
% \begin{exercise}
%   There does not exist a computable enumeration of the randomized
%   machines with error probability bounded away from $\frac{1}{2}$.
% \end{exercise}
% \noindent Therefore, typical hierarchy arguments fail for randomized
% machines.  The typical hierarchy arguments can be tailored to prove a
% hierarchy theorem for a modified model of randomized computation
% called promise-$\BPP$.  See HW 1.2.  However, no hierarchy result is
% known for $\BPP$.

% \section{$\BPP$ and the Poly Time Hierarchy}

% Although we don't know if $\BPP = \P$, or even if $\BPP \subseteq
% \NP$, we do know that $\BPP \subseteq \PH$:

% \begin{theorem}\label{12.thm.BPPInSecondLevelOfPH}
%   $\BPP \subseteq \Sigma^{p}_{2} \cap \Pi^{p}_{2}$
% \end{theorem}
% \begin{proof}
%   Fix $M$ a randomized polytime machine that accepts $L \in \BPP$, and
%   let $r$ be the number of random bits $M$ uses when running on an
%   input $x$ of size $n = |x|$.  We will now show that $\BPP \subseteq
%   \Sigma^p_2$.  Since $\BPP = co\BPP$ it follows that $\BPP
%   \subseteq co\Sigma^p_2 = \Pi^p_2$, completing the proof.

%   \begin{figure} [h]
%     \centering
%     $x \in L$
%     \input{rej-acc-ellipse.pstex_t}
%     \quad    
%     \input{acc-rej-ellipse.pstex_t}
%     $x \notin L$
%     % This caption was causing a latex error, but the DVI was being
%     % generated just fine.  I eventually discovered that problem was
%     % due to \notin, which I use above with no trouble.  Replacing
%     % \notin with \not\in fixed the problem.  I'm quite confused ...
%     \caption{If the error rate $\varepsilon$ of $M$ is small, then
%       $Acc(x)$ will be much larger than $Rej(x)$ when $x \in L$, and
%       $Acc(x)$ will be much smaller than $Rej(x)$ when $x \not\in
%       L$. }
%     \label{12:fig:acceptReject}
%   \end{figure}

%   We need to remove randomness from the computation.  For a given
%   input $x$, the space $\{0,1\}^r$ of $r$-bit strings gets partitioned
%   into two pieces: $Acc(x)$, the set of random strings on which $M$
%   accepts $x$, and $Rej(x)$, the set of strings on which $M$ rejects
%   $x$.  If the error rate $\varepsilon$ of $M$ is small, then $Acc(x)$
%   will be much larger than $Rej(x)$ when $x \in L$, and $Acc(x)$ will
%   be much smaller than $Rej(x)$ when $x \not\in L$. See Figure
%   \ref{12:fig:acceptReject}. If our random computation has
%   a small error rate then it will be correct on most random bit
%   strings.  We'll turn ``most'' into ``all.'' The idea is that when $x
%   \in L$ a few ``shifts'' of $Acc(x)$ will cover the whole space
%   $\{0,1\}^r$ of $r$-bit random sequences, while the same few shifts
%   of $Acc(x)$ will fail to cover the whole space when $x \not\in L$.

%   % I've left the footnote in since I, for one, wouldn't otherwise know
%   % what \oplus was denoting.
%   If $S \subseteq \{0,1\}^r$ and $\sigma \in \{0,1\}^r$, then $S \oplus
%   \sigma = \{ s \oplus \sigma \mid s \in S \}$, the \emph{shift} of $S$ by
%   $\sigma$.\footnote{The symbol ``$\oplus$'' denotes XOR, or,
%     equivalently, addition in $\mathbb{Z}^r_2$} Since shifting is
%   invertible we see that $|S \oplus \sigma| = |S|$.

%   We will use shifts to give a $\Sigma_2$-predicate using the
%   intuition discussed above.  Namely, consider
%   \begin{equation}\label{12.eqn.AccXShiftCoversWholeSpace}
%     x \in L \iff \exists \sigma_1, \dots, \sigma_t \forall \rho \in
%     \{0,1\}^r [ \rho \in \bigcup_{i=1}^t Acc(x) \oplus \sigma_i] .
%   \end{equation}
%   Now, $r$ is poly in $n$, and so if we can pick $t$ poly in $n$ as
%   well, then the above will be a $\Sigma^p_2$-predicate, provided that
%   we can verify the membership of $\rho$ in $\bigcup_{i=1}^t Acc(x)
%   \oplus \sigma_i$ in time poly in $n$.  The membership check is no
%   problem since we can equivalently check that $\rho \oplus \sigma_i
%   \in Acc(x)$ for some $i$, which is polytime since it corresponds to
%   running $M$ on $x$ with the random bit string $\rho \oplus
%   \sigma_i$, for poly-many $\sigma_i$.

%   We'll show we can pick $t$ a suitable poly to make
%   \eqref{12.eqn.AccXShiftCoversWholeSpace} true by showing that we can
%   choose the $\sigma_i$s randomly with a high rate of success.  There
%   are two cases to consider:
%   \begin{enumerate}
%   \item $x \in L$: For $\rho$ fixed,
%     \[
%     \Pr_{\sigma_1,\dots,\sigma_t}[ \rho \notin 
%       \bigcup_{i=1}^t Acc(x) \oplus \sigma_i] \le (\mu(Rej(x))^t
%     \le \varepsilon^t,
%     \]
%     where $\mu(Rej(x))$ is the probability that $M$ rejects when it
%     should accept, and is hence not larger than $\varepsilon$.  So, by
%     union bound,
%     \[
%     \Pr[ \{0,1\}^r \not\subseteq \bigcup_{i=1}^t Acc(x) \oplus
%     \sigma_i] \le |\{0,1\}^r|\varepsilon^t = 2^r\varepsilon^t ,
%     \]
%     and hence if $2^r\varepsilon^t < 1$ then some choice of the
%     $\sigma_i$s must work.
    
%   \item $x \notin L$: We need to be sure that for any choice of the
%     $\sigma_i$s that some bit string $\rho$ makes $M$ reject $x$.  But
%     \[
%     \mu(\bigcup_{i=1}^t Acc(x) \oplus \sigma_i) \le \sum_{i=1}^t \mu(Acc(x)
%     \oplus \sigma_i) = t\varepsilon,
%     \]
%     since $\mu(Acc(x)) \le \varepsilon$ when $x \not\in L$, and hence
%     we need $t\varepsilon < 1$.
%   \end{enumerate}
%   So, we need to choose $t$ so that both $t\varepsilon$ and
%   $2^r\varepsilon^t$ are less than $1$.  So $t = r$ works, provided
%   that $\varepsilon < \frac{1}{r}$.  From the definition of $\BPP$ we
%   only know that $\varepsilon < \frac{1}{3}$, but, using the
%   ``majority vote'' trick introduced during the last lecture, we can
%   in $k$ runs reduce the error to $e^{-2k\delta^2}$, where $\delta =
%   \frac{1}{2} - \varepsilon > 0$.  Since $k$ runs will need $kr$
%   random bits, we see that it suffices to choose $k$ poly and large
%   enough that $e^{-2k\delta^2} < \frac{1}{rk}$.  In fact, $k = O(\log
%   r)$ works, and so an examination of the above shows that the time
%   complexity is $O(T^2\log T)$ if the original run-time was $T$.
% \end{proof}

\section{Expanders}

The proof of Theorem 2 in Lecture 12 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%  this should be a reference!!!! Except this doesn't work:
%%%%%%%%%  \ref{12.thm.BPPInSecondLevelOfPH} 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
used the ``majority vote'' trick to get an exponential in $k$ increase in
accuracy using a linear in $k$ increase in random bit usage.  Expander
graphs, which are introduced here, lead to an ``amplification''
technique that gives an exponential in $k$ accuracy improvement using
only a constant increase in random bit usage ($rk$ versus $r + k$).
Expanders have many other uses, some of which we mention in this
lecture, and others we may see in later lectures.

\begin{definition} [$(k,c)$-expanding]
  An (undirected) graph $G = (V,E)$ is \emph{$(k,c)$-expanding} 
  if every $S \subset V$ with $|S| \le k$ 
  satisfies $|\Gamma(S)| \ge c|S|$, where
  $\Gamma(S) = \{ v \in V \mid \exists s \in S, (s,v) \in E \}$ is the
  \emph{neighborhood} of $S$ in $G$.
\end{definition}

Notice that any graph is trivially $(k,c)$-expanding for all $k$ 
with $c \le 1$. Since $\Gamma(V) = V$ one can see it is important 
not to let $|S|$ be too large. In other words, for 
$k \ge |V|$ and $c > 1$, no graph is $(k,c)$-expanding. 

\begin{definition} [Expander family] \label{13.def.expanderFamily}
  An \emph{expander family} is an infinite sequence of graphs
  $G_1 = (V_1, E_1),G_2 = (V_2, E_2),\dots$ with a fixed integer $d$ 
  real number $c > 1$ such that each $G_n$
  has degree $\le d$ and is $(\frac{N}{2},c)$-expanding, where $N =
  N_n = |V_n|$ is the number of vertices in $G_n$.
\end{definition}

Intuitively, expander graphs are ``very connected'' in that the number
of vertices reachable from a given subset of vertices is proportional
to the size of that subset, at least when the subsets aren't so large
as to make this impossible. 

\subsection{Graph Theoretic Properties}

From now on we'll assume all our graphs are $d$-regular (each vertex has 
$d$ edges) and have $N$ vertices. We describe
a $d$-regular graph using a normalized adjacency matrix.  This view of 
a expander graphs proves useful in the analysis of their properties and 
randomized algorithms that make use of them.

\begin{definition} [Normalized adjacency matrix]
  Given a $d$-regular $G = (V,E)$ its $N \times N$ \emph{normalized adjacency
    matrix} $A$ is defined by
  \[
  A_{ij} =
  \begin{cases}
    \frac{1}{d}, & (i,j) \in E\\
    0, & \text{otherwise}
  \end{cases}
  \]
\end{definition}

The normalized adjacency matrix describes the Markov chain of a random
walk on $G$:
\[
A_{ij} = \Pr[\text{go to state $i$ from state $j$}],
\]
and if $p$ is a column vector with $p_i$ the probability of being at
vertex $i$, then $(Ap)_i$ gives the probability of being at vertex $i$ 
after one random step in $G$.  

The normalized adjacency matrix has a number of properties that can be 
proved using basic linear algebra.  We will use the following properties, but 
we do not prove them here.
\begin{proposition}
  $A$ is real-symmetric and so: all eigenvalues of $A$ are real, and
  $A$ has a basis of orthogonal eigenvectors.
\end{proposition}

It turns out that the eigenvalues of the normalized adjacency matrix of an 
expander are closely connected to the expander graph properties.  The 
following are some basic properties, whose proofs we omit.

\begin{proposition}
  Let $A$ be a normalized adjacency matrix of a graph on $N$ vertices.
  \begin{enumerate}
  \item Each eigenvalue $\lambda$ of $A$ satisfies $|\lambda| \le 1$.
  \item 1 is an eigenvalue of $A$, with corresponding eigenvector given 
    by the uniform distribution on $(1/N,1/N,...,1/N)$.
  \item The multiplicity of the eigenvalue $1$ is greater than $1$ iff
    $G$ is disconnected.
  \item $A$ has $-1$ for an eigenvalue iff $G$ is
    bipartite.
  \end{enumerate}
\end{proposition}

As the uniform distribution is always an eigenvector for eigenvalue 1, 
we look only at the 
remaining eigenvector/values.  It turns out that the second largest 
eigenvalue in absolute value is often useful to work with.

\begin{definition} [$\lambda(G)$]
  Let $\Lambda = \{ \lambda \mid \exists e_\lambda \neq 0 
  \text{ such that } (Ae_\lambda = \lambda
  e_\lambda) \wedge (\langle e_\lambda, u \rangle = 0)  \}$. Then
  \[
  \lambda(G) = \max_{\lambda \in \Lambda} |\lambda|
  \]
  is the largest eigenvalue corresponding to an eigenvector
  orthogonal to $u$.
\end{definition}

So, if $G$ is connected and not bipartite then $\lambda(G) < 1$. The 
value $1 - \lambda(G)$ is called the \emph{spectral gap} of the graph. The
next two theorems will not be proved here.  See the notes for CS 880
Spring 2006 for proofs.

\begin{theorem}\label{13.thm.expandingImpliesSpectralGap}
  There exists a function f(x,y) such that if $x > 1$ then $f(x,y) < 1$ and 
  for any graph $G$ that is $(\frac{N}{2}, c)$-expanding, 
  $\lambda(G) < f(c,d)$. Note that the value of $f(c,d)$ 
  depends only on $c$ and $d$.
\end{theorem}

\begin{corollary}
  Expanders have positive spectral gaps and expander families have
  spectral gaps bounded away from 0. 
\end{corollary}

\begin{theorem}\label{13.thm.spectralGapImpliesExpanding}
  If $\lambda(G) \le \lambda < 1$ then $G$ is $(\frac{N}{2},
  g(\lambda))$-expanding where $g(x)$ is a function satisfying 
  $g(\lambda) > 1$ when $\lambda < 1$. Note that $g(x)$ does not depend 
  on $G$, only on $\lambda$. 
\end{theorem}

In light of Theorems \ref{13.thm.expandingImpliesSpectralGap} and
\ref{13.thm.spectralGapImpliesExpanding} we see that we could just
as well have defined expander families in Definition
\ref{13.def.expanderFamily} as $d$-regular families with positive
spectral gaps.

So, for expanders, the uniform distribution $u$ is the only fixed
point.  The following Lemma tells us that the larger the
spectral gap the faster an arbitrary probability distribution
converges to the uniform distribution via a random walk on
$G$.

\begin{lemma} \label{13.lem.spectralConvergenceRelationship}
  For any probability distribution vector $p$
  \[
  \| A^t p - u \|_1 \le \sqrt{N}\lambda^t,
  \]
  where $\| v \|_q = \big[\sum_i |v_i|^q \big] ^{1/q}$ is the $q$-norm
  of $v$ and $\lambda = \lambda(G)$.
\end{lemma}
One way to understand this lemma is: given an initial distribution for a 
random walk $p$, after $t$ steps of the random walk $A^tp$, we will be 
exponentially closer to the random distribution, $u$, provided 
$\lambda < 1$, that is $G$ is connected and not bipartite. 

\begin{proof}
  Since $Au = u$ we have $A^t p - u = A^t (p - u)$.  Now, $(p - u)
  \bot u$ because
  \[
  \langle p - u, u \rangle = \langle p, u \rangle - \langle u, u \rangle
  = \sum_{i=1}^N p_i/N - \sum_{i=1}^N 1/N^2 = 1/N - 1/N,
  \]
  which follows since $p$ is a probability distribution.

  Write $p - u = \sum_i a_i e_i$ for some real $a_i$, 
  where the $\{e_i\}$ form an
  orthogonal basis of eigenvectors for $A$ with $A e_i = \lambda_i e_i$.  Then,
  since the $u$-component of $p - u$ is $0$, we have the following inequality 
  \[
  \| A^t (p - u) \|_2 = \left\|A^t \sum_i a_i e_i \right\|_2 
  = \left\| \sum_i \lambda^t_i a_i e_i \right\|_2 \le
  \lambda^t \left\| \sum_i a_i e_i \right\|_2 = \lambda^t \| p - u \|_2
  \]

  Now
  \[
  \| p - u \|^2_2 + \| u \|^2_2 = \| p \|^2_2 \le \| p \|^2_1 = 1,
  \]
  where $(p - u) \bot u$ implies the first equality, and 
  $\| v \|^2_2 \le \| v \|^2_1$ for all vectors $v$ implies the inequality. 
  By the Cauchy-Schwartz inequality, we have
  \begin{align*}
    \|A^t (p - u)\|_1 &= | \langle
    (-1^{\sigma_1},\dots,-1^{\sigma_N}),
    A^t (p - u) \rangle |\\
    &\le \|(-1^{\sigma_1},\dots,-1^{\sigma_N})\|_2 \| A^t (p - u) \|_2\\
    &= \sqrt{N} \|A^t (p - u) \|_2 ,
  \end{align*}
  where $\sigma_k =
  \begin{cases}
    0, & (A^t (p - u))_k \ge 0\\
    1, & \text{otherwise}
  \end{cases}$.  Combining all of the above we get
  \begin{align*}
    \|A^t (p - u) \|_1 &\le \sqrt{N} \|A^t (p - u) \|_2\\
    &\le \sqrt{N} \lambda^t \| (p - u) \|_2\\
    &\le \sqrt{N} \lambda^t,
  \end{align*}
  completing the proof.
\end{proof}

This lemma can be used to prove that the random walk algorithm for the 
undirected path problem needs only polynomially many steps, i.e. that 
PATH $\in\BPL$.  The proof uses Lemma 
\ref{13.lem.spectralConvergenceRelationship} and the following
exercise.

\begin{exercise} \label{13.claim.spectralBound} If $G$ is connected and
  not bipartite then $\lambda(G) \le 1 - \frac{1}{dN^2}$.
\end{exercise}



\subsection{Constructions}
To prove that expanders exist one can argue that a randomly chosen
$d$-regular graph $G$ (for each vertex, choose $d$ edges randomly) 
has a high probability of being an expander (with $c$ close to $d$).
However, we want to use expanders to reduce the amount of randomness
needed in random algorithms, so using randomness to construct
expanders won't help us.  We want explicit constructions for which
given a vertex $v$ and index $i$ we can compute $v$'s $i$th neighbor
in time $\poly(|v|,|i|)$.

\begin{example}
  For any integer $m \ge 2$, we can get an expander $G$ on vertices $V =
  \mathbb{Z}_m \times \mathbb{Z}_m$ with edges given by the relations
  \[
  \Gamma(\{(x,y)\}) = \{ (x, y \pm x), (x, y \pm (x+1)), (x \pm y, y), (x
  \pm (y+1), y) \}.
  \]
  If $m \ge 4$ then the graph has degree $8$.  The proof that this
  construction works is non-trivial.  See the notes for CS 880 for a
  partial proof using harmonic analysis.
\end{example}

There are other efficient constructions of expanders, but the aforementioned
expander suffices for our needs.  See the notes for CS880 Spring 2006 for
other constructions.

% \section{Next Time}

% In the next lecture we prove the correctness of the algorithms given in the 
% applications above.  To accomplish this, we also prove
% \begin{lemma} [Expander mixing lemma]
%   Given a $d$-regular expander graph $G = (V,E)$ and $S,T \subseteq V$
%   \[
%   \left| \frac{|E(S,T)|}{dN} - \mu(S)\mu(T) \right| \le
%   \lambda(G)\sqrt{\mu(S)(1 - \mu(S))}\sqrt{\mu(T)(1 - \mu(T))}
%   \]
%   where $E(S,T)$ is the set of edges connecting $S$ to $T$.
% \end{lemma}
% Notice that $dN$ is the number of ways to choose an edge in $G$: $N$
% choices of starting vertex and then degree $d$ choices for ending
% vertex.  Also, $\mu(S)\mu(T)$ is the probability that, given two
% randomly chosen vertices, one lies in $S$ and the other in $T$.

% Last lecture we introduced expander graphs.  From a combinatorial
% point of view these were relatively sparse graphs where each vertex
% had a constant degree and were non-trivially expanding.  From an
% algebraic point of view expanders are graphs of constant degree which
% have $\lambda(G) < 1$.  Recall $\lambda(G)$ is the largest absolute value
% eigenvalue of the normalized adjacency of $G$ corresponding 
% to an eigenvector which is perpendicular to the all ones vector.

% Today we discuss two applications of expanders for performing
% correctness amplification on randomized algorithms.  The first
% application decreases the error rate of an algorithm using no
% additional randomness.   The second application decreases the
% error rate of an algorithm even further using only slightly more
% randomness. 

\section{Expander Properties}

The second largest eigenvalue, $\lambda(G)$, 
determines how quickly random walks converge to the
uniform distribution.  Lemma \ref{13.lem.spectralConvergenceRelationship}
showed that for a probability distribution $p$, a normalized matrix $A$ 
and the uniform distribution $u$:

\begin{equation}
\label{13.eq.spectralConvergenceRelationship}
\|A^tp-u\|_1 < \sqrt{N}(\lambda)^t.
\end{equation}

If $\lambda < 1$, the distance between the uniform distribution and a
random walk from an arbitrary distribution $p$ decreases exponentially
with the number of steps $t$.  The left hand side of equation 
\eqref{13.eq.spectralConvergenceRelationship} can be
written using the definition of the 1-norm:

\begin{equation}
  \|A^tp-u\|_1 = 2 \cdot \max_{B\subseteq V} | \Pr[A^tp\in B] -
  \Pr[u\in B]|
\end{equation}

Think of $\|A^tp-u\|_1$ as twice the distance between a uniform
distribution and a random walk for any set of vertices $B$.
Normally it would take $\log V$ random bits to select a vertex at
random from $V$, using this property a vertex can be selected almost
uniformly using $t \log d$ bits by fixing some start vertex and
performing a random walk for $t$ steps.  

\medskip

The next important algebraic property of expanders is called the
\emph{expander mixing lemma}:

\begin{lemma}[Expander Mixing Lemma]
  For every pair of subsets $S,T \subseteq V$,
  \begin{equation}
    \left|\frac{|E(S,T)|}{dN} -
    \mu(S)\mu(T)\right| \le \lambda \sqrt{\mu(S)(1-\mu(S))\mu(T)(1-\mu(T))}.
  \end{equation}
\end{lemma}

This lemma bounds the difference in the distributions of picking two
vertices uniformly at random (second term on left) with picking one
vertex and one neighbor of the vertex at random (first term on left).
In the first case $2 \log V$ random bits are used, in the second case
only $\log V + \log d$ random bits are used. This idea allows the
amount of randomness to be reduced at the cost of producing a
distribution that is not quite uniform.

\begin{proof}{Expander Mixing Lemma}

  Recall that $A(G)$ is symmetric and real implying that it has a full
  orthonormal eigenbasis.  The number of edges between two sets $S$
  and $T$ can be written in terms of their relative characteristic
  vectors (i.e. $\chi_{S,i} = 1$ iff $v_i \in S$ and $\chi_{S,i} = 0$
  otherwise):

  \begin{equation}
    \label{13.eqn.eml.1}
    |E(S,T)| = \chi_S^T (dA) \chi_T.
  \end{equation}

  By definition of $A$, $dA$ is the standard adjacency matrix for $G$.  
  Both $\chi_S$ and $\chi_T$ can be rewritten in terms of their
  components parallel and perpendicular to the uniform vector $u$.
  Recall $u = (\frac{1}{N},\frac{1}{N},...,\frac{1}{N})$ is an 
  eigenvector corresponding to the eigenvalue 1.  The inner product of
  $\chi_S$ with $u$ is $(\chi_S,u) = \frac{|S|}{N} = \mu(S)$ and
  $\chi_S^\parallel = (\chi_S,\hat{u})\hat{u} = 
  (\chi_S,\sqrt{N}u)\sqrt{N}u = |S|u$.  Equation \ref{13.eqn.eml.1}
  can be rewritten: 

  \begin{equation}
    \begin{aligned}
    |E(S,T)| &= (\chi_S^\parallel + \chi_S^\perp)(dA)(\chi_T^\parallel
     + \chi_T^\perp) \\
     &= \chi_S^\parallel dA \chi_T^\parallel + \chi_S^\perp dA
     \chi_T^\perp \\
     &= d \chi_S^\parallel \chi_T^\parallel + \chi_S^\perp dA
     \chi_T^\perp \\
     &= d \frac{|S||T|}{N} + \chi_S^\perp dA \chi_T^\perp. \\
     \end{aligned}
  \end{equation}

  The second and third lines follow from the first because
  $\chi^\parallel$ is an eigenvector of $A$ causing the first term to
  simplify and the cross terms to vanish.  Dividing by $dN$, moving
  terms around and taking the absolute value gives:

  \begin{equation}
    \begin{aligned}
    \left|\frac{|E(S,T)|}{dN} - \mu(S)\mu(T)\right| &=
    \left|\frac{\chi_S^\perp(dA)\chi_T^\perp}{dN}\right| \\
    &\le \frac{\|\chi_S^\perp\|_2 \cdot \lambda d \cdot
    \|\chi_T^\perp\|_2}{N} \\
    \end{aligned}
  \end{equation}
  
  The second line is reached by applying Cauchy-Schwarz to the RHS and
  using the fact that $A$ decreases the magnitude of $\chi_T^\perp$ by
  at least $\lambda$ as there is no component of $\chi_T^\perp$ along
  $u$.  Applying the Pythagorean theorem and some
  simple algebra to $\|\chi_S\|_2$ we can derive the value of
  $\|\chi_S^\perp\|_2$:

  \begin{equation}
    \begin{aligned}
      \|\chi_S\|_2^2 &= \|\chi_S^\parallel\|_2^2 +
      \|\chi_S^\perp\|_2^2, \rm{~therefore}\\
      |S| &= |S|^2 \frac{1}{N} + \|\chi_S^\perp\|_2^2, \rm{~and} \\
      \|\chi_S^\perp\|_2^2 &= |S|(1-\mu(S)), \\
      \|\chi_S^\perp\|_2 &= \sqrt{|S|(1-\mu(S))}. \\
      \end{aligned}
  \end{equation}

  Substituting this back in for $\|\chi_S^\perp\|_2$ and
  $\|\chi_T^\perp\|_2$ and pulling the factor of $N$ into the
  square root completes the proof.

\end{proof}

% \section{Deterministic Amplification}

% Using expanders it is possible to transform a random algorithm, $R$,
% that takes $r$ random bits and errs with probability $\epsilon_0 \le
% \frac{1}{3}$, into an equivalent algorithm, $R'$, also using $r$
% random bits which errs with probability $\le \epsilon$ by calling $R$
% poly($\frac{1}{\epsilon}$) times.   

% The idea behind this transformation is to consider an explicit
% expander graph $G$ with $2^r$ vertices with each vertex corresponding
% to a random bit string in $\{0,1\}^r$.  Pick one vertex, $\rho'$, at
% random from $G$ using $r$ random bits.   Look at all neighbors,
% $\rho \in V$ at distance $t$ from $\rho'$.  Run $R$ on each of these
% neighbors and return the majority vote of these neighbors as the output of
% $R'$.   

% Intuitively, because $G$ is expanding the set of neighbors at distance
% $t$ will be ``spread out'' on the graph and close to uniformly
% distributed.  

% \subsection{Analysis}

% Let $S$ be the set of the bad random strings for $R$ (the strings
% that give the wrong answer for an input $x$).   Let $T$ be the set of
% bad random strings $\rho'$ for $R'$ (the strings whose majority of
% neighbors at distance $t$ give the wrong answer).

% \begin{equation}
%   \begin{aligned}
%     S &= \{ \rho \mid R \rm{~gives~incorrect~answer}\} \\
%     T &= \{ \rho' \mid \rm {~majority~of~} N^t(\rho') \rm{~give~the~incorrect~answer}\} \\
%   \end{aligned}
% \end{equation}

% By construction of $S$, $\mu(S) \le \epsilon_0 \le \frac{1}{3}$.  In
% order for $R'$ to have the required error we want $\mu(T) \le
% \epsilon$.  Consider fixing $t = 1$ then:

% \begin{equation}
%   \frac{|E(S,T)|}{dN} \ge \frac{|T|\frac{d}{2}}{dN} = \frac{\mu(T)}{2}.
% \end{equation}

% This is because at least half of the immediate neighbors ($t=1$) of
% each $\rho'$ in $T$ are in $S$ (they were incorrect for $R$).
% Applying the expander mixing lemma to this gives:

% \begin{equation}
%   \mu(T) |\frac{1}{2} - \mu(S)| = |\frac{\mu(T)}{2} - \mu(S)\mu(T)|
%   \le \left|\frac{|E(S,T)|}{dN} - \mu(S)\mu(T)\right| \le \lambda \sqrt{\mu(S)\mu(T)}.
% \end{equation} 

% The two extra terms in the last expression were dropped because they
% must be less than or equal to 1.  Rearranging terms and bounding
% $\mu(S) \le \epsilon_0$:

% \begin{equation}
%   \label{13.eqn.2}
%   \mu(T) \le \frac{\lambda^2\mu(S)}{(\frac{1}{2}-\mu(S))^2} \le
%   \frac{\lambda^2\epsilon_0}{(\frac{1}{2}-\epsilon_0)^2} \le
%   \epsilon.
% \end{equation} 

% This only gives a constant decrease in error.  Consider the effect of
% taking neighbors at distance $t$ instead of only immediate neighbors.
% Replace $G$ with $G'$ where $G'$ has the same vertices as $G$ but
% there is an edge between two vertices $u$ and $v$ in $G'$ if and only
% if there is a path of length $t$ between $u$ and $v$ in $G$ (allow
% multiple edges between two vertices in $G'$).  This has the effect of
% increasing the degree of $G'$ to $d^t$ and decreasing second
% eigenvalue $\lambda(G') = \lambda^t(G)$, and the prior analysis still
% holds with $\lambda(G')$ substituted for $\lambda(G)$  This changes
% the result of the Equation \ref{13.eqn.2} to give:

% \begin{equation}
%   \mu(T) \le \frac{\lambda^{2t}\epsilon_0}{(\frac{1}{2} -
%   \epsilon_0)^2} \le \epsilon.
% \end{equation}

% Selecting $t = O(\log\frac{1}{\epsilon})$ gives error less than
% $\epsilon$.  The number of neighbors of $\rho'$ at distance $t$ is
% $d^{O(\log\frac{1}{\epsilon})}$ which is polynomial in
% $\frac{1}{\epsilon}$.  The complexity of determining the neighbors is
% polynomial because $G$ is an explicit expander construction.
% Therefore we can reduce the error to $\frac{1}{\rm{poly}}$ from a
% constant using polynomial time and the same randomness as the original
% algorithm.  

% \section{Randomness Efficient Amplification}

% The idea of this second application is to reduce the error further by
% using a little additional randomness.  

% Given a random algorithm $R$ which uses $r$ random bits and errs with
% probability $\epsilon_0 \le \frac{1}{3}$, it can be transformed
% into an equivalent algorithm $R'$ which uses $r +
% O(\log\frac{1}{\epsilon})$ random bits with error $\le \epsilon$ by
% calling $R$ only $O(\log\frac{1}{\epsilon})$ times in total.  Notice
% this construction allows us to acheive exponentially small error
% ($\frac{1}{2^n}$) in polynomial time with only slightly more randomness.

% Such a transformation can be done trivially using
% $O(r\log\frac{1}{\epsilon})$ bits by sampling $\log\frac{1}{\epsilon}$
% random strings, but does not achieve the additive result suggested.
% In order to improve on the trivial result, we perform a variation on
% approach in the previous application.

% Again, pick $\rho' \in \{0,1\}^r$ uniform at random.  Consider all
% $\rho$ on a random walk of length $t$ starting at $\rho'$.  Run $R$ on
% all such $\rho$ and take the majority vote.  

% The idea here is that we can do a better job than in the previous
% application because there is more randomness to work with and
% neighbors at a further distance can be visited (because only one
% neighbor at each distance is visited).  

% \subsection{Analysis}

% There is one key lemma that will allow us to bound the error of this
% approach:

% \begin{lemma}
%   Let $P$ be a projection on those $\rho$ for which $R$ errs, then
%   for any vector $x$:
%   \begin{equation}
%     \|PAx\|_2 \le \sqrt{\epsilon_0+\lambda^2}\|x\|_2.
%   \end{equation}
% \end{lemma}

% \begin{proof}
%   Consider the representation of $x = x^\parallel + x^\perp$ in the
%   eigenbasis with respect to $u$ as stated earlier.  Then by the triangle inequality:

%   \begin{equation}
%     \|PAx\|_2 \le \|PAx^\parallel\|_2 + \|PAx^\perp\|_2.
%   \end{equation}
  
%   Using the facts that the bad set is small, $P$ projects onto the bad
%   set, and that $x^\perp$ contracts by at least $\lambda$:

%   \begin{equation}
%     \begin{aligned}
%       \|PAx^\parallel\|_2 = \|Px^\parallel\|_2 \le \sqrt{\epsilon}
%       \|x^\parallel\|_2, \\
%       \|PAx^\perp\|_2 \le \|Ax^\perp\|_2 \le \lambda \|x^\perp\|_2. \\
%     \end{aligned}
%   \end{equation}
    
%   Substituting back into the original equation we have:

%   \begin{equation}
%     \|PAx\|_2 \le \sqrt{\epsilon_0} \|x^\parallel\|_2 + \lambda
%     \|x^\perp\|_2 = (\sqrt{\epsilon_0},\lambda) \cdot
%     (\|x^\parallel\|_2,\|x^\perp\|_2)^T \le \sqrt{\epsilon_0+\lambda^2}\|x\|_2
%   \end{equation}

%   The last inequality follows from Cauchy-Schwarz.
% \end{proof}

% This lemma can be used to bound the error probability of $R'$.
% Consider the probability that $R'$ errs, this is the same as the
% probability that at least half of the steps in the random walk fell in
% the set where $R$ errs.

% \begin{equation}
%   \begin{aligned}
%     \Pr[R' \rm{~errs}] &= \Pr[\rm{At~least~} \frac{t}{2} \rm{~of~} \rho
%       \rm{~fall~in~the~set~on~which~} R \rm{~errs}] \\
%     &\le \sum_{B \subseteq [t], |B| \ge \frac{t}{2}} \Pr[(\forall i
%       \in B) i^{\rm{th}} \rm{~step~lies~in~the~bad~set~for~} R] \\
%     &= \sum_{B \subseteq [t], |B| \ge \frac{t}{2}} \|M_tAM_{t-1}A
%     ... M_2AM_1Au\|_1 \\
%     &\le \sum_{B \subseteq [t], |B| \ge \frac{t}{2}} \sqrt{N} \|M_tA
%     ... M_1Au\|_2 \\
%     &\le \sum_{B \subseteq [t], |B| \ge \frac{t}{2}} \sqrt{N}
%     \left(\sqrt{\epsilon_0 +\lambda^2}\right)^{|B|}\|u\|_2 \\
%     &= \sum_{B \subseteq [t], |B| \ge \frac{t}{2}} \left(\sqrt{\epsilon_0
%       +\lambda^2}\right)^{|B|}  \\
%     &\le 2^t \cdot \left(\sqrt{\epsilon_0 +\lambda^2}\right)^{\frac{t}{2}} \\
%     &= (4\sqrt{\epsilon_0+\lambda^2})^{\frac{t}{2}} \le \epsilon. \\
%   \end{aligned}
% \end{equation}

% The second line upper bounds the actual
% probability because it is over counting the bad strings.  This
% probability is rewritten as a matrix in the third line, where $M_i =
% P$ if and only if $i \in B$ and $M_i = I$ otherwise.  The fifth line
% follows from repeated applications of Lemma 2.  A constant number of
% iterations can decrease $\sqrt{\epsilon_0+\lambda^2}$ to less than
% $\frac{1}{4}$, making $|B| = \frac{t}{2}$ maximize the RHS.  We can
% use the deterministic amplification procedure to acheive this.  If
% $4\sqrt{\epsilon_0+\lambda^2} < 1$ walking for $t =
% O(\log\frac{1}{\epsilon})$ steps will give error less than
% $\epsilon$.  This procedure uses $r$ random bits to pick the starting
% vertex, and $\log d$ bits for each of the $\log\frac{1}{\epsilon}$
% steps in the random walk for a total of $r + O(\log\frac{1}{\epsilon})$
% random bits for $R'$.

% \section{Other Results}

% A stronger result which considers the variance of random walks is
% known a the Expander Chernoff Bound.  It states that if you take a
% random walk that the fraction of times the walk lands in the bad set
% does not vary much from the expected number.  Let $X_i$ be an
% indicator variable that indicates the event that the $i^{\rm{th}}$
% step lies in some set $A_i \subseteq V$.  Then the probability that
% the walk varies from the the expected number of steps in the bad sets
% can be written as: 

% \begin{equation}
%   \Pr[\sum_i^t (X_i - \mu(A_i)) \ge at] \le e^{-b(1-\lambda)a^2t}.
% \end{equation}

% where $a \ge 0$ and $b$ is some universal constant.  The probability
% that the walk varies from expected for a constant $a$ decreases
% exponentially as $t$ increases.  This inequality reduces to the
% standard Chernoff Bound if $G$ is a complete graph ($\lambda(G) = 0$
% because rank($G$) = 1 and all the $X_i$ are independent).   

% \section{Next Time}

% Next lecture we will discuss pseudorandom generators for space-bounded
% computation, as another application of expanders.  This application
% leverages the power of being able to pick one vertex and a neighbor at
% random instead of picking two vertices uniformly at random.  We will
% apply this procedure recursively to construct pretty good (though
% perhaps not the best) pseudorandom generators.  Pseudorandom number
% generation is the dual of amplification.  In pseudorandom number
% generation we will try to reduce the amount of randomness used 
% while not making the error grow by too much more.


\subsection{Next Time}

The expander mixing lemma can be used to reduce the error probability
of randomized algorithms while using little or no extra random bits.  
In particular, we'll use expanders for two purposes:

\begin{enumerate}
\item \label{13.enum.deterministicAmplification} \emph{Deterministic
    amplification}: Given a randomized algorithm $R$ that uses $r$
  random bits we reduce the error to be less than some arbitrary
  $\varepsilon$, without using any additional random bits.  This
  requires running $R$ $\poly(1/\varepsilon)$ times.
\item \label{13.enum.randomnessEfficientAmplification}\emph{Randomness
    efficient amplification}: With $R$ and $r$ as
  above we reduce the error to be less than some arbitrary
  $\varepsilon$ using $r + O(\log(1/\varepsilon))$ additional random
  bits and running $R$ $O(\log(1/\varepsilon))$ times.
\end{enumerate}

To accomplish the above amplifications we use an expander graph
$G$ whose vertices are in one-to-one correspondence with the bit strings in
$\{0,1\}^r$.  For application \ref{13.enum.deterministicAmplification} we run
$R$, look at the vertex $v$ in $G$ corresponding to the random bits
used by $R$'s run, and then run $R$ once for each neighbor $v'$ of
$v$, with random bits corresponding to $v'$.  For
application \ref{13.enum.randomnessEfficientAmplification} we use the 
same $G$ and $v$.  Then, starting from $v$, we perform a random walk in $G$
starting from $v$ of length $t = \log(1/\varepsilon)$.  This gives $t$
vertices $v_1,\dots,v_t$.  We then run $R$ once for each vertex $v_i$
with random bits corresponding to $v_i$.  In both cases, we finish by
taking the ``majority vote'' of $R$'s runs.  We discuss these applications
and prove their correctness in the next lecture.


\end{document} 
