\documentclass[11pt]{article}

\usepackage{lecture}

\begin{document}

\lecture{5}{2/6/2013}{Applications of $k$-Wise Uniform Generators}{Lubos Krcal}

In this lecture we discuss two applications of $k$-wise uniform 
generators, namely (i) list decoding for the Hadamard code, which 
hinges on the linear structure of the simple pairwise uniform 
generator from the previous lecture, and (ii) hashing.
Before embarking on those, we review a probability concentration 
result known as Chebyshev's inequality, which has a close connection 
to pairwise independence.

\section{Chebyshev's Inequality}

A good randomized algorithm should behave well not just in 
expectation, but with high probability. For example, if the outcome of 
the algorithm is a random variable whose expected value exceeds a 
desired threshold but the probability with which it exceeds that 
threshold is only 1\%, then the algorithm is no good.

Concentration results in probability theory provide confidence levels
with which a random variable $X$ attains values around its expectation
$\Exp[X]$. There are many such results for various types of random
variables. Chebyshev's inequality is a concentration result that
applies to every random variable $X$, and yields confidence levels as a
function of its variance $\Var[X]$. Recall that 
\[ \Var[X] \doteq \Exp\left[ (X - \Exp[X])^2 \right] = \Exp[X^2] -
\Exp[X]^2, \]
where the first equality is by definition, and the last equality
follows from linearity of expectation. Note that since $\Var[X] \geq
0$ by definition, the last equality implies that $\Exp[X^2] \geq
\Exp[X]^2$, which is the Cauchy-Schwarz inequality in disguise.
For an indicator variable $X$, we have $\Var[X] = p \cdot
(1-p)$, where $p$ denotes the probability of the underlying event.

The following proposition bounds the probability that the deviation
of a random variable from its expectation exceeds a certain threshold
$a$ as a function of the variance of the random variable.
\begin{proposition}[Chebyshev's inequality]\label{chebyshev}
For every random variable $X$ and real $a>0$,
\[ \Pr\Big[ |X-\Exp(X)| \geq a \Big] \leq \frac{\Var[X]}{a^2}\,. \]
\end{proposition}
The proof of Chebyshev's inequality is an application of Markov's
inequality, which is a concentration result that applies to every
nonnegative random variable $Y$, and yields confidence levels as a
function of its expected value.
\begin{proposition}[Markov's inequality]\label{markov}
For every nonnegative random variable $Y$ and real ${b>0}$,
\[ \Pr\big[ Y \geq b \big] \leq \frac{\Exp[Y]}{b}. \]
\end{proposition}
An instantiation of Markov's inequality states that the 
probability that a nonnegative random variable exceeds twice its 
expectation is at most 50\%.
\begin{proof}[of Markov's inequality]
Recall that $\Exp[Y] = \sum_{y} y\cdot \Pr[Y=y]$, where $y$ ranges over
all possible values of $Y$. We can split the sum into two sums with
the given value $b$ as the pivot:
\[ \sum_{y} y \cdot \Pr[Y=y] = \sum_{y < b} y \cdot \Pr[Y=y] + \sum_{y
  \geq b} y \cdot \Pr[Y=y]. \]
Since $Y$ is nonnegative, the first sum (for $y<b$) is always
nonnegative, whereas we can lower bound the second sum as follows:
\[ \sum_{y \geq b} y \cdot \Pr[Y=y] \leq \sum_{y \geq b} b \cdot \Pr[Y=y]
= b \cdot \sum_{y \geq b} \Pr[Y=y] = b \cdot \Pr[Y \geq b]. \]
Combining the lower bounds for both sums yields
\[ \Exp[Y] \geq b \cdot \Pr[Y \geq b]\,, \]
which is equivalent to the stated inequality.
\end{proof}
The proof of Chebyshev's inequality is now a direct invocation of 
Markov's inequality.
\begin{proof}[of Chebyshev's inequality]
Apply Markov's inequality to the random variable $Y
\doteq (X - \Exp[X])^2$, which is non-negative and has expectation
$\Exp[Y] = \Var[X]$. For $b=a^2$ this shows that
\[ \Pr\Big[ (X- \Exp[X])^2 \geq  a^2 \Big] \leq \frac{\Var[X]}{a^2}, 
\]
which is equivalent to the stated inequality.
\end{proof}

\paragraph{Connection with pairwise independence.}
Many concentration results apply to random variables $X$ that are
built up as the sum $X = \sum_{i=1}^s X_i$ of some components $X_i$
that possess some degree of independence.
The central limit theorem can be seen as a concentration result for a 
sequence of random variables $X_i$ that are fully independent.
Chebyshev's inequality applies to the random variable $X = 
\sum_{i=1}^s X_i$ no matter what the dependencies between the $X_i$'s 
are, but yields particularly interesting bounds when the $X_i$'s are 
pairwise independent.
This is because the variance is additive when the $X_i$'s are pairwise 
independent, as shown in the following proposition.  
\begin{proposition}\label{prop:variance:pairwise}
If the random variables $X_1,\dots,X_s$ are pairwise
independent, then
\begin{equation*}
  \Var\left[\sum_{i=1}^s X_i\right] = \sum_{i=1}^s \Var[X_i]\,.
\end{equation*}
\end{proposition}
\begin{proof}
\begin{align*}
\Var\left[\sum_{i=1}^s X_i\right]
& = \Exp\left[
  \left(\sum_{i=1}^s X_i -
  \Exp\left[\sum_{i=1}^s X_i\right]\right)^2 \right]
   \qquad\textrm{ (by the definition of variance)} \\
& = \Exp\left[ \left(\sum_{i=1}^s \left(X_i - 
      \Exp[X_i]\right)\right)^2 \right] \qquad\textrm{ (by
  linearity of the inner expectation)} \\
& = \Exp\left[ \sum_{i,j=1}^s (X_i - \Exp[X_i])(X_j - \Exp[X_j])
\right] \qquad\textrm{ (by spelling out the square)} \\
& = \sum_{i,j=1}^s \Exp\Big[ (X_i - \Exp[X_i])(X_j - \Exp[X_j])
\Big] \qquad\textrm{ (by linearity of the outer expectation)}\\
& = \sum_{i=1}^s \Exp\Big[ (X_i - \Exp[X_i])^2 \Big]
 + \sum_{\substack{i,j\in[s]\\i\neq j}}  \Exp\Big[ (X_i - 
 \Exp[X_i])(X_j - \Exp[X_j]) \Big]
 \textrm{ (by splitting the sum)} \\
& = \sum_{i=1}^s \Exp\Big[ (X_i - \Exp[X_i])^2 \Big]
 + \sum_{\substack{i,j\in[s]\\i\neq j}}  \Exp\big[X_i - \Exp[X_i]\big] 
 \cdot \Exp\big[X_j -
 \Exp[X_j]\big] \qquad\textrm{ (see below)} \\
& = \sum_{i=1}^s \Exp\Big[ (X_i - \Exp[X_i])^2 \Big] \qquad\textrm{ 
  (see below)} \\
& = \sum_{i=1}^s \Var[X_i]
\qquad\textrm{ (by definition of variance)}\,,
\end{align*}
where the third-to-last equality uses the fact that, for every $i \neq 
j$, the variables $Y_i \doteq X_i - \Exp[X_i]$ and $Y_j \doteq
X_j - \Exp[X_j]$ are independent and therefore satisfy $\Exp[Y_iY_j]
= \Exp[Y_i]\Exp[Y_j]$, and the second-to-last equality follows because
$\Exp[Y_i] = \Exp[X_i] - \Exp[X_i] = 0$.
\end{proof}
Note that the expected value is always additive: $\Exp[\sum_{i=1}^s
X_i] = \sum_{i=1}^s \Exp[X_i]$ holds irrespective of the dependencies
between the $X_i$'s. In contrast, the variance is not always
additive. For example, if all~$s$ variables $X_i$ are identical, then 
$\Var[X] = \Var[s X_1] = s^2 \Var[X_1]$, whereas additivity would
require $\Var[X] = s \Var[X_1]$. But if the variables are pairwise
independent, then the variance is additive. 
% More generally, if the variables $X_i$ are $k$-wise independent,
% then the $k$th central moment of their sum, $\Exp[(X-\Exp[X])^k]$,
% equals the sum of the individual $k$th central moments. 

\paragraph{Confidence boosting.}
We can reduce the probability of error of a randomized algorithm by 
running it $s$ times and outputting the plurality answer, that is, the 
answer that occurred most frequently.  Proposition
\ref{lect03:prop:confidence-boosting} of Lecture 3 analyzed the 
technique when the $s$ runs are fully 
independent. In that case, the error probability decreases
exponentially with $s$. When the runs are only pairwise independent,
we can use Chebyshev's inequality to show that the error decreases
inversely proportional with $s$. 
\begin{proposition}[Confidence boosting]\label{prop:confidence}
Suppose that a randomized process outputs $y$ with probability 
$\frac{1}{2}+\eta$.
Then the plurality vote of $s$ pairwise independent runs equals $y$ 
with probability at least $1 - \delta$ for $s \geq \frac{1}{\delta 
  \eta^2}$.
\end{proposition}
\begin{proof}
Let $X_i$ be the indicator that the $i$th run produces an answer other
than $y$. Note that $\Exp[X_i] = \frac{1}{2}-\eta$ and $\Var[X_i] =
\frac{1}{4}-\eta^2$. The only way the plurality vote can be different
from $y$ is when $X \doteq \sum_{i=1}^s X_i \geq \frac{s}{2}$, which
implies that $|X - \Exp[X]| \geq \frac{s}{2} - s \cdot (\frac{1}{2}-\eta) =
s \cdot \eta$. Since the runs are pairwise independent, 
Proposition~\ref{prop:variance:pairwise} tells us that
$\Var[X] = \sum_{i=1}^s \Var[X_i] = s \cdot (\frac{1}{4}-\eta^2)$.
Thus, by Chebyshev's inequality, we can upper bound the probability 
that the plurality vote differs from $y$ by
\begin{align*}
\Pr\Big[ | X - \Exp[X] | \geq s \eta \Big]
\leq \frac{s \cdot  (\frac{1}{4}-\eta^2)}{(s \eta)^2}
\leq \frac{1}{s\eta^2}\,,
\end{align*}
which is at most $\delta$ if we set $s\geq 1/(\delta \eta^2)$.
\end{proof}
Let us compare the two ways we have seen for reducing the error from
$\frac{1}{2}-\eta$ down to $\delta$, namely taking the plurality
output of a number of (i) fully independent runs and (ii) pairwise
independent runs. We compare the number $s$ of runs needed, as well as
the number of truly random bits assuming a single run of the original
algorithm uses $r$ random bits.
This number equals $r\cdot s$ in case (i).
In case (ii), we can use the pairwise independent generator given by
Theorem \ref{lect04:thm:k-wise} from Lecture 4, which uses $2\cdot 
\lceil\log_{2^r}(s)\rceil \cdot r = O(r+\log(s))$ truly random bits.\\

\begin{tabular}{|l|c|c|}
\hline
method & number of runs & number of random bits
\\ \hline
(i) fully independent runs & $O(\log(\frac{1}{\delta}) \cdot 
\frac{1}{\eta^2})$ & $O(r \cdot \log(\frac{1}{\delta}) \cdot 
\frac{1}{\eta^2})$
\\ \hline
(ii) pairwise independent runs & $O(\frac{1}{\delta \eta^2})$ &
  $O(r + \log(\frac{1}{\delta}) + \log(\frac{1}{\eta}))$
  \\ \hline
\end{tabular}
\\

For fixed $\eta$, the required number of runs is exponentially
smaller in $\frac{1}{\delta}$ for (i) than for~(ii), but~(ii) needs fewer
random bits. Using expanders we will later see how to get the best of
both worlds.

For fixed $\delta$, the required number of runs for (i)
and (ii) is the same up to constant factors, but~(ii) only needs an
additive number of extra random bits, whereas (i) needs a
multiplicative extra. Using expanders we will later see how to do the
same with no additional random bits and the same order of runs. 

\section{List Decoding the Hadamard Code}

We are now ready to discuss our first application of pairwise
independent generators.
This is not a ``standard'' application, in which we would use a good 
pairwise independent pseudorandom generator for derandomization 
purposes.
In fact, we'll be looking at a context in which full derandomization is impossible.
Moreover, the argument will hinge on a special property of one of the 
pairwise independent PRGs we constructed, namely $\GF{2}$-linearity.

The context we are looking at is that of {\em list decoding} the
Hadamard code. Recall that, as long as the number of errors is less
than half the distance of a code, we can uniquely recover the
correct encoding and thus the message. If the number of errors is
somewhat larger, we can no longer guarantee that there is a unique
codeword from which the received word can be obtained by introducing
at most that many errors, but we can still hope that there are few
such codewords. If so, we can try to list all of them, or all of the
corresponding messages. The latter is referred to as list decoding.
\begin{definition}[List decoding]
Let $\Enc: \Sigma^k \to \Sigma^n$ be a code. List decoding $\Enc$ up
to relative error $\epsilon$ is the following problem: Given a
received word $z \in \Sigma^n$, find all the messages $x \in \Sigma^k$
such that $\hdist\big(\Enc(x),z\big) \leq \epsilon n$.
\end{definition}

The Hadamard code $\Enc$ works over the binary alphabet $\Sigma = \zo$
and encodes $x \in \zo^k$ as a string of length $n = 2^k$, namely the inner
product $\langle x,a \rangle$ of $x$ with all strings $a \in
\zo^k$. All codewords are at relative distance $\frac{1}{2}$ from each
other, which means that we cannot uniquely decode if the fraction
$\epsilon$ of errors is $\frac{1}{4}$ or more, and we cannot
efficiently list decode if the fraction of errors is $\frac{1}{2}$ or
more (since on input a valid codeword, the list would need to contain
all messages of length~$k$). In Lecture 3 we saw how to efficiently
uniquely decode the Hadamard code for $\epsilon<\frac{1}{4}$. We now
show how to efficiently list decode the Hadamard code for
$\epsilon<\frac{1}{2}$. In order to do so, we model the received word
$z \in \zo^{2^k}$ as a function $z: \zo^k \to \zo$, where the value of
the function at $a \in \zo^k$ equals the bit $z(a)=z_a$ at position 
$a$ of the string $z$.
\begin{theorem}\label{thm:list-decoding}
There exists a randomized algorithm that, on input
$\eta>0$ and with oracle access to a function $z: \zo^k \to \zo$, 
produces a list that, with 99\% certainty, contains all strings $x \in
\zo^k$ that satisfy
\begin{equation}\label{eq:agreement}
\Pr_{a \in_u \zo^k}\Big[ \langle x,a \rangle = z(a)\Big] \geq 
\frac{1}{2} +
\eta.
\end{equation}
The algorithm runs in time polynomial in $\frac{k}{\eta}$.
\end{theorem}
A few remarks are in order. First, note that the algorithm is
randomized and has oracle access to the received word $z$. Just like
for the unique decoding procedure from Lecture 3, both are necessary
in order to have any hope of achieving a running time that is
polynomial in $k$. 

Second, the algorithm only guarantees (with high
probability) that the list {\em contains} all the solutions to the
list decoding problem, but not that it {\em only consists of}
solutions. Again, this is the best that we can hope for with a running
time that is polynomial in $k$, for a similar reason: There may be
messages $x$ whose encoding agrees with the received word $z$ in only
slightly less than a fraction $\frac{1}{2}+\eta$ of the
positions. Such an $x$ should be included in the list for a slightly
modified received word $z \in \zo^{2^k}$, and there is no way for an
algorithm that runs in time polynomial in~$k$ -- even a randomized one
-- to detect this difference with reasonable confidence. What we {\em
  can} efficiently guarantee is that no $x$ on the list has an
encoding that agrees with $z$ in significantly less than
$\frac{1}{2}+\eta$ of the positions. 

Finally, the existence of the algorithm implies that the number of
solutions to the list decoding problem is polynomial in
$O(\frac{k}{\eta^2})$, which is a nontrivial result by itself. In
fact, the following lemma, which captures the crux of the proof of
Theorem \ref{thm:list-decoding}, implies an upper bound of
$O(\frac{k}{\eta^2})$ on the number of solutions. In a later lecture
we will see how to improve this upper bound to $\frac{1}{4\eta^2}$,
which is tight.
\begin{lemma}\label{lemma:list-decoding}
There exists a randomized algorithm that, on input
$\eta>0$ and oracle access to a function $z: \zo^k \to \zo$, produces 
a list of size $O(\frac{k}{\eta^2})$ such that, for every string $x 
\in \zo^k$ satisfying \eqref{eq:agreement}, the probability
that $x$ is included in the list is at least 50\%. 
The algorithm runs in time polynomial in $\frac{k}{\eta}$.
\end{lemma}
\begin{corollary}\label{cor:list-decoding}
For every $\eta>0$ and function $z: \zo^k \to \zo$, the number of $x 
\in
\zo^k$ satisfying \ref{eq:agreement} is $O(\frac{k}{\eta^2})$. 
\end{corollary}
\begin{proof}[of Corollary \ref{cor:list-decoding}]
Let $S$ denote the set of solutions and $L$ the list that the
algorithm of Lemma~\ref{lemma:list-decoding} outputs. The statement of
the lemma implies that \[ |S| \cdot \frac{1}{2} \leq \sum_{x \in S} 
  \Pr[x \in L] \leq \sum_x
\Pr[x \in L] = \sum_x \Exp[ I[x \in L] ] = \Exp[|L|] =
O(\frac{k}{\eta^2})\,, \]
which implies that $|S| = O(\frac{k}{\eta^2})$.
\end{proof}
The proof of Theorem \ref{thm:list-decoding} is immediate from Lemma
\ref{lemma:list-decoding} and Corollary \ref{cor:list-decoding}.
\begin{proof}[of Theorem \ref{thm:list-decoding}]
Let $S$ denote the set of solutions.
We run the algorithm of Lemma \ref{lemma:list-decoding} a number~$t$ 
times and output the concatenation of the $t$ lists.
This yields a list $L$ such that, for every fixed $x \in S$, $\Pr[ x 
\not\in L] \leq \frac{1}{2^t}$.
Hence, by the union bound,
$\Pr[ (\exists x \in S) \, x \not\in L ] \leq |S|/2^t$, which by the
upper bound on $|S|$ given by Corollary \ref{cor:list-decoding} is at
most 1\% for $t \geq \log(\frac{k}{\eta^2}) + O(1)$. 
\end{proof}
We now get to the crux of the argument, which is 
Lemma~\ref{lemma:list-decoding}.
Let us fix a solution $x$, i.e., a string
$x \in \zo^k$ satisfying \eqref{eq:agreement}. For starters, let us
recall the basis for the local decoding procedure from Lecture 3: In
order to find the $i$th bit of $x$, pick $a \in \zo^k$ uniformly at
random and output $z(a) + z(a+e_i)$. We argued that
\begin{equation}\label{eq:correct}
    \Pr_{a \in_u \zo^k}[ z(a) + z(a+e_i) \neq x_i] \leq 1-2\eta.
\end{equation}
The reason was that the predicate underlying \eqref{eq:correct} holds
if  $z$ agrees with the encoding of $x$ at the two positions~$a$ and 
$a+e_i$, and by the union bound this fails with probability at most
$(\frac{1}{2}-\eta) + (\frac{1}{2}-\eta)$. For $\eta>\frac{1}{4}$,
\eqref{eq:correct} shows that the probability of error remains bounded
below $\frac{1}{2}$, after which we can apply the usual confidence
boosting techniques to obtain the correct result with high
probability. When $\eta \leq \frac{1}{4}$, however, the upper bound in
\eqref{eq:correct} becomes meaningless. 

In order to get around this issue, consider the following 
``procedure'' for
finding the $i$th bit of $x$: Pick $a \in \zo^k$ uniformly at
random,  and output $\langle x,a \rangle + z(a+e_i)$. We write
``procedure'' between quotes because it isn't clear how we can find
the value $\langle x,a \rangle$ of the correct encoding of $x$ at $a$,
but let us assume for now that we can. Since 
\[    \Pr_{a \in_u \zo^k}[ \langle x,a \rangle + z(a+e_i) \neq x_i]
\leq \frac{1}{2}-\eta, \]
we are in a position to apply confidence boosting for any $\eta >
0$. In particular, if we take the majority vote for $t$ choices of 
$a$ produced by a pairwise uniform generator $G: \Sigma^\ell \to
\Sigma^t$ for $\Sigma = \zo^k$, Proposition \ref{prop:confidence}
guarantees us that  \[    \Pr_{\sigma \in_u \Sigma^\ell}\Big[ \maj_a 
  (\langle x,a \rangle +
z(a+e_i)) \neq x_i\Big] \leq \frac{1}{t\eta^2}, \]
where the majority is taken over all the $t$ components $a$ of
the random variable $G(\sigma)$.
By the union bound, we get
\[    \Pr_{\sigma \in_u \Sigma^\ell}\Big[ \big(\maj_a (\langle x,a 
  \rangle +
z(a+e_i))\big)_{i=1}^k \neq x\Big] \leq \frac{k}{t\eta^2}. \]
Thus, for any $t \geq \frac{2k}{\eta^2}$, the probability that our ``procedure''
outputs $x$ is at least 50\%. 

How do we obtain the values $\langle x,a \rangle$ we need? Consider
the simple pairwise independent generator that generalizes Lemma
\ref{lect04:lemma:pairwise} from Lecture 4:
$G(\sigma) = (\sum_{j=1}^\ell c_j \sigma_j)_c$, where the arithmetic
is over $\GF{2}^k$ and $c$ ranges over $\zo^\ell \setminus
\{0^\ell\}$, so $t=2^\ell-1$. Note that
for $a_c \doteq \sum_{j=1}^\ell c_j \sigma_j$,
\begin{equation}\label{eq:sum}
 \langle x,a_c \rangle = \sum_{j=1}^\ell c_j \langle x,\sigma_j
\rangle, 
\end{equation}
which means that in order to know all $2^\ell-1$ values $\langle x,a_c
\rangle$ it suffices to know the $\ell$ values $\langle x,\sigma_j
\rangle$ for $j \in [\ell]$.
We do not really know those values, but since there are so few of them, we can just try all possibilities.
That is, for a given seed $\sigma$, we range over all $2^\ell$ 
possible values for $(\langle x,\sigma_j\rangle)_{j=1}^\ell$, use 
those values to compute $\langle x,a_c \rangle$ via \eqref{eq:sum} for 
all $c \in \zo^\ell \setminus \{0^\ell\}$, and include $(\maj_c 
(\langle x,a_c \rangle + z(a_c+e_i)))_{i=1}^k$ in our list of 
candidate solutions.
By the above analysis, the list contains $x$ with probability at least 
50\% if we set $t \geq \frac{2k}{\eta^2}$.
The size of the list equals $2^\ell=t+1$. If we choose the smallest  
$t \geq \frac{2k}{\eta^2}$ of the form $t=2^\ell-1$, the list size is 
$O(\frac{k}{\eta^2})$.
The running time of the resulting procedure is polynomial in $k$ and 
$\frac{1}{\eta}$.
This finishes the proof of Lemma \ref{lemma:list-decoding}.

Note that the resulting algorithm uses a lot of randomness, namely $k
\cdot \ell = O(k \log(\frac{k}{\eta}))$ truly random bits. Indeed, the
reason we use the (particular) pairwise uniform PRG is not to save on
random bits, but because every pseudorandom sequence can be written as
a $\GF{2}$-linear combination of a number of sequences that is
logarithmic in the length of the sequence. This allows us to produce a
polynomial number of pairwise independent positions $a$ which we can
use for boosting \`a la Chebyshev and for which we can generate in
polynomial time all possible restrictions of a valid codeword to the
subset of those positions $a$.

\section{Hashing}

Hashing is a technique to deal with the dictionary problem, in which
we want to store $n$ (key,value) pairs where the keys come from a
universe $[u]$ that is much larger than $n$. A typical setting is a
compiler keeping track of the variables that are in use; in this case
the keys are the variable names. In the static version of the problem,
the set $S$ of stored keys is fixed and we only allow look-ups, i.e.,
retrieving the value corresponding to a given key, or reporting that
there is no value stored for the given key. In the dynamic version, 
the set $S$ can change over time via additions and deletions of 
(key,value) pairs. 

One possible solution is to create a table with one entry for each
possible key value in $[u]$, and use the key value as the index into
that table. This allows us to perform look-ups in constant time, but
requires an exorbitant amount of memory space and set-up time 
because $u \gg n$. Another possibility is to store $S$ in a balanced
search tree, in which case we only need about the memory space to
write down the $n$ (key,value) pairs but the operations now take
$\Theta(\log n)$ time. Hashing is an attempt to combine the best of
both worlds. It uses a table of size $m \approx n$, and stores a
(key,value) pair $(x,v)$ in slot $h(x)$ of the table. The name of the
game is to find a mapping $h: [u] \to [m]$, called the hash
function, that is simple to compute and avoids collisions, i.e., no
two distinct keys $x, y \in S$ map to the same slot $h(x)=h(y)$. This
results in constant look-up time and only requires memory comparable
to that of a balanced search tree.

Having no collisions at all may be a lot to ask, but finding a hash
function that results in few collisions is easy, at least in a
randomized setting -- a random choice will do. More precisely, if we
pick $h$ uniformly from the class $\mathcal{H}$ of all functions from
$[u]$ to $[m]$, then the expected number of collisions with any given
key $x \in S$ is no more than the load factor $\alpha \doteq n/m$ of
the table. This is because  
\begin{equation}\label{eq:hashing}
E_{h \in_u \mathcal{H}}[ |\{ y \in S \text{ s.t. } h(x)=h(y) \} |] 
= \sum_{x \not= y \in S} \Pr_{h \in_u \mathcal{H}}[h(x)=h(y)] =
\frac{n-1}{m} \leq \alpha.
\end{equation}
However, a random hash function $h$ is not simple -- it takes 
$2^u \cdot \log m$ bits to specify. Instead, we'd like to use
functions from a much smaller family $\mathcal{H}$ of hash functions
that can be described and evaluated efficiently.

Note that the only property of $\mathcal{H}$ we need for
\eqref{eq:hashing} to hold is that
\begin{equation}\label{eq:hash-cond}
(\forall x, y \in [u] \text{ with } x \not=y) \,
\Pr_{h \in_u \mathcal{H}}[h(x)=h(y)] = \frac{1}{m}.
\end{equation} 
A family $\mathcal{H}$ with property \eqref{eq:hash-cond} is called
{\em universal}, and it suffices that the marginal distribution of the
values of $h$ on any two distinct inputs $x, y \in [u]$ is uniform. A
family with the latter property is called 2-uniform. More generally,
we have the following definition for any positive integer $k$.
\begin{definition}\label{definition: hash function}
A family $\mathcal{H}$ of functions $h: [u] \to [m]$ is
$k$-uniform if for every $k$ distinct elements $x_1, x_2, \ldots, x_k
\in [u]$, the distribution of $(h(x_i))_{i=1}^k$ is uniform over
$[m]^k$ when $h$ is chosen uniformly from $\mathcal{H}$. 
\end{definition}
There is a close connection between $k$-uniform families of hash
functions $\mathcal{H}$ and $k$-wise uniform generators $G:
\{0,1\}^\ell \to [m]$, where the seed of $G$ is used to describe
a hash function, and the value of the hash function at $x \in [u]$ is
the $x$th component of $G(\sigma)$, i.e., $h_\sigma(x) =
(G(\sigma))_x$. Using the $k$-wise uniform generators given by Theorem
\ref{lect04:thm:k-wise} from Lecture 4,
we obtain seed length $\ell = O(k \cdot \log u)$, so
the corresponding $k$-uniform family $\mathcal{H}$ of hash functions
has size $|\mathcal{H}| = u^{O(k)}$. For example, when $m$ is a power
of 2 and $u$ is a power of $m$, the following family is 2-uniform:
$\mathcal{H} = \{ h_{a,b} \, | \, a.b \in \mathbb{F}_u \}$, where
$h_{a,b}(x)$ is the last $\log m$ bits of $ax+b$ when we interpret $x
\in [u]$ as an element of $\mathbb{F}_u$, and the arithmetic is carried
out over that field. Each function from this family is described by
$2\log(u)$ bits. For $m \ll u$ we can save almost a factor of 2 by
using Exercise \ref{lect04:ex:1} of the previous lecture. It yields a
2-uniform family where $m$ and $u$ are powers of 2, and each function
from the family is described by $\log(u)+2\log(m)$ bits.

In lecture 1 we used a family of hash functions $h_p(x) = x \bmod p$,
where $p$ is a prime consisting of $c\log\log(u)$ bits, and therefore
$m=\log(u)^c$. The resulting family is not universal stricto sensu
because the probability that two distinct elements collide is more
than $\frac{1}{m}$. However, the collision probability is only
slightly more than $\frac{1}{m}$, and therefore the family pretty much
shares the properties of universal families.

When we use a universal family, the expected number of collisions for
any $x \in S$ is less than $\alpha \doteq \frac{n}{m}$. Even when we
keep $\alpha$ under control, some collisions will occur
and we need a way to deal with them. Various strategies have been
considered, including the following.
\begin{itemize}
\item Chaining. \\
The keys in $S$ that hash to the same slot in $[m]$ are concatenated
in a linked list, to which the entry in the hash table points. For the
class $\mathcal{H}$ of all hash functions, \eqref{eq:hashing} implies
that the expected look-up time is bounded by $1+\alpha$, and our
analysis shows that the same holds for any universal family
$\mathcal{H}$.
\item Linear probing. \\
This is a particular method of open addressing, which means that all
keys are stored in the hash table itself. Specifically, when we want
to place a key $x$ into a slot of the hash table that is already in
use, we search the successive slots in the hash table until we find an
empty one, and store $x$ there. It was known for a long time that for
the class $\mathcal{H}$ of all hash functions, the expected look-up
time is $O(\frac{1}{(1-\alpha)^2})$. A few years ago it was shown that
the same holds for every 5-uniform family $\mathcal{H}$ (but not for
every 4-uniform one) \cite{PaghPaghRuzic09}. 
\end{itemize}
Note that we only made claims about the expected look-up times,
whereas we really want good behavior with very high
probability. In addition, there are many practical considerations that
play a role in the choice of the hash functions. For example, linear
probing has good cache behavior because of the fact that it accesses
successive cells, which may counterbalance its worse expected look-up
time compared to chaining.

\bibliographystyle{alpha}
\bibliography{references}

\end{document}

