\documentclass[11pt]{article}

\usepackage{lecture}

\begin{document}

\lecture{8}{2/18/2013}{Applications of Small-Bias Generators}{Gautam Prakriya}

%\draft % put this here only if you want to indicate this is a draft

In the previous lecture, we developed the basics of harmonic analysis
and used them to quantitatively show that every small-bias generator is
almost $k$-wise uniform.
This led to an $\epsilon$-PRG for $k$-wise uniformity with seed 
length $O(k + \log(1/\epsilon) + \log r)$, where $r$ denotes the 
output length.
In this lecture we give a construction that improves the dependence on 
$r$ to $O(\log \log r)$, and for this we again use small-bias 
generators.
In general, it is a good idea
to consider small-bias generators whenever harmonic analysis is
involved in the correctness of the construction. This is because of
the close connection between the Fourier coefficients of a 
distribution and its bias that we mentioned in the last lecture.

We also present two other applications of small-bias generators,
namely in the context of approximation algorithms: a hardness result
and a (derandomized) algorithm for the maximum satisfiability problem
of a set of quadratic equations over $\GF{2}$. 

\section{Almost k-Wise Uniform Generators}

In this section, we develop an alternate way to construct an almost $k$-wise uniform generator.
Theorem \ref{lect04:thm:k-wise} from Lecture 4 gives a
construction of a perfect $k$-wise uniform generator $G: \zo^\ell \to
\zo^r$ with seed length $\ell = O(k \cdot \log r)$. We can try to
further reduce the need for truly random bits by derandomizing $G$,
i.e., by using another PRG $G'$ to generate the seed of length $\ell$
for $G$.

What second PRG $G'$ should we use? The answer depends on 
(i) what properties of the final pseudorandom distribution $D$ matter,
and (ii) how $G$ uses its seed. As for (i), by the analysis of last
lecture, we know that it is sufficient for $D$ to have small bias in
order to get a distribution that is almost $k$-wise uniform. As for
(ii), the PRG $G$ in Theorem \ref{lect04:thm:k-wise} of Lecture 4
(which is based on the one in Lemma \ref{lect04:lemma:k-wise} of that
lecture) is a linear mapping, and in that case the following lemma
shows a simple connection between the biases of $D = G(D')$ and the
biases of $D'$. Facts (i) and (ii) combined then suggest the use of a
small-bias generator $G'$ to generate $D'$. 
\begin{proposition}\label{prop:bias}
Let $D'$ be a distribution on $\zo^\ell$ and let $D \doteq
M \cdot D'$, where $M \in \zo^{r \times \ell}$. For every $a \in 
\zo^r$, we have
\[ \bias_a(D) = \bias_{M^T \cdot a}(D')\,. \]
\end{proposition}
\begin{proof}
We have
\[ \bias_a(D)
= \Exp\left[ (-1)^{\langle a,D \rangle} \right]
= \Exp\left[ (-1)^{\langle a,M \cdot D' \rangle} \right]
= \Exp\left[ (-1)^{\langle M^T \cdot a, D' \rangle} \right]
= \bias_{M^T \cdot a}(D')\,,
\]
where the third equality follows because
\[ \langle a,M D' \rangle = a^T (M D') = (a^T M) D' =
(M^T a)^T D' = \langle M^Ta,D' \rangle. \]
\end{proof}
Proposition \ref{prop:bias} shows that, if $D'$ has small bias, then 
all
the biases of $D = M \cdot D'$ are equally small, except
with respect to those $a$ for which $M^T a = 0$ (for which it is 1).  
In case the linear mapping~$M$ defines a $k$-wise
uniform generator, we argued in Lecture 4 (cf.\ the proof of
Proposition~\ref{lect04:prop:connection}) that every $k$ rows of $M$
are linearly independent, or equivalently, every $k$ columns of
$M^T$ are linearly independent. Thus, the only vector $a \in
\zo^r$ with at most $k$ nonzero entries that satisfies $M^T a=0$ is
the zero vector. In combination with Proposition
\ref{lect07:prop:almost:k-wise} from Lecture 7, this means that
$D$ is close to being $k$-wise uniform in the following quantitative
sense.
\begin{proposition}\label{prop:almost:k-wise}
Let $D'$ be a $\beta$-bias distribution on $\zo^\ell$, and let
$G: \zo^\ell \to \zo^r$ be a linear $k$-wise uniform
generator. Then $D \doteq G(D')$ has the following properties
for every $I \subseteq [r]$ with $|I|=k$:
\begin{itemize}
\item[(i)] $(\forall y \in \zo^k) \quad \Big| \Pr[D|_I = y] - \Pr[U_k 
  = y] \Big|
  \leq \left(1-\frac{1}{2^k}\right) \cdot \beta\,$, and
\item[(ii)] $\sdist(D|_I,U_k) \leq \sqrt{2^k-1} \cdot
  \frac{\beta}{2}$.
\end{itemize}
\end{proposition}
Recall that Theorem \ref{lect06:thm: construction} from Lecture 6
yields a generator $G'$ with seed length $O(\log(\ell/\beta))$ that
produces a $\beta$-bias distribution $D'$ on $\zo^\ell$. The
construction of Theorem \ref{lect04:thm:k-wise} from
Lecture 4 gives us a linear $k$-wise uniform generator on $\zo^r$ with
seed length $\ell = O(k \cdot \log r)$. Combining those two
constructions as in Proposition \ref{prop:almost:k-wise} with $\beta =
\epsilon/\sqrt{2^k-1}$ yields the following result.
\begin{theorem}\label{thm:almost:k-wise}
For all integers $r$ and $k \in [r]$, and every real $\epsilon >0$,
there exists an efficiently computable $\epsilon$-pseudorandom 
generator $G: \zo^\ell \to \zo^r$ for $k$-wise uniformity with 
$\ell = O(k + \log(1/\epsilon) + \log\log r)$. 
\end{theorem}
In many settings where $k$-wise uniformity is sufficient, almost
$k$-wise uniformity is also sufficient. This allows us to reduce the
seed length from $O(k \cdot \log r)$ down to $O(k + \log(1/\epsilon) +
\log\log r)$, where~$\epsilon$ denotes the allowed statistical
distance from $k$-wise uniform. Note that this improves over the seed
length $O(k + \log(1/\epsilon) + \log r)$ from last lecture. For an
example of an application where the reduction from $\log(r)$ down to
$\log\log(r)$ matters, we refer to \cite[Section 8]{MayerNaorStockmeyer}. 

Finally, let us point out that, in some settings, having {\em perfect}
rather than {\em almost} $k$-wise uniformity is critical. An example
is the additivity of the variance given by Proposition
\ref{lect05:prop:variance:pairwise} of Lecture 5 and the applications
that depend 
on it. The variance of the sum of $t$ samples of a random variable
grows only linearly with $t$ when the samples are pairwise
independent, but quadratically when the samples are merely almost
pairwise independent. This is because the $O(t^2)$ cross terms in the
proof of Proposition \ref{lect05:prop:variance:pairwise} no longer
vanish. 

\section{Hardness of Approximation}

In this section we study the problem of finding approximate solutions
to a system of quadratic equations over $\GF{2}$ in the following
sense: Given a list $S$ of $m$ polynomials $Q_i$ of degree at most two 
in $n$ variables over $\GF{2}$, our goal is to find an assignment $x
\in \zo^n$ that satisfies many of the equations $Q_i(x) = 0$. The best
we could hope for is to find a solution that satisfies a fraction
\[ \maxqe(S) \doteq \frac{1}{m} \max_{x \in \zo^n}
  \sum_{i=1}^m I[ Q_i(x) = 0] \]
of the equations.
Finding a solution that realizes $\maxqe(S)$ is \NP-hard.
In fact, it is even \NP-hard to decide whether $\maxqe(S)=1$ holds.
\begin{proposition}\label{prop:NP-hard:exact}
  Given a system $S$ of  quadratic equations over $\GF{2}$, deciding
  whether there exists an assignment to the variables that satisfies
  all the equations is \NP-hard.
\end{proposition}
\begin{proof}
To show that the problem is \NP-hard, we reduce 3-SAT to it based on
the following observation. Every clause in a $3$-CNF formula $\varphi$
can represented by a polynomial of degree 3 over $\GF{2}$, which in
turn can be represented by a pair of polynomials of degree 2 with the
introduction of a new variable. For instance, consider the clause $(x_1
\vee \overline{x_2} \vee x_3)$. An assignment satisfies this clause
if and only if it satisfies $(1-x_1)\cdot x_2 \cdot (1-x_3) = 0$. This 
equation
is equivalent to the system of two quadratic equations
\[ \left\{ \begin{array}{l} 
       (x_2 - y_{1,2})(1-x_3) = 0 \\
       y_{1,2} - x_1 x_2 = 0,
\end{array} \right. \]
where $y_{1,2}$ is an auxiliary variable. By collecting these
equations for all clauses of $\varphi$, we obtain a system $S$ of
quadratic equations that is satisfiable if and only if $\varphi$ is
satisfiable. The transformation from $\varphi$ into $S$ only takes 
polynomial time.
\end{proof}

Given Proposition \ref{prop:NP-hard:exact}, we relax our goal to
finding an assignment that satisfies at least a fraction $\alpha \cdot
\maxqe(S)$ of the equations for some constant $\alpha>0$. We
will use small-bias generators to investigate (1) for which $\alpha$
the relaxed goal remains \NP-hard, and (2) for which $\alpha$ we can
realize the goal in polynomial time. Results of type (1) are called
inapproximability results, and results of type (2)
$\alpha$-approximation algorithms.

\subsection{Inapproximability} 

We show that finding an assignment that satisfies a fraction $\alpha
\cdot \maxqe(S)$ of the equations is \NP-hard for every
constant $\alpha > \frac{1}{2}$. In fact, we prove a somewhat
stronger statement.
\begin{theorem}\label{thm:inapprox}
For every $\alpha \doteq \frac{1}{2} + \epsilon$ where $\epsilon$ is a
positive real, it is \NP-hard to distinguish between the cases
$\maxqe(S')=1$ and $\maxqe(S') < \alpha$, where $S'$ is the input, a 
list of polynomials of degree at most two over $\GF{2}$.
\end{theorem}
\begin{proof}
Given Proposition \ref{prop:NP-hard:exact}, it suffices to construct a
polynomial-time transformation $f$ that takes a system $S$ of quadratic
equations and produces a system $S' = f(S)$ of quadratic equations
with the following properties:
\begin{itemize}
  \item Completeness: $\maxqe(S) = 1$ implies $\maxqe(S') = 1$.
  \item Soundness: $\maxqe(S) < 1 $ implies $\maxqe(S') < \alpha$.
\end{itemize}
Here is how we construct $S'$. Let the system $S$ consist of
the equations $Q_i = 0$ for $i \in [m]$, where~$Q_i$ denotes a
polynomial of degree at most 2 in $n$ variables over $\GF{2}$.
Fix an assignment $x \in \zo^n$, and define
\[ I_i \doteq I[Q_i(x) \neq 0] \]
to be the indicator that the $i$th equation is not satisfied. Note
that $x$ satisfies all the equations of~$S$ if and only if $I \doteq
(I_1, \cdots, I_m) = 0$. 

Consider the linear combination $\sum c_i Q_i$ for some $c \in
\zo^m$. Since we are working over $\GF{2}$, a sum of equations is
satisfied by $x$ if and only if the number of equations that are not
satisfied by $x$ is even. In terms of the above linear combination, we
have that 
\[ \sum_{i=1}^m c_i Q_i(x) = 0 \iff \sum_{i=1}^m c_i I_i = 0 \iff  
\langle c,I \rangle = 0, \]
where all the arithmetic is over $\GF{2}$. 
Therefore, for a uniform linear combination we have
\[
\Pr_{c \gets U_m}[\sum_{i=1}^m c_i Q_i(x) = 0] = \Pr_{c \gets
  U_m}[\langle c,I \rangle = 0] = \left\{ \begin{array}{ll}
 1  &\mbox{ if $I = 0$} \\
  \frac{1}{2} &\mbox{ otherwise.}
       \end{array} \right.
\]
If we defined $S'$ to consist of all equations of the form
$\sum_{i=1}^m c_i Q_i(x) =0$ for all possible $c \in \zo^m$, we would
be close to our goal; the only issue is that $S'$ is too large to
be constructed in polynomial time. To make the size of $S'$ polynomial
in $m$, we let $c$ range over the image of a small-bias generator,
i.e., over $G(U_\ell)$ where $G: \zo^\ell \to \zo^m$ is a $\beta$-bias
generator. By the definition of bias and the connection with PRGs for
polynomials of degree 1, for $D_m \doteq G(U_\ell)$, we have that \[
\Pr_{c \gets D_m}[\sum_{i=1}^m c_i Q_i(x) = 0] = \Pr_{c \gets D_m}[
\langle c,I \rangle = 0]  \left\{ \begin{array}{ll}
 = 1  &\mbox{ if $I = 0$} \\
 \leq \frac{1}{2} + \frac{\beta}{2} &\mbox{ otherwise.}
       \end{array} \right.
\]
If we set $\beta < 2\epsilon$, we meet the completeness and
soundness conditions of the transformation. Moreover, if we use the
small-bias generator from Theorem \ref{lect06:thm: construction} of
Lecture 6, the
number of equations in $S'$ is $2^\ell = O((\frac{m}{\epsilon})^2)$,
which is is polynomial, and we can construct each individual equation
in polynomial time. Thus, we have realized the transformation $f$ we
needed. This completes the proof.
\end{proof}
Theorem \ref{thm:inapprox} immediately yields the following corollary
about the hardness of approximating the value $\maxqe(S)$.  
\begin{corollary}\label{cor:inapprox}
For every $\alpha \doteq \frac{1}{2} + \epsilon$ where $\epsilon$ is a
positive real, the following problem is \NP-hard: Given a list $S'$ of
polynomials of degree at most 2 over $\GF{2}$, find a value $v$ such that
\[ \alpha \cdot \maxqe(S') \leq v \leq \maxqe(S'). \]
\end{corollary}
\begin{proof}
  Given an approximation algorithm as in the statement of the corollary, 
  we can distinguish between the two cases of systems $S'$ in 
  Theorem~\ref{thm:inapprox} as follows:
  Run the approximation algorithm on $S'$ and check whether the value 
  $v$ it outputs is at least $\alpha$.
  If so, we are in the first case, and otherwise in the second case. 
\end{proof}
Corollary \ref{cor:inapprox} implies that it is \NP-hard to find an 
assignment
for the variables of a given system $S$ of quadratic equations that
satisfies a fraction at least $\alpha \cdot \maxqe(S)$ of the
equations for any $\alpha$ of the form $\alpha = \frac{1}{2} +
\epsilon$ where $\epsilon$ is a positive constant. 

In fact, the proofs show that we can pick $\epsilon$ to be a decreasing
function of the input size -- as long as $1/\epsilon$ is polynomially
bounded, the proofs of Theorem \ref{thm:inapprox} and Corollary
\ref{thm:inapprox} carry through.

One can do substantially better using a considerably more involved
approach based on harmonic analysis. That approach shows that $\alpha$
in Theorem \ref{thm:inapprox} can be chosen as $\alpha = \frac{3}{8} +
\epsilon$, where $1/\epsilon$ is polynomially bounded
\cite{Hastad2011}. The same improvement to Corollary
\ref{cor:inapprox} follows. 

Corollary \ref{cor:inapprox} can be
further improved by considering as the first case in Theorem
\ref{thm:inapprox} systems $S$ that are highly satisfiable but not
necessarily fully satisfiable. The approach based on harmonic
analysis then shows that $\alpha$ in Corollary \ref{cor:inapprox} can
be chosen as  $\alpha = \frac{1}{4} + \epsilon$, where $1/\epsilon$ is
polynomially bounded \cite{Hastad2001}. The latter result is tight in
the sense that $\alpha = \frac{1}{4}$ can be realized in polynomial
time, as we will see next. 

\subsection{Approximation algorithms}

In order to construct a good approximation algorithm, we first analyze
how well a random assignment fares, and then show how to
deterministically find an assignment that does at least as well.

Let us first focus on a single equation $Q=0$, where $Q$ is a
polynomial of degree at most 2 in $n$ variables over $\GF{2}$. What is
the minimum probability that a uniform assignment $x \in_u \zo^n$
satisfies $Q(x)=0$? 

Well, if $Q \equiv 1$, then the probability is zero. Such
unsatisfiable equations can be recognized in deterministic polynomial
time, even when they are given in the form of an arithmetic formula or
circuit. Recognizing such $Q$ is equivalent to the polynomial identity
testing problem from Lecture 1. In general, the only efficient algorithms for
this problem are randomized. However, for degree two there is a simple
randomized algorithm: Evaluate the formula or circuit symbolically,
keeping track of the list of all monomials, and check whether at the
end only the constant monomial is present. Since there are only
$O(n^2)$ monomials of degree at most 2, this can be done in
deterministic polynomial time.

How low can the probability be for a satisfiable equation? For $Q(x) =
x_1x_2 -1$, the probability that a uniform assignment satisfies $Q=0$
equals $\frac{1}{4}$. We leave it as an exercise to argue that this is
as low as that probability gets.
\begin{exercise}
Prove that a uniform assignment to the variables of a satisfiable
quadratic equation over $\GF{2}$ is satisfying with probability at
least $\frac{1}{4}$.
\end{exercise}

Suppose $S$ contains $m$ equations that are individually
satisfiable. By linearity of expectation, the expected number of
equations that a uniform assignment satisfies is at least
$\frac{m}{4}$. This implies the existence of an assignment $x$ that 
satisfies at least that many equations. Since no assignment can
satisfy any of the other equations, that assignment satisfies at least
a fraction $\frac{1}{4}\maxqe(S)$ of the equations. What
remains to show is to find such an assignment $x$ in deterministic
polynomial time. 

Suppose that we generate the assignment $x$ using an $\epsilon$-PRG
$G$ for polynomials of degree 2. For each of the $m$ individually
satisfiable equations $Q=0$, the probability that $x$ satisfies it is
at least $\frac{1}{4}-\epsilon$. Thus, the expected number of
satisfied equations among those $m$ is at least
$(\frac{1}{4}-\epsilon)m$, and therefore there exists at least one $x$
in the image of $G$ that satisfies $\lceil (\frac{1}{4}-\epsilon)m
\rceil$ of the equations. Now, if $\epsilon$ is sufficiently small,
the latter quantity is at least $\frac{m}{4}$. We leave it as an
exercise to show that $\epsilon < \frac{1}{4m}$ suffices to guarantee
that property.
\begin{exercise}
Prove that if $\epsilon < \frac{1}{4m}$ then 
$\lceil (\frac{1}{4}-\epsilon)m \rceil \geq \frac{m}{4}$.
\end{exercise}
Corollary \ref{lect06:thm: construction} from Lecture 6 gives us an
efficiently computable $\epsilon$-PRG $G$ for degree 2 polynomials
with seed length $\ell = O(\log(m/\epsilon))$. For $\epsilon =
\frac{1}{5m}$, we can cycle over all assignments $x$ in the range of
$G$ in time polynomial in $m$. Trying out all those assignments and
selecting the one that satisfies the most equations yields the
following result.
\begin{theorem}\label{thm:approx}
There exists a deterministic polynomial-time algorithm that takes a
list $S$ of polynomials $Q$ of degree at most 2 over $\GF{2}$ and outputs
an assignment that satisfies at least $\frac{1}{4} \cdot
\maxqe(S)$ of the equations $Q=0$.
\end{theorem}
Theorem \ref{thm:approx} shows the optimality of the strengthening of
Corollary \ref{cor:inapprox} we mentioned, in which $\frac{1}{2}$ is
replaced by $\frac{1}{4}$. 

In the case where the system $S$ is fully satisfiable, it turns out
that one can
efficiently reduce to a fully satisfiable system $S'$ in which every
equation is individually satisfiable with probability at least $\frac{3}{8}$
\cite{Hastad2011}. The same approach as above then yields a 
deterministic polynomial-time algorithm to find an assignment that
satisfies a fraction at least $\frac{3}{8}$ of the equations. This shows the
optimality of the strengthening of Theorem \ref{thm:inapprox} we
mentioned, in which $\frac{1}{2}$ is replaced by $\frac{3}{8}$. 

\bibliographystyle{alpha}
\bibliography{references}

\end{document}
