\documentclass[a4paper]{article}
\usepackage{geometry}
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{paralist}
\usepackage{epstopdf}
\usepackage{tabularx}
\usepackage{longtable}
\usepackage{multirow}
\usepackage{multicol}
\usepackage[hidelinks]{hyperref}
\usepackage{fancyvrb}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{float}
\usepackage{paralist}
\usepackage[svgname]{xcolor}
\usepackage{enumerate}
\usepackage{array}
\usepackage{times}
\usepackage{url}
\usepackage{fancyhdr}
\usepackage{comment}
\usepackage{environ}
\usepackage{times}
\usepackage{textcomp}
\usepackage{caption}


\urlstyle{rm}

\setlength\parindent{0pt} % Removes all indentation from paragraphs
\theoremstyle{definition}
\newtheorem{definition}{Definition}[]
\newtheorem{conjecture}{Conjecture}[]
\newtheorem{example}{Example}[]
\newtheorem{theorem}{Theorem}[]
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}

\floatname{algorithm}{Procedure}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
\newcommand{\abs}[1]{\lvert#1\rvert}
\newcommand{\norm}[1]{\lVert#1\rVert}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\CC}{\mathbb{C}}
\newcommand{\Nat}{\mathbb{N}}
\newcommand{\br}[1]{\{#1\}}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\renewcommand{\qedsymbol}{$\blacksquare$}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}

\newcommand{\vc}[1]{\boldsymbol{#1}}
\newcommand{\xv}{\vc{x}}
\newcommand{\Sigmav}{\vc{\Sigma}}
\newcommand{\alphav}{\vc{\alpha}}
\newcommand{\muv}{\vc{\mu}}

\newcommand{\red}[1]{\textcolor{red}{#1}}

\def\x{\mathbf x}
\def\y{\mathbf y}
\def\w{\mathbf w}
\def\v{\mathbf v}
\def\E{\mathbb E}
\def\V{\mathbb V}

% TO SHOW SOLUTIONS, include following (else comment out):
\newenvironment{soln}{
    \leavevmode\color{blue}\ignorespaces
}{}


\hypersetup{
%    colorlinks,
    linkcolor={red!50!black},
    citecolor={blue!50!black},
    urlcolor={blue!80!black}
}

\geometry{
  top=1in,            % <-- you want to adjust this
  inner=1in,
  outer=1in,
  bottom=1in,
  headheight=3em,       % <-- and this
  headsep=2em,          % <-- and this
  footskip=3em,
}


\pagestyle{fancyplain}
\lhead{\fancyplain{}{Homework 4: Written Exercise Part}}
\rhead{\fancyplain{}{CS 760 Machine Learning}}
\cfoot{\thepage}

\title{\textsc{Homework 4: \\ Written Exercise Part}} % Title

%\newcommand{\outDate}{Aug. 31, 2016}
%\newcommand{\dueDate}{5:30 pm, Sep. 7, 2016}


%%% NOTE:  Replace 'NAME HERE' etc., and delete any "\red{}" wrappers (so it won't show up as red)

%\author{
%\red{$>>$NAME HERE$<<$} \\
%\red{$>>$ID HERE$<<$}\\
%} 

\date{}

\begin{document}

\maketitle 


\section{Balls and Bins [25/3 pts]}
Suppose we throw balls into $n$ bins. Each ball is thrown independently and uniformly at random. 

(1) [Birthday Paradox] Suppose we throw $m$ balls. What is the probability that at least one bin has more than one balls? Write down the expression and then use the inequality $1-x \le e^{-x}$ to give a lower bound.

\begin{soln}  Solution goes here. \end{soln}

(2) [Coupon Collecting] Let $X$ denote the number of balls thrown until every bin has at least one ball. What is the expectation of $X$? Express it using the harmonic number $H_n = \sum_{i=1}^n 1/i$.

\begin{soln}  Solution goes here. \end{soln}


\section{VC-dimension of Rectangles [25/3 pts]}
What is the VC-dimension $d$ of axis-parallel rectangles in $R^3$? Specifically, a legal target function is specified by three intervals $[x_{\min}, x_{\max}], 
[y_{\min}, y_{\max}], [z_{\min}, z_{\max}]$, and classifies an example $(x,y,z)$ as positive if and only if $x \in [x_{\min}, x_{\max}], 
y \in [y_{\min}, y_{\max}]$, and $z \in [z_{\min}, z_{\max}]$. Justify your answer.

\begin{soln}  Solution goes here. \end{soln}

\section{Mistake Bound Model [25/3 pts]}
CNF is the class of Conjunctive Normal Form formulas in the form $C_1 \land C_2 \land \ldots$, where each clause $C_i$ is in the form $L_1 \lor L_2 \ldots $, and each Boolean literal $L_i$ is either a boolean feature $x$ or its negation $\neg x$. $k$-CNF is the class of CNF in which each clause has size at most $k$. For example, $x_4 \land (x_1 \lor x_2) \land (x_2 \lor \neg x_3 \lor x_5)$ is a $3$-CNF. Give an algorithm
to learn $3$-CNF formulas over $n$ boolean features in the mistake-bound model. Your algorithm should run in polynomial-time per example (so the ``halving algorithm" is not allowed). How many mistakes does it make at most?
(Hint: modify the FIND-S algorithm.)

\begin{soln}  Solution goes here. \end{soln}


\section{Extra Credit: VC-dimension of Linear Separators [20 pts]}
In this problem, you will prove that the VC-dimension of the class $H_n$ of halfspaces (another term for linear threshold functions $f_{w,b}(x) = \text{sign}(w^\top x + b)$) in $n$ dimensions is $n+1$. We will use the following definition: The convex hull of a set of points $S$ is the set of all convex combinations of points in $S$; this is the set of all points that can be written as $\sum_{x_i \in S} \lambda_i x_i$, where each $\lambda_i \ge 0$, and $\sum_i \lambda_i = 1$. It is not hard to see that if a halfspace has all points from a set $S$ on one side, then the entire convex hull of $S$ must be on that side as well. 

(a) [lower bound] Prove that $\text{VC-dim}(H_n) \ge n+1$ by presenting a set of $n+1$ points in $n$-dimension space such that one can partition that set with halfspaces in all possible ways, i.e., the set of points are shattered by $H_n$. (And, show how one can partition the set in any desired way.)

(b) [upper bound part 1] The following is Radon's Theorem, from 1920's. 
\begin{theorem} 
Let $S$ be a set of $n+2$ points in $n$ dimensions. Then $S$ can be partitioned into two (disjoint) subsets $S_1$ and $S_2$ whose convex hulls intersect. 
\end{theorem}
Show that Radon's Theorem implies that the VC-dimension of halfspaces is at most $n+1$. Conclude that $\text{VC-dim}(H_n) = n+1$.

(c) [upper bound part 2] Now we prove Radon's Theorem. We will need the following standard fact from linear algebra. If $x_1, \ldots, x_{n+1}$ are $n+1$ points in $n$-dimensional space, then they are linearly dependent. That is, there exist real values $\lambda_1, \ldots, \lambda_{n+1}$ not all zero such that $\lambda_1 x_1 + \ldots + \lambda_{n+1} x_{n+1} = 0$.
You may now prove Radon's Theorem however you wish. However, as a suggested first step, prove the following. For any set of $n+2$ points $x_1, \ldots, x_{n+2}$ in $n$-dimensional space, there exist $\lambda_1, \ldots, \lambda_{n+2}$ not all zero such that $\sum_i \lambda_i x_i = 0$ and $\sum_i \lambda_i = 0$. (This is called affine dependence.)

\begin{soln}  Solution goes here. \end{soln}

\bibliographystyle{apalike}


%----------------------------------------------------------------------------------------


\end{document}
