In designing the internet, the original arpanet/designed placed an
emphasis on recovery from disasters as the foremost concerns.  An
implication of which lead them towards a distributed control plane --
robust to the failure an individual nodes.  With the changing trends
such as faster/lower latency networks, we have since revisited this
decision and moved towards the notion of a centalized controller.  A
notion embodied by Software Defined Networking (SDN).

SDN is growing in popularity and poised to dominate the following
environments; Data centers (many examples), Clouds (google), enteprise
networks (any examples), and WAN (internet2 and exchange point).
\theo{The paragraph above is a place holder. it needs to be polished
with: (1) citations for deployments, (2) citation for money value of SDN
companies -- and acquisitions. also needs a brief primer on SDN so
readers know what controllers are.s}

%However, another trend that has thus remained ignored is that of the
%pervasive use of commodity devices.
%
While technological trends make SDN viable and extremely attractive,
these trends introduce significant fragility and reliability concerns
into the network.  Two such notable trends are the wide spread and
prevalent use of commodity devices both as network switches and as
server hosting the control plane (Controllers), and the utilization of
untested and relative immature code on the controllers.  Commodity
devices fail with a significantly higher probability than traditional
devices making failure a common case rather than a corner case. 
Futhermore, given the novelty of existing SDN application and the
limited testing performed on them, the problem of failure is further
exhaberated by buggy and unreliable code. XXX~\cite{} observed network
crippling failures in a range of realistic applications.  Both trends
result in an even greater degree of  fragility than before.

Despite acknowledgement of reliability problems, it is surprising that
little work has been done to harden the SDN control plane from failures.
 Existing approaches such as B4 and Onix only address a specific failure
model (controller failures) and assume that paxos~\cite{} can be used to
fix this class of failures.  However, paxos may significant overheads
that limit how paxos can be used. As an example, paxos  using domain
specific knowledge. For example, the storage community does XXX in
XXX~\cite{} \theo{megastore/cassandra} and the distributed systems
community modify paxos by changing XXX in XXX~\cite{}. In addition,
paxos only addresses failures that arise due to commodity devices and not
failure in the applications. For application based failures, approaches
like NICE and XXX can be leverage to detect the root-cause of SDN
application failures but these approach do little to ensure these
applications remain robust in the face of the diagnosed failures.

\theo{What other communities are there}

In this paper, we argue that existing SDN solutions do not degrade
gracefully in the face of certain\fixme{which?} failures and that this lack of
graceful degradation slows adoption and limit deployment scenarios \fixme{
fail stop failures may actually be better on many occasions}.  We
demonstrate that SDN networks can be harden to graceful tolerate failure
by crafting special tailored solutions that leverage domain specific
knowlege about the characteristics of controllers and the applications
that run atop them.  More specifically, knowledge that a large class of
application failures are deterministic and triggered by specific
inputs~\cite{}, knowledge that certain applications and their flows have
a higher priority than others, and knowledge that switches have built in
functionality for performing leader selection and multicast obviating
the need for Paxos.

Building on these insights, our goal is to harden the control plane
such that it degrades gracefully with failures of different components \fixme{
hardening the control plane will prevent failures. For graceful failures
you need something like a recovery solution}. Our
design principles are: 
\begin{itemize} 
\item {\it Keep applications running as long as possible:} \fixme{duh!} 
To overcome applications crashes, \name
discovers events that triggers crashes and transforms either the event
or controller state to eliminate crashes at the expense of correctness. 
This approach provides the network with a unique and principled way to
trade-off between availability and correctness.  Operators can guide
this approach by specifying which policies they are willing to
compromise on. \theo{replace specify with annotate and rewrite sentence.
}

\item {\it Eventually available controllers:}  A controller failures
should be masked from switches and from other controllers in the network
with minimal to little degredation of performance under normal failure
free circurmstances.  Rather than using paxos and incuring latency
penalty for each network event, we argue that by using leader selection
and message multicast functionality available at switches we can ensure
high availability while ensuring latency penalty during failure-free
circurmstances.

\item {\it Selectively ensure availability:} Network partitions and
controller failures reduces the number controller thereby increasing the
load on each.    Rather that providing degraded service to all
applications and all flows, the controller should impose a hierarchy,
selectively running higher priority apps and providing services for
higher priority flows.  Guided by operator preference, the controller is
able to explore trade-offs between network productivity and resource
availability.



\end{itemize}

In developing a system that meets the above design goals, \name needs to
overcome several challenges: discovering application failures and their
root-causes before they occur, dynamically detecting the set of the
minimal set of switches and events that are required for priority
applications to perform, detecting the maximal set of events/switches a
controller can handle, and

We resolve these challenges augment the controller with fault tolerant
code that allows the controller to clone running applications and
evaluate every call into the controller by network events on cloned copies.  
If clone crashes, a software
fault is detected and the input is identified as failure-inducing and
systematic steps are taken to change the controller state such that the
controller can run without failures. Furthermore, this fault tolerant code
learns mappings of flows and switches to applications and when the
controller starts to run out of resources, the code starts to shutdown
lower priority applications ignoring events from switches+flowspace
belong to these applications. \asim{what is this giving you? not clear}

Finally, to provide the illusion of high availability, each controller
binds to a set of any-cast addresses. Each address represents a set of
switches and a controller respond to request made to that address when a
majority of those switches declare it to be their master.  When declared
master the fault tolerant code clones running apps and feeds the new
clone events from both the old and new switches.  The old clone response
to events from existing switches and events from new switches are
buffered until the new clone is ready. at which point the new clone
takes over and starts running. {\asim this is a technique. there is
a mix of goals, problems and mechanisms in this paragraph}


Our work is motivated by fault-tolerant techniques from the operating
systems and the distributed system community.  Our use of cloning
applications and testing input is motivated by work done providing
highly available drivers within the operating system community.  A
distinguishing factor is that work within the driver community focuses
on transient bugs that can be fixed by restarting, where as we focus on
systematic bugs and persist after restart provided the same input is fed
to the applications.  Our use of any-cast address to provide the
illusion of availability is motivated by work within the distributed
systems community to provide highly avaiable servers.


{\bf Road Map}We start, section~\ref{}, by examining the various
deployment scenarios and discuss how utilization of commodity devices
can introduce a range of unexpect failure scenarios. We highlight the
salient features of scenarios relating to failure of switches, of
controllers, and of applications.   Guided by these salient failures, in
section~\ref{} we present a suite of solutions to create harden
controllers.  We evaluate .... in section~\ref{}. While initially
designed for modular controllers such as XXX and XXX, in section~\ref{}
we discuss how the increased power provided by higher level controllers
will allow us to incorporate application information and further
stregnthend/quicken our solution.

\theo{the harkening of SW solutions limits what can be done! the app
level solutions can be a whoel different paper}

The challenges of creating these harden controllers revolve around
consistency in state, limitation of hardware, speed, and network
limitations.  Consistency ..

xxx ...

xx...