\section{design}
In this section, we describe the three key design principles that \name embodies.
\begin{figure}[ht]
	\begin{center}
	\begin{minipage}[t]{0.15\textwidth}
		\begin{center}	\includegraphics[height=.7in]{figs/FloodLightOverview.eps}
		\\
		(a)
		\end{center}
	\end{minipage}
\begin{minipage}[t]{0.15\textwidth}
	\begin{center}	\includegraphics[height=.7in]{figs/SafeFloodLightOverview.eps}
	\\
	(b)
	\end{center}
\end{minipage}
\begin{minipage}[t]{0.15\textwidth}
	\begin{center}	\includegraphics[height=.7in]{figs/SafeModeSDNModules.eps} \\
	(c)
	\end{center}
\end{minipage}
\caption{(a) Overview of the floodlight controller. (b) Overview of a floodlight controller modified to support \name. (c) Architecture of \name.}
\label{f:controller_overviews}
\end{center}
\end{figure}

\name provdies exports programming the application-controller interface with {\it Appvisor}. We demonstrate its usefulness by building {\it Crashpad}
and {\it Netlog}, that provide failure detection and recovery support.

\subsection{AppVisor: Isolating Applications}
To ensure the availability of the control plane, we must isolate applications from each other. Thus ensuring that un-handled-exceptions or segment-faults within one application, do not affect other applications or the controller.  Operating systems deal with a similar issue by modularizing and separating the address space of server-application from one-another.  Similarly, we separate the address space from SDN-application from each other and more importantly from the controller.  This separation, isolates the applications from each other and similarly isolates the controller from them.  

Unlike, with Operating Systems we make the fundamental assumption that applications may become momentarily unavailable due to failures. As such, we require that communication between controller-application include both a time-out/retry mechanism. Apart from improving availability, this separation opens up a wide range of novel use cases: per-application resource limits, application migration, multi-version software testing, and allows for certain seamless controller upgrades. 
%
\usecases{We elaborate on these use-cases in section~\ref{s:usecases}.}

We note that this separation and subsequent interposition, may introduce additional latency into the control-loop but we argue that such additional latency is acceptable as controllers introducing the controller into critical path slows down the network by an factor four~\cite{devoflow:sigcomm11} and thus controllers are not involved performance sensitive operations.

%"We find this slow control-data path adds unacceptable latency to flow setup" --devoflow

\subsection{CrashPad: Detecting and Isolating Failures}
SDN applications are event driven and process a single network event at a time. We assume these network events to be independent of one-another that simplifies the fault tolerance expectations. Faults occur in network due by events either from the network or from services running on the controller.  Thus in many situations detecting the cause of failure can be a simple as determining the last event processed by an application before failure. 
{\asim the event driven confuses me}

CrashPad needs to maintian an uncorrupted state to ensure the application can be rolled back upon failure. To do so, Crashpad clones application logic and feeds it the network events prior to applying them to original applications. If a failure occurs, Crashpad reverts to the clone and applies a range of correction techniques to the {\it offending event} before sending to the clone. \asim {I am confused what is clone and what is original and what is reveretd. please clarify} Concretely, the clone is an in-memory check-point of the application and thus the process of reverting the clone is essentially rolling back to the last check-point.

As such, the first step to recovery is ensuring that potential recovery does not violate operator preferences then applying applying recovery actions. 
%
In our initial straw-man, we propose two recovery actions: The first, a naive and lightweight approach, is to ignore the network event which causes the crash. The second, a heavyweight and invasive approach, is to apply symbolic execution techniques to determine the which changes to the internal state will allow the application to process the event without crashing.  We limit changes to internal state to changes in the topology which can be accomplished by sending the application switch or port drop events. We note that while symbolic execution techniques are time consuming, they can be run apriori and cached for future analysis.

In addition to simply dropping/ignoring the event, we can apply transformations to the event. For example, if we detect that a port-down event is the {\it offending event}, we may transform it to a switch down event. Thus essentially enlarging the scope of the event with the hope that the application will run bug free given the transformed event.

%When \name detects a failure, the \crashpad can use a range of mechanism to recover and prevent the failure. 
%

%



Simultaneously with roll-backing, we generate a problem ticket from the captured  stack-traces or core dump, controller logs and the {\it offending event}. This ticket can in-turn be submitted to the developer forum for future analysis.







We note that dropping or modifying the event does not itself solve the problem but instead creates a temporarily band-aide until a solution can be created. We find this to be acceptable as 80\% of bugs in production quality software do not have fixes at the time they are encountered~\cite{}. Furthermore, we acknowledge that for certain SDN application, such as security applications, network operators may be unwilling to ignore certain events. To account for this, we provide a simple policy language that allows operators to specify, on a per application basis, the set of events, if any, that they are willing to ignore.
 
\subsection{NetLog: Ensuring Safety}
To ensure the safety of the network, in addition to tracking and detecting failure causing input, the system must provide a mechanism that allows recovery. Failure to do this could result in network black-holes, cycles, and violation other network invariants.  Thus, \name includes a rollback module, NetLog, which tracks the network events that an application emits in response to a input event from the controller.

Our recovery solution leverages the insight that each control messages that modify network state is invertible: more precisely, for every state altering control message, A, there exists another control message, B, that undoes A's state change.  Unfortunately undoing a state change is imperfect as certain network state is lost. For example, while it is possible to undo a flow delete event, by adding the flow back to the network, the flow time-out and the flow counters are lost. Consequently, NetLog, stores and maintains the timeout and counter information of a flow table entry before deleting it.  Therefore should, NetLog, need to restore a flow table entry, it adds it with the appropriate time-out information. More specifically, flow counters cannot be inserted into the network.  For counters, NetLog stores the old counter values  in a counter-cache and when a statistics message from the switch to the application is detected, NetLog updates the counter value in the message with the sum of the value within the message and the value in its counter-cache. 

Together these three modules allow us to isolate applications, explicitly eliminating faith-sharing, and to detect and eliminate error inducing events, effectively improving the availability of individual applications. 