\documentclass[11pt]{article}
\usepackage{ifthen}
\usepackage{alltt}
% \usepackage{indentfirst}
\usepackage{latexsym}
\usepackage[pdftex]{graphicx}
\usepackage{gdefs}
% \usepackage{logdef}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{setspace}
\usepackage{rotating}
\usepackage{longtable}
\usepackage{hhline}
\usepackage{refs}
\usepackage{graph}
\usepackage{changebar}
\usepackage{array}
\usepackage{wrapfig}
\usepackage[small,compact]{titlesec}

\usepackage{tweaklist}
% this sets the spacings in the itemize environment
\renewcommand{\itemhook}{\setlength{\topsep}{0pt}%
  \setlength{\itemsep}{0pt}%
  \setlength{\parsep}{0ex}%
}

% this sets the spacings in the enumerate environment
\renewcommand{\enumhook}{\setlength{\topsep}{0pt}%
  \setlength{\itemsep}{0pt}%
  \setlength{\parsep}{0ex}%
}

% this sets the spacings in the description environment
\renewcommand{\descripthook}{\setlength{\topsep}{0pt}%
  \setlength{\itemsep}{0pt}%
  \setlength{\parsep}{0ex}%
}

\setlength{\topmargin}{0.0in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.0in}
\setlength{\textwidth}{6.5in}
\setlength{\oddsidemargin}{0.0in}
\setlength{\evensidemargin}{0.0in}
\setlength{\textheight}{9.0in}

\setcounter{topnumber}{5}
\def\topfraction{0.99}
\def\textfraction{0.05}
\def\floatpagefraction{0.9}
\setcounter{bottomnumber}{5}
\def\bottomfraction{0.99}
\setcounter{totalnumber}{10}
\def\dbltopfraction{0.99}
\def\dblfloatpagefraction{0.8}
\setcounter{dbltopnumber}{5}

\setlength{\marginparwidth}{.95in}
\let\oldmarginpar\marginpar
\renewcommand\marginpar[1]{\-\oldmarginpar[\raggedleft\footnotesize #1]%
{\raggedright\footnotesize #1}}

\let\oldthebibliography=\thebibliography
  \let\endoldthebibliography=\endthebibliography
  \renewenvironment{thebibliography}[1]{%
    \begin{oldthebibliography}{#1}%
      \setlength{\parskip}{0ex}%
      \setlength{\itemsep}{0ex}%
  }%
  {%
    \end{oldthebibliography}%
  }

\title{
       CS701 Guest Lecture 2: \\
       Compilation of GPU Kernels \\
       CS701 \\ 
}

\author{Thomas Reps\\ \small[Based on notes taken by Mason Volk on November 25, 2015]}
\date{}

\begin{document}           % End of preamble and beginning of text.

\maketitle

\begin{abstract}
In this lecture, we see a summary of the basics of GPU architecture and the methodology 
behind various design decisions. We see how GPUs are structured with thought given to 
maintaining high occupancy while at the same time handling issues with concurrency. We 
take a look at GPU programming in the context of CUDA, and get a sense of the concerns 
that must be addressed for compilation. We also see a comparison between GPUs and 
multi-core CPUs. 
\end{abstract}

\section{Introduction}
\paragraph*{GPU Frameworks and Toolkits}
There exist multiple frameworks and toolkits which can be used to generate GPU programs. 
These toolkits operate on different levels of separation from the actual GPU internals. 
\begin{enumerate}
  \item
    The CUDA toolkit provides a basic means to interact with a CUDA-enabled GPU.
    An API allows programmers to specify and launch GPU kernels, with thread 
    dependent functions to be executed in parallel on the GPU.
  \item
    The openCL framework allows for writing programs for heterogenous environments
    that might include GPUs~\cite{AMDOpenCLCompiler}.%[http://llvm.org/devmtg/2010-11/Villmow-OpenCL.pdf]
  \item
    openACC can emit CUDA or openCL.  It abstracts co-processing on heterogenous
    platforms to make programming simpler. openACC is used to accelerate programs
    by transfering parts of them from the CPU to the GPU for computation behind the scenes~\cite{IntroductionToOpenACC}.
   %[https://www.olcf.ornl.gov/wp-content/uploads/2013/02/Intro_to_OpenACC-JL.pdf]
\end{enumerate}

\paragraph*{The CPU ISA}
The ISA of a CPU is a contract between the hardware and what the compiler emits. The 
revealed version of the ISA allows the compiler to assume that the processor executes one
instruction at a time.  In reality, however, the actual ISA is not revealed.  Instead, a virtual
ISA is given.  On top of this abstraction rests further abstraction layers, such as the CUDA API. \\

A virtual ISA has several advantages.  Since it is an abstraction of the actual architecture,
it allows for inovations in ISA without exposing internal details of the hardware which might
complicate things.

FIGURE 1:  Visualization of the Memory Wall.

On the CPU, the job of the ALU is to take out values from the register file, do an operation on the
values, and then put the result back in the register file. It used to be that most operations would 
take one cycle to complete. However, as processing speed increased, memory access speed could 
not keep up. Memory accesses became relatively costly operations, taking upward of 200 cycles 
a piece.  This "memory wall", shown in Figure 1, was a problem since the illusion of single cycle 
instructions was broken.

\paragraph*{Microarchitecture techniques}
The solution to the memory wall problem were microarchitecture techniques created to reduce memory 
access time and smooth overall instruction cost.  These techniques included: 
\begin{enumerate}
  \item
    The use of caches to speed up memory access for select areas of memory
  \item
    Superscalar optimization to fold costly instructions together
  \item
    Speculation to prepare for predicted costly operations (such as pre-loading the cache)
\end{enumerate} 

Microarchitecture optimization techniques were subject to several problems.  First, the techniques
were architecture dependent, necessitating information leaks to the compiler in order to fully take
advantage of the optimization opportunities available.  Old compilers attempting to make compact 
code could actually end up making the situation worse as the output instructions interact badly
with the architecture of the CPU. \\
It was also known that the ALU consumes a relatively small amount of space in the CPU, and has
even less relative power consumption.  The other power required for the chip comes from the
overhead for microarchitecture techniques.   

\paragraph*{GPU Architecture}
The architecture of the GPU is designed so that more resources are distributed to components
such as the ALU(s) that execute instructions and perform operations within the processor. Focus is
placed on making sure that the processor always has something to do.

FIGURE 2: Several Processes Running on a Processor.

\paragraph*{Parallelism Within the GPU}
Suppose that access to memory is relatively slow (say, 200 cycles away).  After an instruction requiring
memory access, the ALU on a CPU has to wait for the access to complete.  However, this is assuming
that the ALU does not have any other work that it can do in the mean time. \\
The implied solution for this problem would be to have many different programs run. This way it would be
likely that the ALU would have processes available to work on if the current process began waiting on a ld
instruction. This situation is played out in Figure 2, where several processes are shown running. Ideally,
as soon as the first ld comes back, the processor should be able to switch back to process P0.
These imaginary processes would be difficult to obtain individually. Each process would need to be retrieved 
from memory, so the situation ends up back at square one.
The situation changes entirely, however, if it is assumed that the programs presented are all identical and small.

\paragraph*{Optimizations for the GPU}
Assuming that the programs running on a GPU are all small and identical, several architectural optimizations
can be applied to minimize the time spent accessing memory and maximize the amount of time spent working on
processes.
First, if the processes on a GPU are all identical, then the processes do not need to be loaded in
from memory individually.  Instead an instruction cache can be added to store a single copy of the program, which
can be looked at and shared by all running processes. \\
Additionally the question arises, what if instead of a single
ALU, the processor had multiple ALUs (say, 16)?  The processor would then be able to fetch just one instruction and
perform that instruction for 16 of the identical programs at once.  In essence, the cost of the memory access
for the instruction would be amortized over the 16 ALUs, greatly reducing its impact.

FIGURE 3: Basic SM Architecture.

Figure 3 shows the architecture described above.  The section of the hardware removed from memory in this figure is
refered to as an SM (Streaming Multiprocessor).  It contains an instruction cache, as well as many RFs (Register Files) 
and PCs (Program Counters) so as to properly keep track of the individual context of each program that it runs. The
multiple ALUs are organized according to the SIMD (Single Instruction, Multiple Data) paradigm, meaning that together,
the ALUs apply a single instruction of a program to multiple RFs at once. \\

It is interesting to note that the GPU still uses the same amount of transistor resources, just with a different way of 
investing those resources throughout the processor. Generally, the CPU spends most of its time in RFs whereas the CPU
spends its time in caches. The architectural requirement in the GPU is typically matched by the number of RFs.  
It is up to the compiler to decide how to allocate the available registers. \\
For example, say you had 5000 registers, and can have 100 programs, each with 50.  The compiler would decide how to allocate those registers
to the programs, giving an error message if no resources are available.  Some high-end GPUs can dynamically allocate SMs
depending on the resource requirements.

FIGURE 4: A Block of Threads.

\paragraph*{Higher Level GPU Organization}
In the GPU SMs are organized into larger structures called blocks, as shown in Figure 4.  Each SM in a block is essentially
a thread which is itself executing 16 parallel instructions. The typical model of a GPU architecture is shown below in Figure 5.

FIGURE 5: Typical GPU Model.
A thread is orginized into groups of 16 threads called warps.  The threads in a warp execute simultaneously, and in this regard
warps are conceptually identical to SMs.  As before, warps are grouped in larger organizational structures called blocks.  These
blocks are then grouped further into grids.  One of the main reasons for this 3-level hierarchy is to allow threads to be mapped
in 3D, as will be explained later.

\paragraph*{Hierarchical Memory}
Memory in the GPU also has a hierarchical organization to reflect the organization of the threads. The levels of memory
organization separate both speed of access and sharing in the GPU. There are generally three levels:
\begin{enumerate}
  \item
    Local memory, which has the fastest access time since it is to the individual thread group.
  \item
    Shared memory, which is shared between all threads in the same block.
  \item
    Global memory, which is reachable by all threads but with the slowest access time.
\end{enumerate} 
Shared memory is important for many applications in the GPU.  As just one example, block matrix multiplication involves threads
needing to obtain results from other threads.  These results can be obtained by reading and writing shared memory. \\
With the availability of shared memory in a typical GPU, there also comes issues with concurrency.  In particular, there are no
guarding guarantees between when two warps or two thread blocks will execute instructions. Without some method of
synchronization, it would be possible for two warps to access the same location in shared memory at the same time, or
for two blocks to try to access the same location in global memory at once.

\section{Programming for the GPU}
\paragraph*{The API}
In order to allow the programmer to create and launch a GPU kernel, CUDA includes a set of APIs to expose connections between
the CPU and the GPU.  One major problem in current computer architecture is that GPU memory is separate from CPU memory.  The
main purpose of GPU memory is storing graphics information, a purpose that departs somewhat from that of CPU memory.  \\
In order to fix the problem of separated memory, CUDA provides an API that allows the user to copy from a pointer in CPU land to a pointer
in GPU land and vice-versa.  This API allows the programmer to supply their threads with data on which to operate.  CUDA also
provides APIs for launching created GPU kernels.  The kernels themselves can be programmed in a high level language and treated
like loops.  The programming model for GPUs is on the whole simpler than that for CPUs.~\cite{GPUOptimizationFundamentals}\\
%[https://www.olcf.ornl.gov/wp-content/uploads/2013/02/GPU_Opt_Fund-CW1.pdf]
There are two outstanding issues left to consider for the GPU programmer:
\begin{enumerate}
  \item
    How do you get low warp divergence?
  \item
    How do you controll access order between blocks?
\end{enumerate}
\paragraph*{Access Control}
The solution to the memory access order issue in CUDA is to provide facilities for shared memory synchronization 
which are usable by the programmer.  Using \textbf{sync\textunderscore threads}, the programmer instructs a block to wait for all of its threads to
reach a certain point before continuing.  This form of synchronization gives control to the programmer by exposing
the underlying parallel hardware of the GPU.
\paragraph*{Thread Numbers}
In the typical GPU, each thread is assigned an explicit number. In the CUDA toolkit, this number is architecturally exposed
to the programmer.  In addition, the 3-level organizational hierarchy discussed earlier allows these thread numbers to be 
mapped into 3D based on grid position. \\
With the exposure of thread numbers, a programmer could program GPU threads 
with behavior that is based on thread number.  This would allow the programmer to diversify the functions of their threads.  
For example, the programmer might choose to make all even numbered threads producers and all odd numbered threads 
consumers over a block of shared memory.  In this instance, the entire piece of work would be one GPU kernel, and the program
point at which the thread decides what to be is the point at which this first kernel launches a "new" kernel, which could be 
either the producer or the consumer kernel.  With this capability, however, there comes the threat of divergent thread behavior 
affecting GPU performance.

\section{Compilation Concepts}
FIGURE 6: Examples of Control Flows that Cause Warp Divergence.

\paragraph*{Warp Divergence}
The threat of warp divergence emerges when threads are made to follow different control-flow paths in a program.  
Control-flow wise, this means that threads can be at different points in the CFG.  We would like to minimize warp divergence,
since this interferes with the SIMD paradigm. Some threads might hit in the instruction cache while some don't, causing 
nonhomogeneity in the SM. \\
In order to minimize warp divergence, the programmer should avoid data dependent branches (where data specific to a certain
thread such as the thread number could effect the branch outcome). Two control flows that can cause warp divergence are 
shown in Figure 6 above. In both cases, the divergence occurs at the if statement, which is dependent on the thread id.

\paragraph*{Performance Techniques}
One possible solution to the problem of warp divergence would be to use larger warps.  This way, more threads could be organized at
once.  However, grouping more threads together will incur additional overhead costs. \\
In Figure 6, it is possible to determine by thread id the result of the expression \textbf{x\%2 == 0} .  One mitigation technique would be to move 
threads so that grids are mostly homogeneous, containing mostly odd or even threads.  By increasing homogeneity, divergence could be reduced. \\
Control flow branch paths can be explicitely encoded by predicating on them.  As an example, upon reaching \textbf{cmp P1, x, y}, the
if and else paths would be predicated on P1 as in Figure 7.

FIGURE 7:  Predication of Control Flow Paths.

If a predicate is always true or false, then a compiler might know to skip the branch test and only include the relevant branch in the output.
A similar concept exists for CPUs: a mask register can be set and then checked for conditional jumps.
\paragraph*{Occupancy}
It is a desired quality of a GPU that as many threads are executing concurrently as possible. Therefore, every effort should be made to ensure
high warp occupancy.  Suppose that you have 8 registers and 64 threads, with 8 threads executing at the same time.  There might be more threads
out of your pool that you coulds squeeze in for execution, and in this sense you might be losing occupancy. \\
One solution to this problem is to allow some threads to spill into shared memory instead of using registers to increase the number of threads.  While
being slower than the registers, shared memory is still fairly fast (it takes about 3 cycles to access).  A small amount of slowdown in certain threads is
a small price to pay for the increased amount of threads that can now be processed simultaneously.\\
There are several possible problems with this technique. First, the remaining shared memory must be taken into account.  If the shared memory runs out,
then an error might be given by the compiler. Also, there might be a "tipping point" where the slowdown caused by spilling into shared memory can
no longer by made up for with increased occupancy. \\
It is also possible to inline all functions, which removes the need for a stack in the warp. Removing the need for stacks also eliminates the associated stack
management overhead.\\
On loads, GPUs can also save energy by detecting what is being loaded and only doing one load for multiple requests.

\section{GPU vs. CPU Comparison}
A multicore CPU hides latency using caching.  At the end of the day, computations per resource in the GPU are not that much higher than the computations per
resource in the CPU.  This is due to the fact that the two processor types have different ways of distributing resources, but ultimately both processor types use
the resources effectively towards their respective goals.  The GPU spends more area and power on important graphics functions, with lots of threads spending time 
in RFs.  The CPU, on the other hand, has far fewer threads, but makes up for the low throughput by mitigating latency through the use of caching.

\paragraph*{The Performance Valley}
Since the GPU spends more area on RFs for many parallel threads, it is generally more suited to be used for processing on a program
that matches these qualities and can be separated.  Likewise, a program that has fewer opportunities for parallelism but lots of
opportunities for caching would perform better on a CPU.  Programs in between the two extremes of this model, however, enter a "valley"
wherein the program will not perform very well on either a CPU or a GPU. ~\cite{StayAwayFromTheValley}%[http://dl.acm.org/citation.cfm?id=1591937]


\paragraph*{The GPGPU}
The result of a breaking point reached with specialized shaders inserted directly into hardware, the GPGPU (General Purpose GPU) has facilities 
to support programmable shaders.  In this sense, the GPGPU can in some ways behave almost like a CPU in that it is able to perform specialized tasks
based around the programmed shaders it is given.~\cite{TheGPUComputingEra} %[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5446251&tag=1&url=http%3A%2F%2Fieeexplore.ieee.org%2Fstamp%2Fstamp.jsp%3Ftp%3D%26arnumber%3D5446251%26tag%3D1]

{\small
  \bibliography{cs701}
  \bibliographystyle{plain}
}

\end{document}
