twepp2015/paper.tex

\documentclass{JINST}

\usepackage[utf8]{inputenc}
\usepackage{lineno}
\usepackage{ifthen}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{textcomp}
\usepackage{booktabs}
\usepackage{floatrow}

\newboolean{draft}
\setboolean{draft}{true}

\newcommand{\figref}[1]{Figure~\ref{#1}}

\title{A high-throughput readout architecture based on PCI-Express Gen3 and DirectGMA technology}

\author{
  L.~Rota$^a$,
  M.~Vogelgesang$^a$,
  L.E.~Ardila Perez$^a$,
  M.~Caselle$^a$,
  S.~Chilingaryan$^a$,
  T.~Dritschler$^a$,
  N.~Zilio$^a$,
  A.~Kopmann$^a$,
  M.~Balzer$^a$,
  M.~Weber$^a$\\
  \llap{$^a$}Institute for Data Processing and Electronics,\\
    Karlsruhe Institute of Technology (KIT),\\
    Herrmann-von-Helmholtz-Platz 1, Karlsruhe, Germany \\
    E-mail: \email{lorenzo.rota@kit.edu}, \email{matthias.vogelgesang@kit.edu}
}

\abstract{%   Modern physics experiments have reached multi-GB/s data rates.
Fast data   links and high performance computing stages are required for
continuous data   acquisition and processing. Because of their intrinsic
parallelism and   computational power, GPUs emerged as an ideal solution to
process this data in   high performance computing applications. In this paper
we present a   high-throughput platform based on direct FPGA-GPU
communication.    The architecture consists of a   Direct Memory Access (DMA)
engine compatible with the Xilinx PCI-Express core,   a Linux driver for
register access, and high-level software to manage direct   memory transfers
using AMD's DirectGMA technology. Measurements with a Gen\,3\,x8 link show a
throughput of up to 6.4 GB/s.  We also evaluated DirectGMA performance for low
latency applications: preliminary results show a round-trip latency of 2
\textmu s for data sizes up to 4 kB. Our implementation is suitable for real-
time DAQ system applications ranging from photon science and medical imaging
to High Energy Physics (HEP) trigger systems. }

\keywords{FPGA; GPU; PCI-Express; OpenCL; DirectGMA}

\begin{document}

\ifdraft
\setpagewiselinenumbers
\linenumbers
\fi


\section{Introduction}

GPU computing has become the main driving force for high performance computing
due to an unprecedented parallelism and a low cost-benefit factor. GPU
acceleration has found its way into numerous applications, ranging from
simulation to image processing. Recent years have also seen an increasing
interest in GPU-based systems for High Energy Physics (HEP)  (\emph{e.g.}
ATLAS~\cite{atlas_gpu}, ALICE~\cite{alice_gpu}, Mu3e~\cite{mu3e_gpu},
PANDA~\cite{panda_gpu}) and photon science experiments. 

In a typical scenario, data are acquired by back-end readout systems and then
transmitted in short bursts or in a continuous streaming mode to a computing
stage.

The data rates of bio-imaging or beam-monitoring experiments running in
current generation photon science facilities have reached tens of
GB/s~\cite{panda_gpu, atlas_gpu}. In order to collect data over long
observation times, the readout architecture must be able to save. The
throughput data transmission link may partially limit the overall system
performance.

Latency becomes the most stringent requirement for time-deterministic
applications, \emph{e.g.} in Low/High-level trigger systems.  

Due to its high bandwidth and modularity, PCIe quickly became the commercial
standard for connecting high-throughput peripherals such as GPUs or solid
state disks. Moreover, optical PCIe networks have been demonstrated a decade
ago~\cite{optical_pcie}, opening the possibility of using PCIe as a
communication link over long distances.

Several solutions for direct FPGA/GPU communication based on PCIe are reported
in literature, and all of them are based on NVIDIA's GPUdirect technology. In
the implementation of bittnerner and Ruf ~\cite{bittner} the GPU acts as
master during an FPGA-to-GPU data transfer, reading data from the FPGA.  This
solution limits the reported bandwidth and latency to 514 MB/s and 40~\textmu
s, respectively.

%LR: FPGA^2 it's the name of their thing... 
%MV: best idea in the world :)

When the FPGA is used as a master, a higher throughput can be achieved.  An
example of this approach is the \emph{FPGA\textsuperscript{2}} framework by Thoma
et~al.\cite{thoma}, which reaches 2454 MB/s using a 8x Gen2.0 data link.
Lonardo et~al.\ achieved low latencies with their NaNet design, an FPGA-based
PCIe network interface card~\cite{lonardo2015nanet}.  The Gbe link however
limits the latency performance of the system to a few tens of \textmu s. If
only the FPGA-to-GPU latency is considered, the measured values span between
1~\textmu s and 6~\textmu s, depending on the datagram size. Moreover, the
bandwidth saturates at 120 MB/s. Nieto et~al.\ presented a system based on a
PXIexpress data link that makes use of four PCIe 1.0
links~\cite{nieto2015high}. Their system (as limited by the interconnect)
achieves an average throughput of 870 MB/s with 1 KB block transfers.

In order to achieve the best performance in terms of latency and bandwidth, we
developed a high-performance DMA engine based on Xilinx's PCIe Gen3 Core.To
process the data, we encapsulated the DMA setup and memory mapping in a plugin
for our scalable GPU processing framework~\cite{vogelgesang2012ufo}. This
framework allows for an easy construction of streamed data processing on
heterogeneous multi-GPU systems. Because the framework is based on OpenCL,
integration with NVIDIA's CUDA functions for GPUDirect technology is not
possible at the moment. Thus, we used AMD's DirectGMA technology to integrate
direct FPGA-to-GPU communication into our processing pipeline. In this paper we
report the performance of our DMA engine for FPGA-to-CPU communication and some
preliminary measurements about DirectGMA's performance in low-latency
applications.


\section{Architecture}

As shown in \figref{fig:trad-vs-dgpu} (a), traditional FPGA-GPU systems route
data through system main memory by copying data from the FPGA into
intermediate buffers and then finally into the GPU's main memory. Thus, the
total throughput and latency of the system is limited by the main memory
bandwidth. NVIDIA's GPUDirect and AMD's DirectGMA technologies allow direct
communication between GPUs and auxiliary devices over PCIe. By combining this
technology with DMA data transfers (see \figref{fig:trad-vs-dgpu} (b)), the
overall latency of the system is reduced and total throughput increased.
Moreover, the CPU and main system memory are relieved from processing because
they are not directly involved in the data transfer anymore.

\begin{figure}[t]
  \centering
  \includegraphics[width=1.0\textwidth]{figures/transf}
  \caption{%
    In a traditional DMA architecture (a), data are first written to the main
    system memory and then sent to the GPUs for final processing.  By using
    GPUDirect/DirectGMA technology (b), the DMA engine has direct access to
    the GPU's internal memory.
  }
  \label{fig:trad-vs-dgpu}
\end{figure}

\subsection{DMA engine implementation on the FPGA}

We have developed a DMA architecture that minimizes resource utilization while
maintaining the flexibility of a Scatter-Gather memory
policy~\cite{rota2015dma}. The engine is compatible with the Xilinx PCIe
Gen2/3 IP-Core~\cite{xilinxgen3} for Xilinx FPGA families 6 and 7. DMA
transmissions to main system memory and GPU memory are both supported. Two
FIFOs, with a data width of 256 bits and operating at 250 MHz, act as user-
friendly interfaces with the custom logic with an input bandwidth of 7.45
GB/s. The user logic and the DMA engine are configured by the host through PIO
registers.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.75\textwidth]{figures/fpga-arch}
  \caption{%
    FPGA AAA
  }
  \label{fig:fpga-arch}
\end{figure}

The physical addresses of the host's memory buffers are stored into an internal
memory and are dynamically updated by the driver or user, allowing highly
efficient zero-copy data transfers. The maximum size associated with each
address is 2 GB. 

\subsection{OpenCL management on host side}
\label{sec:host}

On the host side, AMD's DirectGMA technology, an implementation of the bus-
addressable memory extension for OpenCL 1.1 and later, is used to write from
the FPGA to GPU memory and from the GPU to the FPGA's control registers.
\figref{fig:opencl-setup} illustrates the main mode of operation: to write
into the GPU, the physical bus addresses of the GPU buffers are determined
with a call to \texttt{clEnqueue\-Make\-Buffers\-Resident\-AMD} and set by the
host CPU in a control register of the FPGA (1). The FPGA then writes data
blocks autonomously in DMA fashion (2). Due to hardware restrictions the
largest possible GPU buffer sizes are about 95 MB but larger transfers can be
achieved by using a double buffering mechanism.

Because the GPU provides a flat memory address space and our DMA engine allows
multiple destination addresses to be set in advance, we can determine all
addresses before the actual transfers thus keeping the CPU out of the transfer
loop for data sizes less than 95 MB.

To signal events to the FPGA (4), the control registers can be mapped into the
GPU's address space passing a special AMD-specific flag and passing the
physical BAR address of the FPGA configuration memory to the
\texttt{cl\-Create\-Buffer} function. From the GPU, this memory is seen
transparently as regular GPU memory and can be written accordingly (3). In our
setup, trigger registers are used to notify the FPGA on successful or failed
evaluation of the data.

Using the \texttt{cl\-Enqueue\-Copy\-Buffer} function call it is possible to
write entire memory regions in DMA fashion to the FPGA. In this case, the GPU
acts as bus master and pushes data to the FPGA.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.75\textwidth]{figures/opencl-setup}
  \caption{The FPGA writes to GPU memory by mapping the physical address of a
  GPU buffer and initating DMA transfers. Signalling happens in reverse order by
  mapping the FPGA control registers into the address space of the GPU.}
  \label{fig:opencl-setup}
\end{figure}

To process the data, we encapsulated the DMA setup and memory mapping in a
plugin for our scalable GPU processing framework~\cite{vogelgesang2012ufo}.
This framework allows for an easy construction of streamed data processing on
heterogeneous multi-GPU systems. For example, to read data from the FPGA,
decode from its specific data format and run a Fourier transform on the GPU as
well as writing back the results to disk, one can run the following on the
command line:

\begin{verbatim}
ufo-launch direct-gma ! decode ! fft ! write filename=out.raw
\end{verbatim}

The framework takes care of scheduling the tasks and distributing the data
items to one or more GPUs. High throughput is achieved by the combination of
fine- and coarse-grained data parallelism, \emph{i.e.} processing a single
data item on a GPU using thousands of threads and by splitting the data stream
and feeding individual data items to separate GPUs. None of this requires any
user intervention and is solely determined by the framework in an automatized
fashion. A complementary application programming interface allows users to
develop custom applications written in C or high-level languages such as
Python.

\section{Results}

We carried out performance measurements on two different setups, described in
table~\ref{table:setups}. In Setup 2, a low-end Supermicro X7SPA-HF-D525
system was connected to a Netstor NA255A external PCIe enclosure. In both
cases, a Xilinx VC709 evaluation board was plugged into a PCIe 3.0 x8 slots.
In case of FPGA-to-CPU data transfers, the software implementation is the one
described in~\cite{rota2015dma}.

The resource utilization
on a Virtex 7 device is reported in \ref{table:utilization}.

\begin{table}[]
\centering
\caption{Resource utilization on a Virtex7 device X240VT}
\label{table:utilization}
\tabcolsep=0.11cm
\small
\begin{tabular}{@{}llll@{}}
  \toprule
Resource & Utilization & Available & Utilization \% \\
  \midrule
LUT      & 5331        & 433200    & 1.23           \\
LUTRAM   & 56          & 174200    & 0.03           \\
FF       & 5437        & 866400    & 0.63           \\
BRAM     & 20.50       & 1470      & 1.39           \\
  \bottomrule
\end{tabular}
\end{table}

\begin{table}[b]
\centering
\caption{Hardware used for throughput and latency measurements}
\label{table:setups}
\tabcolsep=0.11cm
\begin{tabular}{@{}llll@{}}
  \toprule
Component & Setup 1 & Setup 2 \\
  \midrule
CPU           & Intel Xeon E5-1630 at 3.7 GHz  & Intel Atom D525   \\
Chipset       & Intel C612                     & Intel ICH9R Express   \\
GPU           & AMD FirePro W9100              & AMD FirePro W9100   \\
PCIe link (FPGA-System memory)    & x8 Gen3                        & x4 Gen1     \\
PCIe link (FPGA-GPU)    & x8 Gen3                        & x8 Gen3     \\
  \bottomrule
\end{tabular}

\end{table}


\subsection{Throughput}

% We repeated the FPGA-to-GPU measurements on a low-end Supermicro X7SPA-HF-D525
% system based on an Intel Atom CPU. The results showed no significant difference
% compared to the previous setup. Depending on the application and computing
% requirements, this result makes smaller acquisition system a cost-effective
% alternative to larger workstations.

\begin{figure}[t]
  \includegraphics[width=0.85\textwidth]{figures/throughput}
  \caption{%
    Measured results for data transfers from FPGA to main memory
    (CPU) and from FPGA to the global GPU memory (GPU).
}
\label{fig:throughput}
\end{figure}

The measured results for the pure data throughput is shown in
\figref{fig:throughput} for transfers from the FPGA to the system's main
memory as well as to the global memory as explained in \ref{sec:host}. 
% Must ask Suren about this

In the case of FPGA-to-GPU data transfers, the double buffering solution was
used. As one can see, in both cases the write performance is primarily limited
by the PCIe bus. Up until 2 MB data transfer size, the throughput to the GPU
is approaching slowly 100 MB/s. From there on, the throughput increases up to
6.4 GB/s when PCIe bus saturation sets in at about 1 GB data size. The CPU
throughput saturates earlier but the maximum throughput is 6.6 GB/s.



% \begin{figure}
%   \includegraphics[width=\textwidth]{figures/intra-copy}
%   \caption{%
%     Throughput in MB/s for an intra-GPU data transfer of smaller block sizes
%     (4KB -- 24 MB) into a larger destination buffer (32 MB -- 128 MB). The lower
%     performance for smaller block sizes is caused by the larger amount of
%     transfers required to fill the destination buffer. The throughput has been
%     estimated using the host side wall clock time. The raw GPU data transfer as
%     measured per event profiling is about twice as fast.
%   }
%   \label{fig:intra-copy}
% \end{figure}

In order to write more than the maximum possible transfer size of 95 MB, we
repeatedly wrote to the same sized buffer which is not possible in a real-
world application. As a solution, we motivated the use of multiple copies in
Section \ref{sec:host}. To verify that we can keep up with the incoming data
throughput using this strategy, we measured the data throughput within a GPU
by copying data from a smaller sized buffer representing the DMA buffer to a
larger destination buffer. At a block size of about 384 KB the throughput
surpasses the maximum possible PCIe bandwidth, and it reaches 40 GB/s for
blocks bigger than 5 MB. Double buffering is therefore a viable solution for
very large data transfers, where throughput performance is favoured over
latency.

% \figref{fig:intra-copy} shows the measured throughput for
% three sizes and an increasing block size.


\subsection{Latency}
\begin{figure}[t]
  \centering
  \begin{subfigure}[b]{.45\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/latency}
    \caption{Latency }
    \label{fig:latency_vs_size}
  \end{subfigure}
  \begin{subfigure}[b]{.45\textwidth}
    \includegraphics[width=\textwidth]{figures/latency-hist}
    \caption{Latency distribution.}
    \label{fig:latency_hist}
  \end{subfigure}
  \label{fig:latency}
\end{figure}

For HEP experiments, low latencies are necessary to react in a reasonable time
frame. In order to measure the latency caused by the communication overhead we
conducted the following protocol: 1) the host issues continuous data transfers
of a 4 KB buffer that is initialized with a fixed value to the FPGA using the
\texttt{cl\-Enqueue\-Copy\-Buffer} call. 2) when the FPGA receives data in its
input FIFO it moves it directly to the output FIFO which feeds the outgoing DMA
engine thus pushing back the data to the GPU. 3) At some point, the host enables
generation of data different from initial value which also starts an internal
FPGA counter with 4 ns resolution. 4) When the generated data is received again
at the FPGA, the counter is stopped. 5) The host program reads out the counter
values and computes the round-trip latency. The distribution of 10000
measurements of the one-way latency is shown in \figref{fig:latency-hist}.
[\textbf{REWRITE THIS PART}] The GPU latency has a mean value of 84.38 \textmu s
and a standard variation of 6.34 \textmu s. This is 9.73 \% slower than the CPU
latency of 76.89 \textmu s that was measured using the same driver and measuring
procedure. The non-Gaussian distribution with two distinct peaks indicates a
systemic influence that we cannot control and is most likely caused by the
non-deterministic run-time behaviour of the operating system scheduler.

\section{Conclusion and outlook}

We developed a hardware and software solution that enables DMA transfers
between FPGA-based readout systems and GPU computing clusters. The software
solution that we proposed allows seamless multi-GPU processing of the incoming
data, due to the integration in our streamed computing framework. This allows
straightforward integration with different DAQ systems and introduction of
custom data processing algorithms.

The net throughput is primarily limited by the PCIe link, reaching 6.4 GB/s
for a FPGA-to-GPU data transfer and 6.6 GB/s for a FPGA-to-CPU data transfer.
By writing directly into GPU memory instead of routing data through system
main memory, the overall latency of the system can be reduced, thus allowing
close massively parallel computation on GPUs. Optimization of the GPU DMA
interfacing code is ongoing with the help of technical support by AMD. With a
better understanding of the hardware and  software aspects of DirectGMA, we
expect a significant improvement in the latency performance.

In order to increase the total throughput, a custom FPGA evaluation board is
currently under development. The board mounts a Virtex-7 chip and features two
fully populated FMC connectors, a 119 Gb/s DDR memory interface and a PCIe x16
Gen3 connection. Two x8 Gen3 cores, instantiated on the board, will be mapped
as a single x16 device by using an external PCIe switch. With two cores
operating in parallel, we foresee an increase in the data throughput by a
factor of 2 (as demonstrated in~\cite{rota2015dma}).

Support for NVIDIA's GPUDirect technology is also foreseen in the next months
to lift the limitation of one specific GPU vendor and compare the performance
of hardware by different vendors. Further improvements are expected by
generalizing the transfer mechanism and include Infiniband support besides the
existing PCIe connection.

%% Where do we get this values? Any reference?
%This allows
%speeds of up to 290 Gb/s and latencies as low as 0.5 \textmu s.

Our goal is to develop a unique hybrid solution, based on commercial standards,
that includes fast data transmission protocols and a high performance GPU
computing framework.


\acknowledgments

This work was partially supported by the German-Russian BMBF funding programme,
grant numbers 05K10CKB and 05K10VKE.


\bibliographystyle{JHEP}
\bibliography{literature}

\end{document}