Kernel Optimization

Transferring data in bursts hides the memory access latency and improves bandwidth usage and efficiency of the memory controller.

If burst data transfers occur, the detailed kernel trace will reflect the higher burst rate as a larger burst length number:

Figure:Burst Data Transfer with Detailed Kernel Trace

In the previous figure, it is also possible to observe that the memory data transfers following the AXI interconnect are actually implemented rather differently (shorter transaction time). Hover over these transactions, you would see that the AXI interconnect has packed the 16x4 Byte transaction into a single package transaction of 1x64 Bytes. This effectively uses theAXI4bandwidth which is even more favorable. The next section focuses on this optimization technique in more detail.

Burst inference is heavily dependent on coding style and access pattern. To avoid potential modeling pitfalls, refer to theSDAccel Environment Programmers Guide(UG1277). However, you can ease burst detection and improve performance by isolating data transfer and computation, as shown in the following code snippet:

void kernel(T in[1024], T out[1024]) { T tmpIn[1024]; T tmpOu[1024]; read(in, tmpIn); process(tmpIn, tmpOut); write(tmpOut, out); }

In short, the functionreadis responsible for reading from the AXI input to an internal variable(tmpIn). The computation is implemented by the functionprocessworking on the internal variablestmpInandtmpOut. The functionwritetakes the produced output and writes to the AXI output.

The isolation of the read and write function from the computation results in:

• Simple control structures (loops) in the read/write function which makes burst detection simpler.
• The isolation of the computational function away from the AXI interfaces, simplifies potential kernel optimization. See theKernel Optimizationchapter for more information.
• The internal variables are mapped to on-chip memory, which allow faster access compared to AXI transactions. Acceleration platforms supported inSDAccelenvironment can have as much as 10 MB on-chip memories that can be used as pipes, local memories, and private memories. Using these resources effectively can greatly improve the efficiency and performance of your applications.

The user data width between the kernel and the memory controller can be configured by theSDAccelcompiler based on the data types of the kernel arguments. To maximize the data throughput,Xilinxrecommends that you choose data types map to the full data width on the memory controller. The memory controller in all supported acceleration cards supports 512-bit user interface, which can be mapped toOpenCL™vector data types, such asint16or C/C++ arbitrary precision data typeap_int<512>.

As shown on the following figure, you can observe burst AXI transactions (Burst Length 16) and a 512 bit package size (Burst Size 64 Bytes).

Figure:Burst AXI Transactions

This example shows good interface configuration as it maximizes AXI data width as well as it shows actual burst transactions.

Complex structs or classes, used to declare interfaces, can lead to very complex hardware interfaces due to memory layout and data packing differences. This can introduce potential issues that are very difficult to debug in a complex system.

TheOpenCLAPI 2.0 specification introduces a new memory object called a pipe. A pipe stores data organized as a FIFO. Pipe objects can only be accessed using built-in functions that read from and write to a pipe. Pipe objects are not accessible from the host. Pipes can be used to stream data from one kernel to another inside the FPGA without having to use the external memory, which greatly improves the overall system latency.

In theSDAcceldevelopment environment, pipes must be statically defined outside of all kernel functions. Dynamic pipe allocation using theOpenCL 2.x clCreatePipeAPI is not currently supported. The depth of a pipe must be specified by using the xcl_reqd_pipe_depth attribute in the pipe declaration.

The valid depth values are as follows:

• 16
• 32
• 64
• 128
• 256
• 512
• 1024
• 2048
• 4096
• 8192
• 16384
• 32768

A given pipe can have one and only one producer and consumer in different kernels.

pipe int p0 __attribute__((xcl_reqd_pipe_depth(32)));

Pipes can be accessed using standardOpenCLread_pipe()andwrite_pipe()built-in functions in non-blocking mode or using theXilinxextendedread_pipe_block()andwrite_pipe_block()functions in blocking mode. The status of pipes can be queried usingOpenCLget_pipe_num_packets()andget_pipe_max_packets()built-in functions. See theOpenCLC Specification, Version 2.0 from Khronos Group for more details on these built-in functions.

The following function signatures are the currently supported pipe functions, wheregentypeindicates the built-inOpenCLC scalar integer or floating-point data types.

int read_pipe_block (pipe gentype p, gentype *ptr) int write_pipe_block (pipe gentype p, const gentype *ptr)

The following “Blocking Pipes Example” fromSDAccel Getting Started Exampleson GitHub uses pipes to pass data from one processing stage to the next usingblocking read_pipe_block()andwrite_pipe_block()functions:

pipe int p0 __attribute__((xcl_reqd_pipe_depth(32))); pipe int p1 __attribute__((xcl_reqd_pipe_depth(32))); // Input Stage Kernel : Read Data from Global Memory and write into Pipe P0 kernel __attribute__ ((reqd_work_group_size(1, 1, 1))) void input_stage(__global int *input, int size) { __attribute__((xcl_pipeline_loop)) mem_rd: for (int i = 0 ; i < size ; i++) { //blocking Write command to pipe P0 write_pipe_block(p0, &input[i]); } } // Adder Stage Kernel: Read Input data from Pipe P0 and write the result // into Pipe P1 kernel __attribute__ ((reqd_work_group_size(1, 1, 1))) void adder_stage(int inc, int size) { __attribute__((xcl_pipeline_loop)) execute: for(int i = 0 ; i < size ; i++) { int input_data, output_data; //blocking read command to Pipe P0 read_pipe_block(p0, &input_data); output_data = input_data + inc; //blocking write command to Pipe P1 write_pipe_block(p1, &output_data); } } // Output Stage Kernel: Read result from Pipe P1 and write the result to Global // Memory kernel __attribute__ ((reqd_work_group_size(1, 1, 1))) void output_stage(__global int *output, int size) { __attribute__((xcl_pipeline_loop)) mem_wr: for (int i = 0 ; i < size ; i++) { //blocking read command to Pipe P1 read_pipe_block(p1, &output[i]); } }

The Device Traceline view shows the detailed activities and stalls on theOpenCLpipes after hardware emulation is run. This info can be used to choose the correct FIFO sizes to achieve the optimal application area and performance.

Figure:Device Traceline View

To leverage computational parallelism during the implementation of an algorithm on the FPGA, it should be mentioned that the synthesis tool will need to be able to recognize computational parallelism from the source code first. Loops and functions are prime candidates for reflecting computational parallelism and compute units in the source description. However, even in this case, it is key to verify that the implementation takes advantage of the computational parallelism as in some cases theSDxtool might not be able to apply the desired transformation due to the structure of the source code.

It is quite common, that some computational parallelism might not be reflected in the source code to begin with. In this case, it will need to be added. A typical example is a kernel that might be described to operate on a single input value, while the FPGA implementation might execute computations more efficiently in parallel on multiple values. This kind of parallel modeling is described in theUsing Full AXI Data Widthsection. A 512-bit interface can be created usingOpenCLvector data types such asint16or C/C++ arbitrary precision data typeap_int<512>.

Loops are the basic C/C++/OpenCL™API method of representing repetitive algorithmic code. The following example illustrates various implementation aspects of a loop structure:

for(int i = 0; i<255; i++) { out[i] = in[i]+in[i+1]; } out[255] = in[255];

This code iterates over an array of values and adds consecutive values, except the last value. If this loop is implemented as written, each loop iteration requires two cycles for implementation, which results in a total of 510 cycles for implementation. This can be analyzed in detail through the Schedule Viewer in the HLS Project:

Figure:Implemented Loop Structure in Schedule Viewer

This can also be analyzed in terms of total numbers and latency through theVivadosynthesis results:

Figure:Synthesis Results Performance Estimates

The key numbers here are the latency numbers and total LUT usage. For example, depending on the configuration, you could get latency of 511 and total LUT usage of 47. As you will see, these values can widely vary based on the implementation choices. While this implementation will require very little area, it results in significant latency.

Task parallelism allows you to take advantage of data flow parallelism. In contrast to loop parallelism, when task parallelism is deployed, full execution units (tasks) are allowed to operate in parallel taking advantage of extra buffering introduced between the tasks.

Look at the following example:

void run (ap_uint<16> in[1024], ap_uint<16> out[1024] ) { ap_uint<16> tmp[128]; for(int i = 0; i<8; i++) { processA(&(in[i*128]), tmp); processB(tmp, &(out[i*128])); } }

When this code is executed, the functionprocessAandprocessBare executed sequentially 128 times in a row. Given the combined latency forprocessAandprocessBin the loop is 278, the total latency can be estimated as:

Figure:Performance Estimates

The extra cycle is due to loop setup and can be observed in the Schedule Viewer.

For C/C++ code, Task Parallelism is performed by adding the DATAFLOW pragma into the for-loop:

#pragma HLS DATAFLOW

__attribute__ ((xcl_dataflow))

Refer toSDx Pragma Reference GuideandSDAccel Environment Programmers Guidefor more details regarding the specifics and limitations of these modifiers.

As illustrated by the estimates in the HLS Report, applying the transformation will considerably improve the overall performance effectively using a double (ping pong) buffer scheme between the tasks:

Figure:Performances Estimates

The overall latency of the design has almost halved in this case due to concurrent execution of the different tasks of the different iterations. Given the 139 cycles per processing function and the full overlap of the 128 iterations, this allows the total latency to be:

(1x only processA + 127x both processes + 1x only processB) * 139 cycles = 17931 cycles

Using task parallelism is a very powerful way improve performance when it comes to implementation. However, the effectiveness of applying the DATAFLOW pragma to a specific and arbitrary piece of code might vary vastly. The coding guidelines for applying DATAFLOW effectively are provided inSDx Pragma Reference GuideandSDAccel Environment Programmers Guide. However, it is often necessary to actually look at the execution pattern of the individual tasks to understand the final implementation of the DATAFLOW pragma. Towards that end, theSDAccelenvironment provides the Detailed Kernel Trace, which nicely illustrates concurrent execution.

Figure:Detailed Kernel Trace

For this Detailed Kernel Trace, the tool displays the start of the dataflowed loop, as shown in the previous figure. It illustrates how processA is starting up right away with the beginning of the loop, while processB waits until the completion of the processA before it can start up its first iteration. However, while processB completes the first iteration of the loop, processA begins operating on the second iteration and so forth.

A more abstract representation of this information is presented in the Application Timeline (Host & Device) and Device Hardware Transaction View (device-only during hardware emulation).