Dense Non-Pyramidal LK Optical Flow

Optical flow is the pattern of apparent motion of image objects between two consecutive frames, caused by the movement of object or camera. It is a 2D vector field, where each vector is a displacement vector showing the movement of points from first frame to second.

Optical Flow works on the following assumptions:

  • Pixel intensities of an object do not have too many variations in consecutive frames
  • Neighboring pixels have similar motion
Consider a pixel I(x, y, t) in first frame. (Note that a new dimension, time, is added here. When working with images only, there is no need of time). The pixel moves by distance (dx, dy) in the next frame taken after time dt. Thus, since those pixels are the same and the intensity does not change, the following is true:

Taking the Taylor series approximation on the right-hand side, removing common terms, and dividing by dt gives the following equation:

Where,,and.

The above equation is called the Optical Flow equation, wheret, fxand fyare the image gradientsand ftis the gradient along time. However, (u, v) is unknown. It is not possible to solve this equation with two unknown variables. Thus, several methods are provided to solve this problem. One method is Lucas-Kanade. Previously it was assumed that all neighboring pixels have similar motion. The Lucas-Kanade method takes a patch around the point, whose size can be defined through the ‘WINDOW_SIZE’ template parameter. Thus, all the points in that patch have the same motion. It is possible to find ( fx, fy, ft) for these points. Thus, the problem now becomes solving ‘WINDOW_SIZE * WINDOW_SIZE’ equations with two unknown variables,which is over-determined. A better solution is obtained with the “least square fit” method. Below is the final solution, which is a problem with two equations and two unknowns:

API Syntax

template void xFDenseNonPyrLKOpticalFlow (xF::Mat & frame0, xF::Mat & frame1, xF::Mat & flowx, xF::Mat & flowy)

Parameter Descriptions

The following table describes the template and the function parameters.

Table 1.xFDenseNonPyrLKOpticalFlow Function Parameter Descriptions
Parameter Description
Type pixel type. The current supported pixel value is XF_8UC1, unsigned 8 bit.
ROWS Maximum number of rows of the input image that the hardware kernel must be built for.
COLS Maximum number of columns of the input image that the hardware kernel must be built for.
NPC Number of pixels to process per cycle. Supported values are XF_NPPC1 (=1) and XF_NPPC2(=2).
WINDOW_SIZE Window size over which optical flow will be computed. This can be any odd positive integer.
frame0 First input images.
frame1 Second input image. Optical flow is computed between frame0 and frame1.
flowx Horizontal component of the flow vectors. The format of the flow vectors is XF_32FC1 or single precision.
flowy Vertical component of the flow vectors. The format of the flow vectors is XF_32FC1 or single precision.

Resource Utilization

The following table summarizes the resource utilization of xFDenseNonPyrLKOpticalFlow for a 4K image, as generated in theVivado HLS 2017.1version tool for the Xilinx Xczu9eg-ffvb1156-1-i-es1 FPGA at 300 MHz.

Table 2.xFDenseNonPyrLKOpticalFlow Function Resource Utilization Summary
Operating Mode

Operating Frequency

(MHz)

 Utilization Estimate
BRAM_18K DSP_48Es FF LUTs
1 pixel 300 182 44 25336 21603
2-pixel 300 264 82 25740 17216

Performance Estimate

The following table summarizes performance estimates of the xFDenseNonPyrLKOpticalFlow function for a 4K image, generated usingVivado HLS 2017.1version tool for the Xilinx xczu9eg-ffvb1156-1-i-es1 FPGA.

Table 3.xFDenseNonPyrLKOpticalFlow Function Performance Estimate Summary
Operating Mode

Operating Frequency

(MHz)

 Latency Estimate
Max (ms)
1 pixel 300 28.01
2-pixel 300 14.01