GPGPU stands for General-purpose computing on graphics processing units. It is the use of a graphics processing unit (GPU), which typically handles computation only for computer graphics, to perform computation in applications traditionally handled by the central processing unit (CPU).
Simply put, it's a kind of parallel processing where we're trying to exploit the data-parallel hardware on GPUs to improve the throughput of our computers.
Moore's law is the observation made by Gordon Moore that the density of transistors in an integrated circuit board doubles approximately every two years. It has long been co-opted by the semiconductor industry as a target, and consumers have taken this growth for granted.
Because it suggests exponential growth, it is unsustainable and it cannot be expected to continue indefinitely. In the words of Moore himself, "It can't continue forever.". There are hard physical limits to this scaling such as heat dissipation rate and size of microprocessor features.
As software engineers, this means that free and regular performance gains can no longer be expected. To fully exploit CPU throughput gains, we need to code differently.
Between CPUs and GPUs, there are differences in scale and architecture.
This means that CPUs and GPUs excel at different tasks. You'll typically want to utilize GPGPU on tasks that are data parallel and compute intensive (e.g. graphics, matrix operations).
Data parallelism refers to how a processor executes the same operation on different data elements simultaneously.
Compute intensive refers to how the algorithm will have to process lots of data elements.
Not all problems are inherently parallelizable.
The SIMT (Single Instruction, Multiple Threads) architecture of GPUs means that they don't handle branches and inter-thread communication well.
The benefits of GPGPU are even more pronounced when it comes to embedded systems and Internet of Things (IoT) applications, where computing power is often compromised for physical space, weight, and power consumption. For instance, NVIDIA's Jetson TX2, a computation processor board that delivers the processing capability of the Pascal GPU architecture in a package the size of a business card. The Pascal architecture is used in many desktop computers, data centres, and supercomputers. Along with its small form factor, this makes the Jetson ideal for embedded systems that require intensive processing power.
Packed with a NVIDIA Pascal GPU with 256 CUDA cores at maximally 1300 MHz, the TX2 is capable of intensive parallel computational tasks such as real-time vision processing or deploying deep neural networks, allowing mobile platforms to solve complex, real-world problems.
Compatible with a range of components and large ecosystem products, the Jetson platform is now widely used for deploying vision and neural network processing onboard mobile platforms like self-driving cars, autonomous robots, drones, IoT, handheld medical devices, etc.