People see you in a different light when they know you can write CUDA kernels.
But what exactly is a CUDA kernel?
Simply put, a kernel is a function that allows you to write data-parallel operations to be executed on a GPU.
If you want to maximize throughput and minimize latency on modern hardware like A100s or H100s, this is where the magic happens.
In this framework, the CPU is the βhostβ and the GPU is the βdevice.β
As you can see in the attached diagram, when you run a program, the sequential parts are executed on the CPU, while the heavy parallel instructions (the kernels) are offloaded to the GPU. This is known as a heterogeneous CPU/GPU architecture.
Here is how the GPU breaks down that parallel workload:
ππ‘π ππ«π’π
Every parallel execution on the GPU creates a βGrid.β You can think of the Grid as the overarching container for your parallel task.
ππ‘π πππ (ππ‘π«πππ ππ₯π¨ππ€)
The Grid is divided into multiple blocks called Cooperative Thread Arrays (CTAs). Each CTA executes independently on the hardware. However, threads inside the same CTA can share fast on-chip memory, which is why a single CTA is always assigned to a single hardware processor core.
ππ‘π ππ‘π«ππππ¬ & πππ«π©π¬
Grids and CTAs are software abstractions. At the hardware level, execution happens in βWarps.β In CUDA, a warp is a group of exactly 32 threads. The hardware scheduler follows a SIMT (Single Instruction, Multiple Threads) model. This means all 32 threads in a warp execute the exact same instruction at the exact same time, just on different pieces of scalar data.
Iβm going to be sharing more technical breakdowns like this as I continue diving deeper into parallel programming architectures.
Let me know in the comments: whatβs the hardest concept youβve encountered when learning GPU architecture?
Image credits: NVIDIA TESLA: A UNIFIED GRAPHICS AND COMPUTING ARCHITECTURE
References
NVIDIA TESLA: A UNIFIED GRAPHICS AND COMPUTING ARCHITECTURE
Scalable Parallel Programming with CUDA
Programming Massively Parallel Processes