Difficulty: Advanced

Reading Time: 35 min read

Last Updated: November 3, 2025

Characteristics of Parallel Computing

Parallel computing refers to the coordinated use of multiple computing resources—such as CPUs, cores, or GPUs—to execute tasks simultaneously. Instead of solving a problem sequentially, it divides the workload into smaller, independent subtasks that can be processed concurrently and later combined to produce the final result. This model relies on concurrency to enable several computations to progress at once, improving overall throughput and responsiveness. It employs decomposition to break complex problems into manageable units, while communication between tasks—through shared memory or message passing—ensures proper data exchange and coordination. To maintain correctness, synchronization mechanisms such as locks, barriers, or semaphores regulate the timing and order of execution. Together, these principles form the foundation of parallel computing, enabling modern systems to achieve higher performance, scalability, and efficiency across diverse workloads.

1. Differences between concurrency, Parallelism, and synchronization

1.1 Concurrency

Concurrency means dealing with multiple tasks at once, but not necessarily running at the same time.

• It's about structuring a program to handle multiple things independently at the same period, as if they're happening simultaneously.
• Often used when tasks are interleaved on the same CPU core (like multitasking).
• It's about designing a system that can handle multiple events or operations without waiting for one to completely finish before starting another.
• Concurrency = Logical simultaneity

Analogy

• You’re cooking and answering messages — not doing both at the exact same time, but switching between them quickly.

1.2 Parallelism

Parallelism means running multiple tasks at the exact same time, on multiple CPU cores or processors.