Accelerating GPU Processing for Deep Learning with Synchronous Data Parallelism

In this article, we present three scenarios for measuring the performance of high-performance computing (HPC) systems. Scenario 1 involves executing a target program for 32 consecutive iterations or until a minimum runtime of 5 seconds is reached. Scenario 2 entails conducting four separate trials with randomized delays between each trial, while post-processing the collected data to remove repetitions occurring during the rise time of the GPU. Scenario 3 proposes executing the target program for a fixed number of iterations and measuring the average performance over time.
To ensure accurate measurements, we recommend implementing controlled delays within each scenario to account for any variations in the system’s activity. Additionally, we suggest discarding repetitions that occur during the rise time of the GPU to prevent data loss due to small averaging windows.
By following these good practices, HPC practitioners can obtain more reliable and accurate measurements of their systems’ performance, which is crucial for optimizing system configuration, evaluating hardware upgrades, and troubleshooting issues.
To better understand these concepts, consider a racecar driver who wants to determine the optimal number of laps around a track to achieve the fastest time. The driver must account for factors like the car’s acceleration, braking, and handling, which can affect performance. By executing multiple laps with controlled delays between each one and discarding any laps where the car is not at its fastest, the driver can gather more accurate data to determine the optimal number of laps for the best time.
In conclusion, measuring the performance of HPC systems involves implementing controlled delays and post-processing the collected data to ensure accuracy. By following these good practices, HPC practitioners can obtain reliable measurements that help them optimize system configuration, evaluate hardware upgrades, and troubleshoot issues more efficiently.