Memory
如何執行 GPGPU 記憶體測試
我們使用了大量的 GPGPU 計算（主要使用 CUDA，但也有一些 OpenCL）。通常，當使用者執行程式碼時，程式碼僅在我們的一台主機上出現記憶體錯誤。我懷疑其中一張卡有問題。有時它會破壞整個系統，有時程序會崩潰。
有哪些最簡單、最快和最徹底的方法可以全面測試 GPU 是否可能出現故障？
我知道有些程序是 nvidia 的 CUDA SDK 的一部分：
deviceQuery nvidia-smi但我需要更徹底的東西。建議？經驗？
事實上的標準似乎是CUDA GPU Memtest。正如@c2h5oh 所提到的，它看起來像是基於 memtest86 測試模式，所以我確信它做得很好。它在我正在測試的高端 GPU 上執行相對較快（在 Quadro 6000 上執行 30 分鐘，在 Tesla C2075 上執行 20 分鐘）。它在作業系統內部執行（與 memtest 不同），因此監控有點不同。您可能希望將 stdout 和 stderr 輸出到文件以供稍後查看。因此，如果您失去終端輸出，請考慮像這樣執行它，您可以查看測試發現的內容：
cuda_memtest 2>cuda_memtest.stderr 1>cuda_memtest.stdout & tail -f cuda_memtest.stdout & tail -f cuda_memtest.stderr &您還需要確保沒有人在使用系統和/或卡。您可以使用以下方法將 GPU 設置為獨占模式：
nvidia-smi --compute-mode=EXCLUSIVE_PROCESS如果您對給出的測試資訊感興趣，以下是 Quadro 和 Tesla 樣本執行的一些輸出：
[09/07/2012 11:56:22][hydro][0]:Running cuda memtest, version 1.2.2 [09/07/2012 11:56:23][hydro][0]:Warning: Getting serial number failed [09/07/2012 11:56:23][hydro][0]:NVRM version: NVIDIA UNIX x86_64 Kernel Module 295.41 Fri Apr 6 23:18:58 PDT 2012 [09/07/2012 11:56:23][hydro][0]:num_gpus=1 [09/07/2012 11:56:23][hydro][0]:Device name=Quadro 6000, global memory size=6441992192 [09/07/2012 11:56:23][hydro][0]:major=2, minor=0 [09/07/2012 11:56:24][hydro][0]:Attached to device 0 successfully. [09/07/2012 11:56:24][hydro][0]:Allocated 6040 MB [09/07/2012 11:56:24][hydro][0]:Test0 [Walking 1 bit] [09/07/2012 11:56:30][hydro][0]:Test0 finished in 5.7 seconds [09/07/2012 11:56:30][hydro][0]:Test1 [Own address test] [09/07/2012 11:56:33][hydro][0]:Test1 finished in 3.5 seconds [09/07/2012 11:56:33][hydro][0]:Test2 [Moving inversions, ones&zeros] [09/07/2012 11:57:05][hydro][0]:Test2 finished in 32.3 seconds [09/07/2012 11:57:05][hydro][0]:Test3 [Moving inversions, 8 bit pat] [09/07/2012 11:57:37][hydro][0]:Test3 finished in 31.9 seconds [09/07/2012 11:57:37][hydro][0]:Test4 [Moving inversions, random pattern] [09/07/2012 11:57:53][hydro][0]:Test4 finished in 15.9 seconds [09/07/2012 11:57:53][hydro][0]:Test5 [Block move, 64 moves] [09/07/2012 11:57:59][hydro][0]:Test5 finished in 6.3 seconds [09/07/2012 11:57:59][hydro][0]:Test6 [Moving inversions, 32 bit pat] [09/07/2012 12:18:46][hydro][0]:Test6 finished in 1246.6 seconds [09/07/2012 12:18:46][hydro][0]:Test7 [Random number sequence] [09/07/2012 12:19:06][hydro][0]:Test7 finished in 19.8 seconds [09/07/2012 12:19:06][hydro][0]:Test8 [Modulo 20, random pattern] [09/07/2012 12:19:06][hydro][0]:test8[mod test]: p1=0x13472f5f, p2=0xecb8d0a0 [09/07/2012 12:20:34][hydro][0]:Test8 finished in 88.0 seconds [09/07/2012 12:20:34][hydro][0]:Test10 [Memory stress test] [09/07/2012 12:20:34][hydro][0]:Test10 with pattern=0x55f6c69858704128 [09/07/2012 12:21:11][hydro][0]:Test10 finished in 36.8 seconds [09/07/2012 12:21:11][hydro][0]:Test0 [Walking 1 bit] [09/07/2012 12:21:16][hydro][0]:Test0 finished in 5.8 seconds [09/06/2012 18:49:07][hydro][0]:Running cuda memtest, version 1.2.2 [09/06/2012 18:49:10][hydro][0]:Warning: Getting serial number failed [09/06/2012 18:49:10][hydro][0]:NVRM version: NVIDIA UNIX x86_64 Kernel Module 295.41 Fri Apr 6 23:18:58 PDT 2012 [09/06/2012 18:49:10][hydro][0]:num_gpus=1 [09/06/2012 18:49:10][hydro][0]:Device name=Tesla C2075, global memory size=5636292608 [09/06/2012 18:49:10][hydro][0]:major=2, minor=0 [09/06/2012 18:49:11][hydro][0]:Attached to device 0 successfully. [09/06/2012 18:49:11][hydro][0]:Allocated 5273 MB [09/06/2012 18:49:11][hydro][0]:Test0 [Walking 1 bit] [09/06/2012 18:49:22][hydro][0]:Test0 finished in 11.1 seconds [09/06/2012 18:49:22][hydro][0]:Test1 [Own address test] [09/06/2012 18:49:25][hydro][0]:Test1 finished in 3.1 seconds [09/06/2012 18:49:25][hydro][0]:Test2 [Moving inversions, ones&zeros] [09/06/2012 18:49:52][hydro][0]:Test2 finished in 27.4 seconds [09/06/2012 18:49:52][hydro][0]:Test3 [Moving inversions, 8 bit pat] [09/06/2012 18:50:20][hydro][0]:Test3 finished in 27.9 seconds [09/06/2012 18:50:20][hydro][0]:Test4 [Moving inversions, random pattern] [09/06/2012 18:50:34][hydro][0]:Test4 finished in 13.7 seconds [09/06/2012 18:50:34][hydro][0]:Test5 [Block move, 64 moves] [09/06/2012 18:50:39][hydro][0]:Test5 finished in 5.5 seconds [09/06/2012 18:50:39][hydro][0]:Test6 [Moving inversions, 32 bit pat] [09/06/2012 19:08:34][hydro][0]:Test6 finished in 1074.9 seconds [09/06/2012 19:08:34][hydro][0]:Test7 [Random number sequence] [09/06/2012 19:08:51][hydro][0]:Test7 finished in 17.1 seconds [09/06/2012 19:08:51][hydro][0]:Test8 [Modulo 20, random pattern] [09/06/2012 19:08:51][hydro][0]:test8[mod test]: p1=0x63136646, p2=0x9cec99b9 [09/06/2012 19:10:10][hydro][0]:Test8 finished in 78.4 seconds [09/06/2012 19:10:10][hydro][0]:Test10 [Memory stress test] [09/06/2012 19:10:10][hydro][0]:Test10 with pattern=0x26341d134a89ac2b [09/06/2012 19:10:39][hydro][0]:Test10 finished in 29.0 seconds