Ubuntu

AWS i3en.3xlarge 非常低的 iops

  • February 1, 2022

我剛剛啟動了一個 i3en.3xlarge 類型的新實例 ec2 實例。作業系統是 Ubuntu。我安裝了 NVMe 實例儲存，但我執行的每個速度測試都非常低，大約 7k iops。我究竟做錯了什麼？

以下是我執行的步驟：

1) 使用 nvme -list 檢查可用的 ssd：

---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     vol012301587a8724842 Amazon Elastic Block Store               1           8.59  GB /   8.59  GB    512   B +  0 B   1.0     
/dev/nvme1n1     AWS16AAAC6C7BFAC4972 Amazon EC2 NVMe Instance Storage         1           7.50  TB /   7.50  TB    512   B +  0 B   0

2）為nvme1n1創建一個新的xfs文件系統：

sudo mkfs -t xfs /dev/nvme1n1

3) 掛載到 /home

sudo mount /dev/nvme1n1 /home

4）檢查df -h：

   ubuntu@ip-172-31-35-146:/home$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.7G  2.8G  4.9G  37% /
devtmpfs         47G     0   47G   0% /dev
tmpfs            47G     0   47G   0% /dev/shm
tmpfs           9.4G  852K  9.4G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            47G     0   47G   0% /sys/fs/cgroup
/dev/loop0       25M   25M     0 100% /snap/amazon-ssm-agent/4046
/dev/loop3       43M   43M     0 100% /snap/snapd/14066
/dev/loop2       68M   68M     0 100% /snap/lxd/21835
/dev/loop1       56M   56M     0 100% /snap/core18/2284
/dev/loop4       62M   62M     0 100% /snap/core20/1242
/dev/loop6       56M   56M     0 100% /snap/core18/2253
/dev/loop5       44M   44M     0 100% /snap/snapd/14549
/dev/loop7       62M   62M     0 100% /snap/core20/1328
tmpfs           9.4G     0  9.4G   0% /run/user/1000
/dev/nvme1n1    6.9T   49G  6.8T   1% /home

5) 使用 fio 執行測試：

fio -direct=1 -iodepth=1 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=iotest -name=Rand_Read_Testing

菲奧結果：

fio-3.16
Starting 1 process
Rand_Read_Testing: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=28.5MiB/s][r=7297 IOPS][eta 00m:00s]
Rand_Read_Testing: (groupid=0, jobs=1): err= 0: pid=1701: Sat Jan 29 22:28:17 2022
 read: IOPS=7139, BW=27.9MiB/s (29.2MB/s)(1024MiB/36717msec)
   slat (nsec): min=2301, max=39139, avg=2448.98, stdev=311.68
   clat (usec): min=32, max=677, avg=137.06, stdev=26.98
    lat (usec): min=35, max=680, avg=139.59, stdev=26.99
   clat percentiles (usec):
    |  1.00th=[   35],  5.00th=[   99], 10.00th=[  100], 20.00th=[  124],
    | 30.00th=[  125], 40.00th=[  126], 50.00th=[  139], 60.00th=[  141],
    | 70.00th=[  165], 80.00th=[  167], 90.00th=[  169], 95.00th=[  169],
    | 99.00th=[  172], 99.50th=[  174], 99.90th=[  212], 99.95th=[  281],
    | 99.99th=[  453]
  bw (  KiB/s): min=28040, max=31152, per=99.82%, avg=28506.48, stdev=367.13, samples=73
  iops        : min= 7010, max= 7788, avg=7126.59, stdev=91.80, samples=73
 lat (usec)   : 50=1.29%, 100=9.46%, 250=89.19%, 500=0.06%, 750=0.01%
 cpu          : usr=1.43%, sys=2.94%, ctx=262144, majf=0, minf=12
 IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  READ: bw=27.9MiB/s (29.2MB/s), 27.9MiB/s-27.9MiB/s (29.2MB/s-29.2MB/s), io=1024MiB (1074MB), run=36717-36717msec

Disk stats (read/write):
 nvme1n1: ios=259894/5, merge=0/3, ticks=35404/0, in_queue=35404, util=99.77%

根據像這裡這樣的基準，iops 性能應該會更好。

那麼我在這裡錯過了什麼嗎？

提前致謝

所以我跨越了這些實例之一來測試自己。我的步驟只是有點不同：

  1. 首先使用分區磁碟parted
  2. 製作文件系統
  3. Mount at /optas/home已經存在，並且我的使用者的主目錄位於 ( ubuntu) 中。
  4. apt update && apt upgrade，然後安裝fio
  5. 執行與您相同的命令：fio -direct=1 -iodepth=1 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=iotest -name=Rand_Read_Testingfrom inside /opt， with sudo。

我得到了類似的結果，與read: IOPS=7147。

然後我進行了另一個測試：

/opt$ sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=332MiB/s,w=109MiB/s][r=85.1k,w=28.0k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=26470: Mon Jan 31 09:14:45 2022
 read: IOPS=91.5k, BW=357MiB/s (375MB/s)(6141MiB/17187msec)
  bw (  KiB/s): min=339568, max=509896, per=100.00%, avg=366195.29, stdev=59791.96, samples=34
  iops        : min=84892, max=127474, avg=91548.82, stdev=14947.99, samples=34
 write: IOPS=30.5k, BW=119MiB/s (125MB/s)(2051MiB/17187msec); 0 zone resets
  bw (  KiB/s): min=111264, max=170424, per=100.00%, avg=122280.71, stdev=20225.33, samples=34
  iops        : min=27816, max=42606, avg=30570.18, stdev=5056.32, samples=34
 cpu          : usr=19.73%, sys=41.60%, ctx=742611, majf=0, minf=8
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
    issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: bw=357MiB/s (375MB/s), 357MiB/s-357MiB/s (375MB/s-375MB/s), io=6141MiB (6440MB), run=17187-17187msec
 WRITE: bw=119MiB/s (125MB/s), 119MiB/s-119MiB/s (125MB/s-125MB/s), io=2051MiB (2150MB), run=17187-17187msec

Disk stats (read/write):
 nvme1n1: ios=1563986/522310, merge=0/0, ticks=927244/24031, in_queue=951275, util=99.46%

…看起來好多了 - read: IOPS=91.5k。

我懷疑這是由於只讀測試的工作原理？或者讀取您正在使用的磁碟的一些細微差別，以及其他一些限制？

我又進行了幾次測試，每次都得到相似的結果。

然後，我使用此處的命令執行了另一個只讀測試，並得到了這個：

/opt$ sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randread
fiotest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=332MiB/s][r=85.1k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=26503: Mon Jan 31 09:17:57 2022
 read: IOPS=88.6k, BW=346MiB/s (363MB/s)(8192MiB/23663msec)
  bw (  KiB/s): min=339560, max=787720, per=100.00%, avg=354565.45, stdev=72963.81, samples=47
  iops        : min=84890, max=196930, avg=88641.40, stdev=18240.94, samples=47
 cpu          : usr=15.37%, sys=31.05%, ctx=844523, majf=0, minf=72
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
    issued rwts: total=2097152,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=8192MiB (8590MB), run=23663-23663msec

Disk stats (read/write):
 nvme1n1: ios=2095751/1, merge=0/0, ticks=1468160/0, in_queue=1468159, util=99.64%

更好的讀取性能。我懷疑您給出命令的參數不允許測試從磁碟獲得最佳性能，可能是由於塊大小、文件大小等原因。我確實注意到它們都是單虛線參數（例如-bs=4k）而不是雙（--bs=4k） ，所以他們甚至可能沒有被正確解析……

引用自：https://serverfault.com/questions/1091607

相關問答

Ubuntu

EC2 實例安裝了 apache2 伺服器，但我無法連接到它

  • October 23, 2021
檢視問答
Ubuntu

無法通過 ssh 連接到 aws ec2 實例，但以前我 100% 能夠

  • August 20, 2020
檢視問答
Ubuntu

哪個 EC2 實例散列大量 25gb 文件

  • August 13, 2020
檢視問答
Ubuntu

無法登錄到 aws ec2 實例

  • July 22, 2020
檢視問答
Ubuntu

如何在 ubuntu 中跟踪導致 postgres 在 Amazon EC2 容器中佔用 100% CPU 使用率的幽靈程序的來源？

  • February 8, 2020
檢視問答
Ubuntu

如果流量通過 CloudFront 路由，網站是否需要在 nginx 文件中配置埠 443？

  • December 27, 2019
檢視問答