chchang/hugo_backup
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

update some content

Browse Source
This commit is contained in:
Eric Chang
2021-10-28 14:13:06 +08:00
parent 97eb780aa4
commit 4f99455d8f
242 changed files with 1826 additions and 2616 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
public/post/10g-lab-using-proxmox-and-mellanox/index.html
View File

@@ -11,9 +11,9 @@
"@id": "https:\/\/h.cowbay.org"
},
"articleSection" : "post",
"name" : "[筆記] 用 proxmox \x26 Mellanox SFP 網卡土炮 10G LAB ",
"headline" : "[筆記] 用 proxmox \x26 Mellanox SFP 網卡土炮 10G LAB ",
"description" : "\x3cp\x3e想做一個 10G 的 LAB 環境出來已經很久了。\x3c\/p\x3e\n\x3cp\x3e只是礙於10G RJ45的卡太貴了，然後光纖的種類又太複雜\x3c\/p\x3e\n\x3cp\x3e如果直接在淘寶購買，很怕會買錯(什麼LC\/FC LC\/LC 多模單模 單芯雙芯 SFP\/SFP\x2b 又是什麼光模塊的一大堆規格)\x3c\/p\x3e\n\x3cp\x3e所以一直沒有付諸行動。\x3c\/p\x3e\n\x3cp\x3e硬體的工作很久沒碰了，剛好在蝦皮看到有個賣家在賣 mellanox 的X2網卡，以在台灣的價格來說，算很便宜的 (550)\x3c\/p\x3e\n\x3cp\x3e聊了一下，跟他請教了關於線材、光纖模塊的問題，回答也都很快很到位\x3c\/p\x3e\n\x3cp\x3e就直接下訂了兩張網卡、兩個光纖模塊、一條LC\/LC 光纖線\x3c\/p\x3e\n\x3cp\x3e就是到貨有點久，等了兩個禮拜左右，一直到昨天東西才寄到\x3c\/p\x3e\n\x3cp\x3e今天就花了點時間測試一下\x3c\/p\x3e",
"name" : "[筆記] 用 proxmox \u0026 Mellanox SFP 網卡土炮 10G LAB ",
"headline" : "[筆記] 用 proxmox \u0026 Mellanox SFP 網卡土炮 10G LAB ",
"description" : "\u003cp\u003e想做一個 10G 的 LAB 環境出來已經很久了。\u003c\/p\u003e\n\u003cp\u003e只是礙於10G RJ45的卡太貴了，然後光纖的種類又太複雜\u003c\/p\u003e\n\u003cp\u003e如果直接在淘寶購買，很怕會買錯(什麼LC\/FC LC\/LC 多模單模 單芯雙芯 SFP\/SFP\u002b 又是什麼光模塊的一大堆規格)\u003c\/p\u003e\n\u003cp\u003e所以一直沒有付諸行動。\u003c\/p\u003e\n\u003cp\u003e硬體的工作很久沒碰了，剛好在蝦皮看到有個賣家在賣 mellanox 的X2網卡，以在台灣的價格來說，算很便宜的 (550)\u003c\/p\u003e\n\u003cp\u003e聊了一下，跟他請教了關於線材、光纖模塊的問題，回答也都很快很到位\u003c\/p\u003e\n\u003cp\u003e就直接下訂了兩張網卡、兩個光纖模塊、一條LC\/LC 光纖線\u003c\/p\u003e\n\u003cp\u003e就是到貨有點久，等了兩個禮拜左右，一直到昨天東西才寄到\u003c\/p\u003e\n\u003cp\u003e今天就花了點時間測試一下\u003c\/p\u003e",
"inLanguage" : "en",
"author" : "Eric Chang",
"creator" : "Eric Chang",
@@ -21,8 +21,8 @@
"accountablePerson" : "Eric Chang",
"copyrightHolder" : "Eric Chang",
"copyrightYear" : "2018",
"datePublished": "2018-11-30 16:05:14 \x2b0800 CST",
"dateModified" : "2018-11-30 16:05:14 \x2b0800 CST",
"datePublished": "2018-11-30 16:05:14 \u002b0800 CST",
"dateModified" : "2018-11-30 16:05:14 \u002b0800 CST",
"url" : "https:\/\/h.cowbay.org\/post\/10g-lab-using-proxmox-and-mellanox\/",
"wordCount" : "583",
"image" : "https://h.cowbay.orghttps://h.cowbay.org/images/post-default-03.jpg"",
@@ -45,9 +45,9 @@
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.3.1/css/all.css" integrity="sha384-mzrmE5qonljUremFsqc01SB46JvROS7bZs3IO2EmfFsd15uHvIt+Y8vEf7N7fWAU" crossorigin="anonymous">
<link href="https://h.cowbay.org/css/style.css?v=1632901488" rel="stylesheet" id="theme-stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/style.css?v=1634607506" rel="stylesheet" id="theme-stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/custom.css?v=1632901488" rel="stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/custom.css?v=1634607506" rel="stylesheet" type='text/css' media='all'>
<link rel="shortcut icon" href="https://h.cowbay.org/img/favicon.ico" type="image/x-icon">
<link rel="icon" href="https://h.cowbay.org/img/favicon.ico" type="image/x-icon">
@@ -331,7 +331,7 @@ if (!doNotTrack) {
<p>設定了新的 vmbridge 之後，就可以在新VM的設定畫面中，指定網卡走這個界面出去</p>
<p>可是這樣做出來的VM ，一直無法開機</p>
<p>錯誤訊息如下</p>
<pre><code>kvm: -drive file=/zp/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on: file system may not support O_DIRECT
<pre tabindex="0"><code>kvm: -drive file=/zp/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on: file system may not support O_DIRECT
kvm: -drive file=/zp/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not open '/zp/images/100/vm-100-disk-1.qcow2': Invalid argument
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 123123 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=da27a9ea-fd55-4542-b2a7-8d5b09bf7611' -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 2048 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:b972d1ad783' -drive 'file=/zp/template/iso/ubuntu-18.04.1-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/zp/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=A2:EA:45:EE:17:25,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
</code></pre><p>當然先去拜google，果然就看到了提示，需要把 磁碟的 Cache 從預設的 Default(No Cache) 改成 write through</p>
@@ -340,14 +340,14 @@ TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 123123 -chardev 's
<p>總之呢，改完之後就可以了 &hellip;</p>
<h3 id="strongfont-colorred必須手動設定路由fontstrong"><!-- raw HTML omitted --><!-- raw HTML omitted -->必須手動設定路由<!-- raw HTML omitted --><!-- raw HTML omitted --></h3>
<h4 id="update">Update</h4>
<pre><code>這邊可能是我有誤解，應該不需要先在pve 本機設定 10G網卡的 IP
<pre tabindex="0"><code>這邊可能是我有誤解，應該不需要先在pve 本機設定 10G網卡的 IP
直接進web console 去設定 vmbr1 就好了
</code></pre><p>設定好新的VM，開機、設定IP、重開機之後，會發現還是ping 不到另一台機器..(翻桌！)</p>
<p>只好又去拜google ，就看到了底下這篇</p>
<p><a href="https://forum.proxmox.com/threads/how-to-add-second-nic.40905/">https://forum.proxmox.com/threads/how-to-add-second-nic.40905/</a></p>
<p>大概點出了方向，必須要手動增加路由(感覺有點蠢)</p>
<p>像我的光纖網卡走的是 192.168.50.0/24 ，就要去把原有的192.168.50.0/24的路由給砍掉，然後再新增(是不是很蠢？)</p>
<pre><code>root@ssd:/etc/network# ip route del 192.168.50.0/24
<pre tabindex="0"><code>root@ssd:/etc/network# ip route del 192.168.50.0/24
root@ssd:/etc/network# ip route add 192.168.50.0/24 dev vmbr1
root@ssd:/etc/network# ip route
@@ -357,7 +357,7 @@ default via 192.168.11.253 dev vmbr0 onlink
root@ssd:/etc/network#
</code></pre><p>OK ping 一下對面看能不能過</p>
<pre><code>root@ssd:/etc/network# ping 192.168.50.10
<pre tabindex="0"><code>root@ssd:/etc/network# ping 192.168.50.10
PING 192.168.50.10 (192.168.50.10) 56(84) bytes of data.
64 bytes from 192.168.50.10: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 192.168.50.10: icmp_seq=2 ttl=64 time=0.067 ms
@@ -374,9 +374,9 @@ PING 192.168.50.10 (192.168.50.10) 56(84) bytes of data.
<p>在linux 上，我習慣用 iperf 來測試兩台主機的連接速度</p>
<p>兩邊都用 apt install iperf 裝好套件</p>
<p>然後找一台作為 server ，執行</p>
<pre><code>iperf -s
<pre tabindex="0"><code>iperf -s
</code></pre><p>然後到另一台，去執行</p>
<pre><code>2018-11-30 15:36:58 [minion@ubuntu ~]$ iperf -d -t 600 -P 10 -c 192.168.50.200
<pre tabindex="0"><code>2018-11-30 15:36:58 [minion@ubuntu ~]$ iperf -d -t 600 -P 10 -c 192.168.50.200
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to 192.168.50.200, TCP port 5001
@@ -389,7 +389,7 @@ TCP window size: 85.0 KByte (default)
</code></pre><p>哈哈哈，有目有！測試速度來到了 9.18 Gbits 啊！ 就是一個爽啊！</p>
<p>記得那個 server IP 是你 VM 裡面設定的 IP，不是 proxmox 上面的</p>
<p>同場加映走 1Gb 網路的測試結果</p>
<pre><code>2018-11-30 16:39:37 [minion@ubuntu ~]$ iperf -d -t 600 -P 10 -c 192.168.11.171
<pre tabindex="0"><code>2018-11-30 16:39:37 [minion@ubuntu ~]$ iperf -d -t 600 -P 10 -c 192.168.11.171
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to 192.168.11.171, TCP port 5001
@@ -410,16 +410,16 @@ TCP window size: 85.0 KByte (default)
<p>在 mapuser/mapgroup這邊要改成 root/wheel 不然會有無法寫入的問題</p>
<p><img src="https://i.imgur.com/Yn7qYdK.png" alt="FREENAS NFS Sharing"></p>
<p>安裝完之後，實際跑一下 dd 看看速度多少</p>
<pre><code>Last login: Mon Dec 3 03:10:54 2018
<pre tabindex="0"><code>Last login: Mon Dec 3 03:10:54 2018
2018-12-03 03:15:03 [administrator@ubuntu ~]$ dd if=/dev/zero of=testfile bs=10240 count=1000000
1000000+0 records in
1000000+0 records out
10240000000 bytes (10 GB, 9.5 GiB) copied, 9.63458 s, 1.1 GB/s
</code></pre><pre><code>2018-12-03 03:17:28 [administrator@ubuntu ~]$ dd if=/dev/zero of=testfile bs=20480 count=1000000
</code></pre><pre tabindex="0"><code>2018-12-03 03:17:28 [administrator@ubuntu ~]$ dd if=/dev/zero of=testfile bs=20480 count=1000000
1000000+0 records in
1000000+0 records out
20480000000 bytes (20 GB, 19 GiB) copied, 16.0786 s, 1.3 GB/s
</code></pre><pre><code>2018-12-03 03:17:50 [administrator@ubuntu ~]$ dd if=/dev/zero of=testfile bs=4096 count=1000000
</code></pre><pre tabindex="0"><code>2018-12-03 03:17:50 [administrator@ubuntu ~]$ dd if=/dev/zero of=testfile bs=4096 count=1000000
1000000+0 records in
1000000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 4.80629 s, 852 MB/s
@@ -723,7 +723,7 @@ title="pinterest icon"></i>
</div>
<script src="https://h.cowbay.org/js/jquery.min.js"></script>
<script src="https://h.cowbay.org/js/jquerymigrate.js"></script>
<script src="https://h.cowbay.org/js/production.min.js?v=1632901488"></script>
<script src="https://h.cowbay.org/js/production.min.js?v=1634607506"></script>
</body>
</html>