proxmox v5.4 buildbook with infiniband and FusionIO cards and Ceph<\/p>\n\n\n\n
first: fix the sources.list<\/p>\n\n\n\n
two commands:
\nrm \/etc\/apt\/sources.list.d\/pve-enterprise.list<\/p>\n\n\n\n
and
\necho “deb http:\/\/download.proxmox.com\/debian\/pve stretch pve-no-subscription” > \/etc\/apt\/sources.list.d\/pve-install-repo.list<\/p>\n\n\n\n
The actual process
\nDownload all the files. Here’s the cheatsheet, so you don’t have to waste plenty of time on this (it creates a folder \/home\/temp where all the files are downloaded automatically:
\nCode:
\nmkdir \/home\/temp && cd \/home\/temp && \\
\nwget -O iomemory-vsl_3.2.16.1731-1.0.tar.gz https:\/\/www.dropbox.com\/s\/up8hrkmk6lhyc6f\/iomemory-vsl_3.2.16.1731-1.0.tar.gz?dl=1 && \\
\nwget -O fio-common_3.2.16.1731-1.0_amd64.deb https:\/\/www.dropbox.com\/s\/pd2ohfaufhwqc34\/fio-common_3.2.16.1731-1.0_amd64.deb?dl=1 && \\
\nwget -O fio-firmware-fusion_3.2.16.20180821-1_all.deb https:\/\/www.dropbox.com\/s\/kcn5agi6lyikicf\/fio-firmware-fusion_3.2.16.20180821-1_all.deb?dl=1 && \\
\nwget -O fio-preinstall_3.2.16.1731-1.0_amd64.deb https:\/\/www.dropbox.com\/s\/iq7pefjdgwtuzf1\/fio-preinstall_3.2.16.1731-1.0_amd64.deb?dl=1 && \\
\nwget -O fio-sysvinit_3.2.16.1731-1.0_all.deb https:\/\/www.dropbox.com\/s\/g39l6lg9of6eqze\/fio-sysvinit_3.2.16.1731-1.0_all.deb?dl=1 && \\
\nwget -O fio-util_3.2.16.1731-1.0_amd64.deb https:\/\/www.dropbox.com\/s\/57huby17mteg6wp\/fio-util_3.2.16.1731-1.0_amd64.deb?dl=1 && \\
\ntar zxvf iomemory-vsl_3.2.16.1731-1.0.tar.gz && cd iomemory-vsl-3.2.16.1731
\nGet all the tools you need in order to compile the drivers from source (please be aware that at the time of reading the kernel is likely to be newer):
\nCode:
\napt-get install gcc fakeroot build-essential debhelper rsync pve-headers dkms
\nCreate the DKMS config file and edit it via the command:
\nCode:
\nnano \/home\/temp\/iomemory-vsl-3.2.16.1731\/root\/usr\/src\/iomemory-vsl-3.2.16\/dkms.conf
\nThe content of the DKMS will be:
\nCode:
\nMAKE=”‘make’ DKMS_KERNEL_VERSION=$kernelver”
\nCLEAN=”‘make’ clean”
\nBUILT_MODULE_NAME=iomemory-vsl
\nBUILT_MODULE_LOCATION=”
\nPACKAGE_NAME=iomemory-vsl
\nPACKAGE_VERSION=3.2.16.1731
\nDEST_MODULE_LOCATION=\/kernel\/drivers\/block
\nAUTOINSTALL=”Yes”
\nREMAKE_INITRD=yes
\nPrepare the source for being compiled, compile the drivers and load the drivers into the kernel:<\/p>\n\n\n\n
you may need to install specific headers if this next step fails:
\nexample apt install pve-headers-4.15.18-12-pve<\/p>\n\n\n\n
Code:
\ncp -r \/home\/temp\/iomemory-vsl-3.2.16.1731\/root\/usr\/src\/iomemory-vsl-3.2.16 \/usr\/src\/ && \\
\nmkdir -p \/var\/lib\/dkms\/iomemory-vsl\/3.2.16\/build && \\
\nln -s \/usr\/src\/iomemory-vsl-3.2.16 \/var\/lib\/dkms\/iomemory-vsl\/3.2.16\/source && \\
\ndkms build -m iomemory-vsl -v 3.2.16 && \\
\ndkms install -m iomemory-vsl -v 3.2.16 && \\
\nmodprobe iomemory-vsl
\nInstall all utilities and firmware:
\nCode:
\ncd \/home\/temp && \\
\ndpkg -i fio-firmware-fusion_3.2.16.20180821-1_all.deb fio-util_3.2.16.1731-1.0_amd64.deb fio-sysvinit_3.2.16.1731-1.0_all.deb fio-common_3.2.16.1731-1.0_amd64.deb
\nYou may need to compile the drivers for the other kernels present on the system:
\nCode:
\nls \/var\/lib\/initramfs-tools | sudo xargs -n1 \/usr\/lib\/dkms\/dkms_autoinstaller start<\/p>\n\n\n\n
Next step get the Infiniband working!
\nrun this
\napt-get -y install debhelper autotools-dev dkms zlib1g-dev python wget quilt python-libxml2 swig dpatch graphviz chrpath<\/p>\n\n\n\n
then install the drivers\/utils<\/p>\n\n\n\n
wget http:\/\/www.mellanox.com\/downloads\/ofed\/MLNX_EN-4.2-1.0.1.0\/mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
\n tar xzvf mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
\n cd mlnx-en-4.2-1.0.1.0-debian9.0-x86_64
\n .\/install –skip-distro-check<\/p>\n\n\n\n
it will warn that other packages will be removed, choose Y and hit enter.<\/p>\n\n\n\n
once it completes. another round of installs and reboots!<\/p>\n\n\n\n
sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev<\/p>\n\n\n\n
add these lines to \/etc\/modules<\/p>\n\n\n\n
mlx4_ib
\nip_ipoib
\nrdma_ucm<\/p>\n\n\n\n
THEN reboot.<\/p>\n\n\n\n
Reboot the OS and make sure the device is attached after the reboot by running
\nCode:
\nfio-status -a
\nFormat the device according to your needs and enjoy.<\/p>\n","protected":false},"excerpt":{"rendered":"
proxmox v5.4 buildbook with infiniband and FusionIO cards and Ceph first: fix the sources.list two commands: rm \/etc\/apt\/sources.list.d\/pve-enterprise.list and echo “deb http:\/\/download.proxmox.com\/debian\/pve stretch pve-no-subscription” > \/etc\/apt\/sources.list.d\/pve-install-repo.list The actual process Download all the files. Here’s the cheatsheet, so you don’t have to waste plenty of time on this (it creates a folder \/home\/temp where all the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4858","post","type-post","status-publish","format-standard","hentry","category-uncategorized","has-post-title","has-post-date","has-post-category","has-post-tag","has-post-comment","has-post-author",""],"builder_content":"","_links":{"self":[{"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/posts\/4858","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/comments?post=4858"}],"version-history":[{"count":2,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/posts\/4858\/revisions"}],"predecessor-version":[{"id":4860,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/posts\/4858\/revisions\/4860"}],"wp:attachment":[{"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/media?parent=4858"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/categories?post=4858"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cdnhost.net\/content\/wp-json\/wp\/v2\/tags?post=4858"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}