End to End Proxmox 5.4 with Infiniband backplane and FusionIO cards for storage.

/ 0 Comments

proxmox v5.4 buildbook with infiniband and FusionIO cards and Ceph

first: fix the sources.list

two commands:
rm /etc/apt/sources.list.d/pve-enterprise.list

and
echo “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription” > /etc/apt/sources.list.d/pve-install-repo.list

The actual process
Download all the files. Here’s the cheatsheet, so you don’t have to waste plenty of time on this (it creates a folder /home/temp where all the files are downloaded automatically:
Code:
mkdir /home/temp && cd /home/temp && \
wget -O iomemory-vsl_3.2.16.1731-1.0.tar.gz https://www.dropbox.com/s/up8hrkmk6lhyc6f/iomemory-vsl_3.2.16.1731-1.0.tar.gz?dl=1 && \
wget -O fio-common_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/pd2ohfaufhwqc34/fio-common_3.2.16.1731-1.0_amd64.deb?dl=1 && \
wget -O fio-firmware-fusion_3.2.16.20180821-1_all.deb https://www.dropbox.com/s/kcn5agi6lyikicf/fio-firmware-fusion_3.2.16.20180821-1_all.deb?dl=1 && \
wget -O fio-preinstall_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/iq7pefjdgwtuzf1/fio-preinstall_3.2.16.1731-1.0_amd64.deb?dl=1 && \
wget -O fio-sysvinit_3.2.16.1731-1.0_all.deb https://www.dropbox.com/s/g39l6lg9of6eqze/fio-sysvinit_3.2.16.1731-1.0_all.deb?dl=1 && \
wget -O fio-util_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/57huby17mteg6wp/fio-util_3.2.16.1731-1.0_amd64.deb?dl=1 && \
tar zxvf iomemory-vsl_3.2.16.1731-1.0.tar.gz && cd iomemory-vsl-3.2.16.1731
Get all the tools you need in order to compile the drivers from source (please be aware that at the time of reading the kernel is likely to be newer):
Code:
apt-get install gcc fakeroot build-essential debhelper rsync pve-headers dkms
Create the DKMS config file and edit it via the command:
Code:
nano /home/temp/iomemory-vsl-3.2.16.1731/root/usr/src/iomemory-vsl-3.2.16/dkms.conf
The content of the DKMS will be:
Code:
MAKE=”‘make’ DKMS_KERNEL_VERSION=$kernelver”
CLEAN=”‘make’ clean”
BUILT_MODULE_NAME=iomemory-vsl
BUILT_MODULE_LOCATION=”
PACKAGE_NAME=iomemory-vsl
PACKAGE_VERSION=3.2.16.1731
DEST_MODULE_LOCATION=/kernel/drivers/block
AUTOINSTALL=”Yes”
REMAKE_INITRD=yes
Prepare the source for being compiled, compile the drivers and load the drivers into the kernel:

you may need to install specific headers if this next step fails:
example apt install pve-headers-4.15.18-12-pve

Code:
cp -r /home/temp/iomemory-vsl-3.2.16.1731/root/usr/src/iomemory-vsl-3.2.16 /usr/src/ && \
mkdir -p /var/lib/dkms/iomemory-vsl/3.2.16/build && \
ln -s /usr/src/iomemory-vsl-3.2.16 /var/lib/dkms/iomemory-vsl/3.2.16/source && \
dkms build -m iomemory-vsl -v 3.2.16 && \
dkms install -m iomemory-vsl -v 3.2.16 && \
modprobe iomemory-vsl
Install all utilities and firmware:
Code:
cd /home/temp && \
dpkg -i fio-firmware-fusion_3.2.16.20180821-1_all.deb fio-util_3.2.16.1731-1.0_amd64.deb fio-sysvinit_3.2.16.1731-1.0_all.deb fio-common_3.2.16.1731-1.0_amd64.deb
You may need to compile the drivers for the other kernels present on the system:
Code:
ls /var/lib/initramfs-tools | sudo xargs -n1 /usr/lib/dkms/dkms_autoinstaller start

Next step get the Infiniband working!
run this
apt-get -y install debhelper autotools-dev dkms zlib1g-dev python wget quilt python-libxml2 swig dpatch graphviz chrpath

then install the drivers/utils

wget http://www.mellanox.com/downloads/ofed/MLNX_EN-4.2-1.0.1.0/mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
tar xzvf mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
cd mlnx-en-4.2-1.0.1.0-debian9.0-x86_64
./install –skip-distro-check

it will warn that other packages will be removed, choose Y and hit enter.

once it completes. another round of installs and reboots!

sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev

add these lines to /etc/modules

mlx4_ib
ip_ipoib
rdma_ucm

THEN reboot.

Reboot the OS and make sure the device is attached after the reboot by running
Code:
fio-status -a
Format the device according to your needs and enjoy.