End to End Proxmox 5.4 with Infiniband backplane and FusionIO cards for storage.
proxmox v5.4 buildbook with infiniband and FusionIO cards and Ceph
first: fix the sources.list
two commands:
rm /etc/apt/sources.list.d/pve-enterprise.list
and
echo “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription” > /etc/apt/sources.list.d/pve-install-repo.list
The actual process
Download all the files. Here’s the cheatsheet, so you don’t have to waste plenty of time on this (it creates a folder /home/temp where all the files are downloaded automatically:
Code:
mkdir /home/temp && cd /home/temp && \
wget -O iomemory-vsl_3.2.16.1731-1.0.tar.gz https://www.dropbox.com/s/up8hrkmk6lhyc6f/iomemory-vsl_3.2.16.1731-1.0.tar.gz?dl=1 && \
wget -O fio-common_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/pd2ohfaufhwqc34/fio-common_3.2.16.1731-1.0_amd64.deb?dl=1 && \
wget -O fio-firmware-fusion_3.2.16.20180821-1_all.deb https://www.dropbox.com/s/kcn5agi6lyikicf/fio-firmware-fusion_3.2.16.20180821-1_all.deb?dl=1 && \
wget -O fio-preinstall_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/iq7pefjdgwtuzf1/fio-preinstall_3.2.16.1731-1.0_amd64.deb?dl=1 && \
wget -O fio-sysvinit_3.2.16.1731-1.0_all.deb https://www.dropbox.com/s/g39l6lg9of6eqze/fio-sysvinit_3.2.16.1731-1.0_all.deb?dl=1 && \
wget -O fio-util_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/57huby17mteg6wp/fio-util_3.2.16.1731-1.0_amd64.deb?dl=1 && \
tar zxvf iomemory-vsl_3.2.16.1731-1.0.tar.gz && cd iomemory-vsl-3.2.16.1731
Get all the tools you need in order to compile the drivers from source (please be aware that at the time of reading the kernel is likely to be newer):
Code:
apt-get install gcc fakeroot build-essential debhelper rsync pve-headers dkms
Create the DKMS config file and edit it via the command:
Code:
nano /home/temp/iomemory-vsl-3.2.16.1731/root/usr/src/iomemory-vsl-3.2.16/dkms.conf
The content of the DKMS will be:
Code:
MAKE=”‘make’ DKMS_KERNEL_VERSION=$kernelver”
CLEAN=”‘make’ clean”
BUILT_MODULE_NAME=iomemory-vsl
BUILT_MODULE_LOCATION=”
PACKAGE_NAME=iomemory-vsl
PACKAGE_VERSION=3.2.16.1731
DEST_MODULE_LOCATION=/kernel/drivers/block
AUTOINSTALL=”Yes”
REMAKE_INITRD=yes
Prepare the source for being compiled, compile the drivers and load the drivers into the kernel:
you may need to install specific headers if this next step fails:
example apt install pve-headers-4.15.18-12-pve
Code:
cp -r /home/temp/iomemory-vsl-3.2.16.1731/root/usr/src/iomemory-vsl-3.2.16 /usr/src/ && \
mkdir -p /var/lib/dkms/iomemory-vsl/3.2.16/build && \
ln -s /usr/src/iomemory-vsl-3.2.16 /var/lib/dkms/iomemory-vsl/3.2.16/source && \
dkms build -m iomemory-vsl -v 3.2.16 && \
dkms install -m iomemory-vsl -v 3.2.16 && \
modprobe iomemory-vsl
Install all utilities and firmware:
Code:
cd /home/temp && \
dpkg -i fio-firmware-fusion_3.2.16.20180821-1_all.deb fio-util_3.2.16.1731-1.0_amd64.deb fio-sysvinit_3.2.16.1731-1.0_all.deb fio-common_3.2.16.1731-1.0_amd64.deb
You may need to compile the drivers for the other kernels present on the system:
Code:
ls /var/lib/initramfs-tools | sudo xargs -n1 /usr/lib/dkms/dkms_autoinstaller start
Next step get the Infiniband working!
run this
apt-get -y install debhelper autotools-dev dkms zlib1g-dev python wget quilt python-libxml2 swig dpatch graphviz chrpath
then install the drivers/utils
wget http://www.mellanox.com/downloads/ofed/MLNX_EN-4.2-1.0.1.0/mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
tar xzvf mlnx-en-4.2-1.0.1.0-debian9.0-x86_64.tgz
cd mlnx-en-4.2-1.0.1.0-debian9.0-x86_64
./install –skip-distro-check
it will warn that other packages will be removed, choose Y and hit enter.
once it completes. another round of installs and reboots!
sudo apt-get install opensm infiniband-diags librdmacm-dev libibverbs-dev libmlx4-dev
add these lines to /etc/modules
mlx4_ib
ip_ipoib
rdma_ucm
THEN reboot.
Reboot the OS and make sure the device is attached after the reboot by running
Code:
fio-status -a
Format the device according to your needs and enjoy.