openindiana srp target

Added by Deniss Cirulis over 2 years ago

Hi,
I've setup openindina 151a (illumos) srp target over infiniband. IB HCA is Mellanox dual port 20Gbit adapter. Second side of this setup is Linux with the same adapter and configured ib in connected mode.
I can freely scan the luns and try to write some data to it. Local writes on zfs pool are about 2.5-3.0GB/s, thus over infiniband I achieve not more than 500MB/s. Both machines were connected through TopSpin 7000D switch. As I wondered that the switch is causing slow down, I connected both hosts back-to-back with opensm running on Linux box. As operational bandwidth of Mellanox HCA is 16Gbps I must achieve more than 500MB/s.

What can be the problem of such a low speed writing to srp target ?

Thanks in advance!


Replies (14)

RE: openindiana srp target - Added by Nole Anderson about 2 years ago

I'm seeing the same performance with my srp setup. Mellnaox HCA 20Gbit - HP IB Switch - Vmware. Might be the old drivers, haven't seen much development on them. I'll keep an eye on this thread if someone comes up with anything.

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

Nole Anderson wrote:

I'm seeing the same performance with my srp setup. Mellnaox HCA 20Gbit - HP IB Switch - Vmware. Might be the old drivers, haven't seen much development on them. I'll keep an eye on this thread if someone comes up with anything.

you're way ahead..., mine it's mindblowing slower. using Ibm Bladecenter H, Cisco 4x HCA (Infinihost III EX with mem), 2 Cisco IB Blade Switch, OpenIndiana 1.5.1 as a srp target and VmVare ESX4.1 as an initiator..... and I get ~40MBps... I feel like my head is on fire and can't find a bucket o water...

RE: openindiana srp target - Added by Nole Anderson about 2 years ago

You must be doing something wrong. I'm running my Vmware 4.1 on HP infinihost iii hcas (c7000 blades). 700 mb/s.

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

i'm trying to figure it out... what driver do you use on vmware? i'm using Rev 1.4.1-2.0.000, cisco 7000 2.9.0, OI is a ConnectX VPI.

is it 700mbps or MBps

RE: openindiana srp target - Added by Nole Anderson about 2 years ago

Same driver as me, latest hp firmware and latest mellanox firmwares, 700 MB/s as in it roughly doing 6 x gbit. Locally it does ~4gb/s (only ssds in this storage server).

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

infinihost firmware is 4.8.917, but I see that mellanox has 4.8.200 ?! SRP target is a vpool made of 3 mirrors of 2 HDD WD-1TB-RE4. locally it does 400-500 MBps write and 1.3-1. read. however on infiniband it won't budge above 40MBps at read, and sometimes 10MBps at write... i'm kinda lost here...

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

Bogdan Busila wrote:

infinihost firmware is 4.8.917, but I see that mellanox has 4.8.200 ?! SRP target is a vpool made of 3 mirrors of 2 HDD WD-1TB-RE4. locally it does 400-500 MBps write and 1.3-1.4GBps read. however on infiniband it won't budge above 40MBps at read, and sometimes 10MBps at write... i'm kinda lost here...

RE: openindiana srp target - Added by Nole Anderson about 2 years ago

What stripe do you use with those disks?

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

it's a standard zfs pool with 3 2-disk mirror vdevs... I didn't change anything

RE: openindiana srp target - Added by Nole Anderson about 2 years ago

Tried changing block size to 128k?

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

yep I tried, same results. right now, I installed on one of the blades a windows 7, got winOFED3.0 on it, and did a test with crystaldisk, on a 2g ramdisk. it gave me abour 480MBps at seq read and 150 at seq write. when I tried with atto disk, the disk dissapeared, and the target was offline.. :

root@XXXX:/# stmfadm list-target -v
Target: eui.0002C9030003FD70
Operational Status: Offline
Provider Name : srpt
Alias : -
Protocol : SRP
Sessions : 0

I had to "svcadm disable ibsrp/target" and "svcadm enable ibsrp/target" again so it would appear online. I need to mention that srpt was installed with packet manager, and I use napp-it for the moment.

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

did you make any change in vmware except the driver installation? is there any tunable setting in mellanox's vmware driver? I set up a zfs folder and tried NFS over IPOIB, and got the same results, on all the blades. I'm beginning to suspect the hardware....

RE: openindiana srp target - Added by Bogdan Busila about 2 years ago

hello again,

I switched to Solaris 11 for testing purpose. same results...

RE: openindiana srp target - Added by Andrew Symonds about 2 years ago

Thanks for one more great article about web based predictive dialer. I am thankful for your great job and for all those useful advices you are giving out in your blog articles. Custom dissertation writing I am looking forward to more other great posts here in the future too! Sincerely Thank you for taking the time to publish this information very useful! I've been looking for books of this nature for a way too long. I'm just glad that I found yours. Looking forward for your next post. Thanks :) Thesis

(1-14/14)