We recently upgraded our LAN network from 1G to 10G but the Oracle OVS servers network cards showed only 1G link even though the advertised link was 10G.

[root@ovs1 ~]# ethtool em1
Settings for em1:
 Supported ports: [ FIBRE ]
 Supported link modes: 1000baseT/Full
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: Yes
 Advertised link modes: 1000baseT/Full
 Advertised pause frame use: No
 Advertised auto-negotiation: Yes
 Link partner advertised link modes: 1000baseT/Full
 10000baseT/Full
 Link partner advertised pause frame use: Symmetric
 Link partner advertised auto-negotiation: Yes
 Speed: 1000Mb/s
 Duplex: Full
 Port: FIBRE
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x00000000 (0)

Link detected: yes

And the OVS had the famous bnx2x Broadcom driver which we have had issues earlier too.

[root@ovs1 ~]# ethtool -i em1
driver: bnx2x
version: 1.712.10
firmware-version: FFV7.10.17 bc 7.10.11

So when I tested the speed using iperf I got the following:

[root@ovs1 ~]# ./iperf64 -c 10.10.10.12
------------------------------------------------------------
Client connecting to 10.10.10.12, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.11 port 45871 connected with 10.10.10.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec

So only 1G link speed as expected when the card is showing 1000Mb/s speed.

After trying few different things the solution was to upgrade to latest kernel and latest bnx2x drivers. Current kernel was:

[root@ovs1 ~]# uname -a
Linux ovs1.local 3.8.13-68.3.3.el6uek.x86_64 #2 SMP Tue Jun 23 23:43:15 PDT 2015 
x86_64 x86_64 x86_64 GNU/Linux

And to upgrade it we added the public yum ovs repo to yum.repos.d

[root@ovs1 ~] cat /etc/yum.repos.d/ovm.new:
[3.3_ovs_repo]
gpgcheck = 0
baseurl = http://public-yum.oracle.com/repo/OracleVM/OVM3/3x_latest/x86_64/
name = 3.3_ovs_repo
enabled = 1

After running yum update kernel and rebooting host new kernel is in use:

[root@ovs1 ~]# uname -a
Linux ovs1.local 3.8.13-118.16.3.el6uek.x86_64 #2 SMP Wed Feb 8 
17:40:13 PST 2017 x86_64 x86_64 x86_64 GNU/Linux

And the drivers are updated as well:

[root@ovs1 ~]# ethtool -i em1
driver: bnx2x
version: 1.713.01
firmware-version: FFV7.10.17 bc 7.10.11

However the link shows up quite weird even though speed is 10000Mb/s with autonegotiation on. It doesn’t have advertised speed anymore as 10000base-T!

[root@ovs1 ~]# ethtool em1
Settings for em1:
 Supported ports: [ FIBRE ]
 Supported link modes: 1000baseKX/Full
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: Yes
 Advertised link modes: Not reported
 Advertised pause frame use: Symmetric Receive-only
 Advertised auto-negotiation: Yes
 Link partner advertised link modes: 1000baseKX/Full
 Link partner advertised pause frame use: Symmetric
 Link partner advertised auto-negotiation: Yes
 Speed: 10000Mb/s
 Duplex: Full
 Port: FIBRE
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x00000000 (0)

Link detected: yes

But now running the same iperf test gives new results:

[root@ovs1 ~]# ./iperf64 -c 10.10.10.12
------------------------------------------------------------
Client connecting to 10.10.10.12, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.11 port 57350 connected with 10.10.10.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 5.34 GBytes 4.58 Gbits/sec

So lot better throughput! I’m not so keen recommending the Broadcom NIC to anyone as we have had lot of issues on setting them up with Oracle VM / OVS. Compared to Intel based 10Gb NIC which has been lot easier to manage.

There is also note on MOS on using iperf: How to Use Iperf to Test Network Performance (Doc ID 1507397.1)

Why the card still doesn’t show correctly the advertised and supported speed I don’t know but need to figure out.

Leave a Reply

Your email address will not be published. Required fields are marked *