Research
I’ve done a quick research on CISCO UCS VIC and RSS.
This is written in for Cisco UCS Virtual Interface Card Drivers Release Notes 3.1
Cisco UCS Manager 3.1(2) release now supports VXLAN with Receive Side-Scaling (RSS) stateless offload on VIC adapters 1340, 1380, 1385, 1387, and SIOC on Cisco UCS C3260 for ESXi 6.0 and later releases. VXLAN offload is not supported for IPv6.
3.1(s) was released at September 6, 2016 so RSS should work with ESXi nowadays. The last time I work / design samething with Cisco UCS was at beginning of 2015, so it is the new information for me.
However, it definitely needs configuration or at least validation on UCS Manager and probably also driver configuration or validation on ESXi host.
See. very nice blog post here
The blog above is not about ESXi but nicely covering “ethernet adapter policy” on Cisco UCS profile.
The profile should have “ethernet adapter policy” with something like
ucs1-osl3-B# scope org
ucs1-osl3-B /org # enter eth-policy default
ucs1-osl3-B /org/eth-policy # set recv-queue count 8
ucs1-osl3-B /org/eth-policy* # set trans-queue count 8
ucs1-osl3-B /org/eth-policy* # set rss receivesidescaling enabled
ucs1-osl3-B /org/eth-policy* # set comp-queue count 16
ucs1-osl3-B /org/eth-policy* # set interrupt count 18
I would double check on UCS if RSS is enabled on “ethernet adapter” because it can be disabled by default.
I would also double check that RSS is enabled on ESXi Driver
esxcli system module parameters list -m <DRIVER-MODULE-NAME>
Historically I wrote the blog post how to identify NIC capabilities.
It is available here ESXi Physical NIC Capabilities for NSX VTEP
You probably found this CISCO white paper
Summary from CISCO
It was really great working with you all today! You had some tricky questions but it looks like we were able to get most of it identified. You are doing application testing in a new data center and were trying to enable RSS but couldn’t verify if it was working or not. VMWare was running this command and said there must be a problem with the driver:
[root@duus-esxvs-05:~] esxcli network nic queue loadbalancer list
NIC RxQPair RxQNoFeature PreEmptibleQ RxQLatency RxDynamicLB DynamicQPool MacLearnLB RSS LRO GeneveOAM
------- ------- ------------ ------------ ---------- ----------- ------------ ---------- --- --- ---------
vmnic0 UA ND UA UA NA UA NA UA UA UA
vmnic1 UA ND UA UA NA UA NA UA UA UA
vmnic10 UA ND UA UA NA UA NA UA UA UA
vmnic11 UA ND UA UA NA UA NA UA UA UA
vmnic12 UA ND UA UA NA UA NA UA UA UA
vmnic13 UA ND UA UA NA UA NA UA UA UA
vmnic14 UA ND UA UA NA UA NA UA UA UA
vmnic15 UA ND UA UA NA UA NA UA UA UA
vmnic2 UA ND UA UA NA UA NA UA UA UA
vmnic3 UA ND UA UA NA UA NA UA UA UA
vmnic4 UA ND UA UA NA UA NA UA UA UA
vmnic5 UA ND UA UA NA UA NA UA UA UA
vmnic6 UA ND UA UA NA UA NA UA UA UA
vmnic7 UA ND UA UA NA UA NA UA UA UA
vmnic8 UA ND UA UA NA UA NA UA UA UA
vmnic9 UA ND UA UA NA UA NA UA UA UA
The first thing I did was use this guide to change your VMWare adapter policy:
I followed this section:
Configuring an Ethernet Adapter Policy to Enable Stateless Offloads with VXLAN
Cisco UCS Manager supports VXLAN TSO and checksum offloads only with Cisco UCS VIC 1340, 1380, 1385, 1387, adapters that are running on ESXi 5.5 and later releases. Stateless offloads with VXLAN cannot be used with NetFlow, usNIC, VM-FEX, Netqueue, or VMQ.
VXLAN with Receive Side-Scaling (RSS) support starts with the Cisco UCS Manager 3.1(2) release. RSS is supported with VXLAN stateless offload on VIC adapters 1340, 1380, 1385, 1387, and SIOC on Cisco UCS S3260 system for ESXi 5.5 and later releases.
VXLAN stateless hardware offloads are not supported with Guest OS TCP traffic over IPv6 on UCS VIC 13xx adapters. To run VXLAN encapsulated TCP traffic over IPV6, disable the VXLAN stateless offloads feature.
To disable the VXLAN stateless offload feature in UCS Manager, disable 'Virtual Extensible LAN’ field in the Ethernet Adapter Policy.
Procedure
Step 1
In the Navigation pane, click Servers.
Step 2
Expand Servers > Policies.
Step 3
Expand the node for the organization where you want to create the policy.
If the system does not include multitenancy, expand the root node.
Step 4
Right-click Adapter Policies and choose Create Ethernet Adapter Policy.
In the Resources area, set the following options:
Transmit Queues = 1
Receive Queues = n (up to 16)
Completion Queues = # of Transmit Queues + # of Receive Queues
Interrupts = # Completion Queues + 2
In the Options area, set the following options:
Receive Side Scaling = Enabled
Virtual Extensible LAN = Enabled
Interrupt Mode = Msi-X
For more information on creating an ethernet adapter policy, see Creating an Ethernet Adapter Policy.
Step 5
Click OK to create the Ethernet adapter policy.
Step 6
Install an eNIC driver Version 2.1.2.59 or later.
For more information, see the Cisco UCS Virtual Interface Card Drivers Installation Guide.
Step 7
Reboot the server.
Once we made this change on host duus-esxvs-05, I was able to confirm RSS was working by using vsish:
cat /net/pNics/vmnic6/stats
device {
-- General Statistics:
Rx Packets:47652
Tx Packets:21
Rx Bytes:3740492
Tx Bytes:1344
Rx Errors:0
Tx Errors:0
Rx Dropped:686
Tx Dropped:0
Rx Multicast:5307
Rx Broadcast:0
Tx Multicast:0
Tx Broadcast:0
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:0
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
tx_frames_ok: 21
tx_unicast_frames_ok: 0
tx_multicast_frames_ok: 0
tx_broadcast_frames_ok: 21
tx_bytes_ok: 1344
tx_unicast_bytes_ok: 0
tx_multicast_bytes_ok: 0
tx_broadcast_bytes_ok: 1344
tx_drops: 0
tx_errors: 0
tx_tso: 0
rx_frames_ok: 47652
rx_frames_total: 48338
rx_unicast_frames_ok: 0
rx_multicast_frames_ok: 5307
rx_broadcast_frames_ok: 43031
rx_bytes_ok: 3740492
rx_unicast_bytes_ok: 0
rx_multicast_bytes_ok: 728011
rx_broadcast_bytes_ok: 3061266
rx_drop: 0
rx_no_bufs: 686
rx_errors: 0
rx_rss: 9238
rx_crc_errors: 0
rx_frames_64: 12204
rx_frames_127: 32935
rx_frames_255: 3162
rx_frames_511: 37
rx_frames_1023: 0
rx_frames_1518: 0
rx_frames_to_max: 0
tx_queue_[0]_frames_ok: 21
rx_rss_queue_[0]_frames_ok: 39155
rx_rss_queue_[1]_frames_ok: 757
rx_rss_queue_[2]_frames_ok: 2338
rx_rss_queue_[3]_frames_ok: 473
rx_rss_queue_[4]_frames_ok: 1192
rx_rss_queue_[5]_frames_ok: 797
rx_rss_queue_[6]_frames_ok: 1234
rx_rss_queue_[7]_frames_ok: 1706
}
/> cat /net/pNics/vmnic6/stats | grep vxlan
device {
-- General Statistics:
Rx Packets:48188
Tx Packets:21
Rx Bytes:3782139
Tx Bytes:1344
Rx Errors:0
Tx Errors:0
Rx Dropped:686
Tx Dropped:0
Rx Multicast:5354
Rx Broadcast:0
Tx Multicast:0
Tx Broadcast:0
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:0
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
tx_frames_ok: 21
tx_unicast_frames_ok: 0
tx_multicast_frames_ok: 0
tx_broadcast_frames_ok: 21
tx_bytes_ok: 1344
tx_unicast_bytes_ok: 0
tx_multicast_bytes_ok: 0
tx_broadcast_bytes_ok: 1344
tx_drops: 0
tx_errors: 0
tx_tso: 0
rx_frames_ok: 48188
rx_frames_total: 48874
rx_unicast_frames_ok: 0
rx_multicast_frames_ok: 5354
rx_broadcast_frames_ok: 43520
rx_bytes_ok: 3782139
rx_unicast_bytes_ok: 0
rx_multicast_bytes_ok: 733433
rx_broadcast_bytes_ok: 3097491
rx_drop: 0
rx_no_bufs: 686
rx_errors: 0
rx_rss: 9340
rx_crc_errors: 0
rx_frames_64: 12308
rx_frames_127: 33343
rx_frames_255: 3186
rx_frames_511: 37
rx_frames_1023: 0
rx_frames_1518: 0
rx_frames_to_max: 0
tx_queue_[0]_frames_ok: 21
rx_rss_queue_[0]_frames_ok: 39598
rx_rss_queue_[1]_frames_ok: 762
rx_rss_queue_[2]_frames_ok: 2354
rx_rss_queue_[3]_frames_ok: 479
rx_rss_queue_[4]_frames_ok: 1194
rx_rss_queue_[5]_frames_ok: 800
rx_rss_queue_[6]_frames_ok: 1256
rx_rss_queue_[7]_frames_ok: 1745
}
VSISHPath_Form():Extraneous '|' in path.
VSISHCmdGetInt():mal-formed path
Error: Error in command cat: Bad parameter
/> cat /net/pNics/vmnic6/stats | grep vxlan
device {
-- General Statistics:
Rx Packets:49024
Tx Packets:21
Rx Bytes:3845846
Tx Bytes:1344
Rx Errors:0
Tx Errors:0
Rx Dropped:686
Tx Dropped:0
Rx Multicast:5413
Rx Broadcast:0
Tx Multicast:0
Tx Broadcast:0
Collisions:0
Rx Length Errors:0
Rx Over Errors:0
Rx CRC Errors:0
Rx Frame Errors:0
Rx Fifo Errors:0
Rx Missed Errors:0
Tx Aborted Errors:0
Tx Carrier Errors:0
Tx Fifo Errors:0
Tx Heartbeat Errors:0
Tx Window Errors:0
Module Interface Rx packets:0
Module Interface Tx packets:0
Module Interface Rx dropped:0
Module Interface Tx dropped:0
-- Driver Specific Statistics:
tx_frames_ok: 21
tx_unicast_frames_ok: 0
tx_multicast_frames_ok: 0
tx_broadcast_frames_ok: 21
tx_bytes_ok: 1344
tx_unicast_bytes_ok: 0
tx_multicast_bytes_ok: 0
tx_broadcast_bytes_ok: 1344
tx_drops: 0
tx_errors: 0
tx_tso: 0
rx_frames_ok: 49024
rx_frames_total: 49710
rx_unicast_frames_ok: 0
rx_multicast_frames_ok: 5413
rx_broadcast_frames_ok: 44297
rx_bytes_ok: 3845846
rx_unicast_bytes_ok: 0
rx_multicast_bytes_ok: 740281
rx_broadcast_bytes_ok: 3154350
rx_drop: 0
rx_no_bufs: 686
rx_errors: 0
rx_rss: 9504
rx_crc_errors: 0
rx_frames_64: 12474
rx_frames_127: 33989
rx_frames_255: 3210
rx_frames_511: 37
rx_frames_1023: 0
rx_frames_1518: 0
rx_frames_to_max: 0
tx_queue_[0]_frames_ok: 21
rx_rss_queue_[0]_frames_ok: 40278
rx_rss_queue_[1]_frames_ok: 769
rx_rss_queue_[2]_frames_ok: 2451
rx_rss_queue_[3]_frames_ok: 482
rx_rss_queue_[4]_frames_ok: 1201
rx_rss_queue_[5]_frames_ok: 806
rx_rss_queue_[6]_frames_ok: 1273
rx_rss_queue_[7]_frames_ok: 1764
}
Once I saw these RSS values in the output, I put these commands together to validate RSS was working:
[root@duus-esxvs-05:~] vsish -e get /net/pNics/vmnic6/stats | grep rss
rx_rss: 23986
rx_rss_queue_[0]_frames_ok: 102199
rx_rss_queue_[1]_frames_ok: 1978
rx_rss_queue_[2]_frames_ok: 6385
rx_rss_queue_[3]_frames_ok: 1252
rx_rss_queue_[4]_frames_ok: 3109
rx_rss_queue_[5]_frames_ok: 2058
rx_rss_queue_[6]_frames_ok: 3221
rx_rss_queue_[7]_frames_ok: 4299
[root@duus-esxvs-05:~] vsish -e get /net/pNics/vmnic7/stats | grep rss
rx_rss: 1863743
rx_rss_queue_[0]_frames_ok: 252893
rx_rss_queue_[1]_frames_ok: 432140
rx_rss_queue_[2]_frames_ok: 367045
rx_rss_queue_[3]_frames_ok: 194409
rx_rss_queue_[4]_frames_ok: 181221
rx_rss_queue_[5]_frames_ok: 164989
rx_rss_queue_[6]_frames_ok: 175065
rx_rss_queue_[7]_frames_ok: 196022
[root@duus-esxvs-05:~]
After we saw RSS was working, you ran another application test but it failed again. I logged into the adapter on this host and saw rx drops due to buffer overflow:
DUUM-FI01-A# connect adapter 8/1
adapter 0/8/1 # connect
No entry for terminal type "dumb";
using dumb terminal settings.
adapter 0/8/1 (top):1# attach-mcp
No entry for terminal type "dumb"
adapter 0/8/1 (mcp):5# lifstats -a 25
DELTA TOTAL DESCRIPTION
160269561 160269561 Tx unicast frames without error
157 157 Tx multicast frames without error
40973 40973 Tx broadcast frames without error
62791977101 62791977101 Tx unicast bytes without error
13476 13476 Tx multicast bytes without error
2787208 2787208 Tx broadcast bytes without error
0 0 Tx frames dropped
0 0 Tx frames with error
174076 174076 Tx TSO frames
247638401 247638401 Tx TSO bytes without error
169439268 169439268 Rx unicast frames without error
14488012 14488012 Rx multicast frames without error
724925 724925 Rx broadcast frames without error
63731052924 63731052924 Rx unicast bytes without error
8444893765 8444893765 Rx multicast bytes without error
50420513 50420513 Rx broadcast bytes without error
0 0 Rx frames dropped
3860 3860 Rx rq drop pkts (no bufs or rq disabled)
0 0 Rx frames with error
183898235 183898235 Rx good frames with RSS
0 0 Rx frames with Ethernet FCS error
22427524 22427524 Rx frames len == 64
85447437 85447437 Rx frames 64 < len <= 127
28201512 28201512 Rx frames 128 <= len <= 255
8806746 8806746 Rx frames 256 <= len <= 511
5568237 5568237 Rx frames 512 <= len <= 1023
2721570 2721570 Rx frames 1024 <= len <= 1518
31479179 31479179 Rx frames len > 1518
50.960kbps Tx rate
58.618kbps Rx rate
To reduce these drops, I went to UCS Central and changed your Tx ring size to 512 and Rx ring size to 2048:
DUUM-FI01-A /org/eth-policy # show expand detail
Eth Adapter Policy:
Name: global-VMWare
Description: Recommended adapter settings for VMWare
Policy Owner: Global
VMMQ Resource Pool: Disabled
ARFS:
Accelarated Receive Flow Steering: Disabled
Ethernet Completion Queue:
Count: 9
Ethernet Failback:
Timeout (sec): 5
Ethernet Interrupt:
Coalescing Time (us): 125
Coalescing Type: Min
Count: 11
Driver Interrupt Mode: MSI-X
NVGRE:
NVGRE: Disabled
Ethernet Offload:
Large Receive: Enabled
TCP Segment: Enabled
TCP Rx Checksum: Enabled
TCP Tx Checksum: Enabled
Ethernet Receive Queue:
Count: 8
Ring Size: 2048
ROCE:
RoCE: Disabled
RoCE QOS priority: Best Effort
Resource Groups: 32
Memory Regions: 131072
Queue Pairs: 256
RoCE Version 1: Disabled
RoCE Version 2: Disabled
VXLAN:
VXLAN: Enabled
Ethernet Transmit Queue:
Count: 1
Ring Size: 512
RSS:
Receive Side Scaling: Enabled
DUUM-FI01-A /org/eth-policy #
After changing the adapter policy to use only 1 transmit queue and increasing the rx buffers, it looks like RSS is working and you no longer have Rx drops on the host. However, your application tests are still failing. Currently you suspect the F5 load balancer VMs are too much of a bottleneck for the network.
No comments:
Post a Comment