Multichannel (UFP and vNIC) on IBM Switches
September 13, 2014
Introduction to multichannel
Hardware density has become increasingly important over the last several years. Being able to do more compute with a smaller physical and energy footprint has become the new focus for IT. In order to do this effectively, more and more businesses are moving traditional workloads to the cloud, whether it be public, private or hybrid.
We often find however that all of this consolidation requires previously separated network and storage fabrics to coexist on the same physical hardware, virtualised, especially in private cloud scenarios. Traditionally that has meant introducing more and more IO adapters into systems to cope with the number of isolated fabrics, but we are trying to make everything smaller and cheaper; This is where converged network adapters and multi-channel networking becomes relevant.
CNAs (Converged Network Adapters)
Network adapters such as the Emulex and Brocade CNAs are physical network adaptors which have the ability to provide multiple virtual network or storage adaptors in a single card. Multiple communications channels are established between the CNA (installed in a server) and a compliant network switch. When working with IBM switches, these virtual NICs or HBAs (channels) are configured depending on the multi-channel mode set in the server’s UEFI menu.
Multi-channel modes: vNIC, Switch-Independent vNIC and UFP - what’s the difference?
There are several ways in which multi-channel can be configured between IBM switches and supported CNAs. The most common configuration modes are Switch-Independent vNIC, vNIC and UFP. These modes are mostly the same in functionality, with the exception of UFP which adds some additional functionality. The main way in which these modes differ, is how and where the channels are configured. Let’s step through the individual modes and how they are configured.
Configuring Multi-Channel Mode
Multi-Channel operation is configured by first entering the adaptor configuration menu, under the server’s UEFI menu. On IBM servers, you can enter UEFI by pressing F1 when prompted during server boot up. It may help to take node of the MAC address and PCI address of the adaptor you would like to configure, prior to entering the setup menu, if you have an OS installed on the server. Once in the UEFI menu, you navigate to System Settings -> Network, and select the device you would like to configure multi-channel mode on. For Emulex adaptors, you can enter the Emulex NIC Setting screen the change the multi-channel mode. Once in this screen, you can select supported Multi-Channel modes by altering the Multichannel setting.
For UFP, you select UFP as the multi-channel mode. With UFP, the remaining configuration is done in the configuration of a supported switch. Negotiation between the NIC in the server and the switch to establish UFP channels and settings normally only happens at server boot, so remember that once the mode is changed in the UEFI menu, and when any changes are made to the UFP configuration on the switch, the server will need a reboot to pick up the changes.
An example switch configuration section is below, which we will walk through.
As you can see from the configuration, virtual NICs or “UFP Channels” and their usage are configured on the switch side. All that needs to be done from the server side is to enable UFP as the multi-channel mode in the UEFI menu for the CNA.
Switch-Independent vNIC Configuration
Switch independent vNIC mode is different to UFP mode, in that the CNA allocates vNICs for each VLAN permitted on the switch port. As many switches are capable of using VLANs, even older ones which are not UFPcapable, this mode is considered switch-independent as the CNA is doing most of the heavy lifting when it comes to detecting and configuring vNICs. The majority of configuration is handled by the CNA. This mode can be selected in the UEFI menu, an example switch config for an EN4093 below could be used to allocate vNIC ports based on VLAN.
The above switch configuration does not carry any vNIC-specific configuration as you can see. The CNA does all the work in finding out how many vNICs need to be mapped to VLANs. Keep in mind, that your second vNIC will typically be used for any FCoE traffic.
Standard vNIC Configuration
Standard vNIC predates UFP, and typically if supported by your CNAs and switches you should go with UFP, as it is more flexible. Configuration-wise, it is very similar to UFP, except some of the configuration is performed in the UEFI menu in addition to the switch configuration. It is essentially in-between UFP and switch-independent vNIC when it comes to the CNA and switch’s responsibility for configuration of the fabric. If you are on older switching or NIC/HBA hardware, or older firmware levels you might need to use this mode for compatibly.
To configure this mode, enter the UEFI menu on your server and enter the configuration for your virtual fabric adapter. Depending on your adaptor, you will be able to see the virtual fabric channels listed and you will be able to change the modes of each channel. This information will propagate to the switching hardware connected to the VFA on next reboot.
Switch configuration can be then used to configure VLAN membership on the virtual ports, which are referred to by their physical port number and channel. Example: INT4.1 Also, when adding vNIC ports to a VLAN, you normally need to use the vmember command instead of the regular member command.
An example for enabling VNIC and the first channel on physical port 4, allocating 50% of the physical bandwidth the port:
Keep in mind, your bandwidth percentages will need to add up to 100. Unlikely UFP, you set the modes of the various channels in UEFI if you need to use things like FCoE.
FCoE and Multichannel
For consistency and stability, when FCoE is in use with multi-channel, ensure you use the second channel for FCoE traffic, unless you have reason to change it. The reason for this is that some CNAs, especially when using less sophisticated multi-channel modes (Switch-Independent vNIC for example) will use the second channel for FCoE by default, and it is non-trivial or not possible to change it, which can lead to some confusion if you are trying to configured a different channel on your switching equipment for FCoE use.