Pages

Saturday, May 9, 2015

vSphere VSAN: Notes From The Field and VMware Support

VMware vSAN 5.5 (and now 6.0) is a software-defined storage solution developed by VMware and integrated into the kernel of its premier virtualization platform, allowing for the creation and management of shared object-based storage using the local solid-state and spinning media in the physical host servers themselves.

Note that vSAN is not the same animal as VMware’s vSphere Storage Appliance (vSA), though the underlying value proposition is the same. The two are implemented and managed very differently. vSA is now end-of-life/availability, though still supported through 2018. vSAN has been integrated directly into the kernel, so it is there whether you use it or not, and no longer requires the deployment of controller appliances. Storage appears as a single unified ‘datastore’ across all hosts in the cluster and is managed entirely through vCenter’s web client.

vSAN is licensed separately from vSphere but in the same familiar fashion, on a per socket basis. When you enable vSAN on the cluster you are initially allowed a 60-day evaluation period but must assign a proper license to the cluster before this evaluation period expires.

The purpose of this email is to provide notes from both field deployments and from working with VMware support.

General notes:

1.       All hosts should be configured to report to a syslog server
2.       All hosts should be configured to synchronize their time to the same valid time source
3.       The minimum number of hosts supported in a vSAN cluster is three
4.       The maximum number of hosts supported in a vSAN cluster is thirty-two (in 5.5)
5.       The maximum number of VMs per host is currently limited to 100 (in 5.5)
6.       The maximum number of VMS per datastore to be protected by HA is 2048.*
7.       The sweet-spot for cluster sizing is up-to sixteen hosts.

*This is important since vSAN storage appears as a single ‘datastore’.

On the host side:

1.       vSAN hosts must be comprised of certified controllers and disks

Note: Make sure and verify that the controller and disks in-use in the design appear on the VMware HCL! This is key to the supportability of the solution and must be followed. Note that the controller must support pass-through or pseudo-pass-through disk access modes, furthermore the controller must have sufficient queue depth. A minimum depth of 256 is required for vSAN (5.5), though a higher queue depth (>512) is recommended.

2.       You can have multiple disk groups per host
3.       A disk group is made up of at-least one SSD and at-least one HDD
4.       A disk group can contain up-to one SSD and up-to seven HDD each
5.       There is a maximum of five disk groups per host
6.       Utilize 10GbE interfaces for the best performance
7.       Dedicate 10GbE interfaces if you can, especially if using Broadcom adapters (see note on Network I/O Control below)
8.       If you do not have 10GbE interfaces, consider physically dedicated 1GbE interfaces for vSAN
9.       SSDs are used for caching – do not count them towards your capacity

10.   When sizing your vSAN cluster, ensure that you take into account the resiliency level (replicas) you intend to support and ensure that your SSD to HDD ratio is at-least 1:10 respectively. SSD capacity should be sized to at-least 10% of the capacity of HDDs in the disk group. An example would be if you are building a disk group of four 1.2TB 10K SAS disks, giving you’re a disk group capacity of 4.8TB, your SSD selection should be at-least 480GB.

11.   Keep in mind that by default 70% of the SSD capacity per disk group will be used as a read cache and 30% will be used as a write buffer. Using SSDs with the right bias (Read or Write Intensive) or a non-bias (Mixed Use) will make a significant difference in performance based on your intended workload so take this into account. For general purpose virtualization, the recommendation would be to use Mixed Use SSDs because of their non/even-bias.

12.   Also note that when sizing your host memory, keep in mind the ideal workload and consolidation ratios you hope to achieve. Given storage is more finite with vSAN clusters, large amounts of physical memory (>256GB) are certainly supported but may be underutilized in many environments. Keep in mind that IF you are sizing a host with 512GB or more of physical memory, the embedded SD cards are not supported and ESXi must be installed on physical media.

On the virtualization side:

1.       Both the standard and distributed virtual switches are supported.
2.       Use of the web client is required. You cannot configure vSAN using the thick client.
3.       Use of vCenter is also required. This will need to be taken into consideration on green field deployments. You will need to format one of the HDD disks on the first host and create a local datastore, install vCenter and configure it, configure vSAN and then use storage vMotion to move the VM to the new vSAN storage. Once storage vMotion is complete, you can then remove the ‘legacy’ datastore and move the disk into the vSAN disk group.

4.       vSAN storage is presented as a single common ‘datastore’ but the utilization and expression of the objects (VMs) on that store are controlled through storage policies. vSAN storage policies must be defined as they control the resiliency level (FTT, number of replicas) and other tuning parameters.

5.       When configuring HA for use with vSAN, choose ‘Power Off’ as your isolation response.
6.       When configuring HA for use with vSAN, ensure that your ‘host failures to tolerate’ setting aligns with your vSAN availability strategy and settings.
7.       vSAN does NOT (in 5.5) support FT, DPM, Storage DRS or Storage I/O Control.
8.       vSAN does support Network I/O Control and if you are using Intel adapters and the distributed virtual switch, the recommendation would be to enable and configure it for optimal performance.

Note: DO NOT enable Network IO Control (in 5.5, with or without vSAN) with Broadcom adapters! http://kb.vmware.com/kb/2065183

On the (physical and virtual) networking side:

1.       Layer-2 Multicase IS required for vSAN.
2.       It is a recommended practice to create a separate, segregated, VMkernel for vSAN data
3.       The VMkernel interface created for vSAN can utilize private IP space
4.       At-least one VLAN per vSAN cluster. vSAN clusters should NOT share the same broadcast domain

5.       It is a recommended practice to create two VLANs per vSAN cluster for maximum performance. It is however not supported to have a VMkernel for vSAN active on more than one NIC, therefore the recommendation is to set this up similarly to iSCSI. It is key that each separate VMkernel have its own IP subnet.

a.       VMkernel called vSAN0 attached to VLAN 92 with IP 192.168.92.10 and vmnic1 as active and vmnic3 as standby.*
b.      VMkernel called vSAN1 attached to VLAN 93 with IP 192.168.93.10 and vmnic3 as active and vmnic1 as standby.*

Note: Because of the Active/Standby (as opposed to Active/Unused) and the use of two different subnets, these physical switch ports must be configured as trunks and be tagged for both VLANs.

6.       The current recommended practice from VMware is to avoid the use of Jumbo Frames with vSAN

Jumbo Frames are officially supported however there was an issue discovered with jumbo frames and multicast, which vSAN makes extensive use of, in vSphere 5.5 update 2. Not sure if this has been fixed in update 3 or not but something to be aware of. The consensus from VMware support is that jumbo frames does not make a significant difference in performance with vSAN. You may utilize Jumbo Frames elsewhere in the environment, however the VMkernel(s) for vSAN should be configured for the default 1500.

7.       IP HASH link aggregation is supported by vSAN but keep in mind that since traffic will be flowing to and from the same IPs, it is unlikely that you will drive the link utilization desired using this method.

8.       For our physical switches, the same quick configuration guides for EqualLogic can be used as reference, the cabling recommendations are the same, however do not enable DCB or iSCSI optimization. You may also need to create additional VLANs and provision switch ports as trunks instead of access (tagged instead of untagged) depending on your host and cluster design.


5.5 Reference:








6.0 Reference:

What’s New in vSAN 6.0?

Configuration Maximums for vSphere 6:

VMware Virtual SAN 6.0 Design and Sizing Guide:

Hope this helps!


No comments:

Post a Comment