Nested VMware VSAN in vSphere 6.5

Steps to prepare a Nested VMware VSAN in vSphere 6.5 Host in my lab

Prepare the following components in a single ESXi 6.7 host with full SSD HDD installed

• 1 x vCenter Appliance 6.5
• 3 x vSphere ESXi 6.5 U2 Hosts with
  • 2 x vCPU (Expose hardware assisted virtualization to the guest OS)
  • 6GB RAM (Minimum 6GB RAM is required)
  • 2 x vNIC
  • 1 x 10GB HDD – OS
  • 1 x 5GB HDD – Cache Tier
  • 1 x 30GB HDD – Capacity Tier

Enable the Expose hardware assisted virtualization to the guest OS for each ESXi 6.5 Hosts

1. Register the Hostname with Reverse Lookup records for vCenter & ESXi hosts in AD DNS Server with PowerShell

#Remote PS Session
$ComputerName = "lab-ad01.mylab.local"
Enter-PSSession -ComputerName $ComputerName

#DNS Records 

$DnsZone = "mylab.local"
Get-DnsServerZone

#List all A records 
Get-DnsServerResourceRecord -ZoneName $DnsZone | Where-Object {$_.RecordType -eq "A"}

#Reverse Lookup Zone
Add-DnsServerPrimaryZone -DynamicUpdate Secure -NetworkId ‘192.168.1.0/24’ -ReplicationScope Domain

#Add DNS Records for all hosts
$Vcsa_Name = "vcsa"
$Vcsa_IP = "192.168.1.170"
$ESX1_Name = "uat-esx1"
$ESX1_IP = "192.168.1.171"
$ESX2_Name = "uat-esx2"
$ESX2_IP = "192.168.1.172"
$ESX3_Name = "uat-esx3"
$ESX3_IP = "192.168.1.173"

Add-DnsServerResourceRecordA -IPv4Address $Vcsa_IP -Name $Vcsa_Name -ZoneName $DnsZone
Add-DnsServerResourceRecordA -IPv4Address $ESX1_IP -Name $ESX1_Name -ZoneName $DnsZone
Add-DnsServerResourceRecordA -IPv4Address $ESX2_IP -Name $ESX2_Name -ZoneName $DnsZone
Add-DnsServerResourceRecordA -IPv4Address $ESX3_IP -Name $ESX3_Name -ZoneName $DnsZone

#Add Reverse Lookup Address
$ReverseLookupZone = "1.168.192.in-addr.arpa"
Add-DnsServerResourceRecordPtr -Name "170" -ZoneName $ReverseLookupZone -AllowUpdateAny -AgeRecord -PtrDomainName "vcsa.mylab.local"
Add-DnsServerResourceRecordPtr -Name "171" -ZoneName $ReverseLookupZone -AllowUpdateAny -AgeRecord -PtrDomainName "uat-esx1.mylab.local"
Add-DnsServerResourceRecordPtr -Name "172" -ZoneName $ReverseLookupZone -AllowUpdateAny -AgeRecord -PtrDomainName "uat-esx2.mylab.local"
Add-DnsServerResourceRecordPtr -Name "173" -ZoneName $ReverseLookupZone -AllowUpdateAny -AgeRecord -PtrDomainName "uat-esx3.mylab.local"

2. Create a Virtual Switch, Port Group with VMKernel Adapter for VSAN in 3 x ESXi 6.5 Hosts

#Import VMware Module
Import-Module VMware.VimAutomation.Core

$ESX1 = "192.168.1.171"
$User = "root"
$Password = "P@ssw0rd!@#$"

Connect-VIServer -Server $vCenter -User $User -Password $Password -WarningAction SilentlyContinue

#List NIC installed
Get-VMHostNetworkAdapter 

$vSwitch = "vS_UAT"
$PortGroup = "vKernel"

#Create new vSwitch
New-VirtualSwitch -VMHost $ESX1 -Name $vSwitch -nic vmnic3

#Create new Port Group 
Get-VirtualSwitch -name $vSwitch | New-VirtualPortGroup -Name $PortGroup

#New Kernel 
New-VMHostNetworkAdapter -VMhost $ESX1 -PortGroup $PortGroup -VirtualSwitch $vSwitch -IP 10.10.10.171 -SubnetMask 255.255.255.0 -VMotionEnabled $true -VsanTrafficEnabled $true -Mtu 9000 

3. Preparation of vCenter

• Deploy VCSA 6.5 with CLI
• Login to https://192.168.1.170:5480 to adjust the timezone

• Create Data Center, Cluster and add 3 x ESXi 6.5U2 Hosts to the Cluster. In Addtional to that, we also enable SSH Access for all ESXi hosts

#Import VMware Module
Import-Module VMware.VimAutomation.Core

$vCenter = "192.168.1.170"
$User = "[email protected]"
$Password = "P@ssw0rd"

$VCSA = Connect-VIServer -Server $vCenter -User $User -Password $Password -WarningAction SilentlyContinue
Disconnect-VIServer * 

$DataCenter = "MYLAB"
$Cluster = "MYLAB-CLUSTER"

#New Data Center
New-Datacenter -Location (Get-Folder -NoRecursion) -Name $DataCenter

#New Cluster without HA & DRS enabled 
New-Cluster -Location $DataCenter -Name $Cluster

#Add Hosts to Cluster
Add-VMHost uat-esx1.mylab.local -location $Cluster -user root -password P@ssw0rd -force:$true
Add-VMHost uat-esx2.mylab.local -location $Cluster -user root -password P@ssw0rd -force:$true
Add-VMHost uat-esx3.mylab.local -location $Cluster -user root -password P@ssw0rd -force:$true

#Enable SSH 
#Start SSH for Host
Start-VMHostService -HostService (Get-VMHost | Get-VMHostService | Where { $_.Key -eq “TSM-SSH” } )

#Supress SSH Warning
Get-VMHost | Get-AdvancedSetting UserVars.SuppressShellWarning |Set-AdvancedSetting -Value 1

Get-Datastore -Name Datastore1 | Set-Datastore -Name LOCAL_ESX1
Get-Datastore -Name "Datastore1 (1)" | Set-Datastore -Name LOCAL_ESX2
Get-Datastore -Name "Datastore1 (2)" | Set-Datastore -Name LOCAL_ESX3

• Create a Distribution Switch, Port Group and assign the VMkernel for VSAN to it

#Distribution Switch
$ESX1 = "uat-esx1.mylab.local"
$ESX2 = "uat-esx2.mylab.local"
$ESX3 = "uat-esx3.mylab.local"

$DS_VSAN = "DS-VSAN"
$DP_VSAN = "DP-VSAN"
$DS_PROD = "DS-PROD"
$DP_PROD = "DP-PROD"

#New Distribution Switch (DS)
New-VDSwitch -NumUplinkPorts 1 -Name $DS_VSAN -mtu 9000 -Location $DataCenter
New-VDSwitch -NumUplinkPorts 1 -Name $DS_PROD -Location $DataCenter

#Assign ESXi Host to DS 
Add-VDSwitchVMHost -vdswitch $DS_VSAN -vmhost $ESX1
Add-VDSwitchVMHost -vdswitch $DS_VSAN -vmhost $ESX2
Add-VDSwitchVMHost -vdswitch $DS_VSAN -vmhost $ESX3

Add-VDSwitchVMHost -vdswitch $DS_PROD -vmhost $ESX1
Add-VDSwitchVMHost -vdswitch $DS_PROD -vmhost $ESX2
Add-VDSwitchVMHost -vdswitch $DS_PROD -vmhost $ESX3

#Create a VDPort Group for the DS
Get-VDSwitch -Name $DS_VSAN| New-VDPortgroup -Name $DP_VSAN -NumPorts 8 
Get-VDSwitch -Name $DS_PROD| New-VDPortgroup -Name $DP_PROD -NumPorts 8

#Assign Physical NIC to DS Switch from ESXi Hosts- VSAN
$Physical_NIC_ESX1_VSAN = Get-VMHostNetworkAdapter -VMHost $ESX1 -Name "vmnic1"
$Physical_NIC_ESX2_VSAN = Get-VMHostNetworkAdapter -VMHost $ESX2 -Name "vmnic1"
$Physical_NIC_ESX3_VSAN = Get-VMHostNetworkAdapter -VMHost $ESX3 -Name "vmnic1"

#Assign Physical NIC to DS Switch from ESXi Hosts - PROD
$Physical_NIC_ESX1_PROD = Get-VMHostNetworkAdapter -VMHost $ESX1 -Name "vmnic2"
$Physical_NIC_ESX2_PROD = Get-VMHostNetworkAdapter -VMHost $ESX2 -Name "vmnic2"
$Physical_NIC_ESX3_PROD = Get-VMHostNetworkAdapter -VMHost $ESX3 -Name "vmnic2"

#Get the VMKernel Port Group
$VMKernel_ESX1 = Get-VMHostNetworkAdapter -VMHost $ESX1 -Name "vmk1"
$VMKernel_ESX2 = Get-VMHostNetworkAdapter -VMHost $ESX2 -Name "vmk1"
$VMKernel_ESX3 = Get-VMHostNetworkAdapter -VMHost $ESX3 -Name "vmk1"

#Assign VMKernel Port Group to Distribution Switch - Port Group -VSAN
Get-VDSwitch $DS_VSAN | Add-VDSwitchPhysicalNetworkAdapter -VMHostVirtualNic $VMKernel_ESX1 -VMHostPhysicalNic $Physical_NIC_ESX1_VSAN -VirtualNicPortgroup $DP_VSAN -Confirm:$false
Get-VDSwitch $DS_VSAN | Add-VDSwitchPhysicalNetworkAdapter -VMHostVirtualNic $VMKernel_ESX2 -VMHostPhysicalNic $Physical_NIC_ESX2_VSAN -VirtualNicPortgroup $DP_VSAN -Confirm:$false
Get-VDSwitch $DS_VSAN | Add-VDSwitchPhysicalNetworkAdapter -VMHostVirtualNic $VMKernel_ESX3 -VMHostPhysicalNic $Physical_NIC_ESX3_VSAN -VirtualNicPortgroup $DP_VSAN -Confirm:$false

#Assign Physical NIC to DS-PROD
Get-VDSwitch $DS_PROD | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $Physical_NIC_ESX1_PROD -Confirm:$false
Get-VDSwitch $DS_PROD | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $Physical_NIC_ESX2_PROD -Confirm:$false
Get-VDSwitch $DS_PROD | Add-VDSwitchPhysicalNetworkAdapter -VMHostPhysicalNic $Physical_NIC_ESX3_PROD -Confirm:$false

#Enable VSAN
Get-Cluster | Set-Cluster -VsanEnabled:$true -VsanDiskClaimMode Automatic -Confirm:$false -ErrorAction SilentlyContinue -Verbose 

5. Enable VSAN

#Enable VSAN
Get-Cluster | Set-Cluster -VsanEnabled:$true -VsanDiskClaimMode Automatic -Confirm:$false -ErrorAction SilentlyContinue -Verbose 

Change Add Disks to Storage from Automatic to Manual

Verify all the disks had been claimed for VSAN

Enable Performance Service

Verify the VSAN Healty – You can ignore the Warning on Hardware Compatibility as we are running VSAN on Nested VM which is NOT officialy supported

VSAN 6.6 is up and running fine now, and you can provision Virtual Machines (VM) on the VSAN Datastore.

New VM provisioned will consume 2x VSAN disk space with the default storage policy.