A small step to a Giant Leap

vCenter 6.5 High Availability - Initiate the Failover

I'm so impressed with this vCenter 6.5 HA feature and I was testing this in my test environment. I thought to write this article while I was testing the vCenter HA Failover. I wrote couple of posts regarding this feature and I'm writing this post to show the way of initiating the Failover. 

If you are doing the patch upgrades or any related maintenance you can use this feature to minimize the downtime of the vCenter. 

So, login to the vCenter and go the vCenter HA option under the "Configuration" tab. You can see the status of the nodes and please note that my active node is node 2 which is 

To initiate the Failover click on the "Initiate Failover" button on the right upper corner 

vCenter 6.5 High Availability - Possible Configuration Issues

Hope you read my previous post with related to the VCSA 6.5 HA configuration. Again!, this is a new feature came with VMware vCenter 6.5 to minimize the down time of the vCenter server. That is one of the cool feature so far. 

Here, I would like to share something that I experienced while I was deploying and configuring the vCenter HA feature. I believe, this will make your life more easier if you are configuring the "HA" feature.

SSH should be enabled ...

First of all, I mentioned that SSH should be enabled on the appliance if you are deploying the nodes, otherwise you will get a notification and deployment get failed. 

vCenter 6.5 - High Availablity Basic configuration

VMware has introduced the vCenter High Availability feature from 6.5 on wards to minimize the significant down time of the vCenter during the Host hardware failure and the patching or any related maintenance.

Basically, it has two configuration options:
  • Basic 
  • Advanced
Here we are discussing only the "Basic" configuration of the High Availability feature

There are few special requirements to enable this feature in the vCenter
  • This feature only comes with the vCenter 6.5 
  • vCenter deployment size should be at least small therefore 4 vCPUs and 16 GB of RAM required
  • A minimum of 3 Hosts required 
  • Hosts should be at least ESXi 5.5
  •  Management network should be configured with a static IP and FQDN should be reachable 
  • SSH should be enabled on the appliance 
  • Separate Portgroup for the HA network is required 
  • HA network must be a different subnet from the management subnet 
  • Net work latency between the hosts must be less than 10ms 
  • This feature is available with embedded and separate PSC deployment 
You can read performance and best practices from this article

vCenter Server Appliacnce (VCSA) 6.5 Deployment

You may already heard of that VMware released the vCenter server 6.5 and you can experience new features with this new release. I'm in the middle of articles writing during my busy schedule and this is the article for the vCenter Server Appliance aka VCSA 6.5 Deployment. 

It is not just a typical appliance deployment and there are few changes when compared to the traditional pre-built OVA or OVF appliances. Basically, it has two main Stages in the deployment.

Let's get started....

Stage 1 Deployment

 First of all you need to setup your DNS entries for your vCenter Appliance as that's a common requirement for every deployment.

ESXi Password reset - Host Profile the life saver

Before I start this post I would like to give you the background of this post. Recently, we had an uncommon issue with one of the VMware environment in my company. There was a VMware environment which was built few years back and it was working fine. All of sudden, we were informed that vCenter service was not starting and we had to pay attention to the issue. 

That was correct, vCenter service was not running as it was having some issues to it's database. After that we found that the vCenter Database server was not running. As the next step, we tried to login to the ESXi host using root credentials to check the DB server status as it was not possible to RDP in to the server. Bad luck!... Someone has reset the root credentials and not updated the password repository and we didn't know who has done the thing, there might be an story behind this and it is not a time to talk about that. 

The big problem here is we lost the entire root access to the VMware environment as well as we didn't have the vCenter server running and DB was down. That's why I said that's an uncommon issue. You might not face these type of issues in your environment but I think, it's worth of sharing my experience and how we fixed it. 

That was a small environment and our management and all the other teams wanted to know the running VMs and Hosts which are holding them. I would say it is an easy task if you have access to the VMware environment. But here that was a challenge. Fortunately, we had access to the storage and we have enough spare server in our warehouse just in case. 

I would like to say that we had seek the advises from VMware to break the ESXi passwords and they didn't recommend to break the passwords and that was not supported by VMware. I know that there many posts in the internet to show the way of breaking the ESXi root passwords. But we didn't use it......

Our initial plan was to setup an stand alone host in the same site and configure the same subnet as the management network and managed to install the ESXi 5.5 hypervisor. Also, we managed to zone and mask the Production LUNs to this host and logged in to the host, accessed the datastores to check the running VMs. That was good to see the running VMs but was not able to identify the hosts aka owners which are keeping the VMs (VMs and applications were running without any issues Production was not impacted). 

We used -flat.VMDK file locks to identify the ESXi hosts owners, we created a ssh session to the stand alone ESXi host and used "vmkfstools -D" switch to identify the MAC address of the owner of the VM. 

vSphere 6.0 - VMware Auto Deploy Configuration

If you have a large VMware environment with hundreds of ESXi hosts, deploying hosts might be a big challenge. VMware introduced this Auto Deploy feature for the bulk ESXI deployment and previously, configuration was bit complex. This feature was a separate component and with vSphere 6.0 it comes as a inbuilt feature of the vCenter, you need only the activation and start the service. 

Here in my post I'm going to show the way that you can configure this feature in the vSphere 6.0 vCenter server, I'm really sorry If I made this post long as I need to show you the every single step that I followed in my deployment, that might be really helpful to a new starter and those who new to Auto Deploy feature.

By default, hosts provisioned by the auto deploy pull down the ESXi image each time at the host boots. If the auto deploy server or the FTP server which is holding the ESXi image is down host will not boot using the auto deploy. You can see a similar error and your host will reboot frequently. 

First of all, you need have a DHCP server in place in your environment and it should support the PXE boot and the it should be able to point the TFTP boot server for your Auto Deploy ESXi server. In my case I'm using a Windows 2008 R2 Server as my DHCP server and SolarWinds free TFTP server to stream the boot image. 

VMware vExpert for the First time...

Yesterday, VMware announced the VMware vExpert for the year 2017, I'm glad to say that my name is also there and I'm really excited with this award as this is the first time I got this award from VMware. 

VMware vExpert 2017

You can find all the VMware vExperts for 2017 here .

VMware vSphere 6 - vCenter Server Installation

I'm writing this article to show you the complete installation of the vCenter Server 6. It's a pretty straight forward and I'm using the Windows based Server (2008 R2) to install my vCenter server which is my favorite. I have installed the Platform Service Controller in a different server and I'm pointing my vCenter server to the installed PSC. You can find my previous PSC installation post here .

Check the vCenter server System requirement according your deployment method from the vSphere 6 Documentation Center .

Note: If you are planing to use a Windows 2016 64 bit server you can install only the vCenter server 6.5. Check the Operating System Compatibility from here .

Once you setup the Server for the vCenter server, mount your vCenter 6.x ISO and run the installer as usual and click on "Install" to start the installation

VMware vSphere 6 - ESXi Syslog and Network Dump collector configuration

Sometimes, you might be aware of this and might be using this in your environment. But I thought it is a good time to write an article about VMware Syslog and Network Dump Collector configurations as I'm doing some Autodeploy configurations. 

Setting up a Syslog collector and Network Dump collector is a must when you are up to use the VMware Autodeploy feature. Bulk ESXi Configuration and Deployment might be a headache when you have more than thousands of ESXi hosts to deal with as your daily operations.

As I mentioned earlier Syslog and Network Core Dump Collector is a must when you have auto deployed hosts which are not having a local disk to store the system files. In a situation like that the log files of these hosts store in the RAM disk which means each time this host reboots log files will be destroyed. That can lead you to a huge problem where you can't find the exact issues such as PSOD incidents. As a precaution of that you need to have a separate log collector of your ESXi Hosts. 

Syslog and Dump Collector services are in built to the vSphere 6.x and you need to setup this separately in vSphere 5.x. Here I'm using my vSphere 6.x environment and I'm focusing the ESXi host level configuration to pass the log files to your remote log collector. 

Setting up the Syslog Collector 

Login to your ESXi host with a SSH session and check the Syslog configurations with the below command 

"esxcli system syslog config get" and check the remote host 

VMware vSphere 6 - Platform Service Controller Installation

In this article I'm focusing on the installation steps of the vSphere 6.0 Platform Service Controller aka PSC before I install the vCenter Server. It is a pretty straight forward installation and You can easily deploy this and integrate to your new vSphere 6.x environment. There are ways we can maintain the High availability of the PSCs, I'm planing to discuss them in a different post. 

Basically, PSC deals with the identity management for the vSphere users and applications. Also, below services are installed with the PSC.
  • VMware Appliance Management Service (only in Appliance-based PSC)
  • VMware License Service
  • VMware Component Manager
  • VMware Identity Management Service
  • VMware HTTP Reverse Proxy
  • VMware Service Control Agent
  • VMware Security Token Service
  • VMware Common Logging Service
  • VMware Syslog Health Service
  • VMware Authentication Framework
  • VMware Certificate Service
  • VMware Directory Service
Below VMware products and components are supported with the PSC
  • VMware vCenter Server
  • VMware vCenter Inventory Services
  • VMware vSphere Web Client
  • VMware Log Browser
  • VMware NSX for vSphere
  • VMware Site Recovery Manager
  • VMware vCloud Air
  • VMware vCloud Director
  • VMware vRealize Automation Center
  • VMware vRealize Orchestrator
  • VMware vSphere Data Protection
  • VMware vShield Manager
You can deploy this as an appliance or a windows based server and it is compatible with both Windows/Appliance based vCenter server.

Requirements when deploying the Appliance-based Platform Services Controller:
  •  Processor - Intel or AMD x64 processor with two or more logical cores, each with a speed of 2 GHz
  •  Memory - 2 GB
  •  Disk storage - 30 GB
  •  Network speed - 1 Gbps
Requirements when deploying the Windows-based Platform Services Controller:
  • Processor - Intel or AMD x64 processor with two or more logical cores, each with a speed of 2 GHz
  •  Memory - 2 GB
  •  Disk storage - 4 GB
  •  Network speed - 1 Gbps
Ok, Let's jump in to the installation straight away...

 I'm using a Windows 2008 R2 server to install the PSC component. Mount your vSphere 6.x ISO and launch the installation. Select the "vCenter Server for Windows" to start the installation.

VMware vSphere 6 - Enable Lockdown Mode in VMware ESXi 6.0

Maintain a high security environment for your Infrastructure is one of the great best practices in many ways. There are many ways which you can restrict access to your infrastructure just avoid any unexpected situations for your sensitive data. 

Here, what I'm going to show is one of the common and best way to restrict access to your VMware environment. We call it "VMware Lockdown Mode". There are two types of Lockdown modes available in VMware so far. 

  • Normal - Host is accessible only through the DCUI (Direct Control User Interface) or the VMware vCenter server. If you lost the access to your vCenter server, still you have a chance to exit the host from the "Lockdown Mode" with a privileged user account
  • Strict - You are not able to access DCUI with the Strict mode and you must use the vCenter to access and remove the "Lockdown Mode". If you lost access to the vCenter server while you enabled the "Strict Lockdown Mode" your host might be unavailable. Only option you have is re-install the ESXi from the scratch
To enable the "Lockdown Mode" Right-click on the host which you want to enable the lockdown mode and select "Settings" (I'm using the web console)

Go to "Manage" tab and select "Security Profile", then you will be able to find the Lockdown mode. Click on "Edit" to change the setting

Then you will get the "Lockdown Mode" window and you can enable the Normal/Strict mode according to your requirement. I'm enabling the "Strict Mode" in my case. Click "OK" to accept the Warning message and enable the Strict Lockdown mode

Click "OK" and note the completed tasks in the "Recent Task" window 

Now you have enabled the "Lockdown Mode" and try to access it through the DCUI. DCUI is stopped in the "Strict Lockdown Mode". You need to only use the vCenter to turn off the Strict Lockdown Mode

Adding Exception Users in to the Lockdown Mode

Still you can provide access to your host while you are in the Lockdown mode for some user accounts to continue the work and access. This might be helpful for third party applications or solutions to continue the functionality while in the Lockdown Mode. 

Once you get the Lockdown Mode window select the "Exception Users" and click on the green "+" sign to add the local users to the lockdown mode

Add the local user to the "Exception Users" list and click "OK" to add to the users

You can see the added users and "OK" to complete the user addition 

Use DCUI to disable the "Normal Lockdown Mode"

Let's say you are on the "Normal Lockdown Mode" still you can use the DCUI to disable the lockdown mode without the vCenter. Again, if you are on the "Strict Lockdown Mode" you are not able to do this. 

I'm enabling the "Normal Lockdown Mode" 

Login to your DCUI with the root credentials and select the "Configure Lockdown Mode", you can see it as enabled, hit "Enter" to change the mode

Use the "Space bar" to toggle the selection 

You can see it as Disabled once you change the Mode


Note : SSH and Shell is not affected by Lockdown Mode and you need to disable/enable it manually

VMware Update Manager Configuration


If you are maintaining a VMware environment, I believe this is not a new thing and you may already using VMware Update Manager aka VUM for your Host upgrades and Patch management. You need to have a separate server for this and with vSphere 6.5 you are getting this as an inbuilt feature with your vSphere vCenter Appliance. If you are interested in reading more about this you can read VMware blog article here

I thought it is a good idea to write this article as there are few interns/freshes in my team and they are also interested in learning these hosts upgrades and patch management tasks. This article is based on the VMware Update Manager 5.5. 

First of all, as I mentioned earlier you need to have a separate Windows based server to install the VUM and there should be a clear connectivity to your vCenter server. Mount your VMware installation ISO to your VUM server and start the installation by clicking the "install" button.