Before I start this post I would like to give you the background of this post. Recently, we had an uncommon issue with one of the VMware environment in my company. There was a VMware environment which was built few years back and it was working fine. All of sudden, we were informed that vCenter service was not starting and we had to pay attention to the issue.
That was correct, vCenter service was not running as it was having some issues to it’s database. After that we found that the vCenter Database server was not running. As the next step, we tried to login to the ESXi host using root credentials to check the DB server status as it was not possible to RDP in to the server. Bad luck!… Someone has reset the root credentials and not updated the password repository and we didn’t know who has done the thing, there might be an story behind this and it is not a time to talk about that.
The big problem here is we lost the entire root access to the VMware environment as well as we didn’t have the vCenter server running and DB was down. That’s why I said that’s an uncommon issue. You might not face these type of issues in your environment but I think, it’s worth of sharing my experience and how we fixed it.
That was a small environment and our management and all the other teams wanted to know the running VMs and Hosts which are holding them. I would say it is an easy task if you have access to the VMware environment. But here that was a challenge. Fortunately, we had access to the storage and we have enough spare server in our warehouse just in case.
I would like to say that we had seek the advises from VMware to break the ESXi passwords and they didn’t recommend to break the passwords and that was not supported by VMware. I know that there many posts in the internet to show the way of breaking the ESXi root passwords. But we didn’t use it……
Our initial plan was to setup an stand alone host in the same site and configure the same subnet as the management network and managed to install the ESXi 5.5 hypervisor. Also, we managed to zone and mask the Production LUNs to this host and logged in to the host, accessed the datastores to check the running VMs. That was good to see the running VMs but was not able to identify the hosts aka owners which are keeping the VMs (VMs and applications were running without any issues Production was not impacted).
We used -flat.VMDK file locks to identify the ESXi hosts owners, we created a ssh session to the stand alone ESXi host and used “vmkfstools -D” switch to identify the MAC address of the owner of the VM.
We were able to create a list of VMs and owners of the VMs to share with the other teams and understand the risk of the VMs if we had to reboot the hosts during the recovery. Fortunately, as I mentioned earlier this was a small site and number of running VMs were less.
We found the ESXi host which was holding the vCenter DB Virtual Machine, obviously there are other VMs in this host. If we want to reset the DB VM we could have easily power reset or power down the host HA will do it for us and DB VM will restart from a different host. But we wanted to check the DB sever status as we were not sure with the VM OS status and there were few things to check in the VM. We have added this VM to the Stand alone hosts inventory and it was successful, but we were not able to perform any other tasks since VMs was residing in a different host. Even though we added to the inventory file lock was with the same host.
Then we manually powered down the ESXi host which was keeping the vCenter DB, all the other VMs restarted in different hosts confirming the vSphere HA theories except the vCenter DB VM. New stand alone host put the file lock in the the vCenter DB VM and we were able to power up the VM. We have created the the virtual port groups and we were able to get the network connectivity to the VM and we were able to login to the VM OS without an issue.
We were able to start the vCenter services without an issue and we got the connectivity back to the VMware environment. As I mentioned earlier we wanted to check few things in the OS level. So now it is time to break the ESXi root passwords, you can lots of blog articles to break the root passwords and keep in mind if you follow the steps which VMware not supported you are not able to get the support in any case. That is fair enough…..
Fortunately, my licenses supported to create a host profile and I have created a host profile using one of the existing host. I set the “Administrator Password” under “Security Configuration” to “Configure fixed administrator password” option and added the password which we wanted.
After applying the host profile to the hosts we got the root access to the hosts. This post is like a story but I wanted to share my experience as I didn’t come up similar situation in my life. I hope this might helpful if you forgot your root ESXi passwords.