vmware expert

VMware vCenter Server 6.7 Update 2

VMware just released a new vCenter Server version: 6.7 Update 2,, build 13010631. In this article I will cover some of the new features and resolved issues. I will also demonstrate how easy is to update from a previous version of vCenter Server 6.7 to VMware vCenter Server 6.7 Update 2.

In case you are looking for a plain installation of vCenter Server 6.7, you can check my other article: How to Install VCSA 6.7 (VMware vCenter Server Appliance).

VMware vCenter Server 6.7 Update 2 New Features

vCenter Server 6.7 Update 2 introduces Virtual Hardware Version 15 which adds support for creating virtual machines with up to 256 virtual CPUs.

There are few changes in vCenter backups: you can use NFS v3 (Network File System) and SMB2 (Server Message Block) protocols for file-based backup and restore operations. Also it adds version details to the “Enter backup details” page that help you to pick the correct build to restore the backup file. You can create alarm definitions to monitor the backup status of your system (using email, SNMP traps or scripts as actions).

vCenter Server 6.7 Update 2 introduces the Developer Center with two new features: API Explorer and Code Capture. This update brings API Explorer (formerly accessible via https://<vCSA-FQDN>/apiexplorer) into the vSphere Client, thus removing the extra steps to authenticate prior to interacting with the REST APIs. If you ever played with the old Onyx flings, you will enjoy Code Capture. Just enable recording, do something in vSphere Client, then end recording and see the equivalent PowerCLI code generated.

VMware vCenter Server 6.7 Update 2 - Code Capture

You can now publish your VM templates managed by Content Library from a published library to multiple subscribers. You can trigger this action from the published library, which gives greater control over the distribution of VM templates.

vCenter Server 6.7 Update 2 Resolved Issues

VMware vCenter Server 6.7 Update 2 resolves plenty of issues with vMotion, backup, auto deploy, VMware tools, storage, management of VMs, and networking.

  • vSphere vMotion operations for encrypted virtual machines might fail after a restart of the vCenter Sever system
  • Power-on or vSphere vMotion operations with virtual machines might fail with an infinite loop error
  • Migrating a virtual machine might fail due to inability to access the parent disk
  • Migrating a virtual machine might fail due to inability to access the parent disk
  • VMware vSphere Auto Deploy Discovered Hosts tab might display an error after creating or editing a deployment rule
  • Customization of virtual machines by using Microsoft Sysprep on vSphere 6.7 might fail and virtual machines stay in customization state
  • The c:\sysprep directory might not be deleted after Windows guest customization
  • You might not see the configured CPU shares when exporting a virtual machine to OVF
  • vCenter Server might stop responding when adding a fault message in the vSphere Storage DRS
  • The vpxd service might fail when the vSphere Storage DRS provides an initial placement operation
  • ESXi hosts with visibility to RDM LUNs might take a long time to start or experience delays during LUN rescans
  • Expanding the disk of a virtual machine by using VMware vRealize Automation might fail with an error for insufficient disk space on a datastore
  • Provisioning of virtual machines might fail if the same replication group is used for some or all virtual machine files and disks
  • You cannot add permissions for a user or group beyond the first 200 security principals in an Active Directory domain by using the vSphere Client
  • User login and logout events might not contain the IP address of the user
  • The vCenter Server daemon service vpxd might fail to start with an error for invalid descriptor index
  • Cloning a virtual machine from a snapshot of a template might fail with a “missing vmsn file” error
  • An internal error might occur in alarm definitions of the vSphere Web Client
  • Attempts to log in to a vCenter Server system after an upgrade to vCenter Server 6.7 might fail with a credentials validation error
  • Migration of vCenter Server for Windows to vCenter Server Appliance might stop at 75% if system time is not synchronized with an NTP server
  • Upgrading vCenter Server for Windows to 6.7 Update 2 from earlier versions of the 6.7 line might fail
  • vCenter Server upgrades might fail due to compatibility issue between VMware Tools version 10.2 and later, and ESXi version 6.0 and earlier
  • You might see a message that an upgrade of VMware vSphere Distributed Switch is running even after the upgrade is complete
  • You cannnot migrate virtual machines by using vSphere vMotion between ESXi hosts with NSX managed virtual distributed switches (N-VDS) and vSphere Standard Switches

VMware vCenter Server 6.7 Update 2 also updates some of the internal packages used.

  • VMware Postgres is updated to version 9.6.11
  • Oracle (Sun) JRE is updated to version 1.8.202.
  • Apache httpd is updated to version 2.4.37
  • The OpenSSL package is updated to version openssl-1.0.2q.
  • The ESXi userworld libxml2 library is updated to version 2.9.8.
  • The OpenSSH is updated to version 7.4p1-7.

For full list of resolved issues you can check the Release Notes.

How to Update to vCenter Server 6.7 Update 2

I will demonstrate an online update from vCenter Appliance Management console. I logged in to https://<vCSA-FQDN>:5480/ using the root appliance password, then I navigated to Update menu. After a short check, I can see my current version is and I have an available update to (which is vCenter Server 6.7 Update 2). I will click on “Stage and install” link.

VMware vCenter Server 6.7 Update 2 - Check Update Availability

Next step is to accept the end user license agreement (EULA). Check the “I accept…” checkbox and click on “Next”.

VMware vCenter Server 6.7 Update 2 - End User License Agreement

The installer will run pre-update checks now. For example, if your root password has expired, you will receive a notice and you will not be able to proceed further before fixing the problem. If everything is allright, the wizard will jump to the next screen. You can see a downtime estimation (which proved to be waaay overestimated in my case). Confirm you have a backup of vCenter Server and click on “Finish”.

VMware vCenter Server 6.7 Update 2 - Backup Server

We can sit down and relax now while the vCenter Server is upgraded.

VMware vCenter Server 6.7 Update 2 - Installation in Progress
VMware vCenter Server 6.7 Update 2 - Stopping Services
VMware vCenter Server 6.7 Update 2 - Installing Packages

After some time we will be logged out from the appliance. Wait few minutes and then you can log back in.

VMware vCenter Server 6.7 Update 2 - Appliance Management Login

Installation is now completed!

VMware vCenter Server 6.7 Update 2 - Installation Completed

Going on the Summary page of the Appliance Management console, you can see the new version:, build 13010631.

VMware vCenter Server 6.7 Update 2 - Status

This article was provided by our service partner :


Get your data ready for vSphere 5.5 End of Support

There have been lots of articles and walkthroughs on how to make that upgrade work for you, and how to get to a supported level of vSphere. This VMware article is very thorough walking through each step of the process.

But we wanted to touch on making sure your data is protected prior, during and after the upgrade events.

If we look at the best practice upgrade path for vSphere, we’ll see how we make sure we’re protected at each step along the way:

vSphere EOL

Upgrade Path

The first thing that needs to be considered is what path you’ll be taking to get away from the end of general support of vSphere 5.5. You have two options:

  • vSphere 6.5 which is now going to be supported till November 2021 (so another 5 years’ time)
  • vSphere 6.7 which is the latest released version from VMware.

Another consideration to make here is support for surrounding and ecosystem partners, including Veeam. Today, Veeam fully supports vSphere 6.5 and 6.7, however, vSphere 6.5 U2 is NOT officially supported with Veeam Backup & Replication Update 3a due to the vSphere API regression.

The issue is isolated to over-provisioned environments with heavily loaded hosts (so more or less individual cases).

It’s also worth noting that there is no direct upgrade path from 5.5 to 6.7. If you’re currently running vSphere 5.5, you must first upgrade to either vSphere 6.0 or vSphere 6.5 before upgrading to vSphere 6.7.

Management – VMware Virtual Center

The first step of the vSphere upgrade path after you’ve decided and found the appropriate version, is to make sure you have a backup of your vCenter server. The vSphere 5.5 virtual center could be a Windows machine or it could be using the VCSA.

Both variants can be protected with Veeam, however, the VCSA runs on a Postgres-embedded database. Be sure to take an image-level backup with Veeam and then there is a database backup option within the appliance. Details of the second step can be found in this knowledge base article.

If you’re an existing Veeam customer, you’ll already be protecting the virtual center as part of one of your existing backup jobs.

You must also enable VMware tools quiescence to create transactionally-consistent backups and replicas for VMs that do not support Microsoft VSS (for example, Linux VMs). In this case, Veeam Backup & Replication will use the VMware Tools to freeze the file system and application data on the VM before backup or replication. VMware Tools quiescence is enabled at the job level for all VMs added to the job. By default, this option is disabled.

vSphere EOL 02

You must also ensure Application-Aware Image Processing (AAIP) is either disabled or excluded for the VCSA VM.

vSphere EOL 03

Virtual Machine Workloads

If you are already a Veeam customer, then you’ll already have your backup jobs created and working with success before the upgrade process begins. However, as part of the upgrade process, you’ll want to make sure that all backup job processes that initiate through the virtual center are paused during the upgrade process.

If the upgrade path consists of new hardware but with no vMotion licensing, then the following section will help.

Quick Migration

Veeam Quick Migration enables you to promptly migrate one or more VMs between ESXi hosts and datastores. Quick Migration allows for the migration of VMs in any state with minimum disruption.

More information on Quick Migration can be found in our user guide.

During the upgrade process

As already mentioned in the virtual machine workloads section, it is recommended to stop all vCenter-based actions prior to update. This includes Veeam, but also any other application or service that communicates with your vCenter environment. It is also worth noting that whilst the vCenter is unavailable, vSphere Distributed Resource Scheduler (DRS) and vSphere HA will not work.

Veeam vSphere Web Client

If you’re moving to vSphere 6.7 and you have the Veeam vSphere Web Client installed as a vSphere plug-in, you’ll need to install the new vSphere Veeam web client plug-in from a post-upgraded Veeam Enterprise Manager.

vSphere EOL 04

More detail can be found in Anthony Spiteri’s blog post on new HTML5 plug-in functionality.

You’ll also need to ensure that any VMware-based products or other integrated products vCenter supports are the latest versions as you upgrade to a newer version of vSphere.

Final Considerations

From a Veeam Availability perspective, the above steps are the areas that we can help and make sure that you are constantly protected against failure during the process. Each environment is going to be different and other considerations will need to be made.

Another useful link that should be used as part of your planning: Update sequence for vSphere 5.5 and its compatible VMware products (2057795)

One last thing is a shout out to one of my colleagues who has done an in-depth look at the vSphere upgrade process.

This article was provided by our service partner : 

Good Bye, VMware vSphere Web Client

VMware has announced to deprecate the Flash-based vSphere Web Client with the next numbered release (not update release) of vSphere. The next version of vSphere will be the terminal release for which vSphere Web Client will be available.

Since vSphere web client is based on Adobe flash technology, It results in less than ideal performance as compared to HTML5 based vSphere client and also has constant update requirements. Additionally, Adobe also has recently announced plans to deprecate Flash.

vsphere web client

Currently we have two variants of the vSphere GUIs which includes the vSphere Web Client and HTML5-based vSphere Client in vSphere 6.5 to manage the operation of virtual datacenter.

With the decommissioning of windows based vSphere client, VMware also introduced the HTML5 based vSphere client with vSphere 6.5. Which provides the solid performance as compared to the vSphere web client. The vSphere Client was introduced first in the Fling, then supported with vSphere 6.5. Since its introduction, the vSphere Client has received positive responses from the vSphere community and customer base.

With the recently released vSphere 6.5 Update 1, the vSphere Client got even better and is now able to support most of the frequently performed operations. With each iteration of the vSphere Client additional improvements and functionality are being added.

By the time the vSphere Web Client is deprecated, the vSphere Client will be full featured but with significantly better responsiveness and usability.

The HTML based vSphere Client will be the primary GUI administration tool for vSphere environments starting in the next release. It is recommended that customers should start transitioning over to the HTML5 based vSphere Client as the vSphere Web Client will no longer be available after the next vSphere release. This announcement from VMware gives ample time to customers to prepare for the eventual vSphere Web Client deprecation.

VMware vCenter Converter

VMware vCenter Converter : Tips and Best Practices

Vmware vCenter converter can convert Windows and Linux based physical machine and Microsoft hyper-v systems into Vmware virtual machines.

Here are some tips and suggested best practices

Tasks to perform before conversion :

  • Make sure you know the local Administrator password! If the computer account gets locked out of the domain – you are likely going to need to login locally to recover
  • Ensure you are using the latest version of Vmware vCenter converter.
  • If possible, install Vmware vCenter Converter locally on the source (physical machine) operating system.
  • Make a note of the source machine IP addresses. The conversion will create a new NIC and having those IP details handy will help.
  • Disable any anti-virus
  • Disable SSL encryption – this should speed up the conversion ( described here )
  • If you have stopped and disabled any services – make sure to take a note of their state beforehand. A simple screenshot goes a long way here!
  • If converting from hyper-v -> vmware. Install the Converter on the host and power down the converter before starting the conversion.
  • Uninstall any hardware specific software utilies from the source server
  • If the source system has any redundant NICs – I would suggest removing them in the Edit screen on the converter ui.
  • For existing NICs – use the VMXNET3 driver and set it to not connected.

Special considerations for Domain Controllers, MS exchange and SQL servers.

Although – You tend to get warned off converting Domain controllers, they do work OK if you take some sensible precautions:

  • Move FSMO roles to Another Domain Controller
  • Make another Domain Controller PDC
  • Stop Active Directory services
  • Stop DHCP service ( if applicable )
  • Stop DNS service ( if applicable )

For SQL and Exchange, you should stop and disable all Exchange and SQL services on the source machine and only start them back up on the target VM once you are happy the server is successfully back on the domain.

( note these steps are not necessary for V2V conversations and you should have the system powered off!)


Tasks to perform after conversion :

  • Once the conversion has successfully completed, get the source physical machine off the network. You can disable the NIC, pull the cable and/or power it down. It should not come up again.
  • For V2V conversion, delete the NIC from the systems hardware properties completely.
  • Once the physical machine is off the network, bring the virtual machine up (ensure network is not connected initially )
  • Install VMwares and set the ip config ( that you noted during the pre-conversion steps )
  • Shutdown and connect the network and bring your Virtual system back up
  • Uninstall VMware vCenter Converter from the newly converted Virtual macine

Special considerations for Domain Controllers, MS exchange and SQL servers.

  • Create test user on DC and ensure he gets replicated to the other ones.
  • Delete this test and ensure that gets replicated
  • Create test GPO policy and ensure it replicates across all domain controllers
  • Check system, application and importantly the File Replication Service logs to ensure that their is no issues with replication.


For SQL and Exchange : double check that their is no trust issues on the virtual machine. Try connecting to the ADMIN$ share from multiple locations. If you do find the computer account locked out. Taking the machine in and out of the domain normally fixes it.

Once happy the machine is on your domain without any trust issues – restart and reconfigure the SQL/Exchange services as per how they originally were.



Open Source Hypervisors and Hyperconverged Environments

We recently started looking at some of the open source solutions such as KVM/QEMU offered by RedHat and Proxmox to replace Microsoft Hyper-V and VMWare vSphere. So far they do appear somewhat feature-full especially for smaller environments. It appears though they do fall short of Enterprise features.

The performance and simplicity were definitely appealing with these solutions. Some of our staff was really into the Linux aspect of them since the Hypervisors have a full Linux shell. Controlling the enviornment easily from a cli was definitely a plus along with the common feel of logfiles and Linux kernel options.

Everything was promising but we got to the point of backing up multi-terabyte VM environments and the flexibility offered by common tools wasn’t working well enough for what we wanted to do. Products such as Veeam really do make it easy for even entry level administrators to use the complex environments.

For now we’ll be sticking with the big boys and keeping a close eye on the developments of Change Block Tracking in libvirt and the user-space tools in the coming year.

Remote Desktop Services

With businesses attaining more WAN bandwidth and businesses trust with hosted services increasing, Microsoft is investing heavily in Remote Desktop Services.  Renamed from Terminal Services to Remote Desktop Services, it encompasses multiple ways to deliver application access from any location.  Below, you will find information on some of the features and requirements in an RDS deployment.

WAN Optimization
RDP Client / Server features cross reference

Why RDS?

  1. Local-lan connectivity when using applications (e.g. Quickbooks) and when accessing the LAN resources (i.e. loading large files)
  2. Improved security for remote users
    1. Data is stored on the servers, not on laptops. This also means data is backed up consistently.
  3. New user setup is quickly done and without the need to “reimage” existing computers
  4. Portability for remote work
  5. Thin Client support
  6. Business Continuity and Disaster Recovery
  7. Green computing (more effective use of resources)
  8. Non-compliant PCs can connect with minimal security compromises
  9. Encrypted connectivity and application-level access limitation for compliance purposes or restricted access for external partners
  10. Centralize application management (updates, configuration is done in one place)


On the surface, RDS can be broken down into 2 Functions: Session Hosts and Virtual Desktop Infrastructure (VDI).  When breaking down the session hosts function further, we can include features such as RemoteApps and Remote Session Host (Terminal Services).  Similarly, VDI provides us with Personal Virtual Desktops and Pooled Virtual Desktops.

Virtual Desktop Infrastructure

Personal Desktops
This is geared for full desktop replacement deployments. The user will treat this is as their own personal computer in a VM.

Pooled Desktops
Pooled desktops are similar to deploying VMs in an academic environment. This usually means the VMs are preinstalled with generic applications and users have full administrative access to install their custom applications.  Of course, after they log off, the VM is reverted to it’s original state for the next user. An example usage would be to provide a pool of 10 Windows XP VMs for users to use intermittently due to legacy software incompatibilities.

Remote Session Host (aka Terminal Services)

Web Access – Single sign-on web portal showing RemoteApps

RemoteApp  – A more seamless integration between remote applications and local desktop

    1. Does not require Windows 7 computer to be joined to domain
    2. Updates automatically when the feeds are updated by administrators
    3. Users have to log on only once to create the connection
    4. XML – so can be used in other ways

Capacity Planning

It’s better to purchase 2 Server than it is to purchase 1 loaded with more memory. The reason is you can load balance between 2 RDS servers and the cost of smaller memory modules is a lot less than of larger ones. Scaling OUT instead of UP is more cost effective, increases Disk IO paths, and creates redundancy.

Unfortunately, adding processors isn’t a 1:1 improvement. Usually, going from 1 to 2 processors will achieve a 1.8:1 gain, while going from 2 to 4 processors will achieve a 1.65:1 improvement.
If you have each user session taking up 10% of CPU, then the server’s CPU can handle up to 10 users at full load. If you added more CPUs to get a total of 4 CPUs, it would be 10*1.8 (1 => 2 cpu)*1.65 (2 => 4 cpu) = 30 users total. As you can see, it’s not 40 users.

  • Use a processor with SLAT support

Usually, allocate about 500MB per session for a 64-bit OS. Of course, the best thing to do is to find the working set of a user’s session.

Hardware Integration


This feature in Windows Server 2008, Vista+ coordinates actions with the hypervisor to make sure that they’re interacting with the hardware as efficiently as possible.  The kernel basically only asks for instructions to be carried out within the confines of it’s child partition instead of all the partitions.  It reduces wasted CPU usage.

VM integration components

These components accelerate VM access to devices.  Without it, the VM will configure hardware device drivers with the emulated devices that the hypervisor presents to it.


AMD-V Rapid Virtualization Indexing (RVI) and Intel VT Extended Page Tables (EPT)

Although running RDS in a VM isn’t a problem, it does take up additional CPU cycles to maintain a “shadow” page table.  When this is updated in the VM, the Hypervisor has to update it’s “shadow” page table also.  This can take away precious CPU cycles that will slow down your server.  This is where SLAT-enabled processors mitigate this issue.  It maintains the address mappings in hardware, not software.  Just as hardware raid is file management using hardware, SLAT provides memory address management using hardware.  In the end, both memory usage and processor overhead will decrease.  This enables you to host more VM sessions by a factor of 1.6-2.5 times.  It’s highly recommended to have this for memory intensive workloads like RDS, SQL, IIS, Exchange, etc.

Improved Application Compatibility

  1. MSI package installation – Prevention of simultaneous first-time uses of applications based on MSI installs from blocking each other
  2. Dynamic Fair Share Scheduling – A better way of preventing a single session from starving other sessions for processor cycles
  3. IP Virtualizaton – Allows a session or application within a session to have a unique IP.  Applications with requirements of a discreet IP address can be used.

High-Fidelity User Experience

  1. True multi-monitor support, including varying layouts and landscape/portrait orientations
  2. Aero remoting for single-monitor sessions on Windows 7
  3. Cilent-side rendering of multimedia and audio Windows Media Player files
  4. Improved display of video from Silverlight and WIndows Media Foundation
  5. Bi-directional audio remoting, including sound recording to a remote session

What the buzz is about: Virtualization and Consolidation

I know everyone has been hearing this continuously for the last 3 years or so, but what does it actually mean?  How does it help the IT department and how does that help the business?  While the two words of Virtualization and Consolidation are separate topics, they often go hand in hand.  I believe the reason for this are to take advantage of new hardware capabilities and new virtualization technologies.  Lets say you bought a new server and wanted to consolidate and migrate all the data from your file server and your mail server.  You’ll essentially end up with a server with more utilization.  Sure, you’ll save a little on time and the electric bill, but that isn’t going to give you the “WOW” factor when it comes to analyzing your ROI. You’ll soon realize that consolidation usually will not fully utilize the full capabilities of your new hardware.  Unlike oil and water, implementation of Virtualization creates a symbiotic relationship with Consolidation.  One could also say it’s the catalyst to a higher ROI.

Why is there a push now?

  • 64-bit operating systems allow for a significant increase in memory
  • Multi-core CPUs creates effective use of processing power
  • VT-enabled CPUs support virtualization specific instruction sets which increase the effectiveness of hardware.
  • Microsoft is now in the market, which usually means it’s growing and here to stay.

Summary of benefits of Consolidation and Virtualization:

  • Lower TCO (power and cooling requirements, less physical assets, reduce time and resources need by IT staff, reduce licensing requirements)
  • Increased flexibility (backups, snapshots, migration, quicker provisioning of new servers)
  • Space savings
  • Makes use of the full capabilities of your new hardware
  • You already have the capabilities, you just need to implement it.
  • Fail-overs can be performed practically instantaneously
  • Upgrades to a new server in the future is greatly simplified
  • It’s much easier  to make  the  resulting  infrastructure resilient  for  business  recovery  and  continuity  solutions

When will you know it’s “GO TIME”?

  • You are trying to cut costs
  • You are trying to increase performance
  • You will be purchasing a new server
  • You spend too much time focused on maintenance
  • You are implementing a software “refresh”
  • Your servers have multiplied to where you have a management problem
  • You need IT to work projects that drive the business
  • Your backup solution is inefficient and ineffective
  • You have a need for a development environment

Virtualization for applications…

In brief, Microsoft SoftGrid helps IT shops lower costs and enhance service by reducing the complexity and labor involved in deploying applications to desktops, portable computers, and terminal servers, as well as the complexity of keeping applications up-to-date. The fact that SoftGrid runs applications locally is a key difference from past attempts at virtualization, based on technologies such as Microsoft Windows Server® Terminal Server or Citrix. Because that approach required applications to be centrally hosted and centrally run, it led to mushrooming growth in servers, along with an increase in both hardware purchase costs and hardware maintenance costs. Furthermore, to prevent application conflicts in multi-tenancy situations, IT shops had to create separate instances of servers to host the various applications—a practice that further encouraged the growth of server silos and that resulted in increased management complexity and higher administrative costs.

In contrast, SoftGrid technology offers the ability to deliver and manage applications centrally while allowing client computers to run the applications locally. This approach reduces both hardware proliferation issues and multi-tenancy application conflicts. It also provides all the benefits of SaaS—such as ease of access, ease of
distribution, and ease of management—to earlier client-server applications.