Navigation
Search
Recent Articles
Wednesday
Feb102016

Error 12700 in VMM 2012 R2 doing a shared nothing live migration between Hyper-V Clusters

I recently got the following error message in VMM 2012 R2 doing a live migration which I had never seen before in our VMM Hyper-V environment.

Error (12700)
VMM cannot complete the host operation on the VM01.domain.com server because of the error:
Virtual machine migration operation for 'VM01' failed at migration source 'HOST21'.
(Virtual machine ID 6DD5F4D2-48F1-4B8A-803B-5275981EF995)

The Virtual Machine Management Service
failed to establish a connection for a Virtual Machine migration with host 'HOST41.domain.com':

The credentials supplied to the package were not recognized (0x8009030D).

Failed to authenticate the connection at the source host: no suitable credentials available.
Unknown error (0x8000)

Recommended Action
Resolve the host issue and then try the operation again.

A quick research on the web brought up articles saying Kerberos should be used as authentication protocol for live migrations between Hyper-V Cluster. However, I had done many live migration between hosts before with out getting this error.

Then I found following post: http://vniklas.djungeln.se/2014/10/17/live-migrating-vms-between-clusters-in-vmm-2012-r2/ There Niklas Akerlund points to the VMM Host Access setting and the missing Run As Account for host management. This was exactly the issue I was experiencing.

I used following PowerShell to see the Run As Account of the hosts in a Hyper-V cluster:

Get-SCVMHostCluster -name Cluster2 | Get-SCVMHost | select name, runasaccount 

On the cluster node where the Run As Account was missing, I run the PowerShell from Niklas post:

$runas = Get-SCRunAsAccount "SCVMM Admin"
$clu = Get-SCVMHostCluster -Name HYPCLU
Set-SCVMHostCluster -VMHostCluster $clu -VMHostManagementCredential $runas

After that, Shared-Nothing Live Migration from that specific cluster node was working again. Even with CredSSP as authentication protocol.

Monday
Feb012016

Session "" failed to start with the following error: 0xC0000022

Recently Microsoft and HP responded to this harmless event log entry and they provided a workaround. You can find the response and the workaround here:

Kernel event ID 2 is logged when the MSFT_NetLbfoTeamNic class is called in Windows Server 2012 R2
https://support.microsoft.com/en-us/kb/3087042

Advisory: Microsoft - Hundreds or Thousands of Kernel Event ID:2 Error Messages Will Be Logged in the Event Viewer While Using HP Insight Management WBEM Provider with the NIC Teaming Feature Enabled on Microsoft Windows Server 2012 R2
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04803280&sp4ts.oid=3659620

Here a copy and past from the MS KB:

Cause

This problem is not caused by the third-party WMI provider. This problem is being tracked for consideration in a future version of Windows Server.
The Windows Server 2012 R2 MSFT_NetLbfoTeamNic WMI class uses the iNetCfg interface. The iNetCfg interface is associated with the NetCfgTrace kernel logging provider, which is automatically enabled by Windows. Trace data is automatically saved to the C:\Windows\inf\netcfgx.0.etl file. All WMI providers run under the context of the WMIPRVSE process, which in turn runs under the NETWORK SERVICE account. However, if a WMI provider calls theMSFT_NetLbfoTeamNic class, the NETWORK SERVICE account does not have authority to write trace data to this file.

Workaround

If the scenario that's described in the "Symptoms" section triggers the kernel event ID 2 error, the error is harmless and can be safely ignored.
If you want to prevent this error from being logged, open an administrative command prompt, and then run the following commands:

Takeown /f c:\windows\inf
icacls c:\windows\inf /grant "NT AUTHORITY\NETWORK SERVICE":"(OI)(CI)(F)"
icacls c:\windows\inf\netcfgx.0.etl /grant "NT AUTHORITY\NETWORK SERVICE":F
icacls c:\windows\inf\netcfgx.1.etl /grant "NT AUTHORITY\NETWORK SERVICE":F

These commands grant the necessary file permissions to prevent the error logging in this scenario.
You may also want to reset the owner of C:\Windows\inf back to the NT SERVICE\TrustedInstaller account (the default setting). To do this, follow these steps:

  1. In Windows Explorer, navigate to C:\Windows , right-click the C:\Windows\INF directory, and then select Properties. On the Security tab, clickAdvanced.
  2. Next to Owner, click Change.
  3. Click Location, and then select the local computer (you may have to scroll up if you’ve joined a domain).
  4. In the Object name, enter NT SERVICE\TrustedInstaller, select Check Names, and then click OK.

Finally, run the icacls command to make sure that the NETWORK SERVICE account is set for inheritance, as in the following example:

C:\>icacls c:\windows\inf

Wednesday
Apr222015

VMM Error 2916 - The WinRM protocol operation failed due to the following error: The WS-Management service cannot process the request. This service has exceeded its maximum of 100 concurrent operations per user allowed for plugin "WMI Provider".

We have a couple of Hyper-V Clusters and a System Center Virtual Machine Manager (VMM) 2012 R2 Cluster. From time to time I got following error when executing a VMM operation.

Error (2916)
VMM is unable to complete the request. The connection to the agent vmm11.domain.com was lost.

WinRM: URL: [http://vmm11.domain.com:5985], Verb: [INVOKE], Method: [ExportCertificateByName], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Unknown error (0x80338012)

Recommended Action
Ensure that the Windows Remote Management (WinRM) service and the VMM agent are installed and running and that a firewall is not blocking HTTP/HTTPS traffic. Ensure that VMM server is able to communicate with VMM11.domain.com over WinRM by successfully running the following command:

winrm id –r:VMM11.domain.com

This problem can also be caused by a Windows Management Instrumentation (WMI) service crash. If the server is running Windows Server 2008 R2, ensure that KB 982293 (http://support.microsoft.com/kb/982293) is installed on it.

If the error persists, restart VMM11.domain.com and then try the operation again. /nRefer to http://support.microsoft.com/kb/2742275 for more details.

 

When I looked into the Event log on the active VMM Server service node I saw the log Applications and Services Logs - Microsoft - Windows - Windows Remote Management which contained lots of the following error:

Event 49 - The WinRM protocol operation failed due to the following error: The WS-Management service cannot process the request. This service has exceeded its maximum of 100 concurrent operations per user allowed for plugin "WMI Provider". Retry the request later or raise the Maximum Operations per User quota. .

 

So I used the following command to increase some WINRM settings on both VMM Cluster nodes:

winrm set winrm/config @{MaxTimeoutms = "1800000"}
winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="1500"}

Then I restarted both VMM Cluster nodes after one another, using Pause Node. So far the error in the Windows Remote Management Event log did not reappear.

Tuesday
Feb032015

VMM Error (12700) VMM cannot complete the host operation

I recently got following error when moving a VM from one Hyper-V 2012 Cluster to another Hyper-V 2012 R2 Cluster. The VM was shutdown at the time of moving. The error occured at the step where VMM imports the moved VM into the target host:

Error (12700)
VMM cannot complete the host operation on the host.domain.com server because of the error: 'MV-1' could not be realized due to errors in its configuration file which could not be resolved. (Virtual machine ID C81AB5CD-1254-4D71-8021-7D866A6D8C8C)
Unknown error (0x8000)

Recommended Action
Resolve the host issue and then try the operation again.

 

In the event log of the target Hyper-V server there are following entries in the Hyper-V-VMMs log:

Error    03.02.2015 14:01:39    Hyper-V-VMMS    24004   
The virtual machine 'VM-1 Clean' is using processor-specific features not supported on physical computer 'host'. To allow for migration of this virtual machine to physical computers with different processors, modify the virtual machine settings to limit the processor features used by the virtual machine. (Virtual machine ID FBB3175E-3951-4589-9495-45E2E9018776)

Error    03.02.2015 14:01:39    Hyper-V-VMMS    19610  
'VM-1' could not be realized due to errors in its configuration file which could not be resolved. (Virtual machine ID C81AB5CD-1254-4D71-8021-7D866A6D8C8C)

Information    03.02.2015 14:01:39    Hyper-V-VMMS    13003  
The virtual machine 'VM-1' was deleted. (Virtual machine ID C81AB5CD-1254-4D71-8021-7D866A6D8C8C)

 

The first event showed that there was a snapshot on the VM I tried to move. I should have checked this first. After merging the snapshot and starting the move again, VMM was able to move the VM and import it into the target host. There might be other issues that are causing this VMM error. In my case the merging the snapshot fixed it.

Friday
Jan302015

VMM – Operations Manager Connection – Error 25904 A Connection Already Exists

When you try to add a System Center Virtual Machine Manager connection to System Center Operations Manager, you might get the following error in VMM 2012 (R2):

image

This means that an connection already exists. However when you try to verify the existing connection it looks like there is no connection:

image

To be able to create a new connection, you need to remove the existing connection. The only way to do this is by using PowerShell:

Remove-SCOpsMgrConnection –force

Now you can use the Add Operations Manager dialog again to create the connection.

Wednesday
Jan072015

VMM Error 23008 - The VM Server1 cannot be migrated to Host Host2 due to incompatibility issues

I was recently doing a Live Migration of some VMs from a Windows Server 2012 Hyper-V cluster to a Windows Server 2012 R2 Hyper-V cluster using Virtual Machine Manager 2012 R2 UR4 and got the following error message:

Error (23008) 

The VM Server1 cannot be migrated to Host Host1 due to incompatibility issues. Virtual machine migration operation for Server1 failed at migration source Host1.

Recommended Action

Resolve the incompatibility issue and try the migration again.

 

clip_image001

There are other reports that state that they had the same issue. One solution (from Benedict Berger) pointed to a conflict with the Hyper-V virtual switch name of the source and destination host. In my situation, the name of the virtual switch on the destination host was different then the name on the source host.

When doing a live migration between hosts, VMM gives you the possibility to select the VM Network and the virtual switch of the destination host. However, choosing a correct switch did not make a difference.

clip_image002

Renaming Hyper-V Virtual Switches, specially when using Hyper-V Clusters and Logical Switches is not really fun. But there is another option. I created a Private Switch on the destination host using the same switch name as on the source Hyper-V host. After that, I was able to migrate the VM between the 2012 and 2012 R2 Hyper-V cluster.

Once the VM was on the other side, the VM lost it’s network connection. Checking the VM network configuration, every thing was configured correctly and the correct Hyper-V switch was configured on the destination host. To get the VM connected again, I had to select a standard Hyper-V switch for the VM and apply the setting. Checking the VM network configuration again, I saw that the correct logical switch was selected again. Strange… However, the network connectivity of the VM was restored.

Monday
Dec012014

System Center Service Manager 2012 R2 – Edit Service Offering is blank

The System Center Service Manager 2012 R2 Console with UR4 still has issues when running in full screen mode. When you try to edit a service offering in full screen mode, you will see an blank dialog box:

image

When you resize the Console, you will be able to see the content of the service offering.

Wednesday
Nov192014

Connection issues to Azure Operational Insight

When configuring the connection from SCOM 2012 to Azure Operational Insight, there are some steps that are not that obvious. SCOM will alert you that the connection has issues, but it’s not that clear why. Here are some alerts I noticed when I initially configure Azure Operational Insight:

 

Description: Advisor is failing to send operational data to service.

A proxy server or your network settings might be misconfigured or there is a temporary network problem. Check the Knowledge Tab for possible resolutions. The context of where the alert appears might indicate whether the problem is due to a proxy server configuration or some other

 

Alert: Advisor failed to download the latest advisor knowledge management packs

Description: Event Description: Failed to synchronize the latest Management Package information from Advisor Cloud service. Wait for the next cycle to retry

 

Alert: Advisor connector failed to send analysis data to management server

Connector Framework Alert Write Action Last modified time

 

Description: One or more data batch was dropped dropped after exceeding retry time.

This indicates that some data was not uploaded to Advisor for some time and was lost ; there should be additional Alerts pointing at the root cause as to why data was failing to be sent in the first place. Typical reasons include but are not limited to problems with DNS name resolution, failed SSL connection, network timeout and proxy settings. Check the Knowledge Tab for possible resolutions.

SatyaVel from Microsoft postet some very use information on how to troubleshoot Azure Operational Insight connection issues here: http://blogs.technet.com/b/momteam/archive/2014/05/29/advisor-error-3000-unable-to-register-to-the-advisor-service-amp-onboarding-troubleshooting-steps.aspx

Wednesday
Jun252014

Upgrading a Windows Server 2012 Hyper-V Cluster to Windows Server 2012 R2

Ok, it´s a bit late for such a post since Windows Server 2012 R2 is out for over six month now, but still here are some rough steps from the upgrade of my demo 2012 Hyper-V cluster to Windows Server 2012 R2.

High Level Steps

To do this, I need to remove a existing node from the current 2012 cluster, install 2012 R2 on it and build a new single node cluster. Then move the VMs from the old cluster to the new cluster. Once finished, destroy the old cluster, install 2012 R2 on the free node and join it to the R2 cluster. Since I use VMM, I will try to leverage VMM as much as possible to make the transition easy. These steps are done in VMM:

  • Add the R2 server as a host to VMM, which also installs the Hyper-V role
  • Setup the virtual switch using a logical switch in VMM, which configures all the converged network settings for me
  • Setup the new R2 cluster using the VMM cluster wizard

It´s important to notice that you need additional shared storage for the new cluster. At least one drive for Quorum and one for a CSV disk.

The cluster notes use a converged network of three NICs for Live Migration, Cluster CSV, Management and VM traffic. iSCSI is connected via separate physical NICs.

In this environment I am also using System Center Virtual Machine Manager 2012 SP1. So this needs to be upgraded as well. This post captures the high level steps I performed:

 

Upgrade of System Center Virtual Machine Manager 2012 SP1 to 2012 R2

  1. Uninstalled VMM 2012 SP1 using Retain Database from the VMM server
  2. Uninstalled the ADK 8 from the VMM server
  3. Upgraded the VMM server to Windows Server 2012 R2
  4. Installed ADK 8.1 on the VMM server (Deployment Tools and Windows Preinstallation Environment features)
  5. Install VMM 2012 R2 using the retained VMM database
  6. Follow the post-upgrade tasks found here: http://technet.microsoft.com/en-us/library/dn469629.aspx

 

Fresh install of Windows Server 2012 R2 on the first cluster node (HV01)

  1. Evicted cluster node1 from the cluster (in my case the rest of the cluster run on just one node)
  2. Remote the cluster node1 from VMM
  3. (Remove the virtual switch and the virtual management OS network adapter from the host)
  4. Started Windows Setup from the DVD
  5. Renamed the network adapter to make sense and set up TCPIP correctly on one of the NICs
  6. Renamed the server and joined the server to the domain.
  7. Run Windows update
  8. Configured iSCSI NIC adapter  (for the new cluster

Bring first cluster node (HV01) under control of VMM

  1. Added the new Windows Server 2012 R2 cluster node to the same VMM server as the other node, so the previous VMM server. Now, I have a cluster with one node and a non-clustered Hyper-V node.
    image
  2. Configured the networking on the host. Since I used logical switch configuration I could apply the already created logical switch to the new Hyper-V host quite easily. First I configure the NICs in the team, then I configured the virtual network adapters:

    image         image

    In the Logical Network view I can now see that the HV01 host has been configured with the Logical Network configuration and that the host is compliant.

    image

    On the host itself, there is now a NIC team and the virtual network adapters as specified in the Logical Switch.

    image     image
  3. Note, that although I name the host management adapter in the logical switch configuration “HostManagement”, in the Network Connection panel it´s named vEthernet (Converged Switch), which is the name of the switch not the adapter. The other adapters like ClusterCSV or LiveMigration are named according to the name that I specified in VMM. This is by design, I guess.

    Also, it´s interesting how VMM 2012 and VMM 2012 R2 name the team adapter in Windows. In VMM 2012 the team adapter is named “ConvergedSwitch6b858dbe-fefc-4ecc-994e-e1587551422b” in VMM 2012 R2 the adapter is simply named “ConvergedSwitch”. I always thought that the VMM 2012 naming was a bit strange.

Create a New Hyper-V Cluster from VMM

  1. I created two new LUNs on the storage for the new cluster (called R2Cluster) that I am going to create using HV01.
  2. Connected the HV01 host to the iSCSI target and set up a LUN for the Quorum and a LUN for VM storage (I called it CSV3)
  3. Run the create cluster wizard from VMM. Add HV01, the R2 server to the new cluster. Configure a IP address for the cluster and configured the two storage LUNs for the new cluster. I only marked the CSV drive as CSV. The other drive will be used as a witness drive.
    image        image   image
  4. After this I have now two clusters in Hyper-V, the old cluster (HVCLUSTER01) and the new cluster (R2CLUSTER):
    image
  5. On the new cluster I renamed the network according to there usage:
    image

 

Copy the VM Role from the current Cluster to the new Cluster

  1. I went to the cluster node of the new cluster and started the cluster manager. I used the Copy Cluster Role link to start copying the VM roles.
    image      image

    The copying of the roles is very quick. At this point the roles are still running on the current Cluster.
  2. Now I stopped the VMs on the current cluster. I put the CSV where the VMs are located in Offline mode. Next I had to connect the new cluster node to the iSCSI Target where the CSV was located. Once that was connected, I brought the CSV online on the new cluster node.

Once all the rules have been moved to the new cluster, I started removing the cluster service from the old cluster and clean up all the relevant VMM and Hyper-V components. Then the old cluster node was reinstalled with Windows Server 2012 R2 and joint the new R2 Cluster.

Monday
Jun092014

VMM 2012 R2 - Error (2941) / Error (2927)

I recently got the following error in a VMM job when creating a new VMM template or when deploying a VM from a VM template.

A Hardware Management error has occurred trying to contact server hvhost02.demo.lab  .

WinRM: URL: [http://hvhost.contoso.com:5985], Verb: [INVOKE], Method: [GetError], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/microsoft/bits/BitsClientJob?JobId={26F5564E-4816-4F87-90D5-913B12E9C48E}]

Unknown error (0x803381fa)

Recommended Action
Check that WinRM is installed and running on server hvhost02.demo.lab. For more information use the command "winrm helpmsg hresult" and
http://support.microsoft.com/kb/2742275 .

Or following error:

VMM is unable to complete the request. The connection to the agent on machine vmmserver.contoso.com has been lost.
Unknown error (0x80072efe)

Recommended Action
Ensure that the WS-Management service and the agent are installed and running and that a firewall is not blocking HTTPS traffic.

To fix this, first check the normal network connection issues to the VMM server and the Hyper-V host. Use following link http://support.microsoft.com/kb/2742275 to further dig into the issue. One think that is not mentioned in this KB is the required VMM certificates. In my case the issue was a missing/deleted certificate on the VMM server itself. Following blog post explains who to correct this: http://blogs.technet.com/b/scvmm/archive/2013/05/14/i-accidentally-deleted-the-vmm-self-signed-certificate-from-the-vmm-server-now-what-do-i-do.aspx

Hope that helps other that experience similar issues.