I took the plunge for Server 2019, created a Storage Pool of 16 disks and created a REFS Deduped Volume.

Installed DPM 16, SQL 16, on DPM UR 7

I am experiencing consist dreadful network performance on 10Gbit network on dedicated network.

Backup mostly from 16 hosts using S2D, hyper converged network.

BITS not in use.

In the past I have been trying to protect a Hyper-V cluster with DPM 2012 and 2016 but the backup jobs were failing continuously. I suppose this could be due to live migration that has been executed from time to time. Then I read this article which states that in order for DPM to maintain protection within a cluster, VMM must be used. Is this correct? I don't have VMM but would like to protect the VMs somehow, what is the recommended method in this case?

Hi there

Does anyone know if backing up 2012 R2 failover clusters is supported using DPM 2019. I am unclear on this after reading documentation I have found from Microsoft on this. I am hoping to buy a new server and would like to use Windows Server 2019 and DPM 2019 if possible.

The reason I ask this is because I have found that in VMM 2019 for instance you cannot add 2012 R2 Clusters or hosts into it as it only supports OS versions that are in the initial support phase not extended support which 2012 R2 is now in so you can only add 2016/2019 hosts or servers.

Hello. We're trying to backup a VM running on a host (host-level backup). The operation always fails in DPM with the following error:

'An unexpected error occurred while the job was running. (ID 104 Details: The filename, directory name, or volume label syntax is incorrect (0x8007007B))'

Consistency check/reusme backup always ends up with this error. Removing the protected member and recreating it fails too.

This is the only VM for which the backup is failing. It's running Linux OS. VM doesn't have any checkpoints (even the hidden backup ones). All VSS writers on the host are in stable/no error state.

Monitoring the DPMRACurr.errlog on the host gives some insight, but no hints on how to solve this:

(first there's a production checkpoint creation failure, which is to be expected, followed by successful crash-consistent checkpoint creation)

2F941990 12/0508:23:48.47322hypervutil.cpp(268)FE7CA82A-5171-4FF6-9990-EB38A3C59E7B WARNINGErrorDescription Checkpoint operation for '<VMNAME>' failed. (Virtual machine ID 21A51FEB-2C44-499E-B9AD-253834ED2FF6)
2F941990 12/0508:23:48.47322hypervutil.cpp(268)FE7CA82A-5171-4FF6-9990-EB38A3C59E7B WARNINGProduction checkpoints cannot be created for '<VMNAME>'. (Virtual machine ID 21A51FEB-2C44-499E-B9AD-253834ED2FF6) ErrorCode 32770
2F941990 12/0508:23:48.47331hypervrcthelperplugin.cpp(92)[000001A3D7D8E2E0]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGApp Consistent snapshot failed, trying crash consistent snapshot
2F941990 12/0508:23:53.73922hypervutil.cpp(1761)FE7CA82A-5171-4FF6-9990-EB38A3C59E7B NORMALSnapshot got created with Id: Microsoft:277CB209-2D60-43E0-B9C2-1C51A9F4C643

(then there are more messages, some errors - not of consequence in my opinion, but later there are these errors):

2F941C08 12/0508:24:12.47318diskfixupreader.cpp(530)[000001A3D7E84D40]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x80070057] Error in Getting checksum buffers, previous failure Hr: = [0x80070057]. Can not continue on failure
2F941C08 12/0508:24:12.47318checksumprovider.cpp(223)[000001A3D801FCF0]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x80070057] : Unexpected end of buffers while reading checksums: lVal : E_INVALIDARG
2F941C08 12/0508:24:12.47318checksumprovider.cpp(223)[000001A3D801FCF0]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x80070057] : Unexpected end of buffers while reading checksums: lVal : E_INVALIDARG
2F941C08 12/0508:24:12.47318diskfixupreader.cpp(414)[000001A3D7E84D40]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x80070057] : Encountered Failure: : lVal : m_pChecksumProvider->GetChecksums(&m_pbInChecksumBuffer, &m_dwInChecksumBufferSizeInBytes)
2F941C08 12/0508:24:12.47318mtfdisk.cpp(665)[000001A3D7FEF700]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x809909d3] GetNextStream failed on disk(D:\vms\<VMNAME>_disk.vhdx)
2F941C08 12/0508:24:12.47318dsmsendersubtaskbase.cpp(588)[000001A3D8109100]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNINGFailed: Hr: = [0x809909d3] SendData() failed
2F941C08 12/0508:24:12.47318dsmsubtaskbase.cpp(265)[000001A3D8109100]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BNORMALHr: = [0x809909d3] CDsmSubTaskBase::ErrorCleanup: subtask state: 1

and more errors:

2F941990 12/0508:24:12.47303cbackupsubtask.cpp(653)[000001A3D5585A20]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNING<?xml version="1.0"?>
2F941990 12/0508:24:12.47303cbackupsubtask.cpp(653)[000001A3D5585A20]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BWARNING<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147014842" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="26b774a3-c5ee-44bb-b285-690b31e23ac0" GuidWorkItem="d67448fb-6a0b-40b8-8c4b-b43418c987fa" TETaskInstanceID="fe7ca82a-5171-4ff6-9990-eb38a3c59e7b"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147014842" DetailedSource="2"/><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1844" NumberOfFilesTransferred="10" NumberOfFilesFailed="0" DataCorruptionDetected="false"/><FixupStatus BytesTransferred="75694080" NumberOfFilesTransferred="8" NumberOfFilesFailed="1" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>
2F941990 12/0508:24:12.47303runtime.cpp(1373)[000001A3D5658ED0]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BFATALSubtask failure, sending status response XML=[<?xml version="1.0"?>
2F941990 12/0508:24:12.47303runtime.cpp(1373)[000001A3D5658ED0]FE7CA82A-5171-4FF6-9990-EB38A3C59E7BFATAL<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147014842" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="26b774a3-c5ee-44bb-b285-690b31e23ac0" GuidWorkItem="d67448fb-6a0b-40b8-8c4b-b43418c987fa" TETaskInstanceID="fe7ca82a-5171-4ff6-9990-eb38a3c59e7b"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="536872931" DetailedCode="-2147014842" DetailedSource="2"><Parameter Name="AgentTargetServer" Value="AOF0048.ad.apponfly.com"/></ErrorInfo><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1844" NumberOfFilesTransferred="10" NumberOfFilesFailed="0" DataCorruptionDetected="false"/><FixupStatus BytesTransferred="75694080" NumberOfFilesTransferred="8" NumberOfFilesFailed="1" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>

Any idea what the actual problem is and how to fix it?

Thanks

Is there a PS command I could run to determine which VM's are not part of a protection group?

Thanks

Hello,

I am having problems trying to backup a VM, which is running on a Hyper-V Cluster and stored on a SOFS cluster (SMB).  This is a bran new DPM 2019 server install.  The DPM server is installed and successfully backing up file servers which have the DPM agent installed.  However, I am unable to backup VMs as a whole.  When I add the VM to a protection group, the initial replica fails.  And now, I also cannot perform a consistency check.  It fails with this error: 

An unexpected error occurred while the job was running. (ID 104 Details: Access is denied (0x80070005))

Here's what I have already done:

• Installed DPM agent on all Hyper-V nodes in the Hyper-V cluster
• Installed DPM agent on all SOFS nodes in the SOFS cluster
• Installed the File Server VSS Agent Service on all SOFS nodes
• Added the following computer accounts to the "Backup Operators" group on all SOFS nodes (DPM$, HYPERV_NODE1$, HYPERV_NODE2$, HYPERV_NODE3$ and HYPERV_CLUSTER$)
• Gave all the computer accounts mentioned above, "Full Control" access to the file share and NTFS permissions on the folder tree, where the VM is stored on the SOFS.
• After making all of these changes, I restarted all the SOFS nodes and the DPM server.

I should mention that my new DPM server is running DPM2019, on Windows Server 2019, with SQL Server 2017.  My SOFS nodes and my Hyper-V nodes, are all running Windows Server 2016.

Does anyone have any ideas?

Hello everybody,

a fresh install of DPM 2019 on Windows Server 2019 Datacenter shows all VMs on a freshly installed Windows Server 2019 Datacenter Hyper-V host, and it backs them up just fine. However, DPM offers a strange VM named "Th-VM" which does not exist, which we did never create, and whose name would be far away from our VM naming standards. So if we choose Hyper-V auto-protection for that host, DPM tries to backup that VM and, of course, fails doing so. We can run backups fine if we uncheck both Auto-Protection and that strange VM. A refresh of the host in the DPM protection group wizard (having the host not expanded in the tree) still shows up this "Th-VM" VM.

Why does DPM offer us a VM which does not and did never exist, and how can we get rid of it? Thanks!

Best Regards, Stefan Falk

Hi,

We have DPM 2016 running on WS2016 the major PG are VM running on hyper-v cluster 2012 R2, I install DPM, create PG and online protect, all run ok, for prower problems (Puerto Rico hurricane Maria), we need off the datacenter every 2 weekends, but the Monday when the datacenter power on, DPM cannot run any PG and this error is show for the jobs:

 The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)

 More information
Recommended action: Ensure that the storage containing the VHD is healthy, mounted, and the VHD is not in use by other processes, and retry the operation.

Isaias

Hi

I had receive error as below when backup my Hyper 2016 VM

pls advise.

either the servername needs reboot , is part of a cluster then some/all nodes in the cluster need t be rebooted. 

Hi..

we are running DPM 2016 RU6 to protect our clustered Hyper-V (2016) VMs.

Now I've tried to restore some of them and I'm wondering why the recovered VM lost all her Checkpoints...

Only one vhdx file was restored. I've tried also a network folder target. One vhdx and two config files (vmcx, vmrs) are the only one that were restored.

Does somebody know if this is the default behavior, that only the current selected/running checkpoint will be used for the backup?

Is it possible to enable DPM to backup any VM Source?

Use a file system backup to backup the VM folder are no option for us because the VMs are running on a S2D Cluster and so the storage location of the VM can dynamically changed...

Thanks

Alex

Hi,

I have newly deployed, 3 node, Windows 2016 (Build 2724) Hyper-Converged Hyper-V/S2D cluster which is operating normally. The intention is to use DPM 2016 LTSB UR6 to perform VM level backups hosted on the cluster.

After deploying the DPM agents (5.0.375.0) to the cluster nodes, i've started to see cluster errors generated when rebooting a node, CSV auto pause errors for each CSV in the cluster:

"CSVName has entered a pause state because of 'STATUS_UNEXPECTED_NETWORK_ERROR (c00000c4)"

When rebooting a node, there are no backup jobs running, if I remove the DPM agents the nodes reboot without error.

Also, it may be a red herring, but there are entries in the cluster logs generated during a node reboot that relate to an error closing a file named ' DpmFilterTrace.txt' which is stored on the CSVs. Not sure if this is normal or an indication of an issue.

Would appreciate any input from people protecting workloads on HCI clusters using DPM.

Thanks in advance,

James

Hello.

New day, new DPM problem:

Cannot protect one particular VM using host-level protection. It's the only VM having this problem. The error message (redacted names):

Affected area: RCT\<VMNAME>
Occurred since: 24.05.2019 15:52:46
Description: The replica of Microsoft Hyper-V RCT\<VMNAME> on <HVHOST> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
 consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
The DPM service was unable to communicate with the protection agent on <HVHOST>. (ID 52 Details: The remote computer refused the network connection (0x800704C9))
More information
Recommended action: 
1) Restart the DPM Replication Agent service on <HVHOST>.
2) If <HVHOST> is configured using certificates, ensure that the DPM CPWrapper service is running on DPM Server and aof0078.ad.apponfly.com. Also ensure that certificates used by both the computers are valid.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution: To dismiss the alert, click below

Can backup all other VMs on the Hyper-V host without issues. Hyper-V 2016 Server protected by DPM 2019. Previously protected by DPM 2016 UR7, where backup of this particular VM worked. HV hosts and DPM are in the same domain.

What I've tried:

 - CC job - no good

 - removed the protected DS and re-add it to DPM - no good. Actually the initial backup 'finishes' without a problem, but the backup has ~100MB (instead of ~40GB). RP count 0. CC job doesn't help.

 - migrated the VM to different host - same issue

 - 'reset' RCT (by shutting down the VM and deleting the .RCT .MRT files) - no good

 - tried monitoring the DPMRA log during the backup attempt: this error comes out, however doesn't makes it any clearer:

1250 16A4 
05/24 13:51:13.117
03 cbackupsubtask.cpp(653)
[000001F7BE491E40] C4FBC1EF-FB64-41D5-9F10-5A44C6B4D7B0
WARNING <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147023671" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="044cde5b-e0d9-4fd3-8c37-a787048583e2"
 GuidWorkItem="e8436d7b-8204-4910-b7a6-d08637ec1f45" TETaskInstanceID="c4fbc1ef-fb64-41d5-9f10-5a44c6b4d7b0"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147023671"
 DetailedSource="2"/><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1272" NumberOfFilesTransferred="7" NumberOfFilesFailed="0"
 DataCorruptionDetected="false"/><FixupStatus BytesTransferred="0" NumberOfFilesTransferred="0" NumberOfFilesFailed="0" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>

Any ideas are welcome

Hi,

we've been (mostly) following this article to set up deduplication for DPM, adapted for

standalone machines. The article also contains instructions to set a few registry values:

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name DeepGCInterval -Value 0xFFFFFFFF

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name HashIndexFullKeyReservationPercent -Value 70

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name EnablePriorityOptimization -Value 1
Trouble is, "HKLM:\Cluster" doesn't exist when you don't have a cluster. Searching 
the web for the settings' names showed me a few people who seem  to have a similar 
problem, but no definitive solution, only a guess.
"DeepGCInterval" is the only setting that appears in a Microsoft document. The 
setting would be made in "HKLM\System\CurrentControlSet\Services\ddpsvc\Settings".
My question: can I assume that this is the right place to do ALL three of the 
above registry settings? Is this the place where the dedup service looks for 
settings when it's not running on a clustered machine? The path seems to imply that.
--
Cheers,
pk 

Hi Everyone!

I am kinda stuck on an issue and have not been able to find an answer to that solves the problem. I was wondering if anyone can point me in the right direction.

I currently have a Hyper-V 2016 server that I am backing up using DPM 2016, Now the Hyper-V host its self has roughly about 50 VM's and about 35 of them are backing up without issue however I have about 15 guests that are all in a "Replica is inconsistent" State with the following error in even log :

The replica of <Guest> on <Server> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
An unexpected error occurred while the job was running. (ID: 104)

Performing a consistency check does nothing and the alert pops back into the console. Any ideas or help ?

Stuart.

Hi,

I have a file server backed up using DPM 2019, that is in an inconsistent state. When I run the consistency check, after a while access to shares on the server is interrupted to the point that files cannot be opened. When I stop the check, access is fine again.

I have ran 2010 and 2019 and have had no issue with this before. 

The server that I am backing up is on a 2012 Hyper V Fail-over Cluster connected to iSCSI SAN. My backup server is on a dedicated server.

Any help with this would be much appreciated.

Thanks,

Mark

Hi,

I have been having a recent issue with DPM 2016 UR7 where my VM backups have been failing and leaving behind "checkpoints" that don't actually exist in Hyper-V.

If I then try to run the backup job again I get faced with Unknown error ID 104. I've seen a few posts regarding this however no fixes that have worked for me.

I have tried rebooting the virtual machines in question to find that they fail there merge. I have updated all protection agents to use UR7 agent (5.0.409.0). Some of the virtual machines will merge once shutdown and backup again successfully but a day or two later I will be faced with the same issue.

Has any experienced anything similar recently?

I would love some ideas on this one, in some cases if I shut down the VM to merge and try to start again I will be faced with power on error, virtual hard disk chain is corrupt of which I then run SET-VHD command to point back to the parent disk. It will then boot and backup.

Just seems like a really odd issue and has only been happening over the course of the past week or 2. I have also noticed that DPM has stopped deleting expired recovery points outside of the retention range but I believe that is a different issue.

Cheers.

Hi @ all,

[all servers involved are 2016, patched up to July CU]

trying to back up Hyper-V VMs running on three separate clusters. All VM configs and disks reside on the same SOFS living on an S2D cluster. The clusters are as follows:

Production: 10 nodes on HP Synergy blades, production VLAN and SMB VLAN sharing the same converged networking infrastructure.

DMZ: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic

Management: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic. This is where the DPM VM resides.

Shares on the SOFS, permissions, delegation and so on are configured in an identical way and differ only in regard to the machine accounts actually granted permissions. All permissions are granted via Active Directory groups.

Virtual machines can be created, managed and live migrated within their respective clusters without any issues.

DPM 2019 10.19.58.0 on a VM with a local SQL2016 instance.

DPM agent is installed on all hosts in all clusters, connectivity shows OK. Replicas of both Windows and Linux VMs can be created without warnings. And here's where it gets interesting:

• Any VMs on the Management Cluster (where the DPM VM is running) can be backed up without issue regardless of OS or power state.
• On the DMZ cluster, an initial replica can be created and shows OK. On attempt to create a recovery point, there is an error saying "Replica is inconsistent". Consistency check ends in error state saying ID 2033 (a required privilege is not held by the client (0x80070522). The UNC path and host shown in the error message are where the VM is stored and registered. This behaviour does not change with the power state of the VM.
• On the production cluster, an initial replica stays green for several seconds then goes red. From there on, the behaviour is identical to taht on the DMZ cluster.

All hosts have been rebooted at least once since DPM agent rollout.

What am I missing?

• added the machine account of the DPM server to Full Access on one of the shares, to no avail
• enabled File Server VSS writer on S2D cluster nodes, to no avail (backup of Management cluster worked before I did that)

Evgenij Smirnov

http://evgenij.smirnov.de

Dears,

can someone advise on how to configure hyperv replication between a cluster of 2 hyper-v in main site and one standalone hyperv in the DR.

many thanks

I have a 4 Node Cluster with 2 Shared Cluster Volumes on an ISCSI SAN.  Each node is up-to-date with Updates and is running Server 2012 DataCenter.  I have a seperate 2012 server with Microsoft DPM on it I use to backup my VMs.  The cluster has 24 diffrent VMs on it all diffrent sizes, OSs,  and configurations.  Recently DPM has been getting hung on one of the VMs.  When the backup gets hung, I get events

5217

Software snapshot creation on Cluster Shared Volume(s) ('\\?\Volume{f3c25587-f296-481e-b2fc-606c4eef1e92}\') with snapshot set id '{dd4a47b9-e645-4303-9df6-c978b29b3cd5}' failed with error 'HrError(0x00000001)(1)'. Please check the state of the CSV resources and the system events of the resource owner nodes.

and

5120

Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_FILE_CLOSED(c0000128)'. All I/O will temporarily be queued until a path to the volume is reestablished.

And naturally I cannot backup the VM.  All of the other VMs backup fine.

Anyone got any Ideas?