How to detect and recover from a USN rollback in Windows Server 2003 (875495)


The information in this article applies to:

  • Microsoft Windows Server 2003, Datacenter Edition for Itanium-based Systems
  • Microsoft Windows Server 2003, Enterprise Edition for Itanium-based Systems
  • Microsoft Windows Server 2003, Standard Edition
  • Microsoft Windows Server 2003, Datacenter Edition
  • Microsoft Windows Server 2003, Enterprise Edition

SUMMARY

This article describes the operations that Active Directory-aware backup programs and the Windows operating system perform to maintain consistent copies of Active Directory partitions when you restore the system state on a domain controller in a common Active Directory forest.

To roll back the contents of an Active Directory database, restore the system state by using an Active Directory-aware backup utility. If you use any other method, replication partners in the forest may not be notified that your domain controller has started its operating system by using an earlier version of the Active Directory database.

When such "Update Sequence Number (USN) rollbacks" occur, modifications to objects and attributes that occur on one domain controller do not replicate to other domain controllers in the forest. However, no Active Directory replication errors are reported in the event logs of the affected domain controllers. Additionally, replication-monitoring utilities such as Repadmin.exe do not detect any replication errors.

USN rollback may prevent the replication of any object or attribute in any replicated partition in Active Directory. The most frequent side effects are that user accounts and computer accounts exist on one domain controller but do not exist on another. Alternatively, the passwords for a user account may be inconsistent between domain controllers in a common domain, and logon operations may fail, even though replication appears to be working correctly.

After hotfix 875495 is installed, a Microsoft Windows Server 2003 domain controller logs Directory Services event 2095 when it encounters a USN rollback. The text of the event message directs administrators to this article for recovery options.

Because it is difficult to detect and recover from a USN rollback, we recommend that administrators install hotfix 875495 on all Windows Server 2003 domain controllers, especially those in virtualized hosting environments.

For a Microsoft Windows 2000 Server version of this article, see 885875.

INTRODUCTION

Over a domain controller's life cycle, you may have to restore, or "roll back," the contents of the Active Directory directory service. Alternatively, you may have to roll back elements of a domain controller's host operating system, including Active Directory, to a "known good" point in time.

The following are the two supported methods that you can use to roll back the contents of Active Directory or the local state of an Active Directory domain controller:
  • Use an Active Directory-aware backup and restoration utility that uses Microsoft-provided and Microsoft-tested APIs. These APIs non-authoritatively or authoritatively restore a system state backup. In this case, the backup originates from the same operating system installation and from the same physical or virtual computer that is being restored.
  • Use an Active Directory-aware backup and restoration utility that uses Microsoft Volume Shadow Copy Service APIs. These APIs back up and restore the domain controller system state. The Volume Shadow Copy Service supports creating single point-in-time shadow copies of single or multiple volumes on Windows Server 2003 computers. Single point-in-time shadow copies are also known as snapshots. For more information, visit the following Microsoft Web site and search for "Volume Shadow Copy Service":

    http://support.microsoft.com/

This article discusses the following topics:
  • The behavior that typically occurs when you restore an Active Directory-aware system state backup
  • How copying a previously saved Active Directory database into the folder that contains the current Active Directory database without restoring the system state can lead to an Update Sequence Number (USN) rollback
  • How Active Directory replication is affected when a Microsoft Windows Server 2003-based domain controller experiences a USN rollback
  • Ways to recover an Active Directory domain controller after it experiences a USN rollback
  • Enhancements that hotfix 875495 provides to detect USN rollbacks and quarantine affected domain controllers

MORE INFORMATION

Typical behavior that occurs when you restore an Active Directory-aware system state backup

Windows Server 2003 domain controllers use USNs in combination with the invocation IDs of source domain controllers to track updates to Active Directory that must be replicated. When USNs and invocation IDs are used, all domain controllers maintain consistent copies in Active Directory of the directory database partitions that are replicated. The invocation ID identifies the version of the directory database that is running on the domain controller. When the system state is correctly restored on a domain controller, the invocation ID is reset before Active Directory starts. Therefore, the domain controller is identified to its replication partners as a new instantiation of an existing domain controller. This situation prompts other domain controllers to reset their replication high-water marks and update the restored domain controller with changes that occurred because its system state backup was performed.

System state restorations that Active Directory-aware backup programs perform use APIs and methods that Microsoft has designed and tested. These APIs and methods make sure that local and replicated Active Directory databases are consistent when the restoration is finished. These APIs and methods also make sure that other domain controllers in the forest are notified that invocation IDs have been reset.

Software and methodologies that cause USN rollbacks

When the following environments, programs, or subsystems are used, administrators can bypass the checks and validations that Microsoft has designed to occur when the domain controller system state is restored:
  • Virtualized hosting environments, including but not limited to Microsoft Virtual Server 2005 and EMC VMWARE
  • Software that backs up and restores an Active Directory operating system installation or a hard disk volume that contains that installation

    Note Such software includes but is not limited to Norton Ghost.
  • Advanced disk subsystems that can selectively copy a volume that contains an Active Directory operating system installation that was saved in the past
Operations that are not supported include the following:
  1. Starting an Active Directory domain controller whose operating system was restored to a hard disk by using an imaging program such as Norton Ghost
  2. Starting an Active Directory domain controller whose operating system resides in a virtualized hosting environment such as Microsoft Virtual PC or EMC VMWARE.
  3. Starting an Active Directory domain controller that is located on a volume where the disk subsystem loads using previously saved images of the operating system without requiring a system state restoration of Active Directory.
The only supported way to roll back the contents of Active Directory or the local state of an Active Directory domain controller is to use an Active Directory-aware backup and restoration utility to restore a system state backup that originated from the same operating system installation and the same physical or virtual computer that is being restored.

Microsoft does not support any other process that takes a snapshot of the elements of an Active Directory domain controller's system state and copies elements of that system state to an operating system image. Unless an administrator intervenes, such processes cause a USN rollback. This USN rollback causes the direct and transitive replication partners of an incorrectly restored domain controller to have inconsistent objects in their Active Directory databases.

The effects of a USN rollback

The following steps show the sequence of events that may lead to a USN rollback. A USN rollback occurs when the domain controller system state is rolled back in time using an unsupported system state restoration.
  1. An administrator promotes three domain controllers in a domain. (In this example, the domain controllers are DC1, DC2, and DC2, and the domain is Contoso.com.) DC1 and DC2 are direct replication partners. DC2 and DC3 are also direct replication partners. DC1 and DC3 are not direct replication partners but receive originating updates transitively through DC2.
  2. An administrator creates 10 user accounts that correspond to USNs 1 through 10 on DC1. All these accounts replicate to DC2 and DC3.
  3. An operating system image is created on DC1. This image has a record of objects that correspond to local USNs 1 through 10.
  4. The following changes are made in Active Directory:
    • The passwords for the user accounts that were created in step 2 are reset on DC1. These passwords correspond to USNs 11 through 20. All 10 updated passwords replicate to DC2 and DC3.
    • 10 new user accounts that correspond to USNs 21 through 30 are created on DC1. These 10 user accounts replicate to DC2 and DC3.
    • 10 new computer accounts that correspond to USNs 31 through 40 are created on DC1. These 10 computer accounts replicate to DC2 and DC3.
    • 10 new security groups that correspond to USNs 41 through 50 are created on DC1. These 10 security groups replicate to DC2 and DC3.
  5. DC1 experiences a hardware or software failure. The administrator copies the operating system image that was created in step 3 into place. DC1 uses a database that has a record of USNs 1 through 10 to start Active Directory.

    Because the operating system image that was created in step 3 was copied into place, and the supported method of restoring the system state was not used, DC1 maintains its original invocation ID, and DC2 and DC3 maintain their original up-to-date vector of USN 50 for DC1. (An up-to-date vector is the current status of the latest originating updates to occur on all domain controllers that store a replica of a specific directory partition.)

    Unless an administrator intervenes, DC1 will not inbound-replicate the changes for local USN 11 through 50 that it originated in step 4 and replicated to DC2 and DC3. (These changes correspond to the newly created objects, deleted objects, and existing objects that are modified in this example.) Because the changes in step 4 do not exist on DC1, logon requests fail with an "access denied" error. This error occurs either because passwords do not match or because the account does not exist when the newer accounts randomly authenticate with DC1.
  6. Administrators who monitor replication health in the forest note the following situations:
    • The Repadmin /showreps command-line tool reports that two-way Active Directory replication between DC1 and DC2 and between DC2 and DC3 is occurring without error. This situation makes any replication inconsistency difficult to detect.
    • Replication events in the directory service event logs of domain controllers that are running Windows Server 2003 do not indicate any replication failures in the directory service event logs. This situation makes any replication inconsistency difficult to detect.
    • Active Directory Users and Computers or the Active Directory Administration Tool (Ldp.exe) show a different count of objects and different object metadata when the domain directory partitions on DC2 and DC3 are compared to the partition on DC1. The difference is the set of changes that map to USN changes 11 through 50 in step 4.

      Note In this example, the different object count applies to user accounts, computer accounts, and security groups. The different object metadata represents the different user account passwords.
    • User authentication requests for the 10 user accounts that were created in step 2 occasionally generate an "access denied" or "incorrect password" error. This error may occur because a password mismatch exists between these user accounts on DC1 and the accounts on DC2 and DC3. The user accounts that experience this problem correspond to the user accounts that were created in step 4. The user accounts and password resets in step 4 did not replicate to other domain controllers in the domain.
  7. DC2 and DC3 start to inbound-replicate originating updates that correspond to USN numbers that are greater than 50 from DC1. This replication proceeds normally without administrative intervention because the previously recorded up-to-dateness vector threshold, USN 50, has been exceeded. (USN 50 was the up-to-dateness vector USN recorded for DC1 on DC2 and DC3 before DC1 was taken offline and restored.) However, the new changes that corresponded to USNs 11 through 50 on the originating DC1 after the unsupported restore will never replicate to DC2, DC3, or their transitive replication partners.
While the symptoms that are mentioned in step 6 represent some of the impact that a USN rollback can have on user and computer accounts, a USN rollback can prevent any object type in any Active Directory partition from replicating, including the following object types:
  • The Active Directory replication topology and schedule
  • The existence of domain controllers in the forest and the roles that these domain controllers hold

    Note These roles include the global catalog, relative identifier (RID) allocations, and operations master roles. (Operations master roles are also known as flexible single master operations or FSMO.)
  • The existence of domain and application partitions in the forest
  • The existence of security groups and their current group memberships
  • DNS record registration in Active Directory-integrated DNS zones
While the image is stored on the backup media, the forest continues to work and also stores information about the DC that an image was created from. When the image is brought back, you rollback this DC, and only this DC, to the time the backup was taken and no other DC knows or will know about it. Therefore, the meta-data stored on the rest of the DCs does not match any longer. Any changes you make on this restored DC will be supplied with USNs that have already been used by the previous incarnation of the DC before the restore. All other DCs think they have already inbound-replicated this change. So this DC and all others lose synchronization of their DB contents. The severity of this problem depends on the nature of the changes that do not replicate. If the DC holds an important FSMO role (PDC, RID, DNM, Schema). Then, you'll have serious issues, see list of changes that were discussed earlier.

The size of the USN hole may represent hundreds, thousands, or even tens of thousands of changes to users, computers, trusts, passwords, and security groups. (The USN hole is defined by the difference between the highest USN number that existed when the restored system state backup was made and the number of originating changes that were created on the rolled-back domain controller before it was taken offline.)

Detecting a USN rollback on a domain controller that is running Windows Server 2003

Because errors are not logged in the event log or in the replication engine, a USN rollback can be difficult to detect.

One way to detect a USN rollback is to use the Windows Server 2003 version of Repadmin.exe to run the repadmin /showutdvec command. This version of Repadmin.exe displays the up-to-dateness vector USN for all domain controllers that replicate a common naming context. To detect a USN rollback, compare the output of the repadmin /showutdvec command on the domain controller with the output of the same command on the domain controller's replication partners. If the direct replication partners have a higher USN number for the domain controller than the domain controller has for itself, and the repadmin /showreps command does not report replication errors between direct replication partners, you have compelling evidence of a USN rollback.

Note A correctly restored domain controller resets its local invocation ID attribute when it restarts into Active Directory after its system state is restored by using a supported backup and restore method. When the reset invocation ID is outbound-replicated, remote domain controllers in the forest record the reset invocation ID as a new database instance on the restored DC. Although the restored domain controller is still the same domain controller, the remote domain controllers acknowledge this restored domain controller as a new replication partner because the invocation ID changed. (The invocation ID is the identity of the database instance.) The restored domain controller itself will accept changes from other remote domain controllers that originated on the remote domain controllers and on the domain controller before it was restored.

The following example shows the output of the repadmin /showutdvec command on DC1 and DC2 in the contoso.com domain. In this example, the command is run immediately following the rollback in step 5.

C:\>Repadmin /showutdvec dc1 dc=contoso,dc=com
Caching GUIDs...
Site1\DC1 @ USN 10 @ Time 2004-08-04 15:07:15
Site2\DC2 @ USN 24805 @ Time 2004-08-04 15:06:59
C:\>Repadmin /showutdvec dc2 dc=contoso,dc=com
Caching GUIDs...
Site1\DC1 @ USN 50 @ Time 2004-08-04 15:07:15
Site2\DC2 @ USN 24805 @ Time 2004-08-04 15:06:59

The output from DC1 shows a local USN of 10. DC2 has inbound-replicated USN 50 and will ignore the Active Directory updates that correspond to the next 40 USN numbers from the originating DC1.

Detecting a USN rollback on a Windows Server 2003 domain controller that has Windows Server 2003 SP1 or the 875495 hotfix installed

Because a USN rollback is difficult to detect, a Windows Server 2003 domain controller that has Windows Server 2003 Service Pack 1 (SP1) or the 875495 hotfix installed logs event 2095 when a source domain controller sends a previously acknowledged USN number to a destination domain controller without a corresponding change in the invocation ID.

To prevent unique originating updates to Active Directory from being created on the incorrectly restored domain controller, the Net Logon service is paused. When the Net Logon service is paused, user and computer accounts cannot change the password on a domain controller that will not outbound-replicate such changes. Similarly, Active Directory administration tools will favor a healthy domain controller when they make updates to objects in Active Directory.

On a domain controller that has Windows Server 2003 SP1 or the 875495 hotfix installed, event messages that are similar to the following are recorded when the following conditions are true:
  • A source domain controller sends a previously acknowledged USN number to a destination domain controller.
  • There is no corresponding change in the invocation ID.
Message 1 Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2095
Date: 3/10/2005
Time: 4:26:51 PM
User: USN\2B25VB$
Computer: 2B9A
Description: During an Active Directory replication request, the local domain controller (DC) identified a remote DC which has received replication data from the local DC using already-acknowledged USN tracking numbers. Because the remote DC believes it is has a more up-to-date Active Directory database than the local DC, the remote DC will not apply future changes to its copy of the Active Directory database or replicate them to its direct and transitive replication partners that originate from this local DC. If not resolved immediately, this scenario will result in inconsistencies in the Active Directory databases of this source DC and one or more direct and transitive replication partners. Specifically the consistency of users, computers and trust relationships, their passwords, security groups, security group memberships and other Active Directory configuration data may vary, affecting the ability to log on, find objects of interest and perform other critical operations. To determine if this misconfiguration exists, query this event ID using http://support.microsoft.com or contact your Microsoft product support. The most probable cause of this situation is the improper restore of Active Directory on the local domain controller. User Actions: If this situation occurred because of an improper or unintended restore, forcibly demote the DC. Remote DC: b55ee67f-ed73-4970-b2d4-7dc6f571439f Partition: CN=Configuration,DC=usn,DC=loc USN reported by Remote DC: 24707 USN reported by Local DC: 20485 For more information, see Help and Support Center at http://support.microsoft.com.Message 2Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1113
Date: 3/10/2005
Time: 4:26:51 PM
User: USN\2B25VB$
Computer: 2B9A
Description: Inbound replication has been disabled by the user. For more information, see Help and Support Center at http://support.microsoft.com.Message 3Event Type: Warning
Event Source: NTDS General
Event Category: Replication
Event ID: 1115
Date: 3/10/2005
Time: 4:26:51 PM
User: USN\2B25VB$
Computer: 2B9A
Description: Outbound replication has been disabled by the user. For more information, see Help and Support Center at http://support.microsoft.comMessage 4Event Type: Error
Event Source: NTDS General
Event Category: Service Control
Event ID: 2103
Date: 3/10/2005
Time: 4:26:51 PM
User: USN\2B25VB$
Computer: 2B9A
Description: The Active Directory database has been restored using an unsupported restoration procedure. Active Directory will be unable to log on users while this condition persists. As a result, the Net Logon service has paused. User Action See previous event logs for details. For more information, see Help and Support Center at http://support.microsoft.com.

Recovering from a USN rollback

To recover from a USN rollback, follow these steps:
  • Use the Active Directory Installation Wizard (Dcpromo.exe) to remove and then reinstall Active Directory.

    If you are not interested in the changes that originated on this domain controller, remove Active Directory from the domain controller, and then reinstall Active Directory.

    Method 1
    1. Start the Net Logon service.
    2. Enable inbound and outbound replication by using the following command:

      repadmin /options DC_Name -disable_inbound_repl -disable_outbound_repl

    3. If the incorrectly restored domain controller hosts operations master roles, transfer these roles to a healthy domain controller. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      255504 Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

    4. Remove Active Directory from the domain controller. Demote the controller to a member server.
    5. Restart the server.
    6. If you are required to, install Active Directory on the member server again. Promote the member server again to a domain controller.
    7. If the domain controller was previously a global catalog, configure the domain controller to be a global catalog For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      313994 How to create or move a global catalog in Windows 2000

    8. If the domain controller previously hosted operations master roles, transfer the operations master roles back to the domain controller. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      255504 Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

    Method 2
    1. Remove Active Directory from the domain controller to force it to be a stand-alone server. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      332199 Domain controllers do not demote gracefully when you use the Active Directory Installation Wizard to force demotion in Windows Server 2003 and in Windows 2000 Server

    2. Shut down the demoted server.
    3. On a healthy domain controller, clean up the metadata of the demoted domain controller. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      216498 How to remove data in Active Directory after an unsuccessful domain controller demotion

    4. If the incorrectly restored domain controller hosts operations master roles, transfer these roles to a healthy domain controller. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      255504 Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

    5. Restart the demoted server.
    6. If you are required to, install Active Directory on the stand-alone server again.
    7. If the domain controller was previously a global catalog, configure the domain controller to be a global catalog. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      313994 How to create or move a global catalog in Windows 2000

    8. If the domain controller previously hosted operations master roles, transfer the operations master roles back to the domain controller. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

      255504 Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

  • Restore the system state.

    Evaluate whether valid system state backups exist for this domain controller. If a valid system state backup was made before the rolled-back domain controller was incorrectly restored, and the backup contains recent changes that were made on the domain controller, restore the system state from the most recent backup.

Hotfix information

A supported hotfix is now available from Microsoft. We recommend that you install this hotfix on existing Windows Server 2003-based domain controllers. In addition, we recommend that you include this hotfix as part of your Windows Server 2003 installation process so that new domain controllers include this hotfix.

Contact Microsoft Product Support Services to obtain the hotfix. For a complete list of Microsoft Product Support Services phone numbers and information about support costs, visit the following Microsoft Web site:

http://support.microsoft.com/default.aspx?scid=fh;[LN];CNTACTMS

Note In special cases, charges that are ordinarily incurred for support calls may be canceled if a Microsoft Support Professional determines that a specific update will resolve your problem. The usual support costs will apply to additional support questions and issues that do not qualify for the specific update in question.

File information

The English version of this hotfix has the file attributes (or later file attributes) that are listed in the following table. The dates and times for these files are listed in Coordinated Universal Time (UTC). When you view the file information, it is converted to local time. To find the difference between UTC and local time, use the Time Zone tab in the Date and Time tool in Control Panel.
   Date        Version            Size  File name     
   -------------------------------------------------
   09-30-2004  5.2.3790.220  1,531,904  Ntdsa.dll
   09-30-2004  5.2.3790.212     32,768  Ntdsatq.dll
   09-30-2004  5.2.3790.220     79,872  Ntdsetup.dll
   07-19-2004  5.5.31.0         21,504  Spcustom.dll
   07-18-2004  5.5.31.0          7,168  Spmsg.dll
   07-19-2004  5.5.31.0        169,984  Spuninst.exe
   09-19-2004  5.2.3790.212     59,392  Ws03res.dll
   07-18-2004  5.5.31.0        654,336  Update.exe
Modification Type:MajorLast Reviewed:10/4/2006
Keywords:kbQFE kbHotfixServer KB875495
©2004 Microsoft Corporation. All rights reserved.