FRS: Disk Defragmentation Causes Excessive FRS Replication Traffic (282791)
The information in this article applies to:
- Microsoft Windows Server 2003, Standard Edition
- Microsoft Windows Server 2003, Enterprise Edition
- Microsoft Windows Server 2003, Datacenter Edition
- Microsoft Windows Advanced Server, Limited Edition
- Microsoft Windows 2000 Server
- Microsoft Windows 2000 Advanced Server
- Microsoft Windows 2000 Datacenter Server
- Microsoft Windows Small Business Server 2003, Premium Edition
- Microsoft Windows Small Business Server 2003, Standard Edition
This article was previously published under Q282791 SYMPTOMS When you use disk defragmentation utilities to optimize
logical drives that host File Replication service (FRS) replicated content, the
following symptoms may occur:
- Files in System Volume (SYSVOL) and Distributed File System
(DFS) shares are replicated excessively and there is no apparent change to the
files.
- Files may replicate at offpeak hours but at regularly
occurring times if you schedule disk defragmentation to run during specific
times or periods of low server usage.
- The number of files in the staging folder constantly grows,
and then empties after the disk defragmentation utility is completed, or the
FRS schedule opens to allow replication.
- The number of files in the staging folder constantly grows
but never empties if changes to downstream partners cannot be replicated either
because of network connectivity or some other error condition.
- Network traffic between replication partners is consuming
excessive network bandwidth as a result of FRS.
CAUSE In Windows 2000, the file system APIs that support
defragmentation use the file cache to do a mapped file write of the file data
to the new set of assigned disk clusters. Because a mapped file may not be
associated with any particular file handle, the NTFS journal code forces the
SourceInfo field in the USN record to zero. This behavior occurs because
defragmentation is an on-line operation (the volume is not locked) and the file
cache may actually contain modified data for a given byte range of the file if
a user was writing to the file while the defragmentation process was underway.
However, NTFS cannot tell that this operation has occurred. FSCTL_MARK_HANDLE
becomes a no-op. When you stop FRS while Diskkeeper is being run, NTFS still
writes the journal records and FRS reads them when it is later
restarted.
In Windows Server 2003, NTFS no longer passes the data
through the file cache and no USN records are generated. RESOLUTION You can use the FRS debug log files and the outbound log of
the FRS database to locate the source of FRS replicating files. To determine if
disk defragmentation utilities are causing excessive FRS replication traffic in
your environment, use either of the methods described in this section. Method One: Searching the FRS Debug Logs To search the FRS debug logs to determine the reason for
replication:
- Open a Windows NT CMD prompt, and then type: cd
/d %systemroot%\debug
- From the %SystemRoot%\Debug folder, use either of the
following steps:
- If you are using FRS in Windows 2000 or Windows 2000
Service Pack 1 (SP1), run the following command:
findstr /i "contentcmd" %systemroot%\debug\ntfrs_00??.log - If you are using FRS in Windows 2000 Service Pack 2
(SP2), Windows Server 2003, increase log level verbosity to 4, and then let the service run for a sufficient period of time to
capture replication events. To filter the FRS change events, run the following
command:
findstr /i "content" %systemroot%\debug\ntfrs_00??.log NOTE: Run this command if you are using the hotfix that is described
in the following Microsoft Knowledge Base article:272567 File Replication Service Improvements in Windows 2000 Service Pack 2
For additional
information about increasing the logging levels, click the article number below
to view the article in the Microsoft Knowledge Base: 221111 Description of FRS Entries in the Registry
Use the Layout tab in the CMD sessions properties to set the window size width
and height to accommodate FINDSTR output. To start, set the width at 110 characters, the height at 45, and the resolution to 1024 x 768. The "Content|ContentCmd" string in the FINDSTR output
displays the change that took place for files on NTFS formatted partitions. The
FINDSTR output for files replicated by disk defragmentation utilities have
lines at the "[DatOvrWrt ]" or the [StrmOvrWrt] string. For example:
d:\>FINDSTR /i "ContentCmd" ntfrs_00??.log
---------- D:\WINNT\DEBUG\NTFRS_0001.LOG
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ]
<ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd Flags set in the NTFS change log describe modifications to
FRS-replicated files. FRS replication is predicated on closed files that reside
on NTFS 5.0 formatted partitions in FRS replicated folders. The reasons for
changes in FRS are displayed in the following table:
Close - Change log close record
Create - File or folder was created
Delete - File or folder was deleted
RenNew - File or folder was renamed
DatOvrWrt - Main file data stream was overwritten
DatExt - Main file data stream was extended
DatTrunc - Main file data stream was truncated
Info - Basic info change (attrib, last write time, etc.)
Oid - Object Id change
StreamNam - Alternate data stream name change
StrmOvrWrt - Alternate data stream was overwritten
StrmExt - Alternate data stream was extended
StrmTrunc - Alternate data stream was truncated
EAChg - Extended file attribute was changed
Security - File access permissions changes
IndexableChg - File change requires re-indexing.
HLink - Hardlink change
CompressChg - File commpression attribute changed
EncryptChg - File encryption changed
Reparse - Reparse point changed To increase the log endurance and verbosity for FRS, modify the
following registry entries:
- Set the Debug Maximum Log Messages (REG_DWORD) registry entry to 20,000 (do not enter the comma).
- Set the Debug Log Files (REG_DWORD) registry entry to a value between 20 and 50.
- For Windows 2000 SP2 and Windows Server 2003 versions of
FRS, set the Debug Log Severity (REG_DWORD) registry entry to 4 to see Content: strings. If you do not to set the log verbosity
to a high enough value, you may be able to filter the debug logs against the
"UsnReason:" string to locate the the type of modification that took
place.
NOTE: Follow the procedure in this step if you used the hotfix
described in the following Microsoft Knowledge Base article:272567 File Replication Service Improvements in Windows 2000 Service Pack 2
For additional
information about increasing the logging levels, click the article number below
to view the article in the Microsoft Knowledge Base: 221111 Description of FRS Entries in the Registry
Method Two: Searching the NTFRS Outbound Log To search the NTFRS Outbound Log by using the Ntfrsutl
(Ntfrsutl.exe) utility and Iologsum.cmd:
- Identify the computer that is originating excessive
updates, and then use Ntfrsutl from the Windows 2000 Resource Kit to dump the
FRS outbound log.
To do so, from the Windows NT CMD prompt, type: ntfrsutl outlog >outlog.txt - Use the Iologsum.cmd FRS troubleshooting utility to
structure the Outlog file that is created in the preceding step. Iologsum
supports the following arguments:
c:\>iologsum.cmd /?
Usage: C:\iologsum.cmd [-sort=xxx] datafile
Process the output of "ntfrsutl inlog" or "ntfrsutl outlog"
and summarize the pending change orders.
Sort Keyword Sort the output by:
-sort=seqnum Sequence Number (default)
-sort=version File Version Number
-sort=filename File Name
-sort=size File Size
-sort=fileguid File Guid
-sort=origguid Originator Guid
-sort=eventtime Event Time You can use the Iologsum switches to identify the sources of
replication. Use the -sort=eventtime switch to identify patterns of replication changes, for example,
changes that occur in a scheduled job or process. Use the following command
line to sort the Outlog.txt file by event time: c:\>iologsum -sort=eventtime outlog.txt The following text is an example of output from an outlog that is
sorted by eventtime:
Event Time Ver OrigiGUID File Replication Reason
11/11/2000 3:03 4 646d2289 FILE001.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 5 646d2289 FILE002.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE003.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE004.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 6 646d2289 FILE005.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 5 646d2289 FILE006.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE007.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 5 646d2289 FILE008.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE009.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE010.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE011.TXT [Content DIR MD5/ StrmOvrWrt ]
11/11/2000 3:03 4 646d2289 FILE012.TXT [Content DIR MD5/ StrmOvrWrt ]
...
11/11/2000 6:00 2 0c88c766 FILE101.TXT [Content MD5/ DatOvrWrt ]
11/11/2000 6:00 4 0c88c766 FILE102.TXT [Content MD5/ DatOvrWrt ]
11/11/2000 6:00 10 0c88c766 FILE103.TXT [Content MD5/ DatOvrWrt ]
11/11/2000 6:00 2 0c88c766 FILE104.TXT [Content MD5/ DatOvrWrt ]
11/11/2000 6:00 3 0c88c766 FILE105.TXT [Content MD5/ DatOvrWrt ]
11/11/2000 6:00 2 0c88c766 FILE106.TXT [Content MD5/ DatOvrWrt ] The Iolog output shows a pattern of replication at 3:03AM and
6:00 AM that originates from replica members 646d2289 and 0c88c766 (these
numbers can be resolved to computer names in the Ntfrsutil sets output). The
reason for the changes are [StrmOvrWrt ] and [DatOvrWrt]. The version columns
show that the same files have been replicated anywhere from two to ten times.
Files that have high version numbers may illustrate that excessive updates are
occurring.
Workarounds to Excessive Replication by Disk Defragmentation Utilities To prevent disk defragmentation problems from causing excessive
replication of FRS files:
- Do not run defragmentation utilities on volumes that
contain FRS replicated files. In the short term, disable disk defragmentation
routines to stop FRS updates from generating and allow the staging folders to
empty.
- Exclude FRS-replicated folders from being optimized by
defragmentation utilities. Diskkeeper allows administrators to exclude folders
from fragmentation.
- Remove the member computer from all replica sets hosted by
the volume that is being defragmented, run the defragmentation utility, and
then add the member computer back to the replica sets. When you complete this
procedure, you force a non-authoritative restore (D2) but no file data is moved
over the network.
FRS can generate the staging file along with a new
MD5 checksum. If the checksum matches the value in the IDTable, then abort the
local change order. FRS does a lot of work to generate a staging file for each
file that gets defragmented, but it does not replicate the file if the file has
not changed.
MORE INFORMATION FRS is a multiple-threaded, multiple-master replication
engine that replaces the LanMan Replication (LMREPL) service in Microsoft
Windows NT versions 3.x and 4.0. Windows 2000 domain controllers and servers
use FRS to replicate system policy and logon scripts for Windows 2000 and
down-level clients that reside in the SYSVOL.
FRS can also replicate
content between Windows 2000-based servers that host the same fault-tolerant
DFS roots or child node replicas. FRS replicates files based on version changes
or changes to permissions on files and replicated folders.
Modification Type: | Minor | Last Reviewed: | 1/20/2006 |
---|
Keywords: | kbnetwork kbprb KB282791 |
---|
|