How to troubleshoot Event ID 1311 messages on a Windows 2000 domain (307593)
The information in this article applies to:
- Microsoft Windows 2000 Advanced Server
- Microsoft Windows 2000 Datacenter Server
- Microsoft Windows 2000 Server
This article was previously published under Q307593 SYMPTOMS This article describes how to troubleshoot event ID 1311
messages in the Directory Service event log on a Microsoft Windows 2000 domain.
The Knowledge Consistency Checker (KCC) constructs and maintains the
replication topology for Active Directory. To do this, the KCC examines the sum
of all naming contexts that reside in the forest and all administrator-defined
constraints for site, site link, and link cost. If an Active
Directory domain, a schema, a configuration, an application partition, or the
global catalog naming contexts cannot be replicated between domain controllers
or sites, an event ID 1311 message similar to the following is logged in the
Directory Service event log: Event Type: Error
Event Source: NTDS KCC Event Category: Knowledge Consistency Checker
Event ID: 1311 Date: MM/DD/YYYY Time: HH:MM:SS AM|PM User:
N/A Computer: <domain_controller_name> Description: The
Directory Service consistency checker has determined that either (a) there is
not enough physical connectivity published via the Active Directory Sites and
Services Manager to create a spanning tree connecting all the sites containing
the Partition CN=<partition name>,DC=<root domain of
forest>,DC=com, or (b) replication cannot be performed with one or more
critical servers in order for changes to propagate across all sites (most often
because of the servers being unreachable). CAUSE This behavior occurs if one or more of the following
conditions are true:
- Site link bridging is enabled on a network that does not
support physical network connectivity between two domain controllers in
different sites that are connected by a KCC link.
- One or more sites are not contained in site links.
- Site links contain all sites, but the site links are not
interconnected. This condition is known as disjoint site links.
- One or more domain controllers are offline.
- Bridgehead domain controllers are online, but errors occur
when they try to replicate a required naming context between Active Directory
sites.
- Administrator-defined preferred bridgeheads are online, but
they do not host the required naming contexts.
- Preferred bridgeheads are defined correctly by the
administrator, but they are currently offline.
- The bridgehead server is overloaded either because the
server is undersized, too many branch sites are trying to replicate changes
from the same hub domain controller, or the site link schedules are too
frequent.
- KCC has built a different path around a site-to-site
connection failure, but it retries the failing connection every 15 minutes
because it is in "connection keeping mode."
The common causes of event ID 1311 messages fall into two
categories: improper logical configuration and infrastructure failure. Event ID
1311 messages are logged when an improper logical configuration or a
replication error occurs.
- Improper logical configuration:
A logical
configuration is improperly configured when information in the Configuration
naming context (NC) (visible in the Sites and Services snap-in) does not match
the physical topology of the network that hosts the Active Directory forest.
For example, a site may not be properly defined, sites that are missing from
site links may be included, site links may not be interconnected, or incorrect
bridgeheads may be selected by the administrator. - Infrastructure failure
An infrastructure failure
occurs as a result of one of more of the following events:
- A wide area network (WAN) link fails.
- A domain controller that hosts a necessary naming
context is offline.
- A replication failure occurs for one or more naming
contexts.
RESOLUTION To troubleshoot event ID 1311 messages, use the following
methods.
- Determine if the event ID 1311 messages are site-specific
or forest-wide.
- Determine if site link bridging is turned on and if the
network is fully routed.
- Verify that all sites are defined in site links.
- Detect and remove preferred bridgeheads.
- Resolve Active Directory replication failures in the
forest.
- Determine if source servers are overloaded.
- Determine if site links are disjointed.
- Delete connections if KCC is in "Connection Keeping" mode.
Determine if the Event ID 1311 messages are site-specific or forest-wide Determine if event ID 1311 messages are logged on all inter-site
topology generator (ISTG) domain controllers in the forest or only on
site-specific ISTG domain controllers. To locate ISTG domain controllers, use
the Ldp.exe tool to search for the following attributes:
Base DN: CN=Sites,CN=Configuration,DC=RootDomainName,DC=Com
Filter: (cn=NTDS Site Settings)
Scope: Subtree
Attributes: interSiteTopologyGenerator
For more information about how to locate ISTG domain controllers, click the following article number to view the article in the Microsoft Knowledge Base:
224599
Determining the Inter-Site Topology Generator (ISTG) of a site in the Active Directory
To determine the scope of the event, use one of the
following methods:
- Examine the Directory Service event logs of an appropriate
number of ISTG domain controllers in the forest.
- Use the Eventcombmt.exe tool (available from Microsoft
Product Support Services) to search for event ID 1311 messages on an
appropriate number of ISTG domain controllers in the forest.
Determine if site link bridging is turned on and if the network is
fully routed When you enable site link bridging in the Active Directory Sites
and Services snap-in, you must make sure that any site defined in Active
Directory has a fully routed network connection to any other site that is
defined by the administrator. If KCC builds a connection link between two
unconnected sites in which site link bridging is enabled, event ID 1311
messages may be logged. Site link bridging is enabled in Active
Directory if the following conditions are true:
- The Bridge all site links check box is
selected for the IP protocol and the SMTP protocol in the Active Directory
Sites and Services snap-in.
- The Options attribute for the IP protocol and the SMTP protocol is NULL or
set to 0 (zero) for the following Domain Name (DN) paths:
- CN=IP,CN=Inter-Site
Transports,CN=Sites,CN=Configuration,DC=root domain of
forest
- CN=SMTP,CN=Inter-Site
Transports,CN=Sites,CN=Configuration,DC=root domain of
forest
To determine if a fully routed network connection exists
between two sites, contact your NOS administrator, network administrator, or
Active Directory architect. If site link bridging is enabled in a
non-routed environment, either make the network fully routed, or disable site
link bridging and then create the site links and site link bridges that you
must use. Wait for two times the longest replication interval in the forest. If
event ID 1311 messages continue to be logged or if site link bridging is
enabled in a fully routed network, continue to the "Verify That All Sites Are
Defined in Site Links" method. By default, site link bridging is
turned on. Additionally, the best practice guidelines recommend that you keep
site link bridging turned on. The following diagram uses plus signs
(+) and minus signs (-) to illustrate physical network connections between
Active Directory sites. Site AZ is listed in site link WEST and site GA is
listed in site link EAST, but sites AZ and GA do not have fully routed network
connections to sites WA and NY in an Active Directory configuration where site
link bridging is enabled.
WA <-- Site Link WANY --> NY
+- +-
+ - + -
+ - + -
+ - + -
CA + + + AZ IL + + + GA
Site Link WEST Site Link EAST
Verify that all sites are defined in site links Every site defined in Active Directory must be hosted or reside
in a site link. For example if sites WA, CA, AZ, NY, IL, and GA are defined,
and site links WEST, EAST and WANY are defined, event ID 1311 messages are
logged if any one site (for example, AZ or GA) is not listed in a site link
where the sites are physically connected. Sites are orphaned when sites in a
deleted site link are not added to an appropriate existing site link.
WA -- Site Link WANY -- NY
/ /
/ /
/ /
CA (AZ) IL (GA)
Site Link WEST Site Link EAST
Because sites AZ and GA are not listed in any site links, they are
orphaned and the KCC does not consider them when it constructs the
replication topology for Active Directory. The repadmin /showism command is useful for locating improperly configured sites.
Output from the repadmin /showism command appears similar to the following example from a forest
named "corp:"
==== TRANSPORT CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=corp,DC=com CONNECTIVITY INFORMATION FOR 3 SITES: ====
0, 1, 2
( 0) CN=US-NC,CN=Sites,CN=Configuration,DC=corp,DC=com
0:0:0, 100:15:0, 200:15:0
( 1) CN=US-TX,CN=Sites,CN=Configuration,DC=corp,DC=com
100:15:0, 0:0:0, 100:15:0
( 2) CN=US-WA,CN=Sites,CN=Configuration,DC=corp,DC=com
200:15:0, 100:15:0, 0:0:0
NOTE: Unlike other arguments for the repadmin command, you cannot run the repadmin /showism command from a remote computer. You must run the repadmin /showism command from the console of the domain controller that you want
to examine (in most cases, this is the ISTG domain controller). For
each site that is configured for IP-based replication or for SMTP-based
replication (not shown), the repadmin /showism command returns a site matrix that represent the connections to
all the sites in the forest. Each entry in the site matrix contains three
numbers delimited by colons (:) that represent the cost, replication interval,
and options for each replication link to another site in the Active Directory
forest. The numbers in an entry appear in the following order: Cost:Replication
interval:Options - The Cost value indicates the
preference for a network link for replicating directory information between
sites. The administrator uses the Active Directory Sites and Services snap-in
to define the Cost value for each site link.
- The Replication interval value
indicates the replication frequency of the link in minutes.
- The Options value indicates the
options for the site link, including site link notification.
NOTE: When you troubleshoot event ID 1311 messages, you can ignore the
Options value.
In the example from the "corp.com" forest, site link bridging
is enabled, and the forest contains three Active Directory sites:
- Site 0: US-NC, an uncovered site that uses the TX<->NC link to
connect to Site 1 (US-TX).
- Site 1: US-TX, which hosts 2 domain controllers.
- Site 2: US-WA, a covered site that uses the TX<->WA link to
connect to Site 1 (US-TX).
Each site matrix contains one "0:0:0" entry that refers to
itself. An entry that contains positive numbers for the cost value and
replication interval value (for example, "200:15:0" or "100:15:0") indicates
that the site connection is good. A "-1:0:0" entry indicates that the site
connection is not working. This occurs if one or more of the following
conditions is true:
- The replication protocol is not used. For example, if SMTP
replication is not configured, the entries in the SMTP portion of the /SHOWISM
matrix all appear as "-1:0:0".
- The site does not host any domain controllers (this is
known as an "uncovered" site).
- The site is not included in a site link.
If site link bridging is enabled and the repadmin /showism command returns "-1:0:0" entries for one or more covered Active
Directory sites, make sure that the affected sites are listed in a site link.
A site with a full complement of "-1:0:0" entries and one "0:0:0"
entry is orphaned unless the site is uncovered (no domain controllers reside in
that site). When you troubleshoot event ID 1311 messages, record the names of
all orphaned sites, but do not record the names of uncovered sites.
If site link bridging is disabled, "-1:0:0" entries are less meaningful. If
this is the case, you must manually determine if each site is included in a
site link. To do so, write down the list of sites and site links, and manually
map each site to a site link. NOTE: The repadmin /showism command always returns "-1:0:0" entries for an uncovered site.
In the following repadmin /showism example, site link bridging is enabled in the "corp.com" forest,
and site link TX<->WA was deleted. Site 2 (US-WA) is orphaned from all
other sites in the forest and must be added to an appropriate site link.
==== TRANSPORT CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=corp,DC=com CONNECTIVITY INFORMATION FOR 3 SITES: ====
0, 1, 2
( 0) CN=US-NC,CN=Sites,CN=Configuration,DC=corp,DC=com
0:0:0, 100:15:0, -1:0:0
( 1) CN=US-TX,CN=Sites,CN=Configuration,DC=corp,DC=com
100:15:0, 0:0:0, -1:0:0
( 2) CN=US-WA,CN=Sites,CN=Configuration,DC=corp,DC=com
-1:0:0, -1:0:0, 0:0:0 Detect and remove preferred bridgeheads Because correct bridgehead selection is difficult in multi-domain
forests, and because Windows 2000 has good fail-over logic in case a
KCC-selected bridgehead goes offline, Microsoft strongly recommends that you do
not define preferred bridgehead servers. To search for preferred
bridgehead servers:
- Use the Ldp.exe command-line tool to perform an LDAP search
for the following criteria:
DN Path: cn=sites,cn=configuration,dc=<root domain of forest>
ObjectClass: server
Attributes: bridgeheadTransportList
- Use the FINDSTR command against an LDIFDE export file from the
CN=Sites,CN=Configuration container:
LDIFDE CN=SITES,CN=CONFIGURATION,DC=<Root domain in forest>
SITEDUMP.LDF
FINDSTR /i "bridgeheadTransportList" SITEDUMP.LDF
If the search returns any results, note the name of server in the
Domain Name path in which the bridgeheadTransportList attribute is populated.
If you find any preferred
bridgehead servers, use the Site and Services snap-in to remove them, and then
wait two times the maximum replication interval in the forest. If event ID 1311
messages continue to be logged, continue to the next method.
Resolve Active Directory replication failures in the forest Active Directory replication requires the transitive replication
of all naming contexts in the forest to all domain controllers that replicate a
common partition. Resolve replication failures for online domain
controllers as quickly as possible, especially those that host one-of-a-kind
naming contexts in a forest (for example, the only domain controller for a
particular domain in the forest). As a last resort, if you cannot make a domain
controller replicate, remove it from the forest. If a domain
controller is offline for fewer days than the tombstone lifetime number (by
default 60), bring the domain controller online and force it to replicate, or
as a last resort, remove it from the forest. If a domain controller
is offline or does not replicate inbound changes for more days than the
tombstone lifetime number, do not resuscitate it. Instead, immediately remove
it from the forest.
For more information
about the TombstoneLifetime value, click the following article numbers to view the articles in the Microsoft Knowledge Base:
216993
Useful shelf life of a system-state backup of Active Directory
314282 Lingering objects may remain after you bring an out-of-date Global Catalog server back online
When you want to discover and troubleshoot
replication failures, the following tools can be useful:
- repadmin /failcache: Run this command from the console of each ISTG domain controller
in the forest to discover replication failures for bridgeheads in the site for
that ISTG.
Note You can also run this command remotely against other ISTG domain
controllers in the forest. - repadmin /showreps: Run this command from the console of each ISTG domain controller
in the forest to analyze replication of specific domain controllers that are
exposed by the repadmin /failcache command.
- dcdiag /test:intersite /e /q: This command tests inter-site connectivity for bridgehead domain
controllers in the forest. The result set is limited to domain controllers that
experience errors with the /q switch.
- dcdiag /test:connectivity /e /q: This command tests name resolution and ldap / rpc connectivity
to all domain controllers in the forest. The result set is limited to domain
controllers that experience errors with the /q switch.
- Examine the Directory Service Event log on ISTG domain controllers and Bridgehead servers, using the
following settings for the NTDS Diagnostic levels:
- 1 Knowledge Consistency Checker: 3
- 5 Replication Events: 3
- Internal Processing: 1
The repadmin /failcache command lists replication failures that KCC knows about. The
output from the repadmin /failcache command is divided into two sections: The "KCC Link
Failures" cache lists errors for existing connection links. The ISTG domain
controller imports showreps ("repsfroms") data for every bridgehead server in
its site. However, the ISTG domain controller does not list errors. The link
failure cache is emptied at the beginning of every KCC run and refilled during
the course of the current run. The "KCC Connection Failures" cache
lists unsuccessful attempts to build connection objects between domain
controllers ("reps from" or "reps to"). When you run the repadmin /failcache command from the ISTG domain controller, it lists entries that
are imported from bridgeheads in the site. At the beginning of each KCC run,
the KCC examines each entry in the connection failure cache and tries to DsBind
to the failing server. If the bind succeeds, the entry is removed.
The repadmin /failcache command differs from the repadmin /showreps command in two ways:
- The repadmin /showreps command shows the naming context that is failing. The repadmin /failcache command does not.
- Data from the repadmin /failcache command is not replicated between domain controllers.
The following example shows sample output from the repadmin /failcache command.
Z:\>repadmin /failcache
==== KCC CONNECTION FAILURES ============================
(none)
==== KCC LINK FAILURES ==================================
USA-WA-24\C-24-DC03
DC object GUID: 134244cd-26be-4944-82a7-ac3eb74fc02f
No Failures.
USA-WA-24\B-24-DC02
DC object GUID: 21b050d6-33b5-424d-aa9b-060fe209233d
No Failures.
USA-WA-24\Z-24-DC-05
DC object GUID: bfb3b008-3849-4e5d-81d8-53dbb76d587a
No Failures. Determine if source servers are overloaded A domain controller that is overloaded with a large number of
direct replication partners or a replication schedule that is overly aggressive
can create a backlog in which some partners never receive changes from a hub
domain controller. In the output from the repadmin /showreps command, partner domain controllers of overloaded source domain
controllers appear with the "at never" status. To resolve this
issue, resize hardware, reconfigure site links and reconfigure site link or
connection schedules as necessary to reduce the load on overloaded domain
controllers. Determine if site links are disjointed "Disjoint site links" is an Active Directory configuration in
which the topology is broken into two parts or in which some sites do not
replicate because site definitions and site link definitions are incorrect. For
example, the following diagram shows a configuration in which Sitelink_ABC
contains sites A, B, and C and Sitelink_DEF contains sites D, E, and F, but no
site link connects any of the sites in Sitelink_ABC to any of the sites in
Sitelink_DEF. To resolve the disjoint site links condition, a new site link
must connect at least one site in Sitelink_ABC with at least one site in
Sitelink_DEF (for example, a new site link between site A and site D).
A D
/ \ / \
/ \ / \
/ \ / \
B C E F
Sitelink_ABC Sitelink_DEF The following diagram shows another possible a disjoint site
links configuration. In this case, a new site link must join any site in
Sitelink_ABDC with at least one site in Sitelink_FG (for example, a new site
link between site A and site F) to resolve the disjoint site links condition.
A F
/ \ \
/ \ \
/ \ \
B C \
\ / \
\ / \
\ / \
D G
Sitelink_ABDC Sitelink_FG Disjoint site links are the most difficult improper configuration
to troubleshoot. Look for disjoint site links only after you rule out all other
known causes. Use a pencil and paper to graph site topology and locate orphaned
sites. Delete connections if KCC is in "Keep Connection mode" If KCC builds a different path around a site-to-site connection
failure, but it retries the failing connection every 15 minutes because it is
in "connection keeping mode," delete all broken connections and let KCC rebuild
them. Wait two times the longest replication schedule in the forest.
MORE INFORMATIONTerminology and concepts- Bridgehead server: Any domain controller in an Active Directory site that
replicates an Active Directory partition (for example, schema, configuration,
domain, application partition or global catalog) to a domain controller in
another Active Directory site.
A bridgehead is selected for each
unique directory partition, domain, or application partition in an Active
Directory site, so a site that hosts three different domains has three in-site
bridgehead servers.
Domain controllers replicate all naming contexts
that are held in common with their direct replication partners, so a domain
controller in the "corp.com" domain replicates CN=SCHEMA and CN=CONFIGURATION
in addition to the "corp.com" domain naming context with its inter-site
bridgehead partner. - Inter-site topology generator (ISTG): For each Active Directory site, a single server, known as the
ISTG, is nominated to build the inter-site replication topology.
For more information about the ISTG, click the following article number to view the article in the Microsoft Knowledge Base:
224815
The role of the Inter-Site Topology Generator in Active Directory replication
- Uncovered Site: An Active Directory site defined in the Sites and Services
snap-in that does not currently contain any Windows 2000 domain controllers. An
uncovered site may be waiting for its domain controller to arrive from a
staging site. Additionally, a site may be defined as uncovered to provide site
preference for client operations.
Truncated output from the REPADMIN /SHOWISM command In some environments, the repadmin /showism command from build 2195 of Windows 2000 quits prematurely during
execution and its output is truncated because of an internal error. For
example, the top portion of this successful /SHOWISM output from a domain
controller in the "corp.com" domain indicates that 128 sites are defined
(0-127).
==== TRANSPORT CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=corp,DC=com
CONNECTIVITY INFORMATION FOR 128 SITES: ====
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58,
59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88,
89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118,
119, 120, 121, 122, 123, 124, 125, 126, 127 In the following example, the repadmin /showism output stops in the middle of the line for site 115,
CN=HeadQuarters.
All DCs in site CN=Headquarters,CN=Sites,CN=Configuration,DC=corp,DC=com (with trans & hosting NC) are bridgehead candidates.
(115) CN=headquarters,CN=Sites,CN=Configuration,DC=corp,DC=com
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0,
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0,
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0,
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0,
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0,
-1:0:0, -1:0:0, -1:0:0, -1:0:0, -1:0:0, 100:0:0, 150:0:0, 150:0:0, 100:0:0, To resolve this truncation problem, obtain an updated version of
the Repadmin.exe file from Microsoft Product Support Services (PSS).
Modification Type: | Minor | Last Reviewed: | 5/24/2005 |
---|
Keywords: | kbprb KB307593 |
---|
|