How to recover from information store corruption (272570)

MORE INFORMATION

When corruption occurs in Exchange Server 5.5, one of the most commonly-affected components is the information store. Corruption typically involves any one of the .Exchange Server database (.edb) files that are located in the Exchsrvr\Mdbdata folder on the server. Either the Priv.edb file or the Pub.edb file may be corrupted.

When the information store is corrupted, you may experience one or more of the following symptoms:

• The Microsoft Exchange Server Information Store service does not start.
• You cannot stop the information store.
• The information store stops responding and the CPU usage level remains at 100 percent.
• Clients cannot retrieve or send e-mail messages. This issue persists even when you stop and restart the information store.
• When you try to restore the Exchange Server databases from an offline backup, and then you run the Information Store Integrity utility (Isinteg) , you receive the following error message:
  Error 4294966746: JET_errDatabaseInconsistent

The procedure that is described in this article is based on a single-server, single-site scenario where the following conditions are true:

• A site connector is not used.
• An offline copy of both Pub.edb and Priv.edb exists.
• A Key Management Server is not used.
• The server is running Microsoft Small Business Server 4.5. However, the principles can apply to most other Exchange Server 5.5 computers that are set up in a single-organization, single-site configuration.

Note This setup is not intended to restore connector settings.

Troubleshoot

To isolate information store corruption and to help you recover the public and private information stores, follow these steps.

Note Before you perform this procedure, make sure that you can recover the data files from another source. For example, if you are using a small Exchange Server database (less than 1 gigabyte [GB]), make a copy of the Pub.edb and Priv.edb files. For larger databases, if you do not have sufficient disk space to copy the files, use a tape backup system or another offline storage utility.

To troubleshoot the information store:

1. Try to start the information store. Make a note of any error messages or any event ID messages that occur.
2. Use the Eseutil.exe command-line tool to check the consistency of the information store databases. If a database is consistent, all the log files are committed to the store. If the database is not consistent, the database may not be corrupted. For example, the log files may not be committed to the database yet. To check the consistency of the public and private information store databases, type the following commands at the command prompt, and then press ENTER after each line:

   c:\exchsrvr\bin>eseutil /mh ..\mdbdata\priv.edb
   
   c:\exchsrvr\bin>eseutil /mh ..\mdbdata\pub.edb

   Note This example assumes that you run the Eseutil.exe command-line tool from the ExchSrvr\Bin folder, and that the .edb files are located in the ExchSrvr\Mdbdata folder.
   
   To verify that the .edb file is consistent, check the output that appears in the "State" line in the output. To send the output of this command directly to a text file, modify the command line as follows:

   c:\exchsrvr\bin>eseutil /mh ..\mdbdata\priv.edb > mypriv.txt
   
   c:\exchsrvr\bin>eseutil /mh ..\mdbdata\pub.edb > mypub.txt

   To display the text directly on the screen, type | more at the end of the command.
   
   If the Pub.edb and Priv.edb files in the ExchSrvr\Mdbdata folder are not located on the same drive as the ExchSrvr\Bin folder that you are running the command from, replace the "..\mdbdata\priv.edb" portion of the command with the full path of the Priv.edb and the Pub.edb files. If both the Pub.edb and Priv.edb files are consistent, go to step 6.
   
   If one of the databases is inconsistent, try to perform a soft recovery of the files to see if there are copies of the log files that are stored in the Mdbdata folder. If you do not have the log files, you cannot complete the soft recovery process. When you perform a soft recovery, uncommitted logs are committed to the information store database.
   
   To initiate a soft recovery, type the following commands at the command prompt. The first command initiates a soft recovery on the private information store. The second command initiates a soft recovery on the public information store:

   c:\exchsrvr\bin>eseutil /r /ispriv
   
   c:\exchsrvr\bin>eseutil /r /ispub

   This command permits you to recover the databases (if the databases are fully recoverable). To confirm this behavior, repeat this step, and then run the eseutil /mh command again. If the "State" line of the output is changed to "Consistent" for both the public and private information store, go to step 6.
3. If either of the databases is inconsistent, and the eseutil /r command does not return the databases to a consistent state, use the Eseutil.exe tool to perform a hard repair.
   
   Note When you use the following command, you may lose some Exchange Server data, including the messages that are contained in the log files that are not yet committed to the information store database. The following command is a hard or forcible state recovery command. Only use this command if the public or private information stores do not return to a consistent state after you perform step 2 earlier in this article.
   
   To perform a hard repair, follow these steps:
   • To return Priv.edb to a consistent state, type the following command at the command prompt, and then press ENTER:

     c:\exchsrvr\bin>eseutil /p /ispriv

   • To return Pub.edb to a consistent state, type the following command at the command prompt, and then press ENTER:

     c:\exchsrvr\bin>eseutil /p /ispub

   For more information about the ramifications of performing a "hard" repair of an Exchange database, click the following article number to view the article in the Microsoft Knowledge Base:

   259851 Ramifications of running the eseutil /p or edbutil /d /r command in Exchange

4. Delete all .log and .chk files in the ExchSrvr\Mdbdata folder on the server. Additionally, if the Temp.edb file exists in the ExchSrvr\Mdbdata folder, delete it.
5. Start the Microsoft Exchange Information Store service to verify that the databases are working. When you confirm that the Exchange Information Store service starts successfully, stop the Exchange Information Store service.
   
   Note If the Exchange Information Store service does not start, verify that the database is in a consistent state. To do so, see step 2 earlier in this article.
6. Defragment the database that you just repaired. To do so:
   
   Note You must have available hard disk space equal to 110 percent of the size of the database that you want to defragment.
   • To defragment Priv.edb, type the following command at the command prompt, and then press ENTER:

     c:\exchsrvr\bin>eseutil /d /ispriv

   • To defragment Pub.edb, type the following command at the command prompt, and then press ENTER:

     c:\exchsrvr\bin>eseutil /d /ispub

     Note If the private and the public information store databases are large, this step may take some time.
     
     If you receive an error message that indicates that there is not sufficient hard disk space to run the command, copy the ExchSrvr\Bin folder to a drive with more space, and then run the command from that location. For additional information about how to resolve this error message, click the following article number to view the article in the Microsoft Knowledge Base:

     192185 How to defragment with the Eseutil utility (Eseutil.exe)

7. Use the Isinteg.exe command-line tool to repair Pub.edb and Priv.edb. This utility runs a test on all areas of each of the databases and reports the results. It also tries to fix any issues that it encounters.
   • To fix Priv.edb, type the following command at a command prompt, and then press ENTER:

     c:\exchsrvr\bin>isinteg -pri -fix -test alltests

   • To fix Pub.edb, type the following command at a command prompt, and then press ENTER:

     c:\exchsrvr\bin>isinteg -pub -fix -test alltests

   You receive a summary report that indicates the number of tests that were run, the number of warnings, the number of errors found, the number of fixes tried, and the time it took for the program to run. If there are any warnings, errors, or fixes, run the command again. Repeat this step until there are no warnings, errors, or fixes.
   
   Note In some situations, the same warnings, errors, or fixes are reported every time you run the program. If you receive the same warnings, errors, or fixes three times in a row, go to the next step. The errors may be caused by a corrupted e-mail attachment or something similar. The repeat errors do not prevent you from starting the information store. If you can start the information store, if it is stable, and if errors and warning messages continue to be reported after you have run Isinteg.exe several times, use the Exmerge.exe tool to repair the information store. For more information about how to use Exmerge.exe, see the "To Use the Exmerge.exe Tool" section later in this article.
8. Try to start the information store database. If the information store starts, go to step 9. If the information store does not start and you receive an error -1011, you must restore Pub.edb and Priv.edb to a state where the information store can recognize them. To do so, type the following command at the command prompt, and then press ENTER:

   c:\exchsrvr\bin>isinteg -patch

   Note The Microsoft Exchange Directory service must be running when you run this command. This command works for both the public and the private databases.
   
   This command works for both the public and the private databases. You do not have to complete this step if you already performed an online restore of the data. However, if you receive an -1011 error message when you try to start the information store, complete this step.
9. Restart the information store.
10. Run the Directory Service/Information Store (DS/IS) Consistency Adjuster. For more information about how to do so, see the "To Use the Directory Service/Information Store (DS/IS) Consistency Adjuster section later in this article. The DS/IS Consistency Adjuster is used in disaster recovery scenarios where you cannot restore the folder to the recovery computer, or where the information store (the Priv.edb or the Pub.edb files) is copied to a recovery computer with a name that is different from that of the original server. If you did not lose the directory service or you did not reinstall Exchange Server to a clean folder, then you do not have to run this tool.
    
    Note If you use this tool and you have multiple sites in your Exchange Server organization or multiple servers in your Exchange Server site, public folders may be rehomed unexpectedly.

For additional information about the DS/IS Consistency Adjuster, click the following article number to view the article in the Microsoft Knowledge Base:

To Use the DS/IS Consistency Adjuster

To run the Exchange Server Consistency Adjuster:

1. Start the Exchange Server Administrator program.
2. Expand Configuration, expand Servers, and then click ServerName.
3. On the File menu, click Properties, and then click the Advanced tab.
4. Click Consistency Adjuster, click to select all the check boxes under Operations, and then click OK.

The DS/IS Consistency Adjuster runs on both the public and private information stores.

If you perform this recovery operation on another server, or if you removed and reinstalled Exchange Server on this server and do not have a working copy of the directory store (Dir.edb), there are two methods for recovering user mailboxes. For both these methods, you must re-create any distribution lists and memberships to those distribution lists that were lost.

Method One

If you can access user accounts, start the Exchange Server Administrator program, and then click Extract Windows NT Account List on the Tools menu. Make sure that you extract the user list from the domain and not from the local computer. This procedure permits you to create a file that contains a list of all the Windows NT users.

After you create the file, on the Tools menu, click Directory Import. When you import this list, all the recipient mailboxes are restored. You may want to run the DS/IS Consistency Adjuster again after you perform this step.

Method Two

If you do not have access to the Windows NT user accounts for these recipients, you must manually re-create all the Windows NT user accounts with the same user names. After you do so, start the Exchange Server Administrator program, and then click Extract Windows NT Account List on the Tools menu. Make sure that you extract the user list from the domain, and not from the local computer. This procedure enables you to create a file that contains a list of all the Windows NT users.

After the file is created, on the Tools menu, click Directory Import. When you import this list, all the recipients' mailboxes are restored.

After you complete either of these methods, if the mailbox has not been restored, you may have created a user account that has a name that is different from the previous Exchange Server recipient.

At this point, you can expect the server to function and the users to be able to retrieve e-mail.

To Use the Exmerge.exe Tool

If the information store starts, but there are other issues that you have to troubleshoot, use the Exmerge.exe tool to export all the user's mailboxes to .pst files. After you do so, merge the .pst files to user mailboxes in a new Priv.edb file. If you use offline storage for mail (.ost files), you have to move the .ost files to a .pst file before you start this procedure to prevent you from losing any .ost data. Note that the Exmerge.exe tool only permits you to move user mailboxes. You cannot use Exmerge.exe to move specific Exchange Server configuration properties such as connector settings, Microsoft Outlook settings, or client-side rules. You cannot use Exmerge.exe to move distribution lists, memberships in those distribution lists, or public folders. If you tried this procedure, you must re-create or reconfigure the distribution lists manually. This procedure also destroys your single instance store for messages up to that point. As a result, the Exchange Server databases run the risk of growing substantially.

For additional information about the Exmerge.exe tool, click the following article number to view the article in the Microsoft Knowledge Base: To download the Exmerge.exe tool, visit the following Microsoft Web site: Note This version of Exmerge.exe was publicly used to deal with the "Love Bug" virus. That is why the download package is named Iloveyouhlpi.zip.

After you download and extract the Exmerge.exe utility, delete the Exmerge.ini file that is located in the folder where you extracted the files. This procedure prevents errors from occurring while you are running the Exmerge.exe utility.

1. Make sure that users are not currently running an e-mail client program and accessing e-mail messages on the server.
2. Run Exmerge.exe using step 1 of the two-step merge process, and then enter all users.
3. When the Exmerge.exe operation is complete, stop the Exchange Server Information Store service.
4. Rename the Pub.edb and Priv.edb files in the ExchSrvr\Mdbdata folder on the server. Delete all other files in this folder. Make sure that you do not delete the Pub.edb and Priv.edb files that you renamed until after you are sure the merge completed successfully.
5. Restart the Exchange Server Information Store service.
6. On a client computer, start Outlook, and then send an e-mail message to all the users in the Exchange Server global address book.
   
   Note If you do not complete this step, step 2 of the Exmerge.exe two-step merge process does not show that there are any available mailboxes to restore.
7. Run Exmerge.exe by using step 2 of the two-step merge process, and then enter all users. When this process is complete, the e-mail messages that were previously stored in the .pst files are moved to Exchange Server mailboxes.

You now have a new Priv.edb file that contains fresh copies of all mail messages.

For additional information about restoring the Exchange Server information store, click the following article numbers to view the articles in the Microsoft Knowledge Base:
For more information about Exchange disaster recovery, see the Exchange 5.5 Disaster Recovery white paper. To do so, visit the following Microsoft Web site: For more information about the Isinteg.exe and Eseutil.exe utilities, see the Eseutil.rtf and Isinteg.rtf documents that are located in the ExchSrvr\Server\Support\Utils folder of the Exchange Server CD.