How to locate and correct disk space problems on NTFS volumes in Windows XP (315688)

MORE INFORMATION

The disk space allocation of an NTFS volume may be misreported for any of the following reasons:

• The cluster size of the NTFS volume is too large for the average-sized files that are being stored.
• File attributes or NTFS permissions prevent files or folders from being either displayed or accessed when you use either Microsoft Windows Explorer or a Windows command prompt.
• The folder path exceed 255 characters.
• Folders or files contain invalid or reserved file names.
• NTFS metafiles (such as the Master File Table [MFT]) have grown and cannot be unallocated.
• Files or folders contain alternate data streams.
• NTFS corruption causes Windows to report free space as being in use.
• Other NTFS features cause file-allocation confusion.

The Cluster Size Is Too Large

Disk space can be consumed only by files and folders that include internal NTFS metafiles, for example the MFT, folder indexes, and so on. Multiples of a cluster consume all file space allocation. A cluster is a collection of contiguous sectors. The cluster size is determined at the time that the volume is formatted and is further determined by the partition size.

For additional information about clusters, click the article number below to view the article in the Microsoft Knowledge Base: After a file is created, it consumes the minimum size of a single cluster of disk space, depending on the initial file size. After you add data to a file, NTFS increases the file's allocation in multiples of the cluster size.

To determine the current cluster size and volume statistics, run the following command from a command prompt: The following text is an example of the output that is displayed if you run this command: NOTE: Multiply each value that is reported in kilobytes (KB) by 1024 to determine the accurate byte size (for example, 2,906,360 x 1024 = 2,976,112,640 bytes).

Review this output to determine the default cluster size and how your disk space is being used. To see if the cluster is using the optimal cluster size, determine the amount of wasted space:

1. Double-click My Computer, and then double-click the drive letter (for example, D) of the volume that you want to check.
2. Click any file or folder, and then click Select All on the Edit menu.
3. Right-click any file or folder, and then click Properties.
4. Click the General tab, and then review the "Size" and "Size on disk" file size values, which calculate the total number of files and folders on the entire volume.

If you are not using NTFS compression for any files or folders that are contained on the volume, the difference between the Size value and the Size on disk value is the wasted space that occurs because the cluster size is larger than necessary. Choose an optimal cluster size so that the Size on disk value is as close to the Size value as possible. An excessive discrepancy between the Size on disk value and the Size value is an indication that the default cluster size is too large for the average file size that you are storing on the volume. In this scenario, it is recommended that you decrease the cluster size. To do so, back up the volume, and then use the format command with the /a switch to specify the appropriate allocation size to reformat the volume. For example, run the following command for a 2-KB cluster size: Alternatively, you can enable NTFS compression to regain space that is lost because of an incorrect cluster size; however, if you do so, you may experience a slight decrease in performance.

File Attributes or NTFS Permissions

You can use either Windows Explorer or the dir /a /s folder list command to report the file and folder statistics for only those files and folders that you have permissions to access. Hidden files and protected operating system files are always excluded from this report by default. Because some folders are excluded, inaccurate file and folder totals and size statistics may be displayed by Windows Explorer or dir command outputs. To include these types of files in the overall statistics, change Folder Options:

1. Double-click My Computer, and then double-click the drive letter of the volume that you want to check.
2. Click Folder Options on the Tools menu, and then click the View tab.
3. Click the Show Hidden Files and Folders check box, and then click to clear the Hide protected operating system files check box.
4. After you receive the warning message, click Apply.

After you unhide the files and folders, you can use either Windows Explorer or the dir /a /s command to generate the total of all files and folders that are contained on the volume to which the user has permissions.

To determine the folders and files that cannot be accessed:

1. From a command prompt, send the output of a dir /a /s to a text file.
   
   For example, run the following command:

   dir d: /a /s >c:\d-dir.txt

2. Start Ntbackup.exe, click Options on the Tools menu, click the Backup Log tab, and then click Detailed.
3. Click the Backup tab, back up the entire volume that is affected (in this example, drive D), and then start the backup procedure.
4. After the backup procedure is complete, view the backup report, and then compare the folders in the Ntbackup log output with the folders in the output that you saved in a text file in step 1.

The backup procedure accesses all files; therefore, its report may contain folders and files that are not seen or counted when you use either Windows Explorer or the dir command. If you are looking for large files or folders that you cannot use Windows Explorer to access, you may find it easier to use the Ntbackup graphical user interface (GUI) to navigate the volume. Use the Ntbackup GUI to view the volume without actually backing up the volume.

After you locate files to which you do not have access, open the file or folder's properties in Windows Explorer, click the Security tab, and then add or change permissions so that the folder is in included in the dir /a /s command output. By default, you do not have access to the System Volume Information folder.

NOTE: You may find that some of the folder or file properties do not contain a Security tab, or you may find that you cannot re-assign permissions to the affected folders and files. You may receive the following error message while you try to access these files: If you find these types of folders, contact Microsoft Product Support Services for additional assistance. To contact Microsoft Product Support Services, visit the following Microsoft Web site:

Invalid File Names

Folders or files that contain invalid or reserved file names may also be excluded from file and folder statistics. Folders or files that contain leading or trailing spaces are acceptable in NTFS; however, these files are not acceptable in the Win32 subsystem. Therefore, neither Windows Explorer nor a command prompt can reliably handle files that have leading or trailing spaces.

For additional information, click the article number below to view the article in the Microsoft Knowledge Base: Typically, it is not possible to rename or delete files or folders that have leading or trailing spaces. If you try to rename or delete these folders of files, you may receive one of the following error messages: If you have folders or files that you cannot delete or rename, contact Product Support Services at the following Microsoft Web site:

NTFS MFT Expansion

After you create and format an NTFS volume, NTFS metafiles are created. One of these metafiles is called the "Master File Table" (MFT). This file is very small when it is created (approximately 16 KB), but it grows as files and folders are created on the volume. When a file is created, it is entered into the MFT as a file record segment, which is always 1024 bytes (1 KB) in size. As files are added to the volume, the MFT grows as required. However, when you delete files, the associated file record segments are marked as free to be reused, but the total file record segments and associated MFT allocation remains the same. This behavior explains why you do not regain the space that is used by the MFT after you delete a large number of files.

To determine the exact size of the MFT, use the built-in disk defragmenter utility to analyze the volume. View the defragmenter report to obtain detailed information about the size and number of fragments in the MFT.

The following text is an example of the defragmenter report: For a more complete picture of how much space (overhead) is being used by the entire NTFS file system, run the chkdsk command, and then view resulting output for the following line: Currently, only third-party defragmenter utilities consolidate unused MFT file record segment records and reclaim unused MFT allocated space.

Alternate Data Streams

NTFS allows files and folders to contain alternate data streams. This feature allows multiple data allocations to be associated with a single file or folder. Please be aware of the following limitations when you use alternate data streams on files and folders:

• Windows Explorer and the dir command do not report the data in alternate data streams as part of the file size or volume statistics. Instead, they show only the total bytes for the primary data stream.
• The output from the chkdsk command accurately reports space used by a user's data files, including alternate data streams.
• Disk quotas accurately track and report all data stream allocations that are part of a user's data files.
• Ntbackup records the number of bytes that are backed up in the backup log report. However, it does not show which files contain alternate data streams, nor does it show accurate file sizes for files that include data in alternate streams.

NTFS File System Corruption

In very rare circumstances, the NTFS $MFT or $BITMAP metafiles may become corrupted and result in lost disk space. To identify and fix this issue, run the chkdsk /F command against the volume in question. Toward the end of chkdsk process, you receive the following message if the $BITMAP metafile needs to be adjusted:

Other NTFS Features That May Cause File Allocation Confusion

NTFS also supports hard links and reparse points that allow volume mount points and folder junctions to be created. These additional NTFS features may cause confusion when you try to determine how much space is being consumed on a physical volume.

A hard link is a folder entry for a file regardless of where the actual file data exists on that volume. Every file is considered to have at least one hard link. On NTFS volumes, each file can have multiple hard links; therefore, a single file can be displayed in many folders (or even in the same folder with different names). Because all of the links reference the same file, programs can open any of the links and modify the file. A file is deleted from the file system only after all of the links to it have been deleted. After you create a hard link, programs can use the link like any other file name. Note that Windows Explorer and a command prompt will show all linked files as being the same size, even though they all share the same data and do not actually use that amount of disk space.

Volume mount points and folder junctions allow an empty folder on an NTFS volume to point to the root or subfolder on another volume. Windows Explorer and the dir /s command follow the reparse point, count any files and folders on the destination volume, and then include them in the host volume's statistics. This behavior may lead you to believe that more space is being used on the host volume than what is actually being used. For additional information about junction points, click the article number below to view the article in the Microsoft Knowledge Base: In summary, use the following methods to correctly determine how the disk space is being used on a volume:

• View the output from the chkdsk command.
• Use the Ntbackup GUI or view the backup logs.
• View the disk quotas.

On the other hand, Windows Explorer and the dir command have some limitations and drawbacks when you use them to determine how disk space is being used.