User:Tudoros1/FileSystems/NTFS

NTFS is a complex file system and there are hundreds of stuff undocumented. However, some things should stay undocumented due to mind boggling and life wasting. This will contain the needed info to work with simple NTFS volumes.

NOTE: Some info here is guesswork. NOTE: This info is incomplete

The Boot sector

Note: This corresponds with the $Boot file, as I will show you later. The fields are:

struct NTFS_BOOT
{
  BYTE unused1[3];  //This is a jmp instruction followed by a nop.
  BYTE name[8];   //Contains the string "NTFS    "
  WORD sectorsize;   //Must be 512
  BYTE clustersize;   //Max cluster size 256KB
  BYTE unused2[7];   //FAT specific stuff
  BYTE media;   //Must be 0xF8(fixed disk)
  WORD unused3[7];
  DWORD signature;   //Must be 0x00800080
  DOUBLE_DWORD length;   //Doesn't include the backup sector;
  DOUBLE_DWORD mft;   //Cluster of the $MFT
  DOUBLE_DWORD mftmirr;   //Cluster of the $MFTMirr
  BYTE filereclength;   //If negative, length = 2^(-(filereclength))
  BYTE unused4[3];
  BYTE indexreclength;
  BYTE unused4[3];
  DOUBLE_DWORD serialnum;
}

File records

Ok, now here is where the fun is, assuming you have parsed correctly you will be here. The Header looks like this:

struct NTFS_FILE_2K
{
  NTFS_RECORD header;    //Common Record Header, id is 'FILE'
  DOUBLE_DWORD unused;   //The $LogFile sequence
  WORD sequence;   //Incremented on every delete, 0 if no checking.
  WORD hardlinkcount;   //Decremented by every hardlink removal and Incremented by every new hardlink
  WORD attroff;   //First Attribute offset
  WORD flags;
  DWORD realsize;
  DWORD allocsize;
  DOUBLE_DWORD base;   //non 0 for extensions
  WORD nextattr;   //The next attribute id to be assigned
}

struct NTFS_RECORD
{
  DWORD id;   //A fourCC code
  WORD upsize;   //Words in the update sequence array
  WORD upoff;   //Offset to Update array
}

Ok, lets explain what they mean:

Flags

There are two known flags and 2 unknown flags. The known flags are:

• 0x1 = In use.
• 0x2 = Directory.

Real/Allocated size

The real size is the offset after the $End attribute padded to a QWORD boundary. The allocated size is the FILE record length.

Update sequence array

(Note this applies not only to FILE records, but also INDX, RSTR and RCRD records that we will look at later) Because each record is more than 1 sector, there has to be a way of knowing if something failed halfway. While I assume single sector transfers are atomic, multi sector ones are not. The array is located at the offset specified in the upoff value. This array begins with 1 WORD, the update sequence. It is a cyclic structure that increments with every transfer. It is followed by a list of WORDs, the amount of which is indicated by the upsize value - 1. When writing a record to disk, the update sequence needs to be incremented, the last 2 bytes of every sector copied into the correct place in the list of WORDs (the offset in the list of words is sector_number / 2) and the update sequence is copied into the last 2 bytes of every sector. When reading a record from the disk, the last 2 bytes of every sector is checked. If any aren't equal to the update sequence the record is considered corrupted and you somehow restore it. If they all are, all the values in the list of WORDs is copied to the last 2 bytes of the corresponding sectors in the in memory copy and then you parse as indicated by this guide. Perhaps a code sample will help:

BOOL CheckUpdateSequence(NTFS_RECORD *record)
{
  DWORD off = 2;
  WORD sequence;
  if (record->upsize < 3)   //3 because there is a minimum of 2 sectors and there is 1 extra word, the update sequence, 2+1=3
  {
    return FALSE;
  }
  sequence = *((record->upoff) + record);
  for (WORD i = 1; i<(record->upsize); i++, off+2)
  {
    WORD *theoff = ((i * 512) - 2) + record;
    if (*theoff != sequence)
    {
      return FALSE;
    }
    *theoff = *((record->upoff) + off + record);   //Why can't we combine 2 passes into 1
  }
  return TRUE;
}

BOOL UpdateUpdateSequence(NTFS_RECORD *record)
{
  DWORD off = 2;
  WORD sequence;
  if (record->upsize < 3)   //3 because there is a minimum of 2 sectors and there is 1 extra word, the update sequence, 2+1=3
  {
    return FALSE;
  }
  sequence = (*((record->upoff) + record)) + 1;
  for (WORD i = 1; i<(record->upsize); i++, off+2)
  {
    WORD *theoff = ((i * 512) - 2) + record;
    *((record->upoff) + off + record) = *theoff;
    *theoff = sequence;
  }
  *((record->upoff) + record) = sequence;
  return TRUE;

Attributes

NTFS file records consist of attributes. These are streams of metadata. The Filesystem is very flexible in this sense. Their header is in this format:

struct NTFS_ATTR
{
  DWORD type;
  DWORD length;   //Length of the attribute on the record;
  BYTE flags;   //Bit 1 means non-resident
  BYTE namelength;   //0 if unnamed
  WORD nameoffset;   //0 if unnamed
  WORD flags;   //Only bit 14(Encrypted) is meaningful, but I'm not going to disclose encryption(some things are better left undocumented)
  WORD attrid;
}

Resident attributes have this format:

struct NTFS_ATTR_RES
{
  NTFS_ATTR header;
  DWORD length;   //a little redundant
  WORD offset;   //also redundant
  WORD flags;   //Bit 1 means indexed
}

And non-resident ones have this format:

struct NTFS_ATTR_NONRES
{
  NTFS_ATTR header;
  DOUBLE_DWORD lowvcn;   //Used for splitting the
  DOUBLE_DWORD highvcn;   //stream across many attributes
  WORD offset;   //a little redundant
  WORD compunitsize;   //0 for uncompressed streams
  DWORD padding;
  DOUBLE_DWORD allocsize;   //Real size rounded to cluster boundary
  DOUBLE_DWORD realsize;
  DOUBLE_DWORD initsize;   //Normally the allocated size
}

Data runs

Every non resident attribute is stored somewhere on the disk. The locations are stored in the record and are called data runs. Every run has 1 byte called a size byte. This byte specifies the size of the VCN and LCN values in the run, in bytes. Bits 0-3 specifies the length of the VCN and bits 4-7 specify the length of the LCN. If both are 0, then this is the last run. The VCN length can't be 0 unless the LCN length is 0 too. The LCN length, however, can be 0 for special reasons. The VCN and LCN follow the size byte and is interpreted as a signed quantity to add to the current values. If the LCN doesn't exist for a compression unit, the unit isn't stored. If the LCN doesn't exist for part of a compression unit, the unit is compressed.

Maybe an example is needed. Here it is:
0x21 0x10 0x89 0x34 0x11 0x07 0x24 0x01 0x19 0x00

The first byte is 0x21, meaning the VCN is 1 byte and the LCN is 2 bytes. The VCN is 0x10(second byte) and the LCN is 0x3489 (third and fourth bytes). When we add 0x10 to the running VCN (currently 0) we get 0x10 and when we add 0x3289 to the running LCN(also 0) we get 0x3489.
The second runs size byte is 0x11, so both VCN and LCN are 2 bytes. The VCN of the second run is 0x07, which when added to the running VCN(0x10) we get 0x17. Same with the running LCN(0x3489), when we add 0x24 we get 0x34AD.
The third runs size byte is 0x01, meaning the VCN is 1 byte long and there is no LCN. The VCN is 0x19, which when added to the VCN equals 0x30. This run crosses one compression unit, so we split it into 2 runs. The first run has a VCN of 0x09 and the second a VCN of 0x10. The first of the 2, along with the second run, is one compression unit, since it contains a stored area and an unstored area, it is compressed. The second of the two is 1 compression unit, so this is a sparse area, and we return all zeros if someone wants the data here.
The forth runs size byte is 0x00, so there are no more runs The final outcome is:

• Unit 1
  • LCN = 0x3489
• Unit 2 (compressed)
  • Compressed Length = 0x7 (we still save that part)
  • LCN = 0x34AD
• Unit 3 (non existent)

Standard attributes

There are 15 standard attribute types as of NTFS 5 that came with Windows 2000. Some are found in every file ($STANDARD_INFO), others in a certain type of file ($INDEX_ROOT, $DATA), some in files with encryption ($LOGGED_UTILITY_STREAM) and some only in one file ($VOLUME_NAME). In a typical file these attributes are present:

• $STANDARD_INFO
• $FILE_NAME
• $DATA
• $END

and in a typical directory these attributes are present:

• $STANDARD_INFO
• $FILE_NAME
• $INDEX_ROOT
• $INDEX_ALLOCATION
• $INDEX_BITMAP
• $END

$STANDARD_INFO (0x00000010)

This is the attribute in every file. It contains all the info FAT would typically store in a directory entry. It's format is:

struct NTFS_ATTR_STANDARD_INFO
{
  NT_TIME time[4];   //The times are in this order: Creation, Last Written, Record Changed, Last Read 
  DWORD dos;   //A superset of the dos permissions
  DWORD maxver;   //Typically 0
  DWORD version;   //Typically 0
  DWORD class;   //Never used, perhaps it will be implemented in future verions
  DWORD owner;   //The index into the $O and $Q indexes in the file $Quota
  DWORD secure;   //An index into the $SII index in the file &Secure
  DOUBLE_DWORD quotacharge;   //How much to charge the owners quota
  DOUBLE_DWORD unused;   //The last update sequence in the file $UsnJrnl
}

DOS Permissions

This is a bit mask with the following bits:

• (0x0001)Read Only(DOS)
• (0x0002)Hidden(DOS)
• (0x0004)System(DOS)
• (0x0020)Archive(DOS)
• (0x0040)Device(DOS)(?)
• (0x0080)Normal(Windows)
• (0x0100)Temp(Windows)
• (0x0200)Sparse(Windows)
• (0x0400)Reparse(Windows)
• (0x0800)Compressed(Windows)
• (0x1000)Offline(Windows)
• (0x2000)Not content indexed(Windows)(?)
• (0x4000)Encrypted(Windows)(?)

$ATTRIBUTE_LIST (0x00000020)

This attribute is used for listing all attributes in a files extension records. It is rare to see this. It is used when there are too many attributes for the record. It is ordered by type, name and VCN. The format is:

struct NTFS_ATTR_ATTRIBUTE_LIST_ENTRY {

 DWORD type;   //The attribute type
 WORD length;   //The length of the entry
 BYTE namelength;   
 BYTE nameoffset;
 DOUBLE_DWORD vcn;   //The lowest vcn exposed by this extent
 DOUBLE_DWORD file;   //The file containing it
 WORD id;   //The attribute id if vcn isn't 0

} </source>

An example on ordering might be needed: Lets say I have 2 $DATA attributes and 1 $STANDARD_INFO attribute in an extension file. The $DATA attributes are names Hi and Bye. The ordering is as follows:

• Type: $STANDARD_INFORMATION VCN: 0 Name: "" Id: 1
• Type: $DATA VCN: 0 Name: "Bye" Id: 2
• Type: $DATA VCN: 0 Name: "Hi" Id: 3
• Type: $DATA VCN: 200 Name: "Hi" Id: 0

$FILE_NAME (0x00000030)

This attribute contains the file name. It also contains other stuff. The format is as follows:

struct NTFS_ATTR_FILE_NAME
{
  DOUBLE_DWORD parent;   //The parent directory
  NT_TIME time[4];   //The times are in this order: Creation, Last Written, Record Changed, Last Read 
  DOUBLE_DWORD allocsize;   //Real size rounded     to cluster boundary
  DOUBLE_DWORD realsize;   //size of file
  DWORD dos;   //A superset of the dos permissions
  DWORD unused;
  BYTE namelength;   //Typically 0
  BYTE namespace;   //The file namespace
}

DOS Permissions

This is a bit mask with the following bits:

• (0x00000001)Read Only(DOS)
• (0x00000002)Hidden(DOS)
• (0x00000004)System(DOS)
• (0x00000020)Archive(DOS)
• (0x00000040)Device(DOS)(?)
• (0x00000080)Normal(Windows)
• (0x00000100)Temp(Windows)
• (0x00000200)Sparse(Windows)
• (0x00000400)Reparse(Windows)
• (0x00000800)Compressed(Windows)
• (0x00001000)Offline(Windows)
• (0x00002000)Not content indexed(Windows)(?)
• (0x00004000)Encrypted(Windows)(?)
• (0x10000000)Directory(Windows)
• (0x20000000)Index view(Windows)(?)

File namespace

Every file has a name space. There are 3 namespaces:

• POSIX(0): all characters except NULL and / are allowed.
• Win32(1): a subset of POSIX, also doesn't allow ", *, :, <, >, ?, \ and | and can't end with a . or a space.
• DOS(2): a subset of Win32, also doesn't allow +,',', ;, and = and names must be in 8.3 format.
• Win32 and DOS(3): This is just the DOS namespace, but it tells the compiler to not look for a Win32 name because the Win32 name is the DOS name.

$OBJECT_ID (0x00000040)

This is a W2K attribute. It contains the record GUID. It is four GUIDs in the order:

• Object ID
• Birth Volume ID(Volume it was created)
• Birth Object ID(First Object ID)
• Domain ID (Unused)

Note: Birth Volume, Birth Object and Domain may not be present.

$SECURITY_DESCRIPTOR (0x00000050)

This attribute will not be present except for special occasions. Normally and index into the $Secure file is all you need. The format is:

struct NTFS_ATTR_SECURITY_DESCRIPTOR
{
  BYTE revision;   //1 for now
  BYTE padding;
  WORD flags;
  DWORD usersid;   //The offset to the SID of the Owner
  DWORD groupsid;   //The offset to the SID of the Group
  DWORD dacl;   //The offset to the DACL
  DWORD sacl;   //The offset to the SACL
}

struct NTFS_ATTR_SECURITY_DESCRIPTOR_ACL
{
  BYTE revision;
  BYTE padding;
  WORD size;
  WORD acecount;   //Amount of ACEs
  WORD padding;
}

struct NTFS_ATTR_SECURITY_DESCRIPTOR_ACE
{
  BYTE type;
  BYTE aceflags;
  WORD size;
  DWORD access;
}

Flags

These flags are defined:

• Bit 0(Owner defaulted) The owner or group is
• Bit 1(Group defaulted) provided by a defaulting mechanism
• Bit 2(DACL present)
• Bit 3(DACL defaulted)
• Bit 4(SACL present)
• Bit 5(SACL defaulted)
• Bit 8(DACL Auto Inherit Req)(?)
• Bit 9(SACL Auto Inherit Req)(?)
• Bit 10(DACL Auto Inherited)(?)
• Bit 11(SACL Auto Inherited)(?)
• Bit 12(DACL Protected)(?)
• Bit 13(SACL Protected)(?)
• Bit 14(RM Control Valid)(?)
• Bit 15(Self relative), All offsets are relative to the offset of the header as opposed to pointers. The form used for on disk files.

Type

The type of the ACE. These types are defined:

• 0x00(Access allowed)
• 0x01(Access denied)
• 0x02(Audit)(?)

ACE flags

The flags in an ACE, of the following bits:

• Bit 0 (Inherit ACE)
• Bit 1 (Container Inherit ACE)
• Bit 2 (Don't propagate Inherit ACE)
• Bit 3 (Inherit only ACE)
• Bit 6 (Audit on success)
• Bit 7 (Audit on failure)

Access

These are a bitmask of the following bits:

• Bit 0 (Read file/directory) Read or list a file
• Bit 1 (Write file/directory) Write a file or add a file
• Bit 2 (Append file/directory) Append a file or add a subdirectory
• Bit 3 (Read EA) Read the EA
• Bit 4 (Write EA) Write the EA
• Bit 5 (Execute file) Execute a file
• Bit 6 (Delete child) Delete a child
• Bit 7 (Read attributes) Read DOS attributes
• Bit 8 (Write attributes) Write DOS attributes
• Bit 16 (Delete)
• Bit 17 (Read Control)
• Bit 18 (Write DAC)
• Bit 19 (Write Owner)
• Bit 20 (Synchronise)
• Bit 24 (Access_System_Security)
• Bit 25 (Maximum allowed)
• Bit 28 (Generic All)
• Bit 29 (Generic Execute)
• Bit 30 (Generic Write)
• Bit 31 (Generic Read)

$VOLUME_NAME (0x00000060)

Note: only found in the $Volume file.

It is just a Unicode string without a terminating NULL.

$VOLUME_INFO (0x00000070)

Note: only found in the $Volume file.

The format of this attribute is:

struct NTFS_ATTR_VOLUME_INFO
{
  DOUBLE_DWORD unused;
  WORD version;
  WORD flags;
}

Flags

Here are the flags in the flags value:

• Bit 0 (Dirty) The chkdsk /F command will be performed on reboot
• Bit 1 (Resize Logfile)(?)
• Bit 2 (Upgrade on mount)(?)
• Bit 3 (Mounted on NT4)(?)
• Bit 4 (Delete USN underway)(?)
• Bit 5 (Repair Object Ids)(?)
• Bit 15 (Modified by chkdsk)(?)

Version

The version value is major version first(Lower offset). Here are the valid version values:

• 1.0 = NT 3.1
• 1.1 = NT 3.5
• 1.2 = NT 4.0
• 3.0 = 2K
• 3.1 = XP, Vista, 7, 8

$DATA (0x00000080)

This attribute is simple, its the file data. It is paired with the $INDEX_ROOT, $INDEX_ALLOCATION and $BITMAP attributes for directories.

$INDEX_ROOT (0x00000090)

This attribute is the root of the B+Tree index. In directories it is named $I30 and is an index of files. The format is:

struct NTFS_ATTR_INDEX_ROOT
{
  DWORD type;   //0x00000030 for the $I30 index, 0 for everything else, its the indexed attribute
  DWORD collation;   //The sorting method
  DWORD sizeofalloc;   //How large an index record is(typically 4096 bytes, same as the entry in the BPB(except its in bytes, not clusters)
  DWORD clusofalloc;   //How many clusters in a record(typically 1, same as the entry in the BPB
  DWORD firstindex;   //Offset from this value to the first index entry
  DWORD totalsize;    //The size of the header from the offset of firstindex on
  DWORD allocsize;    //Unsure why this exists, probably for adding more entries with less overhead. Usually the same as totalsize
  DWORD large;   //1 if the $INDEX_ALLOCATION and $BITMAP attributes with the same name exist
}

Colation

Here are the Collation rules that exist in NTFS:

• 0x00: Binary (A variable length value with the MSB first, smallest to largest).
• 0x01: Filename (Filename attributes sorted alphabetically, in accordance to the $Upcase file, so no two files have the same name even in different case).
• 0x02: Unicode String (Uppercase letters come first).
• 0x10: DWORD (A 32 bit unsigned little-endian value, smallest to largest)
• 0x11: Security Identifier (Similar sorting to binary, except the first 2 bytes don't count)
• 0x12: Security Hash (2 DWORDs, first DWORD is the Hash and its the most significant. The second one is a Security ID(like the one in $STANDARD_INFO) and is the least significant one).
• 0x13: ULONGs (UUIDs, smallest to largest).

The Index entries are in this format:

struct NTFS_INDEX_ENTRY
{
  union form    //Invalid if last entry
  {
    DOUBLE_DWORD mftref;   //The file record number, this form is only used if index type(specified at the root) is 0x00000030.
    struct w2k    //This form is used with all new indexes in NTFS 5
    {
      WORD dataoffset;    //Offset to the data in the record
      WORD datasize;    //Length of data
      DWORD padding;
    }
    WORD length;    //Length of the entry
    WORD keylength;    //Length of the key
    DWORD flags;    //Bit 1 is the sub node flag, Bit 2 is the last entry flag
  }
}

Key

The key immediately follows this header. Its type is specified in the collation rule of the index. The length is specified in the keylength value. Filenames are a type of key.

Data

Some data is located inside the entry. It follows the key and contains the needed info for this index entry. In the $I30 index, the data that FAT stores in a directory entry is stored in the key with the filename, because back then there was no data (Only form.mftref was used). Now you know why there is so much info in the $FILENAME attribute. NOTE: When the sub-node flag is set, the last 8 bytes is the VCN of the INDX record in the corresponding $INDEX_ALLOCATION attribute(not sure whether it is included in the size of the entry data.

$INDEX_ALLOCATION (0x000000A0)

This Attribute is a list of index records. The size of them is declared in the corresponding $INDEX_ROOT attribute. The structure is as follows:

struct NTFS_FILE_2K
{
  NTFS_RECORD header;    //Common Record Header, id is 'INDX'
  DOUBLE_DWORD unused;   //The $LogFile sequence
  DOUBLE_DWORD vcn;   //The VCN of the record
  DWORD entries;   //offset to the Index Entries relative to this values offset
  DWORD entriessize;   //Size of all the Index Entries plus the entries value
  DWORD allocsize;    //Size of Index Record - offset of the entries value
  DWORD node;    //Set to 1 if this isn't a leaf node(i.e. Has children)
}

VCN

I only see one purpose of this value, validating, by comparing this value with the real vcn of the record to see if they match

$BITMAP (0x000000B0)

This is what the name implies.

This attribute is used with indexes, but it is also used with the $MFT

Standard files

In NTFS, everything is a file. Even important filesystem structures. Here is a list of files:

• Index 0: $MFT
• Index 0: $MFTMirr
• Index 0: $LogFile
• Index 0: $Volume
• Index 0: $AttrDef
• Index 0: .
• Index 0: $Bitmap
• Index 0: $Boot
• Index 0: $BadClus
• Index 0: $Secure
• Index 0: $UpCase
• Index 0: $Extend
• Indices 12-15: Reserved for more standard files
• Indices 16-23: Reserved for extensions of $MFT

Others exist in random locations, like $ObjId, $Quota, $Reparse and $UsnJrnl.