File System Forensics

Preface xvii  Acknowledgements xxi  Part I Preliminaries 1  1 Introduction 3  1.1 What is Digital Forensics? 4  1.2 File System Forensics 5  1.3 Digital Forensic Principles 5  1.4 Digital Forensic Methodology 7  1.4.1 Preparation 8  1.4.2 Localisation/Preservation 8  1.4.3 Acquisition 8  1.4.4 Processing 9  1.4.5 Analysis 9  1.4.6 Reporting 9  1.4.7 Quality Assurance 10  1.4.8 Evidence Return 10  1.5 About This Book 10  1.5.1 Who Should Read This Book? 11  1.6 Book Structure 12  1.7 Summary 13  Exercises 13  Bibliography 14  2 Linux as a Forensic Platform 17  2.1 Open-Source Software 17  2.1.1 Advantages of Open-Source Software 19  2.1.2 Open Source ≠ Free 20  2.2 Open-Source Software in Digital Forensics 20  2.3 What is Linux? 21  2.3.1 The Anatomy of the Linux OS 22  2.3.2 Linux Distributions 27  2.3.3 A (very) Brief History of Linux 28  2.4 Using Linux 29  2.4.1 User Accounts 30  2.4.2 Basic Linux Commands 32  2.4.2.1 Navigating the File System 32  2.4.2.2 Getting Help 34  2.4.2.3 Viewing/Editing Text Files 34  2.4.2.4 Managing Directories 35  2.4.2.5 Redirection and Pipes 35  2.5 Linux as a Forensic Platform 36  2.5.1 Commands for Digital Forensics 36  2.5.1.1 Hashing 36  2.5.1.2 Hex Viewers 38  2.5.1.3 Archiving/Compression 39  2.5.1.4 The file Command 40  2.5.1.5 The strings Command 40  2.5.1.6 Text Searching with (e)grep 41  2.6 Summary 42  Exercises/Discussion Topics 42  Bibliography 43  3 Mathematical Preliminaries 45  3.1 Bits and Bytes 45  3.2 Number Systems 48  3.2.1 Notational Conventions 48  3.2.2 Decimal 48  3.2.3 Binary 49  3.2.4 Hexadecimal 50  3.2.5 Number Conversions 51  3.2.6 Number Conversion with Bash 51  3.2.7 Negative Numbers 53  3.2.8 Floating-Point Numbers 53  3.3 Representing Text 56  3.3.1 Ascii 56  3.3.2 Iso- 8859 57  3.3.3 Unicode 59  3.3.4 Utf- 8 60  3.3.5 Utf- 16 61  3.4 Representing Time 62  3.4.1 Unix Time 63  3.4.2 The Linux date Command 64  3.5 Endianness and Raw Data 64  3.6 Summary 66  Exercises 67  Bibliography 68  4 Disks, Partitions and File Systems 69  4.1 Disk Storage 70  4.1.1 Traditional Rotational Hard Drives 71  4.1.1.1 Optical Media 72  4.1.2 Flash Drives 73  4.1.3 Solid-State Drives 73  4.2 Partitions 74  4.2.1 Creating Partitions/File Systems on Linux 74  4.2.1.1 Mounting File Systems on Linux 77  4.2.2 Master Boot Record 78  4.2.3 GUID Partition Table 80  4.3 File Systems 83  4.3.1 File System Concepts 83  4.3.2 Comparison of File Systems 86  4.4 Acquisition of File System Data 88  4.4.1 Logical vs Physical Acquisition 88  4.4.2 Acquisition Under Linux 88  4.4.2.1 The dd Family 89  4.4.2.2 Expert Witness Format (EWF) 90  4.4.2.3 guymager 91  4.5 Analysis of File Systems 92  4.5.1 The Sleuth Kit 92  4.5.1.1 Determine the Partition Layout 93  4.5.1.2 Determine the File System Type 93  4.5.1.3 List the Files 94  4.5.1.4 Recover File Metadata 95  4.5.1.5 Recover File Content 95  4.5.1.6 Other TSK Commands 95  4.5.2 Data Carving 96  4.6 Summary 97  Exercises 97  Bibliography 98  Part II Windows File Systems 99  5 The FAT File System 101  5.1 On-Disk Structures 101  5.1.1 Layout 102  5.1.2 Volume Boot Record 102  5.1.3 File System Information (FSINFO) 102  5.1.4 File Allocation Table 104  5.1.5 Directory Entries 105  5.1.6 FAT Date and Time 108  5.1.7 Mapping Clusters to Sectors 109  5.2 Analysis of FAT 32 109  5.2.1 Creating FAT32 File Systems 109  5.2.2 Supplied FAT32 Image Files 110  5.2.3 FAT32 Manual Analysis 110  5.2.3.1 Process the VBR 111  5.2.3.2 Process the Root Directory 112  5.2.3.3 Process Sub-directories 113  5.2.3.4 Recover Metadata/Content 113  5.3 FAT32 Advanced Analysis 115  5.3.1 Deleted Files 116  5.3.2 The Volume Label 117  5.4 Summary 117  Exercises 118  Bibliography 118  6 The ExFAT File System 121  6.1 On-Disk Structures 121  6.1.1 Volume Boot Record 122  6.1.2 File Allocation Table 123  6.1.3 Directory Entries 125  6.1.3.1 Allocation Bitmap (Type: 0x81) 127  6.1.3.2 Up-Case Table (Type: 0x82) 128  6.1.3.3 Volume Label (Type: 0x83) 128  6.1.3.4 File (Type: 0x85) 129  6.1.3.5 Volume GUID (Type: 0xA0) 130  6.1.3.6 Stream Extension (Type: 0xC0) 130  6.1.3.7 Filename Extension 131  6.1.3.8 Other Directory Entries 132  6.2 Analysis of ExFAT 132  6.2.1 Creating ExFAT File Systems 132  6.2.2 Supplied ExFAT Image Files 132  6.2.3 ExFAT Manual Analysis 132  6.2.3.1 Step 1: Process the VBR 133  6.2.3.2 Step 2: Process the Root Directory 133  6.2.3.3 Step 3: Process Subdirectories 136  6.2.3.4 Step 4: Recover Metadata 137  6.2.3.5 Step 5: Recover Content 137  6.3 ExFAT Advanced Analysis 139  6.3.1 Long File Names 139  6.3.2 Deleted Files 140  6.3.3 Fragmented Files and Large Directories 141  6.4 Summary 142  Exercises 143  Bibliography 143  7 The NTFS File System 145  7.1 On-Disk Structures 146  7.1.1 $Boot 146  7.1.2 Indexes 147  7.1.3 Fixup Arrays 149  7.1.4 Time in NTFS 150  7.1.5 Master File Table 151  7.1.6 MFT Record Structure 152  7.1.6.1 MFT Record Header 152  7.1.6.2 Browsing Attributes 155  7.1.6.3 $STANDARD_INFORMATION (0x10) 155  7.1.6.4 $ATTRIBUTE_LIST (0x20) 156  7.1.6.5 $FILENAME (0x30) 156  7.1.6.6 $OBJECT_ID (0x40) 157  7.1.6.7 $SECURITY_DESCRIPTOR (0x50) 159  7.1.6.8 $VOLUME_NAME (0x60) 162  7.1.6.9 $VOLUME_INFORMATION (0x70) 162  7.1.6.10 $DATA (0x80) 163  7.1.6.11 $INDEX_ROOT (0x90) 163  7.1.6.12 $INDEX_ALLOCATION (0xA0) 165  7.1.6.13 $BITMAP (0xB0) 165  7.1.6.14 $REPARSE_POINT (0xC0) 166  7.1.6.15 $EA_INFORMATION (0xD0) and $EA (0xE0) 167  7.2 Analysis of NTFS 167  7.2.1 Creating NTFS File Systems 168  7.2.2 Supplied NTFS Image Files 168  7.2.3 NTFS Manual Analysis 168  7.2.3.1 Process $Boot 169  7.2.3.2 Recover $MFT 171  7.2.3.3 Process Directories 173  7.2.3.4 Recover File Metadata 177  7.2.3.5 Recover File Content 182  7.3 NTFS Advanced Analysis 185  7.3.1 Further File System Information 185  7.3.2 Deleted Files 186  7.3.3 Fragmented Files 187  7.3.4 Alternate Data Streams 190  7.3.5 Large MFT Records 191  7.4 Summary 194  Exercises 194  Bibliography 195  Part III Linux File Systems 197  8 The EXT2 File System 199  8.1 On-Disk Structures 200  8.1.1 The Superblock 201  8.1.2 The Block Group Descriptor Table 204  8.1.3 The Inode Table 205  8.1.3.1 Mode/Permissions 207  8.1.3.2 Inode Flags 208  8.1.3.3 Block Pointers 208  8.1.4 The Data and Inode Bitmaps 209  8.1.5 Locating an Inode 209  8.2 Analysis of ext 2 210  8.2.1 Creating ext2 File Systems 210  8.2.2 Supplied ext2 Image Files 210  8.2.3 Ext2 Manual Analysis 211  8.2.3.1 Process the Superblock 211  8.2.3.2 Map the Block Groups 213  8.2.3.3 Process Root Directory Inode 216  8.2.3.4 Process the Root Directory 217  8.2.3.5 Process Directories 219  8.2.3.6 Process Files 219  8.3 Ext2 Advanced Analysis 222  8.3.1 Fragmented Files 222  8.3.2 Links 223  8.3.3 Deleted Files 225  8.4 Summary 226  Exercises 226  Bibliography 227  9 The EXT3/EXT4 File Systems 229  9.1 Supplied Image Files 229  9.2 The ext3 File System 229  9.2.1 The Ext Journal 230  9.2.2 HTree Directory Indexing 237  9.3 The Ext4 File System 241  9.3.1 Large Inodes 241  9.3.1.1 Timestamps 241  9.3.2 Ext4 Data Storage 244  9.3.2.1 Extent-Based Storage 244  9.3.2.2 Inline Storage 248  9.3.2.3 Symbolic Links 248  9.3.3 File Deletion in Ext 4 249  9.3.4 Extended Attributes 252  9.3.5 Ext4 Block Group Descriptors 255  9.3.6 Flexible Block Groups 255  9.4 Summary 258  Exercises 259  Bibliography 260  10 The XFS File System 263  10.1 On-Disk Structures 264  10.1.1 Allocation Groups 264  10.1.2 Addressing 266  10.1.2.1 Inode Addressing 266  10.1.3 XFS B+ Trees 267  10.1.4 The Superblock 268  10.1.4.1 Locating Superblocks 268  10.1.5 XFS Signatures 271  10.1.6 XFS Inodes 271  10.1.7 Directories 273  10.1.8 Extents 274  10.1.9 Time in XFS 275  10.2 Analysis of XFS 275  10.2.1 Creating XFS File Systems 275  10.2.2 Supplied XFS Image Files 275  10.2.3 XFS Manual Analysis 276  10.2.3.1 Process the Superblock 276  10.2.3.2 Locate the Root Directory 277  10.2.3.3 Process the Root Directory 279  10.2.3.4 Process the Subdirectories 281  10.2.3.5 Recover File Content/Metadata 281  10.3 XFS Advanced Analysis 282  10.3.1 AG Free Space Management 283  10.3.1.1 AG Free List 285  10.3.2 AG Inode Management 286  10.3.3 Deleted Files 289  10.3.4 Extended Attributes 290  10.3.5 Links 291  10.3.6 The XFS Journal 292  10.4 Summary 300  Exercises 301  Bibliography 301  11 The Btrfs File System 303  11.1 On-Disk Structures 304  11.1.1 The Superblock 305  11.1.2 Btrfs Trees 305  11.1.3 Btrfs Tree Structure 307  11.1.3.1 Node Header Structure 307  11.1.3.2 Internal Node Structure 309  11.1.4 Btrfs Keys 309  11.1.5 Btrfs Items 310  11.1.6 Time in Btrfs 315  11.1.7 Logical and Physical Addressing 315  11.2 Analysis of Btrfs 317  11.2.1 Creating Btrfs File Systems 317  11.2.2 Supplied Btrfs Image Files 318  11.2.3 Btrfs Analysis Methodology 318  11.2.4 Manual Analysis of a Single Device File System 320  11.2.4.1 Process the Superblock 320  11.2.4.2 Process the CHUNK_ARRAY 321  11.2.4.3 Locate the CHUNK_TREE 322  11.2.4.4 Process the CHUNK_TREE 323  11.2.4.5 Locate the Root Tree 326  11.2.4.6 Locate the FS_TREE 327  11.2.4.7 Processing the FS_TREE 328  11.2.4.8 Process Directories 329  11.2.4.9 Recovering Metadata 335  11.2.4.10 Recovering File Contents 336  11.3 Btrfs Advanced Analysis 338  11.3.1 File Deletion 338  11.3.2 Analysis of Internal Nodes 342  11.3.3 Multiple Device Configuration 343  11.3.4 Subvolumes and Snapshots 346  11.4 Summary 350  Exercises 350  Bibliography 351  Part IV Apple File Systems 353  12 The HFS+ File System 355  12.1 On-Disk Structures 355  12.1.1 Forks 357  12.1.2 Time in HFS+ 357  12.1.3 Volume Header 358  12.1.4 B-Trees 358  12.1.5 Catalog File 362  12.1.6 HFS+ Permissions 363  12.1.7 Text Encoding 365  12.1.8 Extents Overflow File 365  12.1.9 Allocation File 366  12.1.10 HFS+ Journal 367  12.2 Analysis of HFS+ 369  12.2.1 Creating HFS+ File Systems 369  12.2.2 Supplied HFS+ Image Files 370  12.2.3 HFS+ Manual Analysis 370  12.2.3.1 Process the Volume Header 370  12.2.3.2 Locate the Catalog File 371  12.2.3.3 Process the Catalog B-Tree 373  12.2.3.4 Gather Metadata 377  12.2.3.5 Recover File Content 377  12.3 HFS+ Advanced Analysis 380  12.3.1 Deleted Files 380  12.3.2 Index Nodes 381  12.3.3 Fragmented Files 383  12.3.4 Links 387  12.4 Summary 390  Exercises 391  Bibliography 391  13 The APFS File System 393  13.1 On-Disk Structures 394  13.1.1 Time in APFS 394  13.1.2 Objects 394  13.1.3 B-Trees 396  13.1.4 Containers and Volumes 399  13.1.5 Container Superblock 400  13.1.6 Volume Superblock 402  13.1.7 Object Maps 404  13.1.8 File-Related Structures 405  13.1.8.1 File System Keys 406  13.1.8.2 Inode 407  13.1.8.3 Directory Record 408  13.1.8.4 Extent 410  13.1.9 Checkpoints 410  13.1.10 Other APFS Structures 412  13.2 Analysis of APFS 412  13.2.1 Creating APFS File Systems 412  13.2.2 Supplied APFS Image Files 413  13.2.3 APFS Manual Analysis 413  13.2.3.1 Process the Container Superblock 414  13.2.3.2 Process the Container Object Map 415  13.2.3.3 Process the Volume Superblock 418  13.2.3.4 Process the Volume Object Map 418  13.2.3.5 Process the File System Tree 419  13.3 APFS Advanced Analysis 425  13.3.1 Deleted Files 425  13.3.2 Checkpoint Recovery 426  13.3.3 Multi-Level B-Trees 427  13.3.4 Multiple Volumes 429  13.3.5 Extended Attributes 430  13.3.6 Links 431  13.4 Summary 433  Exercises 433  Bibliography 434  Part V The Future 435  14 Future Challenges in Digital Forensics 437  14.1 Challenges in Digital Forensics 437  14.1.1 Data Volume 438  14.1.2 Multi-Source Correlation 439  14.1.3 New File Systems 440  14.1.4 Encryption 440  14.1.5 Cloud Storage 441  14.1.6 Lack of Resources 441  14.1.6.1 Human Resources 441  14.1.6.2 Software Resources 442  14.1.6.3 Hardware Resources 442  14.1.7 Tool Validation/Datasets 443  14.1.8 Lack of Standardisation 444  14.1.9 Legal/Scientific Challenges 444  14.1.10 Presentation of Evidence 445  14.1.11 Human Error/Bias 446  14.2 Where Do We Go from Here? 447  14.2.1 Training/Education 448  14.2.2 Free Open-Source Software (FOSS) 448  14.2.3 Triage 449  14.2.4 Artificial Intelligence (AI) 449  14.2.5 Live Data Forensics 450  14.2.6 Legal Solutions 451  14.2.7 Data Set Development/Tool Testing 452  14.2.8 Standardisation 452  14.2.9 Information Sharing 453  14.2.10 Virtualisation 453  14.3 Summary 454  Bibliography 454  Index 457