Advanced Format Hard Drive

Once, while using a Windows 7 hard drive for work, it started making a grinding noise as if it was about to fail. To be safe, I bought a new hard drive and used Norton GHOST to copy the data from the old drive to the new one. It should have been a simple task, but it turned out to be the beginning of a nightmare.

The new hard drive was running slower than a tortoise. How could a new hard drive be as slow as the Stone Age? The sluggish PC forced me to work overtime every day just to get the job done. On top of that, I spent several days investigating the cause of the slow hard drive, depriving myself of proper sleep.

After several days of investigation, I discovered that most of the hard drives currently available are "Advanced Format Hard Drives" (AF). If the partitions are not aligned with the physical sectors of the advanced format hard drive, it severely affects the performance of the hard drive. Windows encounters partition alignment issues with advanced format hard drives, and the same goes for Linux.

"Advanced Format Hard Drives" can increase the capacity of the hard drive by approximately 10% without increasing hardware costs, thanks to the same disk density. Some hard drive manufacturers began experimentally introducing advanced format hard drives in 2011. Since 2014, most hard drives available for purchase should be advanced format hard drives (as for SSDs, based on my testing of several SSDs I have, some are Advanced Format, some are not, so whether an SSD is Advanced Format doesn't seem to have much impact on cost).

Although advanced format hard drives have capacity advantages, improper partitioning without understanding the implications can significantly affect the speed of the hard drive. Before understanding advanced format hard drives, it's essential to understand traditional format hard drives.

In a traditional format hard drive, a sector consists of approximately 512 bytes plus around 50 bytes of error correcting code (ECC) as shown in the diagram below.

Pay attention to traditional format hard drives, and you'll notice that approximately 8% of a 512-byte sector is wasted on ECC. This led some people to think about increasing the sector size while still making the increase in ECC limited to better utilize disk space effectively.

In advanced format hard drives, the sector size is no longer limited to 512 bytes. Instead, advanced formats define 1K/2K and 4K sector sizes. However, the three major mainstream file systems, such as Linux's ext3/ext4, Windows' NTFS, and Apple's HFS Plus, default to using 4K for a cluster or block. Therefore, if a sector is also 4K, it can fill up completely and be more efficient. As a result, advanced format hard drives have evolved to use only 4K sectors.

Taking a 4K (4096-byte) sector advanced format hard drive as an example, the diagram below illustrates that a sector consists of 4K plus an additional 100 bytes of ECC. Just by looking at it, you can see that it can effectively utilize space better than a traditional 512-byte sector hard drive.

• 512e Emulation of Traditional Format Hard Drive
  Due to the traditional standard of having a 512-byte sector in hard drives, these types of traditional 512-byte sector hard drives are called "512n" (512-byte native). Therefore, BIOS, operating systems, and utility programs have always "assumed" that a hard drive has a 512-byte sector and have designed accordingly. As a result, the direct introduction of advanced format hard drives would inevitably cause compatibility issues.

  A workaround method is to appear as if the hard drive has 512-byte sectors (logical sectors) while internally using 4K sectors (physical sectors) in advanced format hard drives to overcome compatibility issues. This "512-byte skin, 4K bone" technology of advanced format hard drives is called "512e" or "512 emulation," which means emulating a 512-byte/sector hard drive.

  Most advanced format hard drives purchased after 2014 are 512e hard drives to avoid compatibility issues.

  Because the operating system (OS) and filesystem treat AF 512e as a traditional hard drive, consider the following diagram as an example: If the OS/filesystem reads only logical block #5, the actual AF 512e hard drive will read the entire 4K physical sector but only retrieve the corresponding part of logical block #5, discarding the rest

  
  
  
  

  The written part is a bit more complex, and poorly planned filesystem allocation can impact performance, especially when it comes to file modifications. The process of modifying a file involves a read-modify-write action. Using the example of modifying only logical block #5, the read operation remains the same as mentioned earlier. However, during the write operation, only logical block #5 is modified, while the rest (logical blocks #0/1/2/3/4/6/7) remain unchanged. Then, the entire 4K physical sector is written back. This additional computation can slightly affect the speed of a 512e hard drive. However, this issue can be resolved by aligning the blocks to multiples of 4K and aligning the partitions accordingly.

  By setting the block size to a multiple of 4K and ensuring proper partition alignment, this problem can be addressed.

  
  

  ^ back on top ^

• 4Kn Native 4K Sector Hard Drive
  "4Kn" is an abbreviation for 4K native, which refers to native 4K sector advanced format hard drives. They no longer need to emulate the traditional 512-byte sector like 512e hard drives.

  However, native 4Kn hard drives may encounter compatibility issues with many older operating systems (OS) and BIOS, particularly during the booting process [Note 1.0]. As OS and motherboard firmware are updated over time, these compatibility issues should gradually be resolved. Eventually, the transitional period with 512e hard drives will no longer be necessary, and native 4Kn hard drives are expected to become more widespread.

  
  
  Last updated : July 12 2021
  To avoid compatibility issues, some hard drive manufacturers offer product options with 512n, 512e, or 4Kn formats. The diagram below illustrates a product catalog from a specific hard drive manufacturer.
  
  
  <圖片來源:Toshiba>
  
  
  
• Partition Alignment for Advanced Format Hard Drive
  Due to the prevalence of AF 512e hard drives in the market, the performance degradation caused by misaligned partitions only occurs in this type of hard drive. The culprit is the 512e format, which emulates the advanced format hard drive with 4K physical sectors as a traditional 512-byte/sector hard drive for compatibility reasons. Fortunately, this necessary compromise in performance for capacity can be resolved by adjusting the starting point of the partitions.

  Returning to the nightmare scenario, why did my Windows 7 system, originally running on a traditional hard drive with 512-byte sectors, become slow when I used Ghost to copy it to the advanced format hard drive? The reason is that if the starting point of the partition does not align with the starting point of the 4K physical sectors of the advanced format hard drive, it severely affects the speed of the hard drive. In the case of an SSD, it can even impact its lifespan.

  Let's assume that the Windows NTFS filesystem has a block size of 4K, which seems to perfectly match the 4K sector of the advanced format hard drive. However, as shown in the diagram below, if this 4K block falls between two physical sectors, an unaligned condition occurs

  
  
  In this scenario, reading requires accessing two physical sectors to retrieve the desired data, and modifying data involves reading and writing two physical sectors after the initial read. This is just for reading or writing a single block. In real-world applications, files are composed of numerous blocks, resulting in a significant impact on hard drive speed. Slower performance is one concern, but the worst-case scenario occurs if this AF 512e hard drive is an SSD. Since NAND flash memory has a limited lifespan in terms of write cycles, it can lead to premature failure of the SSD.

  The diagram below illustrates the performance comparison of different filesystems when performing write operations on an unaligned partition of an AF hard drive. In all cases, the speed of the hard drive is affected to varying degrees.

  
  
  
  1.00 means no penalty; higher values mean worse performance
  
  (Source: IBM :http://www.ibm.com/developerworks/library/l-4kb-sector-disks/ )
  
  If you plan the starting point of each partition to be a multiple of 8 (4K/512 bytes = 8) and then offset it by the alignment_offset provided by the hard drive, you can achieve partition alignment. As shown in the diagram below, this will prevent the situation where every read/write operation of a 4K data requires accessing two physical sectors.
  
  
  
• Identifying Advanced Format Hard Drive
  How do I know if my hard drive is an advanced format hard drive? The simplest method is to check the specifications on the manufacturer's official website. Alternatively, you can look at the label on the hard drive itself. If it has the following symbol, as shown in the diagram below, it indicates that it is an advanced format hard drive (although not all advanced format hard drives may have this specific label).
  
  

  Advanced Format (512 e)	Advanced Format (4K native )

  
  If you prefer not to dismantle your PC to physically examine the hard drive, you can use the parted DEVICE print command in Linux with a kernel version greater than v2.6.31 to view the details of the hard drive.
  
  Example:

  # parted /dev/sdb print Model:ATA TOSHIBA DT01ACA0 (scsi) Disk /dev/sdb: 976773168s Sector size (logical/physical): 512B/4096B ←This line can identify whether it is an advanced format hard disk

  
  In the example mentioned earlier, you can easily identify whether it is an Advanced Format hard drive based on the "logical/physical" output. Here are possible scenarios:
  
  

  logical/physical: 512B/512B	Traditional 512n (512B sector) hard drive
  logical/physical: 512B/4096B	Advanced format 512e hard drive
  logical/physica: 4096B/4096B	Advanced format 4Kn hard drive

  
  Special attention should be given to the Linux Kernel version, which should be v2.6.31 or above, and the version of the parted utility, which should be 2.1 or highe
  
  
  

^ back on top ^

Linux parted provides better support for partitioning advanced formatted hard drives compared to fdisk. However, Linux Kernel versions older than v2.6.31 cannot recognize advanced formatted hard drives. If the Linux Kernel version is 2.6.31 or newer and the parted version is 2.1 or newer, it can correctly identify advanced formatted hard drives and automatically align partitions.

Windows XP and earlier versions, as well as older Linux distributions with a kernel version below 2.6.31, cannot automatically align partitions with advanced formatted hard drives. If users are using these outdated operating systems and experiencing slow disk performance, they should thoroughly check their partition alignent.

The solution is to use a Linux version that supports advanced formatted hard drives and utilize the automatic alignment feature in parted, or manually align partitions using parted. Alternatively, users can use Windows Vista or a later version to partition the drive and then install the older operating systems.

• Automatic Alignment Partitioning with parted
  Linux Kernel (v2.6.31 and above) and parted (v2.1) do support advanced formatted hard drives and automatically align partitions. They also provide warnings if partitions are not properly aligned.

  Here is an example using CentOS 6.6, which fully supports advanced formatted hard drives.

  In the following example, when planning a partition, if it detects that the Physical Sector of the AF hard drive is not aligned, a warning will be issued.
  Example:

  # parted /dev/sdb mkpart p1 777s 100000s Warning: The resulting partition is not properly aligned for best performance. Ignor/Cancel?

  
  parted, which supports AF (Advanced Format) hard drives, automatically aligns partitions as long as you specify the capacity or percentage for partitioning.
  
  Example:

  # parted /dev/sdb mklabel gpt ←Set GPT partition table # parted /dev/sdb mkpart p1 0% 100G ← Create the first partition with a size of 100G # parted /dev/sdb mkpart p2 100G 300G ←Create the second partition with a size of 200G (from 100G to 300G) # parted /dev/sdb mkpart p3 300G 100% ←Create the third partition from 300G to the end # parted /dev/sdb unit s print ←Check the starting sector Model: ATA TOSHIBA DT01ACA0 (scsi) Disk /dev/sdb: 976773168s Sector size (logical/physical): 512B/4096B ←Can correctly recognize the advanced formatted hard drive Partition Table: gpt Number  Start       End         Size        Type  File system  Name Flags 1       2048s       195311615s  195309568s                     p1 2       195311616s  585936895s  390625280s                     p2 3       585936896s  976773119s  390836224s                     p3

  
  Here is an example of using percentages to partition the drive:
  
  Example:

  # parted /dev/sdb mklabel loop ←Clear all partitions # parted /dev/sdb mklabel gpt ←Set GPT partition table # parted /dev/sdb mkpart p1 0% 30% ←Create the first partition with 30% of the entire drive # parted /dev/sdb mkpart p2 30% 100% ←Create the second partition with the remaining 70% of the drive

  
  

  If you would like to further understand the rules/principles of alignment in parted, you can refer to the additional explanation below (you can skip this content if you don't want to spend time understanding the principles).

  Linux Kernel (v2.6.31 and above) relies on the information provided by the disk to guide partition alignment in parted, RAID, or LVM. This information is stored in the directories "/sys/block/<disk>/queue/" and "/sys/block/<disk>/".

  The main files in this directory include:
  

  • alignment_offset:
    The required alignment offset value for the hard drive (in bytes).
    (File is located at “/sys/block/<disk>/alignment_offset”)
    
    
  • logical_block_size
    For hard drives other than 4Kn, it is typically 512 bytes, while for 4Kn drives, it is 4096 bytes.
    (File is located at “/sys/block/<disk>/queue/logical_block_size”)
    
    
  • physical_block_size
    For traditional hard drives, it is generally 512 bytes, while for Advanced Format drives, it is mostly 4096 bytes.
    (File is located at “/sys/block/<disk>/queue/physicall_block_size”)
    
    
  • minimum_io_size
    The smallest file size that does not affect the performance of the hard drive. Usually, it is the size of a physical block. For optimal efficiency, the filesystem block size, LVM stripe size, or RAID chunk size should be a multiple of this size.
    (File is located at “/sys/block/<disk>/queue/minimum_io_size”)
    
    
  • optimal_io_size
    The optimal size for sequential I/O (large files). If the content is 0, it means that the hard drive does not provide this parameter (which is the case for most drives).
    (File is located at “/sys/block/<disk>/queue/optimal_io_size”)
    
  例:

  # cat /sys/block/sdb/alignment_offset ←Query the alignment offset value of the hard disk “/dev/sdb”. 0 # cat /sys/block/sdb/queue/minimum_io_size ←Check the minimum file size that does not affect the performance of the hard disk 4096

  
  In general, using parted -a minimal DEVICE is more suitable for hard disks with more files smaller than 4k or less space wasted (such as SSDs for hard disk space), while parted -a optimal DEVICE is more suitable for hard disks with more large files (this is the default value). If you have no idea, it is recommended to use “optimal” because it is similar to the way Windows plans AF hard disks but will waste more hard disk space (at this time, the first partition starts from sector 2048 and will waste about 1MiB of space. Microsoft calls it “1MiB alignment boundary”).
  
  
  例:

  # parted -a optimal /dev/sdb mkpart p1 0% 40% ←Reference “optimal_io_size” to align

  
  

  You can use parted DEVICE align-check [optimal|minimal] PARTITION # to check whether the planned partition is aligned with the optimal_io_size or minimum_io_size provided by the hard disk.

  Example:

  # parted /dev/sdb align-check optimal 2 ←Check whether partition 2 is aligned with “optimal”

  
  Because not every hard disk will provide hard disk information (or only part of it) for parted to use as a reference for alignment, sometimes you may get a “The resulting partition is not properly aligned for best performance” warning no matter how you use it. In this case or if you don’t want to use parted’s black box operation, you can manually align it! [Note 1.1]
  
  

  ^ back on top ^

• Manual Alignment Partitioning with parted
  Manually aligning partitions means using manual calculations to align the partition with the 4k physical sector of the advanced formatted hard disk. I usually convert all units to sectors.

  There are many articles on the Internet that say “the starting sector of the partition only needs to be a multiple of 8 to align with the advanced formatted hard disk…” This is incorrect! Because it doesn’t consider alignment_offset. (Although most advanced formatted hard disks have an alignment_offset = 0, it is not absolute)

  Since 8 sectors of a traditional hard disk are equivalent to one sector of an advanced formatted hard disk, manually setting the starting sector of the partition should be a multiple of 8 but also add alignment_offset/512 (dividing by 512 is to convert bytes to sectors). At this point, the partition is basically aligned with the advanced formatted hard disk.

  The steps are as follows: (Kernel version must be v2.6.31 or above and parted version must be v2.1 or above to execute correctly)

  
  
  
  
  
  1. The first step is to ask for the “alignment_offset” of the hard disk.
     
     Example:

     # cat /sys/block/sdb/alignment_offset ←Query the alignment offset value of hard disk 3584

     
     In this example, my 2.5" Seagate “ST9640322AS” AF hard disk has an alignment_offset = 3584 (byte), which is 7 sectors (3584/512=7).
     
     
  2. he second step is to start partitioning from 2048 sectors and make sure it is a multiple of 8 and then + alignment_offset/512 (dividing by 512 is to convert bytes to sectors).
     

     Why start partitioning from 2048? In addition to being divisible by 8, another reason is that since Windows Vista, if an advanced formatted hard disk is encountered, the first sector of the first partition starts from 2048. Microsoft calls it “1MiB alignment boundary” (because 2048 sectors x 512B = 1MiB). Microsoft has explained this purpose [Note 1.1a], and major hard disk manufacturers know that Microsoft has this “quirk” and will optimize their own hard disks so that the first partition starts from 2048 sectors + aligment_offset/512 to ensure the best efficiency, including SSDs. So just follow suit!

     In addition, if it’s MBR, Microsoft will secretly reserve 1MiB of space at the end of the hard disk and put its own Dynamic Disk Metadata in it (dynamic disk metadata), so if you plan partitions for Windows use, remember to reserve this space. Microsoft’s partition planning recommendations are shown in the figure below.

     
     
     
     
     例:

     # parted /dev/sdb unit s print free ←List how many sectors the hard disk has (to know the hard disk capacity) Model:ATA ST9640322A (scsi) Disk /dev/sdB: 1250263728S ←This hard disk capacity is 1250263728(sector) x 512(Byte)=640MB (The following is omitted) # parted /dev/sdb mklabel loop ←Clear all partitions of hard disk “/dev/sdb” (for re-partitioning) # parted /dev/sdb mklabel gpt ←Set the hard disk partition table to “gpt” # parted /dev/sdb mkpart p1 2055s 838860793s ←Partition the first partition with a starting sector of 2055 and a size of 400MB [(838860793s-2055s) x 512b] # parted /dev/sdb mkpart p2 838862855s 100% ←Partition another partition with a starting sector of 838862855 to the end

     
     

     The starting sector of the first partition in the above example also starts from 2048, and 2048 plus aligment_offset /512 =2055.

     For the second partition, find an unused sector that is a multiple of 8 + aligment_offset/512 (converted to sector) as the starting point. The principle is the same for further partitioning.

     Finally, enter 100% directly at the end position of the last partition. Parted will automatically reserve space for Backup GPT if it is planned as GPT. If it is planned as MBR and is to be used by Windows, do not enter 100% and manually calculate that at least 2048 sectors (1MiB) of unused space must be left at the end of this partition for Dynamic Metadata.

     
     
     
  3. Finally, you can use print with different units to verify the partitioning situation just now.
     
     例:

     # parted /dev/sdb unit GB print ←Use GB as the unit to verify whether the partition capacity is correct # parted /dev/sdb unit s print ←Use sector as the unit to check whether it is aligned (the starting sector must be a multiple of eight + aligment_offset /512 # parted /dev/sdb unit s print free ←Check if there is any space that has not been partitioned

     
     
  4. If you are not sure, use align-check [optimal|minimal] to check whether the partition is aligned with the AF hard disk.
     
     Example:

     # parted /dev/sdb align-check optimal 1 ←Check whether partition 1 is aligned with the AF hard disk 1 aligned # parted /dev/sdb align-check optimal 2 ←Check whether partition 2 is aligned with the AF hard disk 2 aligned

     
     
     
• Partitioning with Windows
  For Windows Vista and later versions, if you encounter advanced formatted hard disks during partitioning, the partition will automatically align with the Physical Sector of the advanced formatted hard disk. If the Linux kernel version is too old or Windows XP users can use Windows Vista or later versions of Windows to partition and then use it on Linux/Windows XP.

  Although Windows Vista and later versions support advanced formatted hard disks and the partition will automatically align with the 4k physical sector of the advanced formatted hard disk during partitioning, some tool software may not (such as GHOST), so never use “ghost software” to copy partitions from traditional hard disks to advanced formatted hard disks because the partition may not be aligned with the 4k physical sector, and the hard disk will be as slow as a snail. If it is an SSD, it will be even worse! It will also accelerate the wear and tear of NAND-Flash life (commercial software “Acronis True Image” supports copying partitions while aligning with the 4k physical sector of advanced formatted hard disks).

  
  

^ back on top ^