SAS SSDs with 528-byte sectors

The st1600fm0013-528b-sectors repository is a guide to getting Seagate ST1600FM0013 SSDs with IBM firmware, locked to 528-byte sectors, working in Linux. Server SSDs pulled from retired storage arrays are easy to buy second-hand. But anyone who buys a Seagate ST1600FM0013 with IBM firmware and plugs it into a Linux server finds that the system reports a zero-size disk. The drives hold 1.6 TB over SAS, but they are hard-locked to 528-byte sectors. IBM arrays use the extra 16 bytes for metadata. LVM, ext4, XFS and ZFS cannot work with such a disk. It covers reformatting, a kernel patch, what does not work and where the lock sits in the firmware. It was built on eight healthy drives in an HPE server with a Broadcom SAS3216 controller.

The usual tools all hit one error code

For NetApp, EMC, Dell or HPE drives, sg_format --size=512 is usually enough. IBM firmware 6214 rejects the sector size change, just as it rejects a faster link and flashing different firmware, each time with ASC 0x26 / ASCQ 0x99. Everything below was tried on our own drive, and the drive came through every attempt undamaged:

Method	Result
sg_format --size=512 and MODE SELECT via sg_raw	Invalid field in parameter list, only 528 is accepted
openSeaChest, sector change and FORMAT UNIT	not supported, format fails
official Seagate flasher with signed firmware	rejected after 22 segments
PSID revert	the TCG session does not respond; according to the ServeTheHome forum it does not unlock the sector size
sedutil-cli	supports only SATA and NVMe, not SAS

What the repository offers

• An answer in a fraction of a second. The MODE SELECT probe saves nothing and does not touch the media. A drive that answers Good can be reformatted; an Illegal Request answer means a lock and the kernel route.
• Reformatting without losing space. An IBM drive made by HGST accepted sg_format --size=512, recalculated its block count in 36 minutes and kept its capacity at 400.09 GB. Direct writes then ran at 392 MB/s, while user-space emulation through nbdkit gave 118 MB/s.
• A kernel patch for locked drives. With it, the drive shows up as an ordinary block device with 512-byte sectors. The repository ports it to kernels 6.8 and 7.0, and it is measured on all eight drives.
• Ready-made settings. Five kernel parameters with which the emulation does not slow anything down and throughput is limited by the drives themselves.
• An analysis of the lock and a tool to modify it. The lock lives in a configuration record, not in the firmware code, and the script tools/patch_ibm_lock.py rewrites it by changing seven bytes.
• Czech and English. The main documents have a Czech version with the _CZ suffix; only the kernel performance analysis and the ZFS pool design remain English-only.

The probe looks like this:

printf '\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x02\x00' > /tmp/ms512.bin
sg_raw -s 16 -i /tmp/ms512.bin /dev/sdX 55 10 00 00 00 00 00 00 10 00

In-kernel emulation, measured on eight drives

The patch to the sd driver comes from a third party and its origin is unknown. As published, it did not compile, it set the limits for advanced operations too early, so regular code overwrote them right away, and it adjusted the queue depth in the wrong function. On top of that, it freed the same memory twice when an allocation failed. The repository describes all four bugs and fixes them with the port_universal.py script.

Each 528-byte sector maps to one 512-byte sector and the 16 bytes of metadata are discarded. The drive then has 1.55 TB instead of 1.60 TB, which is 3 % less. Data written through the emulation was compared with a direct read of the 528-byte sector and matched byte for byte. TRIM works: after blkdiscard, reads return zeros.

The bottleneck was the emulation buffer shared by all drives. With 128 KiB requests and a larger buffer, eight drives read 4,065 MB/s and write 3,522 MB/s, and 4 KiB random reads hold around 780,000 IOPS. That is roughly 510 MB/s per drive, as much as a single drive manages on its own:

sd_mod.emulate_512_from_fat_sectors=1
sd_mod.emulate_528_queue_depth=32
sd_mod.emulate_528_max_sectors=256
sd_mod.emulate_528_pool_chunks=4608
sd_mod.emulate_528_pool_contexts=512

On top of the emulation runs ZFS as four two-drive mirrors with 5.50 TiB of usable space. It reads 4,129 MB/s and writes 1,231 MB/s sequentially. An fio test with verification over 8 GiB found no errors, the following zpool scrub repaired nothing, and after a cold reboot the pool imported itself.

Where the lock sits in the firmware

Comparing the SPI flash contents of an IBM drive with a drive running stock Seagate firmware showed 688 bytes that Seagate does not have at all. They are four configuration records added by IBM, and record 0xc8 holds the block count and sector size as plain data. The block checksum is a sum of 16-bit words that must come out as zero. The script changes the block count and sector size and recomputes the sum:

python tools/patch_ibm_lock.py dump1.bin patched.bin \
       --mode blocksize --blocks 3125627568

Capacity would stay the same: 3,125,627,568 sectors of 512 bytes is 2,688 bytes more than today's 3,030,911,576 sectors of 528. This requires desoldering the W25Q32 chip and reading it with a CH341A programmer in its 3.3 V version; the README has a step-by-step guide. The dump must come from your own drive, because the configuration block holds the serial number and calibration of that particular unit. As long as the original dump is kept, everything can be reverted.

Who it is for

• Owners of second-hand SAS SSDs with IBM firmware who want to use them in Proxmox or another Linux server.
• Anyone dealing with a controller clone sold as a 9305-16i with the SAS3216 chip, which does not take official firmware. The repository describes how to get it to version P16.12.
• Anyone studying Seagate Koho firmware: the repository has tools for analysing and comparing dumps, and links to the dumps with checksums.

If none of this helps, the README lists other options: use the drives in an IBM array, where 528 bytes is the native size, or sell them to owners of such arrays.

What remains open

Modifying the flash on a real drive has not been tested, and only an actual write will show whether the firmware checks a signature in addition to the checksum. The link stays at 6 Gb/s, because the limit is in the drive firmware; the only known route is a PC-3000 SAS device. A drive with emulation discards the metadata that IBM firmware uses to check integrity, so it should not go back into an IBM array. Proxmox has its own kernel, so the emulation has to be rebuilt there and the pool imported with OpenZFS 2.2 or newer.

Getting started

Clone the repository with git clone https://github.com/nks-hub/st1600fm0013-528b-sectors.git. The scripts run directly in Python 3 and need the sg3_utils package; the flash route also needs flashrom. Flash dumps and the official firmware package are not in the repository; the README links to where they can be downloaded and lists their SHA-256 checksums. The project has no releases or packages.

Source code

The notes and scripts are on GitHub in the nks-hub/st1600fm0013-528b-sectors repository. The repository states no licence. Third-party material, meaning the flash dumps, the Seagate firmware package and the kernel patch, has its origin noted alongside it.