Full format on new drives?

[Julian]

Do you experts tend to do full or quick formatting on a new drive?

I'm rejigging a dual-drive USB enclosure with 2 x 2TB drives in it and decided I might as well do a full format. Then I realised how slowly it was going, probably not helped by the fact that it's actually going through 2 USB hubs to get to the enclosure.

If I'd realised it would be this slow I'm not sure I would have bothered, or I would at least have put it onto a USB3 port connected directly to my PC, but I suppose it's not bad practice to have explicitly written and done a read-back verify on every single sector of the drives which I believe is what it's doing. It would probably also flush out any obviously dodgy electronics as far as thermals are concerned since it's getting warm (but not by any means worryingly hot).

I've started now so I'm not stopping it but it looks as if it's going to be well into tomorrow before it's finished, maybe even into Monday. At least I know the disk activity light is working!

- Julian

[Slarti]

I've always deleted any existing partitions on new drives and created the one(s) that I want and then done a full format.

That is because, a long time ago and far, far away, A customer took a delivery of big name hard drives with malware on them, which caused them no end of problems until the cleaned the drives.

I may be paranoid, but ...


Slarti

[Julian]

Thanks Slarti. I don't think you can be too paranoid on stuff like this. I also consider this a case where even me, frankly a lazy git most of the time, is willing to expend what others might consider an excessive amount of time and/or effort(*) if it is for something that I will only need to do once at the outset but will reap some rewards from, even if those rewards are relatively minor, for a considerable time afterwards. I'm hoping this little portable NAS setup will last me for many many years. I've certainly sized the disks way beyond the amount of data I expect to need to put on them (and with more and more things like video storage moving to the cloud for once the onward march of storage capacity might not make my 2018 storage calculations look laughable in a few years time) so taking the time to set it up as well as I can at the outset seems appropriate.

On the malware-on-big-name-drives thing wasn't there a scandal, also years and years back, where there was concern that malware had somehow found its way into the manufacturing supply chain and was actually in the firmware of the onboard controllers! Stuff like that, malware right in the flash of onboard controllers, is really scary especially if it has slipped past a manufacturer's QA process and is in their official release so that even downloading from their site and reflashing the firmware wouldn't help.

- Julian

(*) In this case just time since I can ignore it as it sits formatting away on my desk. The format of the first drive finished overnight and I just started the format on the second one. My guess is that it's going to complete just in time for the Countryfile weekly weather forecast.

[Slarti]

On the malware-on-big-name-drives thing wasn't there a scandal, also years and years back, where there was concern that malware had somehow found its way into the manufacturing supply chain and was actually in the firmware of the onboard controllers! Stuff like that, malware right in the flash of onboard controllers, is really scary especially if it has slipped past a manufacturer's QA process and is in their official release so that even downloading from their site and reflashing the firmware wouldn't help.

I remember that Dell shipped some computers with pre-installed malware, a lot of years back. As did Gateway and somebody else. But that was in software, somehow and it didn't affect me.

I don't recall anything about problems with onboard controllers, but it is so easy to miss these things in the news.

Slarti

[ReformedCharacter]

Do you experts tend to do full or quick formatting on a new drive?

I'm rejigging a dual-drive USB enclosure with 2 x 2TB drives in it and decided I might as well do a full format. Then I realised how slowly it was going, probably not helped by the fact that it's actually going through 2 USB hubs to get to the enclosure.

If I'd realised it would be this slow I'm not sure I would have bothered, or I would at least have put it onto a USB3 port connected directly to my PC, but I suppose it's not bad practice to have explicitly written and done a read-back verify on every single sector of the drives which I believe is what it's doing. It would probably also flush out any obviously dodgy electronics as far as thermals are concerned since it's getting warm (but not by any means worryingly hot).

I've started now so I'm not stopping it but it looks as if it's going to be well into tomorrow before it's finished, maybe even into Monday. At least I know the disk activity light is working!

- Julian

Interesting to read the discussion. I must admit that I always format a new drive but a quick Google doesn't suggest why that is a good idea. I'm not convinced that it adds much to improve security (protection from pre-installed viruses) because the format doesn't over-write much existing data. Also, if an attacker wanted to infect a manufacturer's production at source they would probably be smart enough to use an infection that would not be removed by formatting. Can anyone tell me why formatting a new drive isn't a waste of time?

RC

[Julian]

Interesting to read the discussion. I must admit that I always format a new drive but a quick Google doesn't suggest why that is a good idea. I'm not convinced that it adds much to improve security (protection from pre-installed viruses) because the format doesn't over-write much existing data. ...

Are you sure about that RC? My quick Google said exactly the opposite regarding not overwriting much existing data. My Google source was this one: https://www.makeuseof.com/tag/differenc ... explained/

My understanding of a full format is that it writes a test block to every sector and then reads it back and verifies that it is the same as the data it wrote. I have always assumed that the test blocks written were all complete sectors although I realise now that is just my assumption. I can't see why it wouldn't completely populate each sector completely with a known (hence verifiable) test pattern though since the drive will overwrite the whole sector anyway. If it is writing then reading every sector and using a simple algorithm to do it it would explain why a full format takes so long on a traditional spinning disk (as opposed to an SSD) because if it is doing it one sector at a time the rotational latency will be absolutely horrible. Once it has written the test block it then needs to wait an entire rotation (minus one sector) until the sector it has just written is back under the head to read it back again. Assuming there is then enough latency in the OS getting the data to the format utility then by the time it has done the bit compare on sector n the heads are probably past the beginning of sector n+1 and assuming the format is following a simple algorithm of incrementing the sectors it probably needs to wait another rotation until sector n+1 is under the head to do the test write, and so on, so essentially near enough 2 complete rotations to write/read/verify each sector of the drive (plus head repositions as necessary).

The fact that it is my understanding that every sector is written and read back is why I am doing a full format, not in my case because I am concerned about malware (although Slarti has a good point), in my case it's because I want to know that every single sector has been written to and then had those contents read back and verified so that I can flush out any bad sectors and have them marked from the outset. (I do understand though that sector failure is a continual process on hard drives so that doesn't mean that sectors found good on the initial format might not go bad at some point in the future hence the bad sector count is unlikely to stay static over the life of the drive.)

- Julian

[ReformedCharacter]

Interesting to read the discussion. I must admit that I always format a new drive but a quick Google doesn't suggest why that is a good idea. I'm not convinced that it adds much to improve security (protection from pre-installed viruses) because the format doesn't over-write much existing data. ...

Are you sure about that RC? My quick Google said exactly the opposite regarding not overwriting much existing data. My Google source was this one: https://www.makeuseof.com/tag/differenc ... explained/

My understanding of a full format is that it writes a test block to every sector and then reads it back and verifies that it is the same as the data it wrote. I have always assumed that the test blocks written were all complete sectors although I realise now that is just my assumption. I can't see why it wouldn't completely populate each sector completely with a known (hence verifiable) test pattern though since the drive will overwrite the whole sector anyway. If it is writing then reading every sector and using a simple algorithm to do it it would explain why a full format takes so long on a traditional spinning disk (as opposed to an SSD) because if it is doing it one sector at a time the rotational latency will be absolutely horrible. Once it has written the test block it then needs to wait an entire rotation (minus one sector) until the sector it has just written is back under the head to read it back again. Assuming there is then enough latency in the OS getting the data to the format utility then by the time it has done the bit compare on sector n the heads are probably past the beginning of sector n+1 and assuming the format is following a simple algorithm of incrementing the sectors it probably needs to wait another rotation until sector n+1 is under the head to do the test write, and so on, so essentially near enough 2 complete rotations to write/read/verify each sector of the drive (plus head repositions as necessary).

The fact that it is my understanding that every sector is written and read back is why I am doing a full format, not in my case because I am concerned about malware (although Slarti has a good point), in my case it's because I want to know that every single sector has been written to and then had those contents read back and verified so that I can flush out any bad sectors and have them marked from the outset. (I do understand though that sector failure is a continual process on hard drives so that doesn't mean that sectors found good on the initial format might not go bad at some point in the future hence the bad sector count is unlikely to stay static over the life of the drive.)

- Julian

Apologies for a late reply, a virus has reduced me to life-support tasks only. You're correct all data is overwritten.

RC