Question about repo Duperemove, my new CLI tool

brent

For those with unhealthy obsessions like keeping your Solus tip top and clean and lean...been playing with this tool today. I love it. You point it to a folder, it kicks down the doors, and shows you the duplicate files across the OS for you to remove.

A command like this: sudo duperemove -r /home/work/Documents gets you a bunch of this:

Showing 2 identical extents of length 279776 with id a97ab384
Start		Filename
0	"/home/work/Documents/more_documents/lipids_garlic.pdf"
0	"/home/work/Documents/lipidsandgaric_peerreview.pdf"

in this case it's the same journal article I renamed once and it found it. But you aim it at a folder (/Pictures or .cachefor instance) it will show you about a dozen to 200 respectively. Scary how much redundancy we accumulate.

Anyways, My Question:
duperemove --help has this oddity at the end:
" What is the cost of deduplication?
Deduplication will lead to increased fragmentation. The blocksize chosen can have an effect on this. Larger blocksizes will fragment less but may not save you as much space. Conversely, smaller block sizes may save more space at the cost of increased fragmentation."

I know how fragmentation works in WIN (i.e. you had to stay on top of it) but does fragmentation effect Linux the same way? will eliminating duplicates create more problems?

I just don't get the disclaimer, that's all. Thanks

Axios

brent Linux is smart where it puts stuff not like win it pretty much handles itself
only time you may start having issues is when a disk gets like 80-90% full lunix doesnt have the space then
to manage it properly. Linux will move stuff on its own to avoid the issue.
(this is for the ext2,3,4 dont know how the newer linux filesystems really work they must
be on par with the older or better maybe)

I am not sure about the statement your asking about not sure what the point of it is.
to me its just like deleting files off your hard drive.(Dunno)
There is no defrag stuff for linux so not sure the point of it.(Dunno)
Interesting.

Sebastian

brent Just to let you know that there is also a good GUI tool for hunting down file duplicates: Czkawka.
Available on flathub and github

Edit: Also available in our repo !

synth-ruiner

brent I know how fragmentation works in WIN (i.e. you had to stay on top of it)

this hasn't been the case since Windows XP, I think. Vista added a background process that would defrag your disks automatically.

brent

Axios Wow I never ran across a linux defrag so I took it their was none.

HAHAHA I had no idea linux has this either!! You are right that

Axios Linux will move stuff on its own to avoid the issue.
(this is for the ext2,3,4 dont know how the newer linux filesystems really work they must
be on par with the older or better maybe)

--I always counted on that. I just wondered if there would be a major effect from clipping small blocks and large blocks here and there...I like to try apps like anyone than uninstall them and tear entire folders out.

Harvey So while the warning is true, its a nothing burger.

thanks Harvey, good.

Harvey EXT4 is better at avoiding fragmentation than NTFS but it still does occur. You're just less likely to notice issues but defrag tools do exist.

good. when you used to call up the defrag tool in windows it looked like a construction relief map...I will continue to prune.
I'm of the mind no amount of recursive backing up will prevent the accumulation to duplicates in other folders. I like the little clean up I've done already with this too.

really great info, both of you

Harvey

Any time you write or delete data from a disk you can create fragmentation. So while the warning is true, its a nothing burger.

EXT4 is better at avoiding fragmentation than NTFS but it still does occur. You're just less likely to notice issues but defrag tools do exist.

Generate fragmentation report on drive mount point
sudo e4defrag -c /mnt/8TB/

It found 4 fragmented files but the report says I do not need to defrag.

 Fragmentation score                            0
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This directory (/mnt/8TB) does not need defragmentation.
 Done.

I have never manually ran defrag and have 4 drives in my system. None need action to be taken.

On an SSD it is flat out not a problem because there is no seek time where it has to move the read head to the correct location on the platter.

Axios

Harvey Wow I never ran across a linux defrag so I took it their was none.
Thanks for correcting me.
I just use never had issues in that department so dont worry about it..lol

Harvey

In case people want more info on why writes / deletes can cause fragmentation and how its unavoidable.

You want every file on the disk to be in a contiguous block (At least for HDDs) so you do not increase seek time by having the head move all over the disk for the same file which would drastically reduce read speed.

It is a bit like Tetris. You have disk capacity and a set filesystem block size. Some files are large an therefore take up multiple blocks of space. If the filesystem block size is 512KiB (It'll be much much smaller than this) and you have a 1MiB file it will take 2 blocks. If its 1.25MiB it will allocate to 3 blocks but the third block is only half full. If the file was 1MiB when first written but with changes over time has grown in size past that, it may no longer exist in a contiguous block as the next block in sequence has already been allocated.

Now imagine in your de-duping efforts you have a 8TB drive full of files w/ zero fragmentation. There are 2,000,000 files of various different sizes and 500,000 of them are duplicates. Deleting those files is going to blow a lot of holes in that stack. Those holes and/or files existing on the disk in non contiguous blocks is called fragmentation. Now you start filling the drive up again with new files of different sizes, some will fill up the holes left behind, some won't fit in those holes so it'll allocate it elsewhere on the disk leaving holes in place.

Additionally the system wants you to have fast write speeds so it will throw data being written to wherever there is free space, contiguous? non contiguous? beginning of the platter? End of the platter it doesn't care just do it quickly for the user.

Some filesystems are better than others, EXT4 is pretty good at avoiding fragmentation. NTFS... not so much. But in my de-duping example EXT4 might need a defrag too, it can only be so smart.

brent

Harvey You want every file on the disk to be in a contiguous block (At least for HDDs) so you do not increase seek time by having the head move all over the disk for the same file which would drastically reduce read speed.

ok I get this. Your Tetris analogy (I still play Tetris) is brilliant--you should be a teacher; you are a teacher.
My application would just be HDD and all of my hdd is ext4. Like you pointed out in an earlier post here: ext4 is good at this and I probably have no need to do this.eopkg it CC andeopkg rm CC are the extent of my moves in solus. although I am going to purge duplicates at a 'spare time' level now..

Harvey Now imagine in your de-duping efforts you have a 8TB drive full of files w/ zero fragmentation. There are 2,000,000 files of various different sizes and 500,000 of them are duplicates. Deleting those files is going to blow a lot of holes in that stack. Those holes and/or files existing on the disk in non contiguous blocks is called fragmentation. Now you start filling the drive up again with new files of different sizes, some will fill up the holes left behind, some won't fit in those holes so it'll allocate it elsewhere on the disk leaving holes in place.

this is a more problematic scenario if one was dupe hunting. For 8TB with half a million dupes maybe leave well enough alone? If this was my 4TB external backup drive I'm not sure I would use this tool at all. Would try to prune at the pre-backup level first.

I'm always amazed by the thoroughness in your explanations, and of the other principals.
edit/typo

Harvey

My intent was to alleviate fears of fragmentation occurring because you can not avoid it entirely, it will happen to some extent. Its just certain scenarios may make it more of an issue while for the average usage on EXT4 you'll never notice. In the 500,000 dupes scenario I gave I would still de-dupe, I would probably just run a defrag check after.

Additionally I would advise people not to run de-duping tools system wide, its only for your own files. I.e do not run it on / but /home/username/Documents/ is fine.

brent

Harvey My intent was to alleviate fears of fragmentation occurring because you can not avoid it entirely, it will happen to some extent. Its just certain scenarios may make it more of an issue while for the average usage on EXT4 you'll never notice. In the 500,000 dupes scenario I gave I would still de-dupe, I would probably just run a defrag check after.

...time to hunt for duplicates....

ReillyBrogan

Fragmentation is entirely a non-issue on any storage device based on flash memory (SSDs) or 3D XPoint (Optane). These devices can and will store data blocks on sectors throughout the disk and can do reads/writes without any penalty from fragmentation. Defragmentation tools should be avoided on SSDs as the only thing they'll accomplish is causing more wear on the disk. Hell, even if the storage was unfragmented it would still be fragmented at the actual chip level as SSD controllers will use different blocks on the device in order to spread out write wears on the disk.

Axios

SSD are 2 tier you have the file system putting stuff where it thinks it should go then you have the
SSD controller putting it where it thinks it should go (for wear leveling and part of the reason for difference in read write speeds).
But like was stated this happens so fast with SSD that its a non issue.
You dont wear out an SSD by reading from it.
SSD get slow when the SSD controller has issues placing data (Good indication you should replace it)
Also dont use and disable any secure delete functions (Many people forget this)
In other words use it till it dies

brent

I did go hunting rabbits.

CONCLUSION: I'm a good housekeeper. My personal files and pdf collections had about 20 instances of duplicates which I hand-deleted.
I only scanned /home and not /filesystem---I refused to mess with the system files.

My bloated/.cache was even clean.

Where things got weird were/.config...

duperemove -r /home/brent/.configshowed 92 mysterious system files(?)/duplicates that were shared between 3 apps: museeks, jitsi, and a very very bloatedBrave browser.

EXHIBIT A:

Showing 8 identical extents of length 8192 with id 3181f604
Start		Filename
0	"/home/brent/.config/Museeks/GPUCache/data_3"
0	"/home/brent/.config/FreeTube/GPUCache/data_3"
0	"/home/brent/.config/FreeTube/DawnCache/data_3"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/ShaderCache/data_3"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/GraphiteDawnCache/data_3"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/Default/GPUCache/data_3"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/Default/DawnCache/data_3"
0	"/home/brent/.config/Jitsi Meet/DawnCache/data_3"

what is this identical "data3" file with no extension that three apps, and only three apps need? curious.
lots of chunks like this.

EXHIBIT B (Brave spawning the love):

Showing 10 identical extents of length 4096 with id daecf672
Start		Filename
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.449/1/scripts/brave_rewards/publisher/reddit/_locales/mr/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.449/1/scripts/brave_rewards/publisher/reddit/_locales/en_GB/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.449/1/scripts/brave_rewards/publisher/reddit/_locales/ml/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.449/1/scripts/brave_rewards/publisher/reddit/_locales/gu/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.449/1/scripts/brave_rewards/publisher/reddit/_locales/te/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.492/1/scripts/brave_rewards/publisher/reddit/_locales/mr/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.492/1/scripts/brave_rewards/publisher/reddit/_locales/en_GB/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.492/1/scripts/brave_rewards/publisher/reddit/_locales/ml/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.492/1/scripts/brave_rewards/publisher/reddit/_locales/gu/messages.json"
0	"/home/brent/.config/BraveSoftware/Brave-Browser/afalakplffnnnlkncjhbmahjfjhmlkal/1.0.492/1/scripts/brave_rewards/publisher/reddit/_locales/te/messages.json"

lots of Brave identical stuff spread out in all the folders.

the rest of .cache is just variations of A) same file with same 3 apps and B) Brave multiple duplicates of same file. no other apps or variations noticed--just these 2.

I didn't mess with the/.config stuff, not sure it's necessary.

any 2nd opinions on my expedition, please share. Thanks!

Harvey

As a general rule, unless you created the file yourself. Assume its supposed to be there duplicate or not.

brent

Harvey As a general rule, unless you created the file yourself. Assume its supposed to be there duplicate or not.

that's the way I played, so thanks for the confirmation. I have a single follow-up question:
are there any exceptions to that general rule?

Harvey

Theoretically some applications could have originally placed their config files in ~/.whatever but in later versions started placing them within ~/.config/ and did not clean up after itself but you will have to be 100% sure that it is the case.

The safest thing is to just adhere to the general rule. Sadly many applications do not adhere to the XDG standards and will continue littering peoples home directory for historical reasons or some other project nonsense such as "supporting that standard adds to code complexity" or "No one has ever asked for it before, so clearly there is no demand for a widely used standard..." *rolls eyes*

brent

Harvey thank for indulging. The relaxing of uniform standards makes me roll eyes too. Discipline or laziness. Adhere to it or reinvent it. Eh.

brent

synth-ruiner I believe it is auto. I tried this in 10 last year but not needed. you can still do this manually , if you want to, as I recall, but I think you are right, it's an auto process now.