anyone know anything about #data #deduplication. i have 1 hard drive where I've gathered everything all my #backups, music, docs, emails etc etc etc. but i have a few copies of everything. want a program that simply scans the #drive, deletes all #duplicates, and keeps a copy of only 1 of everything. does the damn thing exist as i cant find anything simple enough? #techHelp #ComputerHelp #software #tech #techie

The April #syslog_ng #newsletter is now on-line:

- Installing a syslog-ng 4 #development snapshot on #FreeBSD

- Getting data to #Splunk

- Streaming #deduplication in syslog-ng

https://www.syslog-ng.com/community/b/blog/posts/the-syslog-ng-insider-2023-04-freebsd-splunk-deduplication

Ah, my old nemesis: the bits-vs-cycles tradeoff.

#programming #vagueposting #deduplication

Just learned about deduplication and timing attacks on social media with the ThrowBackThursday trend! #ThrowBackThursday #Deduplication #TimingAttacks https://www.GPTrending.com/#/?date=2023-03-17&topic=ThrowBackThursday

How #Git object #deduplication works in #Gitlab

Review
https://forge.etsi.org/rep/help/development/git_object_deduplication.md

TIL about Content Defined Chunking. What a fun rabbit hole!
https://restic.net/blog/2015-09-12/restic-foundation1-cdc/

#golang #deduplication

He previously discussed two techniques for improving the performance of databases, and in this article he discusses another technique: deduplication. Deduplication can be used to replace transactions for eventually consistent use cases with defined primary keys. https://www.infoworld.com/article/3683915/using-deduplication-for-eventually-consistent-transactions.html#tk.rss_all #Deduplication #DatabasePerformance #EventuallyConsistent

Duplicacy - #Backup with cross-computer #deduplication, compression and #encryption

https://github.com/gilbertchen/duplicacy

Hard to believe the gymnastics required to “move” 1,000 files from one folder to another in an #Azure Cold Storage container.
Hoping to tackle 500gb+ of duplicate images in storage but even this tiny attempt to clean up a bit is making me think I’m using the wrong tools.

#azurestorage #deduplication

Did a #borg-compact after the prune 👀
#Deduplication is a neat thing 🥳

@andersbateva @guyjames

Thanks for the hint. Actually the #Proxmox Backup Server already does #deduplication.
Currently I have a factor of 9.68, which is fine I guess.

But I will also look into FSlint 👍

I just wrote an article about how I removed duplicates from my GPS favourites.

https://stuvel.eu/articles/gps-waypoint-deduplication/

It's basically working around a side-effect of Osmand not being able to sync favourites between devices. That's pretty much the only gripe I have with the app, the rest is working so well.

#osmand #GPS #python #deduplication #favourites

After some permission changes inside #macos (full disk access) the new #synology #ActiveBackupForBusiness was completed.

Maxing at over 100MB/s this backup is miles ahead of #timemachine and #deduplication saved storage space on top of it all. Nice!

https://www.blackvoid.club/synology-active-backup-for-business-2-5-support-for-macos-is-here/

After some permission changes inside #macos (full disk access) the new #synology #ActiveBackupForBusiness was completed.

Maxing at over 100MB/s this backup is miles ahead of #timemachine and #deduplication saved storage space on top of it all. Nice!

https://www.blackvoid.club/synology-active-backup-for-business-2-5-support-for-macos-is-here/

The program I used for help trim the fat in my music folder was this one:

https://github.com/qarmin/czkawka/blob/master/instructions/Installation.md

#deduplication #czkawka

Want to free up harddrive space without doing a full "find duplicates and remove"?

Why not just create a link to one of the files and remove the duplicated block of data?

https://www.jodybruchon.com/2015/08/08/finding-duplicates-faster-the-story-of-jdupes-or-how-i-unexpectedly-became-a-better-programmer/

#poormansDeduplication
#jdupes
#storage
#deduplication

@unix_discussions Space concerns aren't anywhere near the main reasons to dislike #flatpak, #compression & #filesystem #deduplication do a pretty good job of limiting that issue.

The dubious #security & packaging practices are of much greater concern (https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/).

#FragFedi #FollowerPower
Weiß jemand, ob es eine elegante Lösung für die Fragestellung "Interessanter Toot - aber bekommen meine Follower das nicht auch so schon mit?" gibt? 🤔

Ist das ein Problem auf Protokoll- oder Plattformebene, dass dann eher @Gargron angehen müsste?
Könnte ein solcher Filter seitens der Instanzen realisiert werden?
Oder wäre das Aufgabe der(Web-)Apps?

Natürlich gilt auch hier: #BoostsWelcome 😉

#ToBoostOrNotToBoost #Filter #Deduplication
@Calypso1

This is rather neat. The "restic" backup utility (which is already awesome) is getting support for compression (using "zstd").

https://forum.restic.net/t/compression-support-has-landet-in-master/4997

#restic #backup #compression #zstd #deduplication

I need out-of-band #deduplication on #ZFS.
Sadly, it seems it won't (can't?) be implemented :-(