Building a Duplicate-File Scanner in .NET 10: Cheap Checks First, Expensive Ones Last Engineering decisions behind dsweep — a three-stage detection pipeline, reversible quarantine, and why I keep rea
By Coderz Club · 2026-07-26 · Tags: ai, go
Building a Duplicate-File Scanner in .NET 10: Cheap Checks First, Expensive Ones Last
Engineering decisions behind dsweep — a three-stage detection pipeline, reversible quarantine, and why I keep reaching for .NET when I want a fast CLI. The obvious approach, and why it's wrong The naive implementation of a duplicate-file scanner goes like this: walk the directory tree, hash every file with SHA-256, group files by hash, done. It works. It's also brutally expensive. If you're scanning 50,000 files and most of them are different, you've computed full SHA-256 hashes of every single one — reading every byte of every file — to find a few hundred matches. The interesting engineering problem isn't "how do I hash files." It's "how do I avoid hashing files I don't need to." This post walks through the design of dsweep, a .NET 10 duplicate-file scanner with a three-stage detection funnel and a reversible quarantine system. I'll focus on the decisions I found genuinely interesting rather than a feature walkthrough. Stage 1: File size — zero hashing, pure filesystem metadata The cheapest possible check: two files with different sizes cannot be identical. No hashing, no reading — just a number from the filesystem stat. List<FileEntry> bySize = entries .GroupBy(e => e.Length) .Where(g => g.Count() > 1) .SelectMany(g => g) .ToList(); This one LINQ expression eliminates the majority of files in a typical scan. A 3 KB config file and a 14 MB video file will never match, and after this stage they never compete for hash time again. The pattern: group, filter groups of 1, flatten back to a list. Only files that share a size with at least one other file move to the next stage. Stage 2: Quick hash — first 64 KB only For files that share a size, we sample the first 64 KB and compute a SHA-256 of that sample. The sample size is deliberate: private const int QuickHashSampleBytes = 64 * 1024; public static string ComputeQuickHash(string path, long fileLength, CancellationToken ct) { using SHA256 sha256 = SHA256.Create(); using FileStream stream = File.OpenRead(path); int sampleSize = (int)Math.Min(fileLength, QuickHashSampleBytes); if (sampleSize == 0) return Convert.ToHexString(sha256.ComputeHash([])); byte[] buffer = new byte[sampleSize]; int read = ReadFully(stream, buffer, ct); return Convert.ToHexString(sha256.ComputeHash(buffer, 0, read)); } Why 64 KB? It's a pragmatic calibration. Large files (videos, disk images, archives) that differ will almost always differ within the first 64 KB — the content starts diverging quickly. The quick hash is cheap enough that a false positive (two different files that happen to share the same first 64 KB) doesn't hurt much: it just means both files go to stage 3 for the full hash. The only cost of a false positive is an unnecessary full-file read. The benefit of a true negative is avoiding that full-file read for every file that doesn't need it. For a 4 GB video file, this is the difference between reading 64 KB and reading 4 GB. For 1,000 video files where 998 are unique, that's roughly 63.9 GB of I/O saved. Stage 3: Full SHA-256 — only on confirmed quick-hash collisions Files that share both size and quick hash are the real candidates. Only these get the full-file SHA-256 treatment: public static async Task<string> ComputeFullHashAsync(string path, CancellationToken ct) { await using FileStream stream = File.OpenRead(path); byte[] hash = await SHA256.HashDataAsync(stream, ct).ConfigureAwait(false); return Convert.ToHexString(hash); } A few things worth noticing: SHA256.HashDataAsync — this is the modern static one-liner available since .NET 5. The old pattern (using var sha = SHA256.Create(); sha.ComputeHashAsync(stream)) is still valid but requires managing the disposable SHA256 instance. The static version is cleaner and does the same thing. Convert.ToHexString — available since .NET 5, replaces the classic BitConverter.ToString(hash).Replace("-", "").ToLower() pattern that I still see in a lot of code. It returns uppercase hex but that's consistent throughout the codebase. await using — the async dispose pattern. FileStream implements IAsyncDisposable so the stream gets closed asynchronously when we're done, which matters when you're running many of these concurrently. The full pipeline in 40 lines The three stages compose cleanly in DuplicateFinder.FindGroupsAsync: // Stage 1: group by size — zero I/O List<FileEntry> bySize = entries .GroupBy(e => e.Length) .Where(g => g.Count() > 1) .SelectMany(g => g) .ToList(); // Stage 2: quick hash (64 KB sample) (Dictionary<string, List<FileEntry>> byQuickHash, List<string> quickWarnings) = await GroupByAsync(bySize, e => Task.FromResult(Hashing.ComputeQuickHash(e.FullPath, e.Length, ct)), ct); List<FileEntry> candidates = byQuickHash.Values .Where(g => g.Count > 1) .SelectMany(g => g) .ToList(); // Stage 3: full hash — only on confirmed quick-hash collisions (Dictionary<string, List<FileEntry>> byFullHash, List<string> fullWarnings) = await Group
Engineering decisions behind dsweep — a three-stage detection pipeline, reversible quarantine, and why I keep reaching for .NET when I want a fast CLI. The obvious approach, and why it's wrong The naive implementation of a duplicate-file scanner goes like this: walk the directory tree, hash every file with SHA-256, group files by hash, done. It works. It's also brutally expensive. If you're scanning 50,000 files and most of them are different, you've computed full SHA-256 hashes of every single one — reading every byte of every file — to find a few hundred matches. The interesting engineering problem isn't "how do I hash files." It's "how do I avoid hashing files I don't need to." This post walks through the design of dsweep, a .NET 10 duplicate-file scanner with a three-stage detection funnel and a reversible quarantine system. I'll focus on the decisions I found genuinely interesting rather than a feature walkthrough. Stage 1: File size — zero hashing, pure filesystem metadata The cheapest possible check: two files with different sizes cannot be identical. No hashing, no reading — just a number from the filesystem stat. List<FileEntry> bySize = entries .GroupBy(e => e.Length) .Where(g => g.Count() > 1) .SelectMany(g => g) .ToList(); This one LINQ expression eliminates the majority of files in a typical scan. A 3 KB config file and a 14 MB video file will never match, and after this stage they never compete for hash time again. The pattern: group, filter groups of 1, flatten back to a list. Only files that share a size with at least one other file move to the next stage. Stage 2: Quick hash — first 64 KB only For files that share a size, we sample the first 64 KB and compute a SHA-256 of that sample. The sample size is deliberate: private const int QuickHashSampleBytes = 64 * 1024; public static string ComputeQuickHash(string path, long fileLength, CancellationToken ct) { using SHA256 sha256 = SHA256.Create(); using FileStream stream = File.OpenRead(path); int sampleSize = (int)Math.Min(fileLength, QuickHashSampleBytes); if (sampleSize == 0) return Convert.ToHexString(sha256.ComputeHash([])); byte[] buffer = new byte[sampleSize]; int read = ReadFully(stream, buffer, ct); return Convert.ToHexString(sha256.ComputeHash(buffer, 0, read)); } Why 64 KB? It's a pragmatic calibration. Large files (videos, disk images, archives) that differ will almost always differ within the first 64 KB — the content starts diverging quickly. The quick hash is cheap enough that a false positive (two different files that happen to share the same first 64 KB) doesn't hurt much: it just means both files go to stage 3 for the full hash. The only cost of a false positive is an unnecessary full-file read. The benefit of a true negative is avoiding that full-file read for every file that doesn't need it. For a 4 GB video file, this is the difference between reading 64 KB and reading 4 GB. For 1,000 video files where 998 are unique, that's roughly 63.9 GB of I/O saved. Stage 3: Full SHA-256 — only on confirmed quick-hash collisions Files that share both size and quick hash are the real candidates. Only these get the full-file SHA-256 treatment: public static async Task<string> ComputeFullHashAsync(string path, CancellationToken ct) { await using FileStream stream = File.OpenRead(path); byte[] hash = await SHA256.HashDataAsync(stream, ct).ConfigureAwait(false); return Convert.ToHexString(hash); } A few things worth noticing: SHA256.HashDataAsync — this is the modern static one-liner available since .NET 5. The old pattern (using var sha = SHA256.Create(); sha.ComputeHashAsync(stream)) is still valid but requires managing the disposable SHA256 instance. The static version is cleaner and does the same thing. Convert.ToHexString — available since .NET 5, replaces the classic BitConverter.ToString(hash).Replace("-", "").ToLower() pattern that I still see in a lot of code. It returns uppercase hex but that's consistent throughout the codebase. await using — the async dispose pattern. FileStream implements IAsyncDisposable so the stream gets closed asynchronously when we're done, which matters when you're running many of these concurrently. The full pipeline in 40 lines The three stages compose cleanly in DuplicateFinder.FindGroupsAsync: // Stage 1: group by size — zero I/O List<FileEntry> bySize = entries .GroupBy(e => e.Length) .Where(g => g.Count() > 1) .SelectMany(g => g) .ToList(); // Stage 2: quick hash (64 KB sample) (Dictionary<string, List<FileEntry>> byQuickHash, List<string> quickWarnings) = await GroupByAsync(bySize, e => Task.FromResult(Hashing.ComputeQuickHash(e.FullPath, e.Length, ct)), ct); List<FileEntry> candidates = byQuickHash.Values .Where(g => g.Count > 1) .SelectMany(g => g) .ToList(); // Stage 3: full hash — only on confirmed quick-hash collisions (Dictionary<string, List<FileEntry>> byFullHash, List<string> fullWarnings) = await Group