Find and remove duplicate documents in MongoDB, safely

Finding duplicate documents in MongoDB is an aggregation. Removing them is a decision, and it is yours to make, because only you know your data. The whole task comes down to one choice: the key that defines what counts as the same document. Get that right and the cleanup is short and safe. Get it wrong and you delete rows that were never duplicates.

Find the duplicates first

Group the collection on the identity field, keep the _id values, and count each group:

db.users.aggregate([
  { $group: {
      _id: "$email",
      ids: { $push: "$_id" },
      count: { $sum: 1 }
  } },
  { $match: { count: { $gt: 1 } } }
])

Every group that comes back has more than one document sharing that email, and the ids array holds their _id values. That is your list of duplicate clusters. Nothing has been deleted; you have only looked. The $group reference covers the operators if you want to group on more than one field.

The dedup key is the whole problem

Before you delete anything, question the key. A collection can look like it has ten thousand duplicate users when you group on email alone. Group on the key that actually defines identity here, email plus tenantId, and it is two hundred, most of them legitimate accounts in different tenants.

{ $group: {
    _id: { email: "$email", tenantId: "$tenantId" },
    ids: { $push: "$_id" },
    count: { $sum: 1 }
} }

A good dedup key is often a composite, and sometimes there is not one. If two documents share an email but differ in every field that matters, they are not duplicates, they are two records. The honest answer is occasionally "leave them." A tool that decides "the same" for you is deciding what to delete for you, and that is not its decision to make.

Remove them, carefully

This is the part where the jokes stop. You are about to write to a production collection, so move slowly.

Decide which document in each cluster you keep (usually the oldest, or the one with the most complete data), and collect every other _id into a list to remove. Then, before deleting, count exactly what the delete will match:

// Preview: how many documents will this remove?
db.users.countDocuments({ _id: { $in: idsToRemove } })

Read that number and make sure it matches what you expect. A delete filter written quickly can match far more than you intended. Only when the count is right do you run the delete:

db.users.deleteMany({ _id: { $in: idsToRemove } })

Keep a backup or a snapshot before the delete, so an undo is possible if the key was wrong after all.

Prevent the next batch

Once the collection is clean, stop the problem at the source with a unique index on the identity key:

db.users.createIndex({ email: 1, tenantId: 1 }, { unique: true })

New inserts that would create a duplicate are rejected at write time. Build it after the cleanup, not before, or the index build fails on the existing conflicts. See MongoDB index optimization for keeping that index lean.

Or resolve them behind a dry run

mongoui's duplicate finder groups documents on the fields you pick, shows you the clusters, and lets you resolve them, never a blind delete. Every removal is a dry run first: the exact before and after counts, on a sample or the full collection, before a single write. Applied changes keep a snapshot, so undo is one click.

The engine runs on your machine, so the documents you are grouping and comparing never leave your network. Choose the key, review the clusters, apply behind the gate. Your duplicates stop multiplying, and nothing legitimate gets caught in the sweep.