Remove duplicates in MongoDB with an aggregation, step by step

The aggregation that finds duplicate documents in MongoDB is short. Removing them is the part you slow down for. If your duplicates have been multiplying like rabbits, the good news is the fix is mechanical; the important news is that it writes to a production collection, so we do it carefully. Here is the whole pattern.

Step 1: group and count

Group the collection on the field that defines identity, push every _id into an array, and keep only the groups with more than one document:

db.users.aggregate([
  { $group: {
      _id: "$email",
      ids: { $push: "$_id" },
      count: { $sum: 1 }
  } },
  { $match: { count: { $gt: 1 } } }
])

Each result is a duplicate cluster: the shared email, an ids array of the documents that share it, and how many there are. Nothing is deleted here. You are looking, not touching. The $group reference has the operators if you want to aggregate more than a count.

Step 2: choose the key before you trust the result

Before going further, question the key, because the key is the entire decision. Group ten thousand "duplicate" users on email alone and you get noise. Group on the key that actually defines identity, email plus tenantId, and the real number is far smaller:

{ $group: {
    _id: { email: "$email", tenantId: "$tenantId" },
    ids: { $push: "$_id" },
    count: { $sum: 1 }
} }

A good dedup key is often a composite, and occasionally there is not one at all. If two documents share an email but differ in everything that matters, they are two records, not a duplicate. The honest answer is sometimes "leave them."

Step 3: keep one, remove the rest

This is where the jokes stop. You are about to delete from a live collection, so move deliberately.

Decide which document in each cluster survives, usually the oldest or the most complete, and gather every other _id into a list to remove. You can do this in application code by taking each ids array and dropping the first element, or in the pipeline with $slice. Either way, before you delete anything, count exactly what the delete will match:

// Preview: how many documents will this remove?
db.users.countDocuments({ _id: { $in: idsToRemove } })

Read that number. Make sure it is the number you expect, not the whole collection. A remove list built from a wrong key or an off-by-one slice will happily match far more than you intended, and MongoDB will do exactly what you asked. Only when the count is right do you run the delete:

db.users.deleteMany({ _id: { $in: idsToRemove } })

Take a backup or a snapshot first, so the door swings both ways if the key was wrong.

Step 4: stop the next batch

A cleanup that does not prevent recurrence is a chore you will repeat. Once the collection is clean, add a unique index on the identity key so new duplicates are rejected at write time:

db.users.createIndex({ email: 1, tenantId: 1 }, { unique: true })

Build it after the cleanup, not before, or the index build fails on the conflicts it finds. More on index options in how to create an index in MongoDB.

Or do it behind a dry run

The manual pattern works, and the risk in it is entirely in step 3: a delete that matches more than you meant. That is exactly the step worth taking out of your hands. mongoui's duplicate finder groups documents on the fields you pick, shows you the clusters, and runs every removal as a dry run first, with the exact before and after counts before a single write. Applied changes keep a snapshot, so undo is one click.

The grouping and comparison happen on your machine, so the documents never leave your network. Pick the key, read the clusters, apply behind the gate. Your duplicates stop breeding, and nothing legitimate gets swept up with them.