If you give your family & friends permission to directly send pictures to your photo frame via email or Dropbox, you will likely end up with duplicate images over time.

To make matters worse, the duplicated images are not easy to spot, as their file names are often different.

In this article, I will describe a script that will run every night to check if you have duplicate images in your Pictures folder.

When a duplicate image is found, it will be moved to a folder called “DuplicatePhotos” in the user directory.

There is an updated script here.

Which of the three duplicate articles should you use?

There are three of these on the blog and I am keeping all three, because each uses a different method and they suit different libraries:

  • This one (2020) compares images by root-mean-square difference. It is the slowest of the three and it does not look inside subfolders — every photograph has to sit at the top level of Pictures.
  • The 2023 update uses SHA-256 hashes of the file contents. It walks subfolders, it is much faster, and it finds byte-for-byte duplicates only. For most people this is the one to use.
  • The 2024 edition uses perceptual hashing, so it also catches a photograph that came back through WhatsApp at a lower resolution, keeps the larger version, and sends you a Telegram message about it.

One thing that has changed on the frame itself: if you are running picframe3 on a Pi 4 or Pi 5, you no longer need the quarantine folder this script creates. picframe3 has a Removed tab — anything you remove is moved aside rather than deleted, every removal is written to a journal saying when it went and where it came from, and a Show this again button puts it back. A removed picture also stays out of the slideshow if the same file turns up again from a sync or a backup. So the review step this article asks you to do by hand now has a place to happen. picframe3 does not find duplicates for you, though — that is still what these scripts are for.

The Root-Mean-Square Difference (RMS)

Python provides several methods of comparing images to determine if they are the same or possibly similar. The method that is being used in this script is the Root-Mean-Square Difference (RMS).

To measure how similar two images are, one can calculate the root-mean-square value of the difference between the images. If the images are exactly identical, this value is zero.

The Python script to find duplicate images

The script takes the RMS value of an image and compares it to another image’s RMS value. If the difference between the two images is less than 1, then the images are considered the same. However, for safety reasons, no files are deleted automatically. They will be quarantined to the DuplicatePictures directory, where you can take a look at them from time to time.

I tested various settings and found that a value of “0.01” in the average_diff line below works best for avoiding false positives. But it still ensures that full duplicates, i.e., not similar but exactly identical images, will be discovered reliably.

Credit for the script goes to Albatrossity on Stackoverflow, I only made some minor modifications.

There is currently one limitation: The script doesn’t scan the Pictures directory recursively. All images have to be on the main level. For an updated version of this script head over here.

Create a new file and paste this code into it.

#!/usr/bin/env python3

import os
from PIL import Image, ImageStat
import shutil

'''
The script takes the RMS value of an image and compares it to another image's RMS value. If the difference between the two images is less than 1 (calculated in the function 'average_diff') then the images are considered the same, and the duplicate is moved to the duplicates folder.
'''
image_folder = os.path.expanduser('~/Pictures') # not yet recursive
duplicate_folder = os.path.expanduser('~/DuplicatePictures')
image_files = []
rms_pixels = []

# create directory for duplicates if it does not exist
if not os.path.exists(duplicate_folder): 
    os.makedirs(duplicate_folder)

#Function calculates the difference between the CURRENT image file RMS value and RMS values calculated at start
def average_diff(v1, v2):
    duplicate = False
    calculated_rms_difference = [v1[0]-v2[0], v1[1]-v2[1], v1[2]-v2[2]]
    if calculated_rms_difference[0] < 0.01 and calculated_rms_difference[0] > -0.01 and calculated_rms_difference[1] < 0.01 and calculated_rms_difference[1] > -0.01 and calculated_rms_difference[2] < 0.01 and calculated_rms_difference[2] > -0.01:
        duplicate = True
    return duplicate

def quick_rms(images, rms):
    image_file_count = 0
    while image_file_count < len(images):
        image_file = images[image_file_count]
        check_duplicates = image_file.endswith('.jpg')
        if check_duplicates:
            rms_file_count = 1
            original_image = Image.open(os.path.join(image_folder, image_file))
            rms_original = ImageStat.Stat(original_image).mean
            while rms_file_count < len(rms):
                rms_file = rms[rms_file_count]
                duplicate_image = average_diff(rms_original, rms_file)
                print('Checking: ', image_file_count, 'against', rms_file_count, 'of', len(rms), end='\r', flush=False)
                if image_file != rms_file[3]:
                    if duplicate_image:
                        source = os.path.join(image_folder, rms_file[3])
                        dest = os.path.join(duplicate_folder, rms_file[3])
                        shutil.move(source, dest)
                        image_to_remove = rms_file[3]
                        images.remove(image_to_remove)
                        rms.remove(rms_file)
                        print('MOVED to ' +  duplicate_folder + ':' + image_to_remove) # comment this out when in production environment
                rms_file_count += 1
        image_file_count += 1
    return
#Creates a list of images to be checked and compared, these are the images stored in the Image folder at top of script
for x in os.listdir(image_folder):
    image_files.append(x)
#Create a list with all the Image RMS values. These are used to compare to the CURRENT image file in list
for x in image_files:
    if x.endswith('.jpg'):
        compare_image = Image.open(os.path.join(image_folder, x))
        rms_pixel = ImageStat.Stat(compare_image).mean
        rms_pixel.append(x)
        rms_pixels.append(rms_pixel)
        print(rms_pixel)
#Driver code, runs the script
quick_rms(image_files, rms_pixels)

Save as “duplicates_finder.py_“_and exit the file.

To test it, put some images into your “Pictures” folder and duplicate some files.

Start the script with

python3 duplicates_finder.py

I ran a few speed tests out of our curiosity.

For 1,000 images with a resolution of 1920 x 1200 px, it took 5 minutes and 30 seconds on a Raspberry Pi 4 and 7 minutes 52 seconds on a Raspberry Pi 3. (Those are 2020 numbers on the hardware of the day. I have no Pi 5 figures for this script — it will be quicker, but I have not measured it.)

My test with 10,000 images took 1 hour and 24 minutes with a Pi 4 at an average CPU load of 25%. On a Pi 3, the time was 2 hours and 49 minutes at 27% CPU load.

The time doesn’t really matter since the script will run in the background during the night. It may depend a little bit on the SD-Cards that you are using.

Autorun the script daily

We will use systemd to run the script once a day to make sure that your image folder is regularly scanned for duplicates.

Let’s first create the service file.

sudo nano /etc/systemd/system/duplicates_finder.service

Copy & paste

[Unit]
Description=Checks for duplicates

[Service]
Type=simple
User=your-username
WorkingDirectory=/home/your-username
ExecStart=/usr/bin/python3 /home/your-username/duplicates_finder.py

Replace your-username with your actual user name. Raspberry Pi OS no longer creates a default pi account — you choose the name when you write the card — so User=pi and /home/pi/… are no longer safe defaults.

Save and close.

Create the corresponding .timer file with

sudo nano /etc/systemd/system/duplicates_finder.timer

Copy & paste

[Unit]
Description=Timer to check for duplicate images

[Timer]
OnCalendar=daily

[Install]
WantedBy=timers.target

Save and close. Instead of “daily”, you can also insert “weekly” or “monthly”. Up to you to decide how often you want to run it.

Tell the system that you have added this file and want to enable this service.

sudo systemctl daemon-reload && sudo systemctl enable duplicates_finder.timer && sudo systemctl start duplicates_finder.timer

To check if everything is ok, type

sudo systemctl status duplicates_finder.timer

You should see something like this:

pi@Pi4:~ $ sudo systemctl status duplicates_finder.timer
● duplicates_finder.timer - Timer to check for duplicate images
   Loaded: loaded (/etc/systemd/system/duplicates_finder.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Thu 2020-12-17 13:11:15 GMT; 7min ago
  Trigger: Fri 2020-12-18 00:00:00 GMT; 10h left

Dec 17 13:11:15 Pi4 systemd[1]: Started Timer to check for duplicate images.

If you don’t update your images often, running it once a month will also be fine but there is no real cost associated with running it more frequently.

Conclusion

I was surprised to learn how many different algorithms Python offers to measure the similarity of images.

The RMS method looks battle-proven. With the above script, you can keep your photo library tidy and avoid that photos are shown over-proportionally often because they happen to reside five times in your Pictures folder.