As part of my efforts to ensure I have a copy of all of my data at any given time, I needed to migrate off of, or add to, Google Photos. Google Photos does not back up your content in a way that is easy for you to access, and if you ever wanted to get all of your content, you must go through a Google Takeout process which is not guaranteed to continue to exist. Unlike Google Drive, where you can sync your data to your desktop computer, Google Photos offers no such thing. Made worse, Photos does not store your content in your Google Drive (they used to but stopped doing this when they began pushing their Photos app and paid add-ons). It’s fine for a business to do as they wish, but for me and my photos, I wanted more security.
I also wanted to find an alternative viewer to my photos in Google Drive that was more feature-rich. The Finder/Folder Explorer experience of looking at either a list or tiles of photos and navigating them one and a time was not ideal. And I wanted access to my photos from multiple devices, which my self-hosted setup would offer. I have included a few screenshots of the product below:
Main Screen
Map View
Labels
Moments
and Videos
Components to this setup
- PhotoSync
- Replaces the need to have Google Photos or any other backup solution backing my photos up to a cloud provider
- a paid, mobile application (the free version is too limited)
- it can send / one-way sync photos from your mobile device to numerous targets
- it can convert HEIC to JPEG (if you wish) and other nice features
- syncs can be triggered by either a time when its charging or based on GPS coordinates
- Google Drive
- Is what it is: raw storage in the cloud
- Google Photos, unfortunately, does not drop files here so you have to find a way to get your originals into Google Drive for safe keeping / to use them outside of the Google Photos app(s)
- PhotoPrism
- a self-hosted photo exploration tool for large photo collections
- can be opened up to the world (i keep mine localhost)
- it’s pretty nice looking and makes browsing photos more blissful
- can tag geo-locations and display on a map
- can recognize people using TensorFlow for people/object detection
- can create preview thumbnails of videos or photo bursts
- can do automated conversions
Initially, I tried to mount Google Drive via rclone to my Unraid machine and have PhotoPrism read from that mounted drive. This was slow in every fashion. That plan was quickly tossed out. I settled on keeping the rclone mount to Google Drive and using rclone to one-way sync from Google Drive to an Unraid network share that will store all of my originals. I am not looking to have PhotoPrism have write access to my Google Drive, so I did make my photo folder read-only to PhotoPrism, but others may wish to change that. That would then change your rclone sync options and other assumptions made in this write-up.
To share the overall process real quick:
- Take photos on my phone
- PhotoSync backs those photos to Google Drive
- Rclone sync script runs daily (but you can do any interval) to pull photos from Google Drive and store them on a local Photo share in Unrai and kicks off a PhotoPrism index upon completion to detect any add/removals of content.
Using the Unraid docker for PhotoPrism, all of the photo metadata, thumbnails, etc (what they call sidecar data) is stored in the AppData folder. This led to my AppData drive getting filled with many GBs of content for my library collection. It’s not data that must be backed up; it can easily be generated in minutes to an hour. So I moved sidecar files to the Photos share in a second folder to house that. As well, PhotoPrism by default, was in write mode, so I disabled that and pointed the main ‘photo storage’ path to my synced Google Drive copy in the Photos share. I had to create a few specific paths and variables to force sidecar content (thumbnails and other data PhotoPrism creates) which I will share. But the result is PhotoPrism’s AppData folder is light and just the critical stuff; my Google Drive photo folder is synced on a parity-protected Unraid share, and the sidecar data sits alongside within the Photos share. Again, the Photo share path that has my Google Drive photos is read-only in the PhotoPrism docker settings.
The follow are the custom paths/variables I added in addition to the default Unraid docker template. You can read more about the variables in the PhotoPrism docs.
Once created, here is how my paths look in the Docker overview tab in Unraid:
Once you load up PhotoPrism, after making all of these changes, you can roll into the Library tab and index away:
The index process is fast. Sidecar data is fast. My Photo share uses Unraid’s cache which means initial indexing and re-indexing are fast. I have found browsing from the array (after the cache move is completed) to be quite zippy enough.
Here is my User Scripts schedule and script:
important caveat: I renamed my photoprism docker image from PhotoPrism to photoprism (all lowercase). If you leave as default, you’ll need to adjust the docker command in this script to reflect that.
#!/bin/bash
CURRENT_DATE=$(date +%Y%m%d)
LOG_FILE=/mnt/user/appdata/app-logs/rclone-photosync-$CURRENT_DATE.log
echo "Logging to $LOG_FILE"
MAX_TRANSFER=600G
# --dry-run \
# "/mnt/user/Backups/GDrivePhotos"
echo Using Syncing Google Drive Photos
rclone sync -v \
--backup-dir=/mnt/user/Backups/GDrivePhotosDeleted/$CURRENT_DATE \
--bwlimit=80M \
--fast-list \
--log-file=$LOG_FILE \
--transfers=8 \
--max-transfer=$MAX_TRANSFER \
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" \
<DriveRcloneMount>:<PathToPhotos> /mnt/user/Backups/GDrivePhotos
# Give us the ability to see the error in our ways
chmod a+r $LOG_FILE
echo "Log file can be found at $LOG_FILE"
echo "Running PhotoPrism index..."
docker exec photoprism photoprism index
One added benefit from this is that I now have a backup of my Google Drive’s photos on my NAS.
While I removed Google Photos from my phone, it’s fine to leave it running in parallel. I know a few from the ReelTalk community do this to make photo sharing and shared albums easier. To me, that’s using Google Photos correctly while still owning your data. Win-win.
Update September 4, 2022:
- I am exploring ways of having PhotoPrism manage my files. I’ll do some test runs but initial signs point to this not being favorable for me in how the source folder is not stored as meta data on the imported files. For many of us, we may have organized our photos in folders like 'Trip to Alaska 2019" and that information seems to be lost.
- If you are letting PhotoPrism write to your Originals folder, you do not want to put sidecar files in another folder. Sidecar files are small .yml files an do not add any real measurable size to your collection. They should remain alongside your originals. As my Originals folder is read-only, I had to store these sidecare files elsewhere.