wiserain/cloud-media-scripts repository overview

⁠Usage

Default settings use ~100GB for local media, remove atleast 80 GB and Plexdrive chunks and cache are removed after 24 hours:

docker create \
	--name cloud-media-scripts \
	-v /media:/local-media:shared \
	-v /mnt/external/media:/local-decrypt:shared \
	-v /configurations:/config \
	-v /mnt/external/plexdrive:/chunks \
	-v /logs:/log \
	--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse \
	madslundt/cloud-media-scripts

If you have more space you can increase REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB, FREEUP_ATLEAST_GB and either increase CLEAR_CHUNK_AGE or add CLEAR_CHUNK_MAX_SIZE.

Example of having REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB set to 2TB, FREEUP_ATLEAST_GB to 1TB and CLEAR_CHUNK_MAX_SIZE to 1TB:

docker create \
	--name cloud-media-scripts \
	-v /media:/local-media:shared \
	-v /mnt/external/media:/local-decrypt:shared \
	-v /configurations:/config \
	-v /mnt/external/plexdrive:/chunks \
	-v /logs:/log \
	-e CLEAR_CHUNK_MAX_SIZE="1000G" \
	-e REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB="2000" \
	-e FREEUP_ATLEAST_GB="1000" \
	--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse \
	madslundt/cloud-media-scripts

⁠Parameters

The parameters are split into two halves, separated by a colon, the left hand side representing the host and the right the container side. For example with a volume -v external:internal - what this shows is the volume mapping from internal to external of the container. Example -v /media:/local-media would expose directory /local-media from inside the container to be accessible from the host's directory /media.

OBS: Some of the volumes need to have :shared appended to it for it to work. This is needed to have the files visible for the host. Example -v /media:/local-media:shared.

:shared is also needed on if you mount these folders to your other Docker containers.

Volumes:

• -v /local-media - Union of all files stored on cloud and local - Append :shared
• -v /local-decrypt - Local files stored on disk - Append :shared
• -v /config - Rclone and plexdrive configurations
• -v /chunks - Plexdrive cache chunks
• -v /data/db - MongoDB database
• -v /log - Log files from mount, cloudupload and rmlocal
• -v /cloud-encrypt - Cloud files encrypted synced with Plexdrive. This is empty if ENCRYPT_MEDIA is 0. - Append :shared
• -v /cloud-decrypt - Cloud files decrypted with Rclone - Append :shared

Environment variables:

• -e ENCRYPT_MEDIA - If media is or should be encrypted. 0 means no encryption and 1 means encryption (default 1)
• -e BUFFER_SIZE - Rclone: Buffer size when copying files (default 500M)
• -e MAX_READ_AHEAD - Rclone: The number of bytes that can be prefetched for sequential reads (default 30G)
• -e CHECKERS - Rclone: Number of checkers to run in parallel (default 16)
• -e RCLONE_CLOUD_ENDPOINT - Rclone: Cloud endpoint (default gd-crypt:)
• -e RCLONE_LOCAL_ENDPOINT - Rclone: Local endpoint (default local-crypt:) - this is ignored when ENCRYPT_MEDIA is 0.
• -e CHUNK_SIZE - Plexdrive: The size of each chunk that is downloaded (default 10M)
• -e CLEAR_CHUNK_MAX_SIZE - Plexdrive: The maximum size of the temporary chunk directory (empty as default)
• -e CLEAR_CHUNK_AGE - Plexdrive: The maximum age of a cached chunk file (default 24h) - this is ignored if CLEAR_CHUNK_MAX_SIZE is set
• -e MONGO_DATABASE - Mongo database used for Plexdrive (default plexdrive)
• -e DATE_FORMAT - Date format for loggin (default +%F@%T)
• -e REMOVE_LOCAL_FILES_BASED_ON - Remove local files based on space, time or instant (default space)
• -e REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB - Remove local files when local storage exceeds this value in GB (default 100) - this is ignored if REMOVE_LOCAL_FILES_BASED_ON is set to time or instant
• -e FREEUP_ATLEAST_GB - Remove atleast this value in GB on removal (default 80) - this is ignored if REMOVE_LOCAL_FILES_BASED_ON is set to time or instant
• -e REMOVE_LOCAL_FILES_AFTER_DAYS Remove local files older than this value in days (default 10) - this is ignored if REMOVE_LOCAL_FILES_BASED_ON is set to space or instant
• -e READ_ONLY If Rclone and Plexdrive should be read only or not. 0 means writeable and 1 means read only (default 1)
• -e PLEX_URL If you want to use empty trash script you have to provide the url for your Plex Media Server (default empty).
• -e PLEX_TOKEN If you want to use empty trash script you have to provide the plex token for your Plex Media Server (default empty).
• -e PGID Group id
• -e PUID User id
• -e CLOUDUPLOADTIME - When to run cloudupload using cron expression (default 0 1 * * * ) set to "0 0 31 2 0" to disable
• -e RMDELETETIME -- When to run cloudupload using cron expression (default 0 6 * * *) set to "0 0 31 2 0" to disable

--privileged --cap-add=MKNOD --cap-add=SYS_ADMIN --device=/dev/fuse must be there for fuse to work within the container.

If using docker-compose:

privileged: true
   devices:
     - /dev/fuse
   cap_add:
     - MKNOD
     - SYS_ADMIN

⁠Setup

After the docker image has been setup and running, Rclone and Plexdrive need to be configured.

⁠Rclone

Setup Rclone run docker exec -ti <DOCKER_CONTAINER> rclone_setup

⁠With encryption

3 remotes are needed when using encryption:

1. First one is for the Google drive connection
2. Second one is for the Google drive on-the-fly encryption/decryption
3. Third and last one is for the local encryption/decryption

• Endpoint to your cloud storage.
  • Create new remote [Press N]
  • Give it a name example gd
  • Choose Google Drive [Press 8]
  • If you have a client id paste it here or leave it blank
  • Choose headless machine [Press N]
  • Open the url in your browser and enter the verification code
• Encryption and decryption for your cloud storage.
  • Create new remote [Press N]
  • Give it the same name as specified in the environment variable RCLONE_CLOUD_ENDPOINT but without colon (:) (default gd-crypt)
  • Choose Encrypt/Decrypt a remote [Press 5]
  • Enter the name of the endpoint created in cloud-storage appended with a colon (:) and the subfolder on your cloud. Example gd:/Media or just gd: if you have your files in root in the cloud.
  • Choose how to encrypt filenames. I prefer option 2 Encrypt the filenames
  • Choose to either generate your own or random password. I prefer to enter my own.
  • Choose to enter pass phrase for the salt or leave it blank. I prefer to enter my own.
• Encryption and decryption for your local storage.
  • Create new remote [Press N]
  • Give it the same name as specified in the environment variable RCLONE_LOCAL_ENDPOINT but without colon (:) (default local-crypt)
  • Choose Encrypt/Decrypt a remote [Press 5]
  • Enter the encrypted folder: /cloud-encrypt. If you are using subdirectory append it to it. Example /cloud-encrypt/Media
  • Choose the same filename encrypted as you did with the cloud storage.
  • Enter the same password as you did with the cloud storage.
  • Enter the same pass phrase as you did with the cloud storage.

⁠Without encryption

1 remote is needed to connect rclone to Google drive:

• Endpoint to your cloud storage.
  • Create new remote [Press N]
  • Give it the same name as specified in the environment variable RCLONE_CLOUD_ENDPOINT but without the colon (:)
  • Choose Google Drive [Press 7]
  • If you have a client id paste it here or leave it blank
  • Choose headless machine [Press N]
  • Open the url in your browser and enter the verification code

Rclone documentation if needed click here⁠

⁠Plexdrive

Setup Plexdrive to the cloud. Run the command docker exec -ti <DOCKER_CONTAINER> plexdrive_setup

Plexdrive documentation if needed click here⁠

⁠Commands

Upload local files to cloud run: docker exec <DOCKER_CONTAINER> cloudupload

Remove local files run docker exec <DOCKER_CONTAINER> rmlocal

Check if everything is running docker exec <DOCKER_CONTAINER> check

Empty trash on Plex Media Server but only if mount is up docker exec <DOCKER_CONTAINER> emptytrash

cloudupload and rmlocal can be ran with arguments. All arguments are passed to rclone. For example it is possible to run docker exec <DOCKER_CONTAINER> cloudupload -v to get verbose on the rclone operations in cloudupload.

⁠Cron jobs

Setup cron jobs to upload and remove local files:

• @daily docker exec <DOCKER_CONTAINER> cloudupload
• @weekly docker exec <DOCKER_CONTAINER> rmlocal

⁠How this works?

Following services are used to sync, encrypt/decrypt and mount media:

• Plexdrive
• Rclone
• UnionFS

When using encryption this gives us a total of 5 directories:

• /cloud-encrypt: Cloud data encrypted (Mounted with Plexdrive)
• /cloud-decrypt: Cloud data decrypted (Mounted with Rclone)
• /local-decrypt: Local data decrypted that is yet to be uploaded to the cloud
• /chunks: Plexdrive temporary files and caching
• /local-media: Union of decrypted cloud data and local data (Mounted with Union-FS)

When NOT using encryption this gives us a total of 4 directories:

• /cloud-decrypt: Cloud data decrypted (Mounted with Plexdrive)
• /local-decrypt: Local data decrypted that is yet to be uploaded to the cloud
• /chunks: Plexdrive temporary files and caching
• /local-media: Union of decrypted cloud data and local data (Mounted with Union-FS)

All Cloud data is mounted to /cloud-encrypt. This folder is then decrypted and mounted to /cloud-decrypt. If ENCRYPT_MEDIA is turned off cloud data is mounted directly to /cloud-decrypt. d A local folder (/local-decrypt) containing local media that is yet to be uploaded to the cloud. /local-decrypt and /cloud-decrypt is then mounted to a third folder (/local-media) with certain permissions - /local-decrypt with Read/Write permissions and /cloud-decrypt with Read-only permission.

Everytime new media is retrieved it should be added to /local-media. By adding files to /local-media it is added to /local-decrypt because of the Read/Write permissions. That is why a cronjob is needed to upload local files from /local-decrypt.

By having a cronjob to rmlocal it will sooner or later move media from /local-decrypt depending on the REMOVE_LOCAL_FILES_BASED_ON setting. Media is only removed from /local-decrypt and still appears in /local-media because it is still be accessable from the cloud.

If REMOVE_LOCAL_FILES_BASED_ON is set to space it will only remove content (if local media size has exceeded REMOVE_LOCAL_FILES_WHEN_SPACE_EXCEEDS_GB) starting from the oldest accessed file and will only free up atleast FREEUP_ATLEAST_GB. If time is set it will only remove files older than REMOVE_LOCAL_FILES_AFTER_DAYS. If instant is set it will remove all files when running.

Media is never deleted locally before being uploaded successful to the cloud.

⁠Rclone

Rclone 1.39 is currently used and tested.

Rclone is used to encrypt, decrypt and upload files to the cloud. It mounts and decrypts Plexdrive to a different folder (/cloud-decrypt) and later encrypts and uploads from a local folder (/local-decrypt) to the cloud.

Rclone creates one config file in /config: config.json. This is used to stored Google Drive api keys and encryption/decryption keys.

⁠Plexdrive

Plexdrive 4.0.0 is currently used and tested.

Plexdrive is used to mount Google Drive to a local folder (/cloud-encrypt).

Plexdrive create two files in /config: config.json and token.json. These are used to store Google Drive api keys.

⁠UnionFS

UnionFS is used to mount both cloud and local media to a local folder (/local-media).

• Cloud storage /cloud-decrypt is mounted with Read-only permissions.
• Local storage /local-decrypt is mounted with Read/Write permissions.

The reason for these permissions are that when writing to the local folder (/local-media) it will not try to write it directly to the cloud storage /cloud-decrypt, but instead to the local storage (/local-decrypt). Later this will be encrypted and uploaded to the cloud by Rclone.

⁠Build Dockerfile

⁠Build

docker build -t cloud-media-scripts .

⁠Test run

docker run --name cloud-media-scripts -d cloud-media-scripts

If you want to support the project or just buy me a beer I accept Paypal and bitcoins.

BitCoin address: 18fXu7Ty9RB4prZCpD8CDD1AyhHaRS1ef3