gemail/rsync-dir repository overview

⁠Rsync-Dir - directory replicator

Rsync-Dir is program (available for Windows and Linux) designed to replicate a specified directory to a data store:

• etcd – distributed key-value data store,
• redis – popular NoSQL database (also distributed key-value cache),
• minio – object storage compatible with Amazon S3. This can be either the AWS (Amazon) S3 cloud service or a Minio S3 server instance (local or cloud),
• dbproxy – any database for which a JDBC driver is available (ssldbproxy.jar component is required).

Only files available in the main directory or the entire tree can be replicated. This is determined by recursive parameter. The application also allows you to specify which types of files are subject to replication and which of them are to be ignored. There are also two replication modes available:

• instant replication (start) – the file is sent to the data store immediately after a change is detected,
• cyclic replication (mtime) – detection of changes in files and their sending to data store occurs cyclically at specified time intervals. Both types of replication can interact with each other and can be activated simultaneously, monitoring the specified directory.

In order to improve the efficiency of sending a file to the data store, the program has implemented a mechanism for dividing files into smaller fragments (called chunks), for which checksums are calculated, thanks to which only those fragments of the file that have changed are sent to the store, not entire files. In the case of deleting a file from a directory, this file can be deleted from the data store, when the assumption is to reflect the state of the monitored directory in the data store. This is decided by the delete action in the actions parameter. The program also supports changing the file name.

The program has implemented several available actions taken by the application:

• start – basic action to start monitoring the specified directory – activating instant replication, in which the file is sent to the data store immediately after detecting a change,

• mtime – activating cyclic replication – changes in files are detected cyclically,

• restore – restoring the state of the directory based on the data store. This action can be standalone and can be the only action taken by the application. It can be used to periodically restore the state of the directory on another server/node/pod,

• delete – deleting all files from the datastore. This action can be standalone and may be the only action taken by the application. It can be used to reset the datastore from previously saved data.

The restore and delete actions can be combined with the start and mtime actions. Then the order of execution of the actions is as follows:

• restore - first, the application recreates the directory based on the records in the data store,
• delete – then the data store is cleared,
• start – the continuous catalog monitoring process is started,
• mtime – the process of detecting changes in the directory is run cyclically.

⁠Application architecture

⁠Usage

You can run the default rsync-dir command simply (synchronization with Amazon S3):

docker run -itd --name rsync-dir \
     --env ENV_RSYNCDIR_BUCKET_S3="AWS_BUCKET_S3" \
     --env ENV_RSYNCDIR_ACCESS_KEY_S3="AWS_ACCESS_KEY_S3" \
     --env ENV_RSYNCDIR_SECRET_KEY_S3="AWS_SECRET_KEY_S3" \
     --env ENV_RSYNCDIR_LICENSE="LICENSE_STRING" \
  gemail/rsync-dir:latest

If the variables indicated above are specified correctly, the correctness of the connection to S3 can be checked with the command:

docker logs rsync-dir

Correct startup looks like this:

Starting Rsync-Dir...
Rsync-DIR v. 1.78 - dir replicator
EoL:20251230
2025-06-27 10:33:08 - Rsync-DIR                         START
2025-06-27 10:33:08 - Action: RESTORE                   START
2025-06-27 10:33:08 - Connected to MINIO[s3.amazonaws.com:443]
2025-06-27 10:33:09 - File:/usr/rsync-dir/data/test.txt checked
2025-06-27 10:33:09 - File:/usr/rsync-dir/data/test.txt read chunk nr: 0
2025-06-27 10:33:09 - File:/usr/rsync-dir/data/test.txt identical - not restored
2025-06-27 10:33:09 - Action: RESTORE                   STOP
2025-06-27 10:33:09 - Action: DELETE                    START
2025-06-27 10:33:09 - Connected to MINIO[s3.amazonaws.com:443]
2025-06-27 10:33:10 - File:rsync-base/test.txt deleted from MINIO
2025-06-27 10:33:10 - Action: DELETE                    STOP
2025-06-27 10:33:10 - Action: START                     START

The most common problem is providing an incorrect license, which is indicated as follows:

Starting Rsync-Dir...
Rsync-DIR v. 1.78 - dir replicator
Error: License not valid or not correct
/usr/rsync-dir

You may also experience authentication issues on the Amazon S3 side if you provide incorrect values ​​for:

• ENV_RSYNCDIR_BUCKET_S3,
• ENV_RSYNCDIR_ACCESS_KEY_S3,
• ENV_RSYNCDIR_SECRET_KEY_S3.

Starting Rsync-Dir...
Rsync-DIR v. 1.78 - dir replicator
EoL:20251230
2025-06-27 11:00:22 - Rsync-DIR                         START
2025-06-27 11:00:22 - Action: RESTORE                   START
2025-06-27 11:00:22 - Connected to MINIO[s3.amazonaws.com:443]
2025-06-27 11:00:22 - S3 operation failed; code: SignatureDoesNotMatch, message: The request signature we calculated does not match the signature you provided. Check your key and signing method.(...)
2025-06-27 11:00:22 - Action: RESTORE                   STOP

⁠Verifying directory synchronization

To verify operation, you can log in to the container:

docker exec -it rsync-dir bash

and change file test.txt in directory /usr/rsync-dir/data (default monitored directory).

After that check log:

docker logs rsync-dir

or log in to Amazon S3 console and check bucket indicated in the variable ENV_RSYNCDIR_BUCKET_S3.

⁠List of environment variables

• ENV_RSYNCDIR_LICENSE - license string. You can get a free license for three days on the website IT-DT store⁠
• ENV_RSYNCDIR_CFG_INI - path to configuration file, for example /config/rsync-dir.ini
• ENV_RSYNCDIR_CMD - list of command to execute before start rsync-dir, for example "ln -sf / <DIR IN ${ENV_RSYNCDIR_CFG_INI}>/DIR2;ln -sf / <DIR IN ${ENV_RSYNCDIR_CFG_INI}>/DIR4;"

List only for default configuration:

• ENV_RSYNCDIR_BUCKET_S3 - name of AWS_BUCKT_S3 in Amazon S3
• ENV_RSYNCDIR_ACCESS_KEY_S3 - access key in Amazon S3
• ENV_RSYNCDIR_SECRET_KEY_S3 - secret key in Amazon S3
• ENV_RSYNCDIR_DIR - dir to monitoring, for example ENV_RSYNCDIR_DIR="\/backup"
• ENV_RSYNCDIR_PROXY - proxy setting, for exmaple ENV_RSYNCDIR_PROXY="http://192.168.1.10:9999⁠"

⁠Kubernetes

In Kubernetes clusters without persistent volumes support - as we know, pods are ephemeral and all data saved by the application on disk is lost in the event of a pod failure. If we want the application to have access to previously created files after a failure, we can use rsync-dir, which will continuously replicate files generated by the application to the selected data store. After a pod failure, by using e.g. an initialization container, we can restore previously replicated files to the state before the failure, and after restoring the files, run the basic container with the appropriate application and with access to previously created files

⁠Kubernetes configuration

⁠Create configmap with own configuration file

Based on documentation of rync-dir (you will receive it after purchasing the license) you need to prepare your own configuration file and create configmap:

kubectl create configmap rsync-dir-cm \
   --from-file=my-rsync-dir.ini \
   --from-literal=ENV_RSYNCDIR_CFG_INI=/config/my-rsync-dir.ini 

⁠Create initial container and add it to pod with container with monitored directory

spec:
     containers:
      - name: rsync-dir
        image: docker.io/gemail/rsync-dir:latest
        env:
          - name: ENV_RSYNCDIR_CFG_INI
            valueFrom:
              configMapKeyRef:
                name: rsync-dir-cm
                key: ENV_RSYNCDIR_CFG_INI
        volumeMounts:
          - name: config
            mountPath: /config
     volumes:
       - name: config
         configMap:
           name: rsync-dir-cm

⁠Examples of applications

The rsync-dir program can be used in many ways. Here are some examples of its use.

• in Kubernetes clusters without persistent volumes support - as we know, pods are ephemeral and all data saved by the application on disk is lost in the event of a pod failure. If we want the application to have access to previously created files after a failure, we can use rsync-dir, which will continuously replicate files generated by the application to the selected data store. After a pod failure, by using e.g. an initialization container, we can restore previously replicated files to the state before the failure, and after restoring the files, run the basic container with the appropriate application and with access to previously created files,
• for replication of transaction files in application servers (e.g. jboss, tomcat, weblogic) working in standalone mode, and after a failure run on another server, access to transaction files can allow for the correct handling of distributed transactions,
• for replication of logical database log files, e.g. in the case of PostgreSQL for current replication of WAL files - especially the current file, to which continuous writing takes place and it is not yet archived with the command defined in the archive_command parameter, in the case of the need to restore the database system from a backup, access to this file can minimize the amount of lost data,
• for backup of Linux server configuration files - monitoring changes in /etc/* files, the backup takes place immediately after the change, so in the event of a server failure we have a complete set of current configuration files,
• for High Availability applications for applications generating files needed for the application to work (e.g. application status files, session files, files with recently handled messages, etc.), after the primary server failure we can first restore files replicated with the rsync-dir program on the backup server, and then run the application that will read the status files and start working from the point shortly before the failure.
• for transferring files between separated systems – files created on the primary server are saved, for example, to the s3 storage, and then read cyclically on another server and made available to another application, e.g. a web application.