Before we added Mitto Rclone jobs in 2.8, users created CMD jobs to use rclone to transfer files in and out of a Mitto instance.
That work flow is still possible in 2.9, but the process is more involved.
One example scenario that might still require the usage of CMD jobs: Searching the Mitto files directory for a file with today’s date in the name, and then uploading that file to an ftp server with another name.
If you wanted to upload the file keeping the same name, you could do this using the rclone --include tag matching on today’s date (ex: 2021_05_07). But using --include with rclone copyto, necessary to save on the destination with a new name, doesn’t work.
In Mitto 2.8+ an rclone job that includes files with 2021_05_07 looks like this:
NOTE it is possible this search could copy multiple files.
{
"command": "copy",
"credentials": "my sFTP",
"rclone_flags": [
{
"flag": "--include",
"value: "filename_2021_05_07*"
]
"targets": {
"source": "/var/mitto/data/"
"destination": ":sftp:/path/to/destination"
}
"timeout_seconds": 18000
}
The same thing in a command job using an rclone remote defined in a rclone config file like this:
[some-stfp]
type = sftp
host = {hostname}
user = {username}
pass = {encrypted password}
… and the command job:
{
"cmd": "rclone copy /var/mitto/data/ --include \"filename_`date --date=today +%Y_%m_%d`*\" some-sftp:path/to/dest/ --config /var/mitto/data/rclone.conf",
"cmd_env": {},
"exec": false,
"shell": true
}
… or without an rclone config file:;
{
"cmd": "rclone copy /var/mitto/data/ --include \"filename_`date --date=today +%Y_%m_%d`*\" --sftp-host={hostname} --sftp-user={username} --sftp-password={password} :sftp:path/to/dest/",
"cmd_env": {},
"exec": false,
"shell": true
}
Since this is a command job, we can inject todays date into the command using back ticks. Essentially the command date --date=today +%Y_%m_%d will be evaluated (today it returns 2021_05_07) before the full command is run.
As mentioned earlier if you want to save a file on the destination with a new name like filename.csv using --include won’t work. However, if you know filename_2021_05_07* will only match one file, it is possible with a little shell scripting.
The command job will look like this (bash script will be explained below):
{
"cmd": "bash /var/mitto/data/rclone-copyto.sh > /var/mitto/data/rclone-copyto-output.log 2>&1",
"cmd_env": {},
"exec": false,
"shell": true
}
The above command will run the following shell script, logging the output/errors to a file named rclone-copyto-output.log:
#!/bin/bash
filename="filename_`date --date=today +%Y_%m_%d`*"
cd /var/mitto/data
filename=`ls $filename`
today="_`date --date=today +%Y_%m_%d`"
new_filename=`echo $filename | sed s/$today//g`
rclone copyto /var/mitto/data/$filename some-sftp:path/to/dest/$new_filename --config /var/mitto/data/rclone.conf
The first 5 lines of this script sets up some variables for the filename of today’s file, and today’s date. (One improvement would be to check if the file exists and exit gracefully.)
new_filename=`echo $filename | sed s/$today//g`
This line sets a variable for the new filename stripping the date out of the filename.
Finally, using the variables we just made, we rclone copyto the file from Mitto to the sftp server with the new filename.