The Data Backup script backs up integration data stored in the /var/pricefx/data and /home/customer directories.

How It Works

The script scans the /var/pricefx/data and /home/customer directories recursively.
The output directory is /var/pricefx/archive/YYYY/MM/DD.
The script preserves the files in archive for 360 days.
The script stores files older than 3 days. Newer files stays in the directories untouched.

	File	Note

	File	Note
script	/home/ops/bin/integration_backup.sh	Backups of 3 days old files.
cron	/etc/cron.d/integration_backup	Runs every day at 3 am.
input	/var/price/data/ /home/customer	Scans all directories and its subdirectories.
output	/var/price/archive/YYYY/mm/dd	Contains all archived files for date minus 3 days.
log	/var/log/integration_backup_YYYY-mm-DD_HH-MM-SS.log	Log file of executed integration_backup.sh

How to Set It Up

Ask Ops team to apply salt state called "im.backup" against the desired server.

OR

Do it yourself:

salt-call state.apply im.backup

Where It Is Applied

Server	Applied

Server	Applied
node1.coes-qa.pricefx.net	yes
node1.tfg.pricefx.net	no

Planned Development

Allow to configure things like files retention or archive retention by parameters provided to script

Examples

Provisioning

root@node1.arm-qa.pricefx.net ~/ops # salt-call state.apply im.backup

local:
----------
          ID: /home/ops/bin/integration_backup.sh
    Function: file.managed
      Result: None
     Comment: The file /home/ops/bin/integration_backup.sh is set to be changed
     Started: 13:01:23.201098
    Duration: 421.757 ms
     Changes:   
              ----------
              newfile:
                  /home/ops/bin/integration_backup.sh
----------
          ID: /home/ops/bin/im_browse_backup_files.sh
    Function: file.managed
      Result: None
     Comment: The file /home/ops/bin/im_browse_backup_files.sh is set to be changed
     Started: 13:01:23.622952
    Duration: 109.189 ms
     Changes:   
              ----------
              newfile:
                  /home/ops/bin/im_browse_backup_files.sh
----------
          ID: /home/ops/bin/im_unpack_backup_file.sh
    Function: file.managed
      Result: None
     Comment: The file /home/ops/bin/im_unpack_backup_file.sh is set to be changed
     Started: 13:01:23.732282
    Duration: 96.032 ms
     Changes:   
              ----------
              newfile:
                  /home/ops/bin/im_unpack_backup_file.sh
----------
          ID: /etc/cron.d/integration_backup
    Function: file.managed
      Result: None
     Comment: The file /etc/cron.d/integration_backup is set to be changed
     Started: 13:01:23.828439
    Duration: 106.861 ms
     Changes:   
              ----------
              newfile:
                  /etc/cron.d/integration_backup

Summary for local
------------
Succeeded: 4 (unchanged=4, changed=4)
Failed:    0
------------
Total states run:     4
Total run time: 733.839 ms
root@node1.arm-qa.pricefx.net ~/ops #

Usage of integration_backup.sh

root@node1.coes-qa.pricefx.net ~ops/bin # ./integration_backup.sh
2019-02-07_12-21-14: STARTING ARCHIVING SCRIPT
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Backup folders: 					/home/customer /var/pricefx/data
2019-02-07_12-21-14: Archive folder: 					/var/pricefx/archive
2019-02-07_12-21-14: Integration backup file: 			/var/pricefx/archive/2019/02/07/integration_backup_2019-02-07_12-21-14.tar
2019-02-07_12-21-14: Log file: 							/var/log/integration_backup_2019-02-07_12-21-14.log
2019-02-07_12-21-14: Retention for integration files: 	30
2019-02-07_12-21-14: Retention for backup files: 		180
2019-02-07_12-21-14: Temporary backup file: 			/tmp/tmp.P6WCpzdVbP
2019-02-07_12-21-14: Temporary remove backup file: 		/tmp/tmp.ks4bdoVxAW
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Checking validity of backuping folders: /home/customer /var/pricefx/data
2019-02-07_12-21-14: Checking folder /home/customer
2019-02-07_12-21-14: /home/customer is a valid folder!
2019-02-07_12-21-14: Checking folder /var/pricefx/data
2019-02-07_12-21-14: /var/pricefx/data is a valid folder!
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Completing files which has modification time over 30 days to temporary file /tmp/tmp.P6WCpzdVbP
2019-02-07_12-21-14: Files to be archived: 
/home/customer/coesia-qa/filearea/gdm-qa/IN/Error/SALESCONDITION_TABLE__20190110-095140.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Error/TRANSACTION_TABLE__20190110-095119.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/CUSTOMER_TABLE__20190110_095019.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/CONVERSION_TABLE__20190110_094939.CSV
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/PRODUCT_TABLE__20190110_095042.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/PRODUCTEXT_TABLE__20190110_095104.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/PARTNER_TABLE__20190110_095019.csv
/home/customer/coesia-qa/filearea/gdm-qa/IN/Archive/2019/01/PURCHASECOST_TABLE__20190110_095102.csv
Files count: 8 /tmp/tmp.P6WCpzdVbP
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Archiving files to: /var/pricefx/archive/2019/02/07/integration_backup_2019-02-07_12-21-14.tar
tar: Removing leading `/' from member names
2019-02-07_12-21-14: Archiving completed!
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Removing already archived files!
2019-02-07_12-21-14: All files removed!
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: Removing backups older than 180 days.
2019-02-07_12-21-14: No backup candidates for removing found older than 180 days!
2019-02-07_12-21-14: -------------------------------------------------------
2019-02-07_12-21-14: END ARCHIVING SCRIPT
root@node1.coes-qa.pricefx.net ~/ops #

In output above you can see some temporary files. For debugging purposes you can see, for example, which files were marked to be backed up.

Another temporary file /tmp/tmp.ks4bdoVxAW is empty because as you can see that script could not find any archives older than 180 days.

Temporary files are generated on the fly, so they are unique for each script run. You can recognize which temp file belongs to which run by the log file. In the log file there is written the same info as you can see in the output, so it can be easily paired if needed. The script does not remove the temporary files. This can change in the future.

Usage of im_browse_backup_files.sh

Help message

Without any parameters it will show all files in all archives.

You can use both parameters or just one. As you can see, parameters are not case sensitive. Of course, you can use another pipe for grep etc.

Usage of im_unpack_backup_file.sh

Help message

Let's say we need to restore files for transactions from 10.11.2018 for gdm-qa customer. Using the previous script we can do this:

Now you can copy your file anywhere you want (for example back for reprocessing). The file has the same permissions as at the time of archiving.

Do not forget to remove the temporary unpacked folder after you are done.

IntegrationManager Documentation

Integration Backup