Integration Backup (New)
- Ondřej Tesař
The Data Backup script backs up integration data stored in the /var/pricefx/data and /home/customer/<customer>/filearea directories.Â
How It Works
The script scans the /var/pricefx/integration/data and /home/customer/<customer>/filearea directories recursively.
If exclude list contains some paths, the backup script will not back up these folders and subfolders.
The output directory is /var/pricefx/integration/data-archive.
The script preserves the files in archive for 400 days.
The script stores files older than 30 days. Newer files stays in the directories untouched.
 | File | Note |
|---|---|---|
script | /home/ops/bin/integration_backup.sh | Backups of 40 days old files. |
cron | /etc/cron.d/integration_backup | Runs every day at 3 am. |
input | /var/price/integration/data/ | Scans all directories and its subdirectories. |
exclude list | /home/ops/bin/integration_backup_exclude_list.txt | Contains folders which are excluded from backup, such as: If such file does not exist, than everything is backed up. Exclude folders are managed by salt and can be fine grained per server. Do not edit this file on the server. If you need to exclude some folder from backup, create a support ticket. Ops team will create salt pillars specially for this server. Example below. |
output | /var/pricefx/integration/data-archive | Contains all archived files for date minus 400 days. |
log | /var/log/integration/backup/integration_backup_YYYY-mm-DD_HH-MM-SS.log | Log file of executed integration_backup.sh. |
How to Set It Up
Ask Ops team to apply salt state called "im.backup" against the desired server.
OR
Do it yourself:
salt-call state.apply im.backupWhere It Is Applied
Server | Applied | Note |
|---|---|---|
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
yes | ||
no | salt-agent is not responding - seems like entire server is in bad condition. Raid is broken etc. Jiri Kokosek is taking care of it. |
Total: 125
Done: 124
Planned Development
Distribute the backup script over salt once per a period of time to be sure all IM servers are compliant with integration backup standard.
Examples
Definition of Backup Folders
We have a map and we want to back up customer and data folders.
BACKUP_FOLDERS=( ["customer"]="/home/customer" ["data"]="/var/pricefx/integration/data" )Definition of Other Properties
ARCHIVE_FOLDER="/var/pricefx/integration/data-archive"
EXCLUDE_LIST_LOCATION="/home/ops/bin"
EXCLUDE_LIST_FILENAME="integration_backup_exclude_list.txt"
RETENTION_FILES_CUSTOMER="30"
RETENTION_FILES_DATA="10"
RETENTION_ARCHIVE_FILES="400"
ACTUAL_TIMESTAMP=$(date +%F_%H-%M-%S)
LOG_FILE_FOLDER="/var/log/integration/backup"
LOG_FILE="$LOG_FILE_FOLDER/integration_backup_$ACTUAL_TIMESTAMP.log"
#prefix will be compound from backup folders key + customer(subfolder) + subsubfolders
INTEGRATION_BACKUP_FILE_SUFFIX="integration_backup_$ACTUAL_TIMESTAMP.tar.gz"RETENTION_FILES_CUSTOMER, RETENTION_FILES_DATA and RETENTION_ARCHIVE_FILES values can be configured per server via salt pillars.
Exception for Customer Folders
Because we do not want to back up the entire filearea folder which is in the /home/customer/<customer>/ folder, we have an exception that we will go right into the filearea folder and start the backup from there.
Exception for Exclude List
Sometimes in the backup folders we have folders which we do not want to back up for some reason.
Example of Pillar File Overriding ValuesÂ
IntegrationManager version 5.8.0