File Archiving (IM Templates-Customization)

Disclaimer: The content provided in this series is formatted as a step-by-step narrative to provide a prescriptive sequence for a common customization. Since your implementation will undoubtedly be different than shown here, everyone should consider this as a conceptual guideline to resolve a custom requirement.

Description

In this scenario, we will illustrate the customization of the generated code from the IM template to accomplish some additional functionality not supported by the templates. In our new scenario, we have two distinct customizations:

First, we have a CSV file coming from the customer and we use it to import data for a single company parameter table. However, the requirement is that this same file could be used for other integrations, for example customer extension table.
Second, do not delete the input CSV file
Third, keep an archive and provide a date and timestamp for each archive.

The following are the list of steps for this scenario:

1 Prerequisites
2 Step 1: Connect to IM Instance
3 Step 2: Clone to Local
4 Step 3: Password Updates
5 Step 4: Start Local Application
6 Step 5: Manage Archive File
7 Step 6: Local Route Execution

Prerequisites

In order to be able to complete this exercise, we will need the following prerequisites before proceeding.

PlatformManager access
An existing IM instance
Access privileges for:
- Partition creation
- Git repository
- Pricefx Maven repository
Partition Connection
SFTP Connection
One functioning Route
WinSCP FTP Client or equivalent
Cloned IntelliJ project

Step 1: Connect to IM Instance

Prior to local development, we will need to follow the path from the IM instance within PlatformManager to the Git repository that contains the contents of our integration project. Our goal will be the to open the Git repository linked to our IM instance.

Go to platform.pricefx.com and choose the Login with O365 option:

2. Click on the Integrations option (left-hand side):

3. From the list of integration instances, create a filter to locate your IM instance:

4. Locate your IM instance and click on it, this will open the IM instance page:

On the left-hand navigation, we will see all options related to this instance:

NOTE: We have selected our single IM instance and all of this options (Routes, Mappers, Connections, etc) are reflective of this single instance.

6. Next, we will open the Git repository linked to this IM instance:

NOTE: The creation of IM instance will automatically build a corresponding Git repository that will then contain all of the generated elements that are linked to this instance.

7. Click on the Git repository link to open the Git project:

Examine what is provided, click on the src/main folder.

Step 2: Clone to Local

Our objective in this step is to customize the generated XML and Json code associated with our IM instance and the first step in that process is to clone this project to our local machine.

Click on the Clone option:

2. Next, click on the Copy URL option to copy the URL path.

3. Then, we will open IntelliJ and use the File | New | Project from Version Control to connect to our remote Git repository:

4. Next, this will display the Get from Version Control panel and we will paste the URL path:

Click on the Clone button.

5. The project has now been cloned to our local machine and this local instance is connected to our remote repository in gitlab.pricefx.eu.

6. Next, we need to switch to the dev branch within Git and in that branch we will perform all of our updates and modifications to the code. In the lower right-hand corner we can select the remote dev branch:

Then we will choose the Checkout option.

7. Next, lets verify that we have successfully cloned the dev branch from the remote repository to our local machines. So, within Intellij lets open folder path of src/resources/repo and it will appear as:

8/. Next, open the Connections folder to view our connection files:

9. As an example, we can open one of the connection Json files. It will appear as:

10. Next, open the Routes folder to view our route files:

11. It does appear that we have successfully cloned our IM instance Git project to our local machine. We can verify this by comparing the Git folders in src/main/resources/repo to those in Git:

Step 3: Password Updates

Before we can run our integration processes locally, we will need to update the encrypted passwords with actual clear text passwords.

1. First, we will need to locate the connections being used by our integration instance. Open the folder path src/main/resources/repo/connections:

2. There are two different types of connection files that will need to be updated. One for the SFTP connection and the other for the partition.

3. When we open our connection JSON file we should find an encrypted password:

4. We will need to update the connection passwords with the clear text versions that were provided (via email) when the integration instance was created.

Step 4: Start Local Application

After the updates to our connections, we can start the local process via IntelliJ that will allow us to execute our routes through our local machine.

First, open the folder main/java/pricefx and locate the LocalAppRunner application:

2. Open the LocalAppRunner file:

3. Prior to launching our local instance, we will need to ensure that any Routes connected to this IM instance have been stopped. Go to Routes for our IM instance:

We should verify that the Status for our routes has been changed to Stopped:

4. Next, we will launch the LocalAppRunner application. Right click on LocalAppRunner and choose the Run option:

5. In the log file, we are looking for the messages related to the successful launch of any routes that are linked to the IM instance:

Step 5: Manage Archive File

One of our requirements was to create an archived copy of the incoming CSV file and then to provide a date/timestamp for that file.

First, we will need to establish an archive folder within SFTP. Therefore, within /filearea folder of our SFTP server we need to create an archive folder.

2. Next, we need to update the application.properties file to refer to this new archiving location for placement of archived files. Thus, open the application.properties file in the config folder:

3. Add the following attribute at the bottom of the file that references our new SFTP archive folder:

pfx\:import-csv-from-ftp-to-pricefx.sftp.archive=/filearea/archive

4. Next, we will make a copy of an existing route that we will modify. Simply, select an existing route in IntelliJ, right click and select Copy option:

5. Update the route name to make it unique, in this case we’ve added multicast to the end of the name. Then, Click OK button.

6. Next, we add the new file to Git by clicking Add button:

7. Next, we will begin the process of adding the move option that will use our SFTP connection and identify the correct directory for our archive. So, open the new route XML file and locate the <route> entry:

8. Next, within the <route>, look for the <from> tag in the route XML file:

9. Now, to the <from> tag we will add a MOVE option that will reference our new archive folder on the SFTP server. If you scan this <from> tag you will see entries for:

a. connection, that looks like this:

connection={{pfx:import-csv-from-ftp-to-pricefx.sftp.connection}}

b. directory, that looks like this:

directory={{pfx:import-csv-from-ftp-to-pricefx.sftp.directory}}

10. Next, after the directory option, we will add the following move command that will reference our archive folder. The statement looks like this:

move={{pfx:import-csv-from-ftp-to-pricefx.sftp.archive}}/${file:name}

--${date:now:yyyyMMdd-HHmmss}&

The entire statement might appear as:

<from uri="pfx-sftp:parameters?connection={{pfx:import-csv-from-ftp-to-pricefx.sftp.connection}}&amp;
directory={{pfx:import-csv-from-ftp-to-pricefx.sftp.directory}}&amp;
move={{pfx\:import-csv-from-ftp-to-pricefx.sftp-archive}}/$(file:name)--${date:now:yyyyMMdd-HHmmss}&amp;
{{pfx:import-csv-from-ftp-to-pricefx.archive.file.clause}}&amp;
moveFailed=.error&amp;
streamDownload=true&amp;
stepwise=false&amp;
useUserKnownHostsFile=false{{pfx:import-csv-from-ftp-to-pricefx.done.file.clause}}"/>

Step 6: Local Route Execution

Start the local instance of IM and test the route to verify that the archive option is working.

Next, we will launch the LocalAppRunner application. Right click on LocalAppRunner and choose the Run option:

2. In the log file, we are looking for the messages related to the successful launch of any routes that are linked to the IM instance:

Now that the local instance is running, we can start the route by adding a new CSV file to our inbound folder that the route is polling.

3. Now that the local instance is running, and the routes on PlatformManager/IntegrationManager are stopped, all polling of the SFTP folders is being performed by our local instance.

4. Go to SFTP folder view in WinSCP for our inbound route:

5. Drag and drop the CSV file associated with this inbound route for products.

6. In IntelliJ and within the log entries for LocalAppRunner instance, we should see the CSV file being consumed and reported:

7. Finally, return to SFTP folders and open the filearea/archive folder to see our archive: