File Multicast (Message Routing-Customization)

Owned by Mike Deskis
Last updated: Nov 23, 2022
12 min read

Disclaimer: The content provided in this series is formatted as a step-by-step narrative to provide a prescriptive sequence for a common customization. Since your implementation will undoubtedly be different than shown here, everyone should consider this as a conceptual guideline to resolve a custom requirement.

Description

In this scenario, we will illustrate the customization of the generated code from the IM template to accomplish some additional functionality not supported by the templates.

 In our new scenario, we have these distinct customizations

  • Have multiple data structures in the partition that require the same CSV data

  • DON’T want to create multiple routes using the same input CSV file

  • Need to provide the ability to move a CSV file and replicate into multiple folders on the SFTP server so additional routes can utilize the data to load other data structures in our partition.  

 

The following are the list of steps for this scenario:

Prerequisites

In order to be able to complete this exercise, we will need the following prerequisites before proceeding.

  • PlatformManager access

  • Existing IM instance

  • Access privileges for:

    • Partition creation

    • Git repository access

    • Pricefx Maven repository

  • Partition Connection

  • SFTP Connection

  • One functioning Route

  • WinSCP FTP Client or equivalent

  • Cloned IntelliJ project

Step 1: Connect to IM Instance

Prior to local development and customization, we will need to follow the path from the IM instance within PlatformManager to the Git repository that contains the contents of our integration project. Our goal will be to open the Git repository linked to our IM instance.

  1. Go to platform.pricefx.com and choose the Login with O365 option:

2. Click on the Integrations option (left-hand side):

3. From the list of integration instances, create a filter to locate your IM instance:

4. Locate your IM instance and click on it, this will open the IM instance page:

On the left-hand navigation, we will see all options related to this instance:

NOTE: We have selected our single IM instance and all of this options (Routes, Mappers, Connections, etc) are reflective of this single instance.

5. Next, we will open the Git repository linked to this IM instance:

NOTE: The creation of IM instance will automatically build a corresponding Git repository that will then contain all of the generated elements that are linked to this instance.

6. Click on the Git repository link to open the Git project:

Examine what is provided, click on the src/main folder.

Step 2: Clone to Local

Our objective in this step is to customize the generated XML and Json code associated with our IM instance and the first step in that process is to clone this project to our local machine.

  1. Click on the Clone option:

2. Next, click on the Copy URL option to copy the URL path.

3. Then, we will open IntelliJ and use the File | New | Project from Version Control to connect to our remote Git repository:

4. Next, this will display the Get from Version Control panel and we will paste the URL path:

Click on the Clone button.

5. The project has now been cloned to our local machine and this local instance is connected to our remote repository in gitlab.pricefx.eu.

6. Next, we need to switch to the dev branch within Git and in that branch we will perform all of our updates and modifications to the code. In the lower right-hand corner we can select the remote dev branch:

Then we will choose the Checkout option.

7. Next, lets verify that we have successfully cloned the dev branch from the remote repository to our local machines. So, within Intellij lets open folder path of src/resources/repo and it will appear as:

8. Next, open the Connections folder to view our connection files:

9. As an example, we can open one of the connection Json files. It will appear as:

10. Next, open the Routes folder to view our route files:

11. It does appear that we have successfully cloned our IM instance Git project to our local machine. We can verify this by comparing the Git folders in src/main/resources/repo to those in Git:

 

 

Step 3: Password Updates

Before we can run our integration processes locally, we will need to update the encrypted passwords with actual clear text passwords.

1.. First, we will need to locate the connections being used by our integration instance. Open the folder path

src/main/resources/repo/connections:

2. There are two different types of connection files that will need to be updated. One for the SFTP connection and the other for the partition.

3. When we open our connection JSON file we should find an encrypted password:

4. We will need to update the connection passwords with the clear text versions that were provided (via email) when the integration instance was created.

Step 4: Start Local Application

After the updates to our connections, we can start the local process via IntelliJ that will allow us to execute our routes through our local machine.

First, open the folder main/java/pricefx and locate the LocalAppRunner application:

2. Open the LocalAppRunner file:

3. Prior to launching our local instance, we will need to ensure that any Routes connected to this IM instance have been stopped. Go to Routes for our IM instance:

We should verify that the Status for our routes has been changed to Stopped:

4. Next, we will launch the LocalAppRunner application. Right click on LocalAppRunner and choose the

Run option:

5. In the log file, we are looking for the messages related to the successful launch of any routes that are linked to the IM instance:

Step 5: Replicate CSV File

Before we can fulfill the multicast requirement to place the incoming data file into multiple folders, we will need to move it to ensure that it isn’t deleted after the route is processed.

 

  1. First, we will need to establish an archive folder within SFTP. Therefore, within /filearea folder of our SFTP server we need to create an archive folder.

2. Next, we need to update the application.properties file to refer to this new archiving location for placement of archived files. Thus, open the application.properties file in the config folder:

3. Add the following attribute at the bottom of the file that references our new SFTP archive folder:

pfx\:import-csv-from-ftp-to-pricefx.sftp.archive=/filearea/archive

4. Next, open the route XML file and locate the <route> entry:

5. Next, within the <route>, look for the <from> tag in the route XML file:

6. Now, to the <from> tag we will add a MOVE option that will reference our new archive folder on the SFTP server. If you scan this <from> tag you will see entries for:

a.      connection, that looks like this:

connection={{pfx:import-csv-from-ftp-to-pricefx.sftp.connection}}

b.      directory, that looks like this:

directory={{pfx:import-csv-from-ftp-to-pricefx.sftp.directory}}

7. First, in the <from> tag scroll to the right and locate the delete=true parameter. We will remove this attribute and replace it with our move option.

8. We will add a MOVE option (in replace of the delete=true option)  that will reference our new archive folder on the SFTP server. Add the following move command that will reference our archive folder and assign a time/date stamp. The statement looks like this:

move={{pfx:import-csv-from-ftp-to-pricefx.sftp.archive}}/${file:name}

--${date:now:yyyyMMdd-HHmmss}&amp;

9. Next, after the directory option, we will add the following move command that will reference our archive folder. The statement looks like this:

move={{pfx:import-csv-from-ftp-to-pricefx.sftp.archive}}/${file:name}

--${date:now:yyyyMMdd-HHmmss}&amp;

 The entire statement might appear as:

<from uri="pfx-sftp:parameters?connection={{pfx:import-csv-from-ftp-to-pricefx.sftp.connection}}&amp; directory={{pfx:import-csv-from-ftp-to-pricefx.sftp.directory}}&amp; move={{pfx\:import-csv-from-ftp-to-pricefx.sftp-archive}}/$(file:name)--${date:now:yyyyMMdd-HHmmss}&amp; {{pfx:import-csv-from-ftp-to-pricefx.archive.file.clause}}&amp; moveFailed=.error&amp; streamDownload=true&amp; stepwise=false&amp; useUserKnownHostsFile=false{{pfx:import-csv-from-ftp-to-pricefx.done.file.clause}}"/>

Step 6: Multicast Setup & Configuration

After the file has been moved, then the requirement for the multicast function for replicating an incoming data file into multiple SFTP folders can be performed.  In this step we identify what needs to be completed for a multicast to work.

1.   First, let’s look at the basic multicast option and it will appear as:

<route>     <from uri="direct:a"/>     <multicast>         <to uri="direct:b"/>         <to uri="direct:c"/>         <to uri="direct:d"/>     </multicast> </route>

2. The URI path that is referenced in the multicast can be different than the path used to find the incoming file for the route. This new path will be defined in application.properties.

3. Now we will update the application.properties file. At the bottom of this file add the following statement:

pfx:\:copy-product-multicast=copy-product-multicast-SFTP-inbound-partition-1

4. Next, ensure that you have created a new SFTP folder that will be linked to our SFTP connection. In this example, we have created a folder called extrafile in the /filearea/ folder:

5. Next, we need to create the SFTP connections in PlatformManager for our IM instance. The SFTP connection should appear as:

6. In the Connections folder (in IntelliJ), we should see our connection Json file:

7. Next, we will need to update the connection Json file to add the clear text passwords. In the connections folder open the copy-product-multicast-SFTP-inbound-partition-1 file:

8. Then, locate the encrypted password in the file:

 

9. Replace the encrypted password with the clear text password received in the SFTP server activation.

10. Now, we return to IntelliJ and we will update our route and add the multicast option to the XML definition of the route.

11. Within our route XML definition, and within the <route> tag, add several blank lines and the add the following set of tags:

<multicast> </multicast>

As you are adding these tags, you will notice the popup for auto-completion:

12. Next, within the <multicast></multicast> tags, we add the <to> tag with the uri parameter:

After adding these new elements, then the entry will appear as:

                                                                     

13. Next, we need to update the URI parameter. First, we will need to provide the SFTP connection information and that string will look like this:

pfx-sftp:parameters?connection={{pfx:copy-product-multicast}}&amp;

14. Then, to this statement we can append the name of the new file in the replicated folder by adding the following:

filename=/productXtra&amp;

15. Next, to the multicast statement, we will add a Done file option. So, after the filename attribute add the following:

doneFileName=%24%7Bfile:name%7D.done

Step 6: Local Route Execution

Start the local instance of IM and test the route to verify that the multicast option is working.

  1. Next, we will launch the LocalAppRunner application. Right click on LocalAppRunner and choose the Run option:

2. In the log file, we are looking for the messages related to the successful launch of any routes that are linked to the IM instance:

Now that the local instance is running, we can start the route by adding a new CSV file to our inbound folder that the route is polling.

3. Now that the local instance is running, and the routes on PlatformManager/IntegrationManager are stopped, all polling of the SFTP folders is being performed by our local instance.

4. Go to SFTP folder view in WinSCP for our inbound route:

5. Drag and drop the CSV file associated with this inbound route for products.

6. In IntelliJ and within the log entries for LocalAppRunner instance, we should see the CSV file being consumed and reported:

7. Finally, return to SFTP folders and open the filearea/extrafile folder to see the file replicated by the multicast operation.