A watch-folder is a very common pattern where a scheduled workflow will check the content of a folder and will trigger a sequence for each pertinent file found.
However, it is very easy to setup when you are scanning one or 2 directories but could be tedious to setup if you are watching let say 20 or 30 directories. In this case 30 schedulers would consume a significant amount of power, would be inefficient, and it would be difficult to maintain. in such cases, it is highly recommended to carry out a multipaths watch-folder.
Image 1 : Multipaths watch-folder
Multipaths watch-folder scans each file in each folder, filters them before triggering a sequence. this sequence allows us to supervise and use forms.
To achieve this project, it is important to consider concurrent executions, useless files and growing files.
This project requires several steps:
- Storing the different paths of different folder
- Creation of workflow and link the workflow to a sequence
- Scheduling the workflow launch
Step1: Storing the paths to watch
Saving login, url, path into your code is not a very good habit and make your code difficult to maintain. This is the reason why we use data source to save the list of paths (It is important to remember that each path represents a folder). So, to create a data source, go to the menu Configure/datasources and press on the button «add »
and fulfill the followings parameters:
- Name: name of the data source
- Type: choose data mapping
- key/value: each key has his own value for example (key: path 1 and its value is \\10.0.4.242\Sources\NENE\01-medias in)
Image 2: Data Source
It is possible to put as many paths as you want, you only have to press the button “+Mapping”
Step2: The workflow
This workflow aims to carry out several watch-folder (watch-folder of several folders) simultaneously with a single scheduler. If in a watch-folder, each file triggers a sequence, they will be multiple sequences as each useful files coming from the different folders will trigger a sequence: this is a real time saver and will improve the productivity.
The workflow represents all the tasks for the resolution of Multipaths watch-folder and is made of two parts:
- pre-scanner phase
- scanner phase
Image 3 : different tasks of the multipaths watch folder
Step2.1: Pre-Scanner Phase
To be able to scan several folders, we must loop on the different folders. So, for each path in data source, we will perform a scanner. Consequently, through this phase we retrieve the stored paths in the data source.
The pre-scanner phase consists of two tasks: the read data source and the iterator.
Image 4 : Pre-Scanner phase tasks
Step2.1.1: Read data source task
The “Read data source” task allows us to retrieve the values of the paths recorded previously by putting the name of the source data.
Image 5 : Read data source task
[Data source name]: is set on the name of our data source (see Image 1)
Step2.1.2: Iterator task
It loops on the different folders of our data source (each path is the path of a folder) by fulfilling input data_source (the object data_source is the output of our previous task).
Image 6: Iterator task
[input]: is set on data_source which is the output of the “Read data source” task (see Image 3)
Step 2.2: The Scanner
For each path found by the iterator, we scan and detects all files from the different folders. Each file will be filtered and moved to a sub-folder “in_process” and finish with launching a sequence. this process includes four tasks.
Image 7: the scanner tasks
Step 2.2.1: Folder Scan task
“Folder Scan” task detect the files located in a folder and for each file found execute a task connected to the yellow port. In addition to that the folder scan task filters useless files with « file filters »
Therefore, for each file we are going to:
- Wait for growing files
- moved the file in the subfolder « in_process »
- execute a sequence
Image 8 : Folder scan task
[Input folder] is set on loop_value (loop_value is the output of the previous task “iterator”).
[Iterate SmartPath] For each file detected, the variable smartPath named « smart_path» will be created by the task.
The output path is exported as "smart_path" and will be used later in the sequence.
Image 9 : Files filters of the "Folder Scan" task
Step2.2.2: Wait for growing file task
Since the files can be very large and copying them can take a long time, it is important to pay attention to growing files (change in files sizes), and use “Wait for growing file” task. This task wait until file size remains unchanged before executing the following task of the scanner.
Image 10 : Wait for growing file task
[Input path] is the variable smartPath named « smart_path» created by the folder_scan task.
Step2.2.3: copy/Move file task
we want to copy the media in the subfolder « in_process » of folder 01-media in, and under the original name of the files to prevent it from being scanned again several times because each time we call folder_scan, System will return all the corresponding files.
Image 11 : Copy/Move file task
[Input path] is the variable smartPath named « smart_path» created by the folder_scan task.
[Method] used is Move
So, output path of Copy/Move file task is then calculated like bellow:
Image 12 : output path of Copy/Move file task
os.path.join("in_process", smart_path.basename())
# os.path.join est une fonction de concaténation de path
# smart_path.basename() retourne le nom du fichier pointé par un smartpath
# le path sera donc égal à :
# /mnt/sharing1/NENE/01-medias in/in_process/nomDuFichier
Step2.2.4: The start sequence task
for each file found, we are going to start one sequence called « valid_media display »
Image 13 : Start sequence task
On Context tag of “Start sequence” task, we are going to define the parameters to transfer to the sequence. we will pass on it the variable smartPath called « smart_path », and the variable file_name. in some cases, it is also important to pass the source of the file on the sequence.
Image 14: Context of Start sequence task
Step3: The launch scheduling
To schedule the workflow, go the monitor/scheduler. In this part of this project, we must check the contents of the folder by running this workflow every 3 minutes from 9 a.m to 6 p.m on weekdays. We will use a scheduler and configure it like this:
Image 15 : Workflow Scheduler
from the Workflow scheduler list, we can verify the right execution of our workflow.
Image 16: Workflow Scheduler List