Solution
The solution finally implemented at the Home Office is a complete system with 7 environments from dev to prod to ensure complete testing of all the workflows.
Activeeon is used to pilot and orchestrate mutlitple scripts and applications such as Talend which is used to perform ETL (Extract, Transform & Load) operations.
The final solution is able to handle 100,000 tasks per day in a fault tolerant way. Indeed, multiple options are available upon task failure such as kill the job, pause the job, pause the dependent tasks or simply ignore. The operators are then able to restart the job where it stopped and recover without complex maitenance.