Mitto - Exit job [SEQUENCE] if underlying file is not there

Runner
#1

Hi Zuar -

We have job [SEQUENCE]s that run every day as generally there is a new XLS file on an SFTP server we have to pick up and ingest every day.

On some occasions (usually a weekend), there is no file waiting for us on the SFTP server. Because of this, the Mitto job fails.

What would be the best way to handle this time of situation? I imagine the ideal situation would be to have some sort of logic in the [SEQUENCE] that would stop the ETL if there is no file available.

1 Like
#2

Hey @BlueFlamingo , great question!
The how to handle this really depends on what you want to happen. It sounds like if there is no file (job failing) you want the entire sequence to stop?

If so, you could create some logic in Mitto that could help accomplish this. I don’t know the full details of how you have your jobs and sequences nested but here is a simple and an advanced version of a potential solution:

Simple
The simplest approach but maybe not feasible depending on your setup would be to put the SFTP job into a Master Sequence, separate from all the other jobs within the sequence and set the Master Sequence “Continue on Error” to “False”. If that SFTP job positioned before the child sequence fails, everything will stop and the child sequence or series of ETL jobs will not run.

Advanced - Uses Webhooks so positioning or locations within a sequence don’t necessarily matter as much.

  1. Create a master sequence that contains the child sequence of jobs that should or shouldn’t run depending on the result of the SFTP job and set the Master Sequence “Continue on Error” to “False”.
  2. In the Master Sequence, before the child sequence, add your SFTP job.
  3. Within your SFTP job, add a webhook that is set to trigger on JOB_SUCCESS. For the url, have it target the child sequence inside of the Master Sequence (or wherever you want it to live) and send the start action to that sequence. Here is an exact example of what I’m referring to: Add Webhooks to Any Individual Job or Sequence in Mitto | Zuar

If the above is built correctly, you’ve essentially implement logic between the SFTP job and the dependent sequence, which is all organized within the Master Sequence. When the SFTP job succeeds, it will fire off the webhook to kick off the child sequence. When it fails, nothing will happen since you have the Master Sequence set to FALSE for continue on error!

1 Like