This post is for consolidating ideas and best practices around flat file use cases and sequences.
Load Flat File(s) onto Mitto
In order for Mitto to interact with a file via a job, the file needs to be loaded into Mitto’s file system. There are several methods of accomplishing this ranging from manual to automatic with jobs. Learn more about how Mitto works with files.
Load Flat Files into a Database
Once the file(s) are loaded onto Mitto, you can use a number of IO job inputs to load the files into a database:
- Single file into a single database table - XSV, Excel, JSON
- Multiple files into a single database table - RegEx
Transform the Data
Once the raw flat file data is loaded into the database, you may want to create data models as new tables in the database.
For most data modeling use cases, use a SQL job.
Sequencing the Jobs
Finally, combine all the flat file related jobs together into a Sequence.
The job order will be:
- (Optional) Jobs related to loading the files onto Mitto
- Jobs related to loading the files into a database
- (Optional) Jobs transforming the data
Run the sequence manually if you are manually loading files into Mitto.
Schedule the sequence if you have jobs automatically loading the files into Mitto.