[Webinar] Unlock the Power of Mitto's Automation

Runner
#1

Thanks for joining our webinar, Unlock the Power of Runner’s Automation! For the on-demand version: Webinar On-Demand

Here are some resources that were discussed during the session:

If you have further questions related to automation, job types, specific use cases, etc., don’t hesitate to ask/reply on this thread or start your own topic! I can almost guarantee that someone has or will have the same question. Your contributions will pave the way for a stronger community around the tool!

As always, thanks for you support in making Mitto the powerful tool that it is today!

#2

Here are some of the jobs that were used during the webinar:

Download file using curl job

Title: [curl] download file - collections.csv
Type: curl
Job Conf:

{
    "args": [
        "-o",
        "collections.csv"
    ],
    "url": "https://raw.githubusercontent.com/zuarbase/data/master/collections.csv"
}

Load CSV file into database using io job

Title: [CSV] collections.csv
Type: io
Job Conf:

{
    "input": {
        "delimiter": ",",
        "encoding": "ASCII",
        "includes_header": true,
        "source": "/var/mitto/data/collections.csv",
        "use": "xsv.iov2#XsvInput2"
    },
    "output": {
        "dbo": "postgresql://localhost/analytics",
        "schema": "csv",
        "tablename": "collections",
        "use": "call:mitto.iov2.db#todb"
    },
    "steps": [
        {
            "transforms": [
                {
                    "use": "mitto.iov2.transform#ExtraColumnsTransform"
                },
                {
                    "use": "mitto.iov2.transform#ColumnsTransform"
                }
            ],
            "use": "mitto.iov2.steps#Input"
        },
        {
            "use": "mitto.iov2.steps#CreateTable"
        },
        {
            "transforms": [
                {
                    "use": "mitto.iov2.transform#FlattenTransform"
                }
            ],
            "use": "mitto.iov2.steps#Output"
        },
        {
            "use": "mitto.iov2.steps#CollectMeta"
        }
    ]
}

Model the data with a sql job

Title: [SQL] Drop/create table example
Type: sql
Job Conf:

{
    "dbo": "postgresql://localhost/analytics",
    "kwargs": {},
    "parameters": {},
    "split": false,
    "sql": [
        "/* Create the database schema if it doesn't exist */",
        "CREATE SCHEMA IF NOT EXISTS model",
        ";",
        "",
        "/* Drop the database table if it exists */",
        "DROP TABLE IF EXISTS model.collections_summary",
        ";",
        "",
        "/* Create the data model table */",
        "CREATE TABLE IF NOT EXISTS model.collections_summary AS",
        "SELECT",
        "    state",
        "  , DATE_TRUNC('month', invoice_date)::DATE AS invoice_month",
        "  , SUM(amount)                             AS amount",
        "FROM csv.collections",
        "GROUP BY",
        "    1",
        "  , 2",
        ";"
    ],
    "transaction": true
}

SQL:

/* Create the database schema if it doesn't exist */
CREATE SCHEMA IF NOT EXISTS model
;

/* Drop the database table if it exists */
DROP TABLE IF EXISTS model.collections_summary
;

/* Create the data model table */
CREATE TABLE IF NOT EXISTS model.collections_summary AS
SELECT
    state
  , DATE_TRUNC('month', invoice_date)::DATE AS invoice_month
  , SUM(amount)                             AS amount
FROM csv.collections
GROUP BY
    1
  , 2
;

Sleep 10 seconds using cmd job

Title: [cmd] Sleep 10
Type: cmd
Job Conf:

{
    "cmd": "sleep 10",
    "cmd_env": {},
    "exec": false,
    "shell": true
}

Note: This was job id 12 and is referenced by other jobs below.

Start a job with another job using curl in a cmd job

Title: [cmd] curl start Mitto job 12
Type: cmd
Job Conf:

{
    "cmd": "curl -X POST -H 'accept:applicaton/json' -H 'Content-Type: applicaton/json' -d '{\"action\": \"START\"}' 'http://localhost:7127/v2/jobs/12/:actions'",
    "cmd_env": {},
    "exec": false,
    "shell": true
}

Start a job with another job using the Mitto cli in a cmd job

Title: [cmd] Mitto cli start Mitto job 12
Type: cmd
Job Conf:

{
    "cmd": "mitto start 12",
    "cmd_env": {
        "LANG": "C.UTF-8"
    },
    "exec": false,
    "shell": true
}