Mitto Salesforce Connector - Custom Fields

Runner
#1

Had the following issue with the Mitto Salesforce Connector

  • There were some fields that the SFDC user connected to Mitto didn’t have access to. These fields didn’t show up in the Salesforce schema (e.g. salesforce.opportunity).

  • The SFDC user permissions to the fields in SFDC was updated.

  • The corresponding job in mitto was started to refresh the table. That didn’t add the field into Mitto.

  • Is this a mitto limitation. What steps do does the user have to take when a new custom field is added into Mitto (either because I just gave permissions to the SFDC user or because we created a new field)?

#2

Great question @gregpowell!

There are several factors that play into your Mitto Salesforce situation:

1 - A Mitto IO job that uses Salesforce as an input uses a specific Salesforce user for authentication. Authorization is handled by the Salesforce API, so only objects/fields that user has access to in Salesforce are able to be piped by Mitto.

Further reading:

Here’s an example input:

"input": {
        "credentials": {
            "password": "thisisapassword",
            "security_token": "thisisasecuritytoken",
            "username": "thisisausername"
        },
        "sobject": "Opportunity",
        "upsert_field": "SystemModstamp",
        "use": "sfdc.iov2#SalesforceInput"
    },

2 - Mitto Salesforce jobs created via the Salesforce plugin wizard are configured for upsert (against both the Mitto store and the output database table).

Here’s how the store is configured for upsert:

    "store": {
        "key": "Id",
        "updated_at": "SystemModstamp"
    }

And there are several steps involved in the table upsert:

"steps": [
        {
            "column": "system_modstamp",
            "use": "mitto.iov2.steps#MaxTimestamp"
        },
        {
            "use": "mitto.iov2.steps.upsert#SetUpdatedAt"
        },
        ...
        {
            "use": "mitto.iov2.steps.upsert#CreateTempTable"
        },
        ...
        {
            "key": "id",
            "use": "mitto.iov2.steps.upsert#SyncTempTable"
        },
        ...
    ],

In the example input above, when Mitto queries Salesforce for the Opportunity object data, it includes a filter using the SystemModStamp. Only records with a SystemModStamp greater than or equal to the maximum SystemModStamp Mitto finds in the store are piped back from Salesforce.

So, with the combination of #1 and #2 above, if a new field is created in Salesforce that the user has access to and that new field updates the SystemModStamp of every record in the table, then the output table will sync the new field and update all records.

However, in the case of a permission change for the Mitto Salesforce user, odds are the SystemModStamp isn’t going to update just because the user has access to a new field. So only records that are added/updated after the permission change will show up in the output table over time.

In this particular case, you should perform a full refresh of the associated Mitto job and output table.

How to do a full refresh of a Mitto Salesforce job with a store and upsert?

  1. Delete the Mitto store
  2. Rename, truncate, or drop the output table.
  3. Rerun the job to recreate the store and output table.

You can also duplicate the Mitto job and change the name of the output table as an alternative. Mitto stores are tied to the name of the Mitto job, so a duplicated job has no store when it is created.