Data Pipeline Latency

#1

Hello

I’ve noticed that my data pipeline in Zuar is experiencing unexpected delays. Normally; data updates are processed within a set timeframe, but recently, I’m seeing long gaps before new data appears in reports. The delay isn’t consistent, making it difficult to troubleshoot. :upside_down_face:

I’ve checked server load, data source connectivity, and scheduling settings, but everything seems normal. Could this be related to data volume, API rate limits / an issue with background processing? :thinking: I’d love to hear if others have faced similar latency problems and how they resolved them. I checked Data Pipeline Architecture: Everything You Need to Know Okta administrator certification related to this and found it quite informative.

If there’s an official guide on diagnosing slow data processing in Zuar, I’d appreciate a link. Any insights on optimizing pipeline efficiency or identifying bottlenecks would be really helpful! :slightly_smiling_face:

Thank you !! :slightly_smiling_face:

#2

Hello @forep!

It will be helpful to know a few things to help diagnose latency issues:

  1. Is your instance Zuar hosted or are you hosting this on-prem?
  2. If on prem, what type of VM are you hosting the instance on? Are the resources of the VM being shared with any other apps?
  3. Which version of Runner are you on and what type of resources are available to the instance? This info can be found on the settings page of the UI:
    image
    image1194×441 13.6 KB
  4. Are you using the internal Runner Postgresql Analytics database as your data warehouse?
  5. If yes, how does your disk space look? Low disk space can certainly contribute to general latency. If no, what are you using for a data warehouse?
  6. Data volume and rate limits could certainly be a factor. What API’s are you hitting with Runner and how much data is flowing via these endpoints (record count should help)?
  7. Are you leveraging upsert when possible?
  8. On slow jobs, are you seeing anything unexpected in the logs? Logs are certainly your friend in these scenarios!

This info should help!