We’ve occasionally had to deal with pillows stalling, particularly xform-pillow and case-pillow. Are there any ways to monitor pillowtops in a more automated fashion, even from bash? I’d love to set up an alert when they stall. Thanks!
Dimagi uses Datadog for this, do you have that or Prometheus set up? Here’s some documentation around pillowtop metrics that are reported:
For example, commcare.change_feed.change_lag reports the number of seconds between when the most recent change was put in the queue and when it was processed. commcare.change_feed.need_processing reports how many pending changes there are.
We have threshold alerting set up around this kind of thing in Datadog, so when it exceeds some value, we get an alert.
1 Like
Awesome, thanks Ethan