New relic
We use new relic for (some) monitoring.
We have 2 dashboards in NewRelic to help here.
Monitoring sourcers
Proxy success / error rates
Shows HTTP error / success rates per sourcer and proxy. Used to tell if we have a proxy blocking issue etc. Note that this shows individual proxy requests, and a Loader may retry several times.
Loader success / error / retry rates
This dashboard shows the per loader success / error and retry rates. Note that this is not the same as the proxy success rate. A loader may retry a proxy request several times until it succeeds, thus the error rate at Loader level may be lower than Proxy sourcer level.