Skip to main content

Engineering support

Each week, one engineer takes ownership of the support rota. The on-call engineer is the first responder for all platform issues, responsible for monitoring Slack channels, triaging incoming requests, and keeping an eye on our logging and security systems.

This is not a passive role. The support engineer should be actively watching channels and dashboards throughout the working day, responding to new issues immediately, and dedicating 10–20% of their time to tackling historic / backlog issues.

Rotation structure​

Length — 1 week. The rota is tracked in the support rota calendar — subscribe to it so you know when your week is coming up.

Document — All issues discovered or reported are created on the PEB board (Product and Engineering Backlog) with relevant details.

Handover — The outgoing engineer briefs the incoming engineer on any open issues or items to follow up on.

Time split — 10–20% proactive work on historic / backlog issues; the rest reactive — triage, monitoring and responding. Jump on new issues as they're raised, while allocating time to reducing the backlog of alerts.

Escalation — If an issue requires significant investigation, escalate to the relevant lead or domain expert. Never let an issue go silent. Keep as much comms as possible in #eng-support.

Responsibilities​

Start of day​

  1. Check all Slack support channels for overnight messages or unresolved threads — see Channels to monitor below.
  2. Review PostHog and New Relic dashboards for error spikes or anomalies.
  3. Check the Aikido security queue for new vulnerabilities.
  4. Review CloudWatch alarms for any triggered alerts.

During the day​

  1. Monitor Slack channels continuously.
  2. Triage and respond to new issues immediately. Even if you can't fix it straight away, acknowledge it and set expectations.
  3. Keep dashboards open, or check them every 1–2 hours.
  4. Update open issue threads with progress so no one is left in the dark.

End of day​

  1. Ensure all new issues raised that day have been acknowledged and either resolved or given a clear next step.
  2. Update any open threads with a status summary.

Channels to monitor​

ChannelWhat's in itPriority
#platform-incidentsActive incidents, outage reports and post-incident updates. Critical during live issues. Any platform-level issue should be reported here.Critical
#platform-help-supportPrimary support channel. Internal teams raise platform issues here.High
#client-supportClient-specific queries and escalations from the CS team.High
#eng-security-alertsNew Aikido security alerts.High
#platform-internal-feedbackInternal team feedback on platform bugs, UX issues and feature requests.Medium
#posthog-alertAlerts raised from PostHog — error trends, and session replays of user-reported issues.Medium

See Slack channels for the full list.

Systems to monitor​

PostHog​

  • Review the main dashboard for error trends and session replays of user-reported issues.
  • Check for failed feature flag evaluations or unexpected drops in key metrics.
  • Use session replay to investigate specific user-reported bugs when needed.
  • Action new issues while allocating time to reducing the backlog of active alerts.

See Posthog.

Aikido (security)​

  • Check the security queue daily for new vulnerability findings.
  • Triage items by severity — critical and high should be addressed during your rotation, or escalated.
  • Mark false positives and document the rationale so they don't resurface.
  • Action new issues while allocating time to reducing the backlog of active alerts.

New Relic​

  • Monitor APM for error rate spikes and slow transactions.
  • Review any triggered alert conditions and investigate root causes.
  • Check infrastructure health metrics (CPU, memory, throughput).
  • Action new issues while allocating time to reducing the backlog of active alerts.

See New Relic.

CloudWatch (AWS)​

  • Review triggered alarms each morning and throughout the day.
  • Check Lambda error rates and timeout metrics.
  • Investigate any log groups flagged by metric filters.
  • Action new issues while allocating time to reducing the backlog of active alerts.

Handover​

At the end of your week, walk the incoming engineer through:

  1. Open issues — any unresolved issues, their current status and expected next steps.
  2. Ongoing incidents — any active or recently resolved incidents, including post-mortem status.
  3. Security items — any open Aikido findings that need follow-up.
  4. Monitoring notes — any patterns spotted in PostHog, New Relic or CloudWatch during the week.
  5. Historic issues tackled — a summary of backlog items worked on and their outcomes.
  6. Anything unusual — gotchas, known flaky areas, or upcoming changes that might affect the platform.

Tips​

  • Communicate proactively. A quick "looking into this now" goes a long way.
  • Don't try to fix everything alone. Escalate early rather than spinning your wheels.
  • Document as you go. If you figure out a tricky issue, update the ticket or leave notes for the next engineer.
  • Be kind to yourself. Support weeks can be intense. Take breaks and ask for help when you need it.