[indexer.runbook] add details around confirmation counts

indexer/README.md

@@ -80,6 +80,7 @@ All security related issues should be filed via github issues and will be triage
 - Since the Indexer API only performs read operations on the database, access to the database for any API instances should be restricted to read-only operations.
 - The API has no rate limiting or authentication/authorization mechanisms. It is recommended to place the API behind a reverse proxy that can provide these features.
 - Postgres connection timeouts are unenforced in the services. It is recommended to configure the database to enforce connection timeouts to prevent connection exhaustion attacks.
+- Setting confirmation count values too low can result in indexing failures due to chain reorgs.
 
 ## Troubleshooting
 Please advise the [troubleshooting](./docs/troubleshooting.md) guide for common failure scenarios and how to resolve them.

indexer/docs/troubleshooting.md

 # Troubleshooting Runbook
-This document provides a set of troubleshooting steps for common failure scenarios.
+This document provides a set of troubleshooting steps for common failure scenarios and how to resolve them.
 
 ## Indexer Troubleshooting
 
+### Postgres failures
+1. Unable to connect to postgres database:
+* Verify that the database is healthy and accessible.
+* Verify that the database credentials are correct.
+* Verify that the database is not rate limiting connections.
+
+2. Unable to insert or read rows:
+* Verify that the database is healthy and accessible.
+* Verify that the database credentials are correct.
+* Verify that most recent`./migrations` schemas have been applied to the database.
+
+### Header Traversal Failure
+Header traversal is a client abstraction that allows the indexer to sequentially traverse the chain via batches of blocks. The following are some common failure modes and how to resolve them:
+1. `the HeaderTraversal and provider have diverged in state`
+* This error occurs when the indexer is operating on a different block state than the node. This is typically caused by network reorgs and is the result of `l1-confirmation-count` or `l2-confirmation-count` values being set too low. To resolve this issue, increase the confirmation count values and restart the indexer service.
+
+2. `the HeaderTraversal's internal state is ahead of the provider`
+* This error occurs when the indexer is operating on a block that the upstream provider does not have. This is typically occurs when resyncing upstream node services. This issue typically resolves itself once the upstream node service is fully synced. If the problem persists, please file an issue.
+
 ### L1/L2 Processor Failures
 The L1 and L2 processors are responsible for processing new blocks and system txs. Processor failures can spread and contaminate other downstream processors (i.e, bridge) as well. For example, if a L2 processor misses a block and fails to index a `MessagePassed` event, the bridge processor will fail to index the corresponding `WithdrawalProven` event and halt progress. The following are some common failure modes and how to resolve them:
 
@@ -40,7 +59,7 @@ no indexed state, starting from genesis
 ### Re-syncing bridge processor
 To resync the bridge processor, the following steps should be taken:
 1. Stop the running indexer service instance.
-2. Delete the database state. This can be done by doing performing cascading delete of both the l1 `l1_transaction_deposits` and the `l2_transaction_withdrawals` tables:
+2. Delete the bridge processor's database state. This can be done by doing performing cascading delete of both the l1 `l1_transaction_deposits` and the `l2_transaction_withdrawals` tables:
 ```sql
 TRUNCATE l1_transaction_deposits CASCADE;
 TRUNCATE l2_transaction_withdrawals CASCADE;
@@ -49,8 +68,9 @@ TRUNCATE l2_transaction_withdrawals CASCADE;
 
 ## API Troubleshooting
 
-### API is returning 500 errors
-The API is a read-only service that does not modify the database. If the API is returning 500 errors, it is likely due to the following reasons:
-* Verify that the API is able to connect to the database. If the database is down, the API will return 500 errors.
-* Verify that http `timeout` env variable is set to a reasonable value. If the timeout is too low, the API may be timing out on requests and returning 500 errors.
-
+### API is returning response errors
+The API is a read-only service that does not modify the database. If the API is returning errors, it is likely due to the following reasons:
+* Verify that the API is able to connect to the database. If the database is down, the API will return errors.
+* Verify that http `timeout` env variable is set to a reasonable value. If the timeout is too low, the API may be timing out on requests and returning errors.
+* Verify that rate limiting isn't enabled in request routing layer. If rate limiting is enabled, the API may be rate limited and returning errors.
+* Verify that service isn't being overloaded by too many requests. If the service is overloaded, the API may be returning errors.