Co-authored-by: Joshua Gutow <jgutow@optimism.io>

feat: bedrock inwards/outwards batch deriv draft v2

skip timed out frames, keep reading tx data

ignore frame if it already exists, do not stop reading tx

update pruning

fix channel closing

fix reorg recover func

misc reorg func fixes

channels for multiplexing submissions, inclusion for ordering of processing, decoded batches can be ordered as necessary after pulling from the stream

ignore timed out frames

fix maxBlocksPerChannel name

fix var name, stop producing output data if no blocks are left

implement channel out reader, start testing, renaming, structure etc.

rename pipeline to channel-in-reader, fix old l2 package imports

close compression stream

improve channel out reader

add de compression and rlp reading to channel-in-reader

channel in reader: l1 origin update

channel in reader updates

move new deriv code into derive package

work in progress integration of batch derivation changes

work in progress, l2 derivation stepper

fix rlp dependency

channel in reader is broken, left todo

update work in progress derivation pipeline with todo spec per function

engine queue todo

work in progress integration with driver

fix channel in reader init

driver event loop and derive loop separation

(WIP) derive: Implement BatchQueue with full window

This implements a simple algorithm for the batch queue. It waits
until it has a full sequence window and then runs the historical
batch derivation process over that data.

The WIP part is that it needs more data that it does not yet have.

derive: Fully derive payload attributes

Also properly slices the queue.

Remove batch bundle, split of reading of data from txs

move engine update/consolidation into derive package

tag channel bank with l1 origin as whole, read frame data may not revert to older l1 origin because out-of-order channel frames

read full channel, forward L1 origin changes in channel-in-reader, don't block on batch reading

engine queue

engine queue work

driver updates

carry data between pipeline stages

log sync progress

wip init pipeline

fetch l1 data as part of derivation pipeline

init fix

work in progress channel bank reset change

channel bank resetting as part of pipeline

define interfaces for stages, clean up l1 interface usage

less trigger happy derivation pipeline resets, just reset when the pipeline says we need to

test utils

update driver snapshot usage, move L1Info interface to eth package, misc driver cleanup

use channel emitter for api, fix build issues

update batch submitter (work in progress, needs more testing)

engine queue fix (@trianglesphere)

find sync start reduce args, just get l2 head directly

fix channel reader: don't attempt to read when there's no channel data yet

log batcher and proposer in e2e

channel emitter / channel out reader fixes

fix channel emitter timeout

fix channel reading end

fix unexpected eof when buffer is not filled due to compressing layer also buffering

add logging to batch filtering

fix batch inputs, don't derive attributes before reading all batches of the origin

all derivation pipeline stages now have the same Step and ResetStep interfaces

misc open/close origin fixes and sync work

fix test

lint

improve testutils, fix l1 mock, implement calldata source tests

more mocking/testing utils, split l1 source/traversal, test first few stages

improve mock test utils, don't use bignum in l2 api

test pipeline per stage

channel timeout config param, test channel bank

fix batcher channel timeout flag

new op-batcher

new batcher in the op-node

logging / disable parts of the op-node for testing

fix off by one in batcher

Close l1src stage

Note: may want to pass the close further out / have more complex
logic about open/close.

logging + hacks to make the sequencer work & verifier half work

change open/close origin api, fix genesis seq nr bug, e2e passing

fix progress/origin naming, avoid engine api linear unwind in consolidation, fix batcher process closing

remove old ChannelEmitter, remove ChannelOutReader in favor of ChannelOut, fix tests, clean up unused l2 engine change, clean up op-batcher flags

fix op-batcher flags / docker compose update

clean up logging

lint

test valid -> if err == nil, not err != nil

L1Source -> L1Retrieval, fix receiver names

wait for derivation to be idle before sequencing new block

implement verifier and sequencer confirmation depth

op-node: Add Epoch Hash to batch

This commits a batch to a specific L1 origin block by hash rather
than just by number. This help in the case of L1 reorgs by stopping
batches from being applied in weird ways.

fix missing epoch block hash

batcher: Handle multiple frames per channel

The batcher is still very simple generating a new channel full of
L2 blocks since the last channel that it created, but it is just
a tad smarter now in that if will handle the case of multiple frames
per channel.

This is the bare minimum functionality to handle happy case batching
on a real network. The only other thing that it can't handle is
reorgs, but it can now handle very larger L2 transactions and blocks.

* op-node: Add eth package changes

* op-node: Add testutils package

* op-node: Pull in L1/L2 source changes

* op-node: Make the refactor build

* Apply suggestions from code review

Use `NewPayloadErr` instead of `ForkchoiceUpdateErr` as pointed out by @protolambda
Co-authored-by: Diederik Loerakker <proto@protolambda.com>
Co-authored-by: Diederik Loerakker <proto@protolambda.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Updates to foundry/hardhat broke the genesis builder, this fixes them

contracts-bedrock: buildinfo set to true

The `debug_standardTraceBlockToFile` RPC was panicing here:

```go
// Line 572 in eth/api_tracer.go
if config != nil && config.Overrides != nil {
```

`config` is an instance of `StdTraceConfig`, which embeds a pointer to `vm.LogConfig`. This pointer is `nil` when the user doesn't pass in any overrides. Since `vm.LogConfig` exposes the `Overrides` parameter, this caused a panic.

`debug_standardTraceBlockToFile` is required for users like Dune, who need access to transaction traces that are too complex to serve over RPC.

fix: oom in hardhat tasks

The `solidity-coverage` package is no longer used.
Don't import it into hardhat to speed up hardhat
and lessen the memory footprint.

Attempt to fix oom issue

Version Packages

Merge develop into master

Published as `ethereumoptimism/deployer-bedrock`.
This will be published along with new releases of the `npm` package.
It wraps `hardhat deploy`. The config file will want to be mounted
in at runtime if deploying to a non default network.

- Outputs the raw GitHub response in check-changed
- Updates the golangci-lint version to the one in `go-builder` that supports 1.18

- Updates CircleCI jobs to exit early if the git diff doesn't require them to run. This will dramatically reduce our CircleCI usage and speed up the pipeline for projects like Bedrock that don't touch legacy code. The entire pipeline will still run on `develop` and `master`. The conditional logic is aware of package dependencies, so changing an upstream package will correctly trigger a build on downstream dependencies.
- Creates a new `ci-builder` image to replace `js-builder` and `go-builder`. The `ci-builder` image contains everything we need to run builds on CircleCI.
- Fixes a bunch of misspellings that should have failed the linter, but didn't.

Adds a metrics server config and tracks metrics for the RPC server and RPC client calls. This PR is just a start - there are many more things we can track. Anything that might be affected by the batch derivation work was deliberately excluded.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Only create the hardhat style `BuildInfo` objects when
compiling through the hardhat interface. Do not build
them when going through `forge build`. This saves
disk i/o when just building using `forge` vs building
using `hardhat` for releases or hardhat tasks.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Remove some of the l2geth packages in place of
using geth packages directly. This begins to
remove l2geth as a dependency from the indexer,
as l2geth will become legacy code after the release
of bedrock.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Fix some very simple, low hanging fruit issues
that semgrep is finding in the indexer.

There are some more issues, but they have a larger
scope.
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* indexer: use op-bindings for address manager

The `Lib_AddressManager` contract was pulled into
`contracts-bedrock` as a legacy contract and can be
imported into `op-bindings` and build the same way that
the other golang contract bindings are maintained.

This is a legacy contract and should NEVER have any changes to it.
It is ok if it is compiled with a different compiler version for
testing, it will never be deployed to a network to be used with an
Optimism system again. It is very important that the ABI never changes.

Any upgrades to the network that touch this contract first must go
through a `hardhat` or `anvil` forked network to ensure correctness
of the upgrade.

* indexer: go mod tidy + dockerfile
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* bedrock: Make portal upgradeable

* bedrock: rebuild bindings and snapshot

* bedrock: update portal storage layout

* bedrock: Test oracle behind proxy

--wip--

* bedrock: Test portal behind a proxy

* bedrock: Set l2Sender in initialize

* bedrock: workarounds for forge test overflow bug

* bedrock: reorder inheritance in portal

This shifts the order of inheritance so that the initializable
values are in the first slot of the portal, which is consistent with
all other contracts we have that inherit from initializable

* Revert "bedrock: workarounds for forge test overflow bug"

Forge seems to be working again!?

This reverts commit bb8cdf2f3ecce1456c01e668f36b1429e02934d1.

* bedrock: Make ResourceMetering non-abstract

* bedrock: Add test for storage access post upgrade

* chore: update gas snapshot and bindings

Enable hardhat style `BuildInfo` generation that integrates
with the hardhat plugin. The debug files (`*.dbg.json`) are
generated by default and are able to be pulled into hardhat
tasks with `Artifacts.getBuildInfo`.