Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
N
nebula
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
exchain
nebula
Commits
39d4f1d9
Commit
39d4f1d9
authored
Oct 24, 2022
by
Joshua Gutow
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
op-node,specs: Clarify channel bank pruning
parent
3498ade5
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
12 additions
and
3 deletions
+12
-3
channel_bank.go
op-node/rollup/derive/channel_bank.go
+4
-3
derivation.md
specs/derivation.md
+8
-0
No files found.
op-node/rollup/derive/channel_bank.go
View file @
39d4f1d9
...
@@ -78,9 +78,6 @@ func (cb *ChannelBank) IngestData(data []byte) {
...
@@ -78,9 +78,6 @@ func (cb *ChannelBank) IngestData(data []byte) {
origin
:=
cb
.
Origin
()
origin
:=
cb
.
Origin
()
cb
.
log
.
Debug
(
"channel bank got new data"
,
"origin"
,
origin
,
"data_len"
,
len
(
data
))
cb
.
log
.
Debug
(
"channel bank got new data"
,
"origin"
,
origin
,
"data_len"
,
len
(
data
))
// TODO: Why is the prune here?
cb
.
prune
()
frames
,
err
:=
ParseFrames
(
data
)
frames
,
err
:=
ParseFrames
(
data
)
if
err
!=
nil
{
if
err
!=
nil
{
cb
.
log
.
Warn
(
"malformed frame"
,
"err"
,
err
)
cb
.
log
.
Warn
(
"malformed frame"
,
"err"
,
err
)
...
@@ -108,6 +105,10 @@ func (cb *ChannelBank) IngestData(data []byte) {
...
@@ -108,6 +105,10 @@ func (cb *ChannelBank) IngestData(data []byte) {
cb
.
log
.
Warn
(
"failed to ingest frame into channel"
,
"channel"
,
f
.
ID
,
"frame_number"
,
f
.
FrameNumber
,
"err"
,
err
)
cb
.
log
.
Warn
(
"failed to ingest frame into channel"
,
"channel"
,
f
.
ID
,
"frame_number"
,
f
.
FrameNumber
,
"err"
,
err
)
continue
continue
}
}
// Prune after the frame is loaded.
// TODO: Ingest a single frame at a time (must enable reads before ingesting the data / after this prune.)
cb
.
prune
()
}
}
}
}
...
...
specs/derivation.md
View file @
39d4f1d9
...
@@ -521,6 +521,14 @@ As currently implemented, each step in this stage performs the following actions
...
@@ -521,6 +521,14 @@ As currently implemented, each step in this stage performs the following actions
frame are discarded.
frame are discarded.
-
Concatenate the data of the
*contiguous frame sequence*
(in sequential order) and push it to the next stage.
-
Concatenate the data of the
*contiguous frame sequence*
(in sequential order) and push it to the next stage.
The ordering of these actions is very important to be consistent across nodes & pipeline resets. The rollup node
must attempt to do the following in order to maintain a consistent channel bank even in the presence of pruning.
1.
Attempt to read as many channels as possible from the channel bank.
2.
Load in a single frame
3.
Check if channel bank needs to be pruned & do so if needed.
4.
Go to step 1 once the channel bank is under it's size limit.
> **TODO** Instead of waiting on the first seen channel (which might not contain the oldest batches, meaning buffering
> **TODO** Instead of waiting on the first seen channel (which might not contain the oldest batches, meaning buffering
> further down the pipeline), we could process any channel in the queue that is ready. We could do this by checking for
> further down the pipeline), we could process any channel in the queue that is ready. We could do this by checking for
> channel readiness upon writing into the bank, and moving ready channel to the front of the queue.
> channel readiness upon writing into the bank, and moving ready channel to the front of the queue.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment