Skip to content

Add more control over loadgen/latencies #394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
drebelsky wants to merge 9 commits into from
Closed

Add more control over loadgen/latencies #394

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a3d6e05
Add option for specify per-edge delay for pubnet data
drebelsky Jun 24, 2026
ad4e6d8
Add flag for enabling e2e metric
drebelsky Jun 25, 2026
282127d
Add flag for setting PEER_AUTHENTICATION_TIMEOUT
drebelsky Jun 25, 2026
d951b16
Address copilot feedback
drebelsky Jun 26, 2026
62f6b28
format
drebelsky Jun 26, 2026
c75ec3d
Replace JSON generatesLoad with --loadgen-keys CLI arg
drebelsky Jul 2, 2026
4b86894
Allow specifying --loadgen-keys on non-delay style pubnet data
drebelsky Jul 14, 2026
30ae4e0
Add check that pubnet data is set when specifying loadgen keys
drebelsky Jul 16, 2026
003e5f4
Update flag name
drebelsky Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions src/App/Program.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,12 @@ type MissionOptions
apiRateLimit: int,
httpProxyReplicas: int,
pubnetData: string option,
pubnetDataDelay: bool,
measureE2eLatency: bool,
peerAuthenticationTimeout: int option,
flatQuorum: bool option,
tier1Keys: string option,
loadgenKeys: string option,
maxConnections: int option,
fullyConnectTier1: bool,
byteCountValues: seq<int>,
Expand Down Expand Up @@ -318,12 +322,32 @@ type MissionOptions
[<Option("pubnet-data", HelpText = "JSON file containing pubnet connectivity graph data", Required = false)>]
member self.PubnetData = pubnetData

[<Option("pubnet-data-delay",
HelpText = "Set to use the new style format for --pubnet-data",

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just update the --pubnet-data format to new style everywhere? That seems simpler than maintaining two formats. If latency isn't configured in the new format, we can fallback to old-style geolocations.

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Friendly reminder than we'll need to check-in the new pubnet data JSON somewhere. I guess it would be in https://github.com/stellar/stellar-supercluster. No worries if you want to wait until this PR lands though, in case there are any last minute changes to the format.

@drebelsky drebelsky Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think about them as serving different purposes. With the existing geoloc-style data, it's easier to support adding/removing nodes and edges with the existing flags while the new style is more useful for running a particular network configuration/experimenting with how we model simulated latencies (e.g., it is easy to check the difference between running using the survey data vs running using our existing geoloc-based delay model).

@marta-lokhova marta-lokhova Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, not sure I understand. Both formats have most of their parameters overlap, right? Is the difference in the new style that we add the delay field?

@drebelsky drebelsky Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As written, the new style change relative to the old style is that geodata field is removed and the per-node peers list becomes a list of {key, owdMs} objects instead of a list of (string) keys. The new-style format makes it hard to support flags like --tier1-orgs-to-add or --non-tier1-nodes-to-add. In the old version, we can just randomly pick geolocations and synthesize the delay, but it's harder to do something similar in the new format (the nodes don't have geolocations in the new model and mixing latency models feels potentially problematic).

Required = false,
Default = false)>]
member self.PubnetDataDelay = pubnetDataDelay

[<Option("measure-e2e-latency",

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh that's a good idea!

HelpText = "Set to enable the loadgen e2e metrics",
Required = false,
Default = false)>]
member self.MeasureE2eLatency = measureE2eLatency

[<Option("peer-authentication-timeout",
HelpText = "Maximum time overlay waits for a peer to authenticate before dropping",
Required = false)>]
member self.PeerAuthenticationTimeout = peerAuthenticationTimeout

[<Option("flat-quorum", HelpText = "Use flat Tier1 quorum", Required = false)>]
member self.FlatQuorum = flatQuorum

[<Option("tier1-keys", HelpText = "JSON file containing list of 'tier-1' pubkeys from pubnet", Required = false)>]
member self.Tier1Keys = tier1Keys

[<Option("loadgen-keys", HelpText = "JSON file containing list of pubkeys to generate load", Required = false)>]
member self.LoadgenKeys = loadgenKeys

[<Option("max-connections",
HelpText = "Maximum number of connections to allow any node in pubnet data to have. When enabled, this option will prune connections for any node with more than this number of connections. (default: no limit)",
Required = false)>]
Expand Down Expand Up @@ -743,6 +767,15 @@ let main argv =
0

| :? MissionOptions as mission ->
if mission.PubnetData.IsNone && mission.PubnetDataDelay then
failwith "Error: --pubnet-data-delay requires --pubnet-data to be set"

if mission.LoadgenKeys.IsSome && mission.PubnetData.IsNone then
failwith "Error: --loadgen-keys requires --pubnet-data to be set"

if mission.MeasureE2eLatency && mission.LoadgenKeys.IsNone then

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should LoadgenKeys also require PubnetData configured? I think otherwise, the keys may not match the auto-generated topology.

@drebelsky drebelsky Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The auto-generated topology uses a different function that doesn't look at the loadgen keys or the tier-1 keys. Updating for the loadgen key case.

failwith "Error: --measure-e2e-latency requires --loadgen-keys"

let _ = logToConsoleAndFile (sprintf "%s/stellar-supercluster.log" mission.Destination)

let ll =
Expand Down Expand Up @@ -848,8 +881,12 @@ let main argv =
apiRateLimit = mission.ApiRateLimit
httpProxyReplicas = mission.HttpProxyReplicas
pubnetData = mission.PubnetData
pubnetDataDelay = mission.PubnetDataDelay
measureE2eLatency = mission.MeasureE2eLatency
peerAuthenticationTimeout = mission.PeerAuthenticationTimeout
flatQuorum = mission.FlatQuorum
tier1Keys = mission.Tier1Keys
loadgenKeys = mission.LoadgenKeys
maxConnections = mission.MaxConnections
fullyConnectTier1 = mission.FullyConnectTier1
byteCountDistribution =
Expand Down
9 changes: 8 additions & 1 deletion src/FSLibrary.Tests/Tests.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,12 @@ let ctx : MissionContext =
apiRateLimit = 10
httpProxyReplicas = 2
pubnetData = None
pubnetDataDelay = false
measureE2eLatency = false
peerAuthenticationTimeout = None
flatQuorum = None
tier1Keys = None
loadgenKeys = None
maxConnections = None
fullyConnectTier1 = false
peerReadingCapacity = None
Expand Down Expand Up @@ -496,7 +500,10 @@ type Tests(output: ITestOutputHelper) =
let Chennai = { lat = 13.08784; lon = 80.27847 }
let dns1 = PeerDnsName "www.foo.com"
let dns2 = PeerDnsName "www.bar.com"
let cmd = getNetworkDelayCommands Ashburn [| (Beauharnois, dns1); (Chennai, dns2) |] None

let cmd =
getNetworkDelayCommands (getPeerDelays Ashburn [| (Beauharnois, dns1); (Chennai, dns2) |]) None

let cmdStr = cmd.ToString()

Assert.Contains(dns1.StringName, cmdStr)
Expand Down
10 changes: 8 additions & 2 deletions src/FSLibrary/MaxTPSTest.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,15 @@ let maxTPSTest (context: MissionContext) (baseLoadGen: LoadGen) (setupCfg: LoadG
List.find (fun (cs: CoreSet) -> cs.name.StringName = "stellar" || cs.name.StringName = "sdf") allNodes

let tier1 = List.filter (fun (cs: CoreSet) -> cs.options.tier1 = Some true) allNodes
let loadGenNodes = List.filter (fun (cs: CoreSet) -> cs.options.generatesLoad) allNodes

let loadGenNodes =
if List.isEmpty loadGenNodes then
// On smaller networks, run loadgen on all nodes to better balance the overhead of load generation
if List.length allNodes > smallNetworkSize then tier1 else allNodes
else
loadGenNodes

// On smaller networks, run loadgen on all nodes to better balance the overhead of load generation
let loadGenNodes = if List.length allNodes > smallNetworkSize then tier1 else allNodes
let isLoadGenNode cs = List.exists (fun (cs': CoreSet) -> cs' = cs) loadGenNodes

// Assign pre-generated transaction information to each load generator node.
Expand Down
7 changes: 6 additions & 1 deletion src/FSLibrary/MinBlockTimeTest.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,8 +337,13 @@ let minBlockTimeTest (context: MissionContext) (baseLoadGen: LoadGen) (setupCfg:
None }

let tier1 = List.filter (fun (cs: CoreSet) -> cs.options.tier1 = Some true) allNodes
let loadGenNodes = List.filter (fun (cs: CoreSet) -> cs.options.generatesLoad) allNodes

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trying to think about the setup and parameters we're going to use to run this test. I believe in this comment #394 (comment) you mentioned wanting to update MinBlockTest mission to output e2e latency. Is that still the plan? (no worries if you want to split work into multiple PRs, and keep this one scoped for now)

@drebelsky drebelsky Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still looking at different options for how to run the test (my current local runs are using a modified slp_eval.sh script and I just get the value from the metrics at the end for the e2e latency); I think we can decide what to do with that and split the work off into a separate PR.

@marta-lokhova marta-lokhova Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, using slp_eval seems fine (might need to rename or refactor the script though since this work isn't related to slps)


let loadGenNodes = if List.length allNodes > smallNetworkSize then tier1 else allNodes
let loadGenNodes =
if List.isEmpty loadGenNodes then
if List.length allNodes > smallNetworkSize then tier1 else allNodes
else
loadGenNodes

let isLoadGenNode cs = List.exists (fun (cs': CoreSet) -> cs' = cs) loadGenNodes

Expand Down
10 changes: 10 additions & 0 deletions src/FSLibrary/StellarCoreCfg.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ type StellarCoreCfg =
automaticMaintenanceCount: int
accelerateTime: bool
generateLoad: bool
measureE2eLatency: bool
updateSorobanCosts: bool option
manualClose: bool
invariantChecks: InvariantChecksSpec
Expand Down Expand Up @@ -317,6 +318,9 @@ type StellarCoreCfg =
t.Add("ARTIFICIALLY_ACCELERATE_TIME_FOR_TESTING", self.accelerateTime) |> ignore
t.Add("ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING", self.generateLoad) |> ignore

if self.measureE2eLatency && self.network.missionContext.measureE2eLatency then
t.Add("LOADGEN_MEASURE_TX_E2E_LATENCY_FOR_TESTING", true) |> ignore

if self.updateSorobanCosts.IsSome then
t.Add("UPDATE_SOROBAN_COSTS_DURING_PROTOCOL_UPGRADE_FOR_TESTING", self.updateSorobanCosts.Value)
|> ignore
Expand Down Expand Up @@ -365,6 +369,10 @@ type StellarCoreCfg =
t.Add("MAX_ADDITIONAL_PEER_CONNECTIONS", self.targetPeerConnections * 3)
|> ignore

match self.network.missionContext.peerAuthenticationTimeout with

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default timeout is a few seconds, which is quite long. Is there a particular reason to bump this timeout?

@drebelsky drebelsky Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to run a simulation with excessively long (more than ~2 seconds) delays (e.g., to simulate a slow node), it is useful to be able to configure the peer authentication timeout.

@marta-lokhova marta-lokhova Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, though I'm not sure that's needed for perf simulations. To avoid noise from slow nodes and properly validate changes to perf, I think all nodes in the simulation have to assume normal latency. Evaluation will be harder if we have outliers that are too slow.

| Some timeout -> t.Add("PEER_AUTHENTICATION_TIMEOUT", timeout) |> ignore
| None -> ()

t.Add("QUORUM_INTERSECTION_CHECKER", false) |> ignore
t.Add("MANUAL_CLOSE", self.manualClose) |> ignore

Expand Down Expand Up @@ -653,6 +661,7 @@ type NetworkCfg with
automaticMaintenanceCount = if opts.performMaintenance then 50000 else 0
accelerateTime = opts.accelerateTime
generateLoad = true
measureE2eLatency = opts.generatesLoad
updateSorobanCosts = opts.updateSorobanCosts
manualClose = false
invariantChecks = opts.invariantChecks
Expand Down Expand Up @@ -698,6 +707,7 @@ type NetworkCfg with
automaticMaintenanceCount = if c.options.performMaintenance then 50000 else 0
accelerateTime = c.options.accelerateTime
generateLoad = true
measureE2eLatency = c.options.generatesLoad
updateSorobanCosts = c.options.updateSorobanCosts
manualClose = false
invariantChecks = c.options.invariantChecks
Expand Down
4 changes: 4 additions & 0 deletions src/FSLibrary/StellarCoreSet.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ type QuorumSetConfiguration =
type CoreSetOptions =
{ nodeCount: int
nodeLocs: GeoLoc list option
edgeDelays: Map<byte [] * byte [], int> option
dbType: DBType
emptyDirType: EmptyDirType
syncStartupDelay: int option
Expand All @@ -211,6 +212,7 @@ type CoreSetOptions =
validate: bool
homeDomain: string option
tier1: bool option
generatesLoad: bool
catchupMode: CatchupMode
image: string
initialization: CoreSetInitialization
Expand All @@ -235,6 +237,7 @@ type CoreSetOptions =
static member GetDefault(image: string) =
{ nodeCount = 3
nodeLocs = None
edgeDelays = None
dbType = Sqlite
emptyDirType = MemoryBackedEmptyDir
syncStartupDelay = Some(5)
Expand All @@ -254,6 +257,7 @@ type CoreSetOptions =
validate = true
homeDomain = Some "stellar.org"
tier1 = None
generatesLoad = false
catchupMode = CatchupComplete
image = image
initialization = CoreSetInitialization.Default
Expand Down
4 changes: 4 additions & 0 deletions src/FSLibrary/StellarMissionContext.fs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,12 @@ type MissionContext =
apiRateLimit: int
httpProxyReplicas: int
pubnetData: string option
pubnetDataDelay: bool
measureE2eLatency: bool
peerAuthenticationTimeout: int option
flatQuorum: bool option
tier1Keys: string option
loadgenKeys: string option
maxConnections: int option
fullyConnectTier1: bool
byteCountDistribution: ((int * int) list)
Expand Down
Loading
Loading