Network Traffic on Default Interface

[bw-solana]

Problem

Even after setting bind-address and gossip-host to some other interface IP, some network egress appears to be sent from the bond0 interface during startup. This subsequently results in traffic being received on this interface.

It is desired to be able to route all traffic to be sent from a specified interface.

Likely related to how this works: https://github.com/anza-xyz/agave/blob/v2.1.7/net-utils/src/lib.rs#L104-L109

Proposed Solution

Need to debug where traffic is originating from and send it from a specified interface. Could piggy back on the bind-address param or create something new, depending on what is appropriate after root cause.

[alexpyattaev]

Metrics traffic seems to go from default gateway no matter what is in bind-address

[bw-solana]

Metrics traffic seems to go from default gateway no matter what is in bind-address

good catch!

Main thing we'll need to update is this:
https://github.com/anza-xyz/agave/blob/master/metrics/src/metrics.rs#L147-L149

to add something like .local_address("".parse().unwrap())

We also have shell scripts, but I think this is more for testing:
https://github.com/anza-xyz/agave/blob/master/scripts/metrics-write-datapoint.sh will have the kernel pick the interface.

need something like --local-address <bind address>

[alexpyattaev]

Assembled a test setup with 2 nodes (bootstrap and 1 validator) with two network interfaces. Specified the bind address for agave in both cases. No traffic found on bond0.

There are some irregularities though, there are a few open listen ports on all interfaces:
UDP 8003 (solana-quic-client)
3 dynamic UDP ports (still tracking down why they are opened there)

[alexpyattaev]

Ok so to trigger the bug the cluster entrypoint needs to be NOT in the DZ network.

[alexpyattaev]

Identified bind on port 8003 -

agave/quic-client/src/nonblocking/quic_client.rs

Line 76 in bf09fd5

fn create_endpoint(&self) -> Endpoint {

 solana_quic_client::nonblocking::quic_client] Local endpoint is : UdpSocket { addr: 0.0.0.0:8003, fd: 29 }

[alexpyattaev]

Need to also test that transactions made via RPC work correctly