fix: unable to establish connection to seed node

[tegefaulkes]

Specification

During the network entry procedure where we attempt to connect to a seed node is failing to connect. The expectation here is that we specify the seed node using the --seed-node CLI parameter and our node connects to said seed node. What is happening is we start the connection but the seed node times out when establishing the reverse connection.

Specifically the connection is being started so the seed node sees this connection. When handling the connection it should create a reverse connection and compose it. We are failing to start the reverse connection during ReverseConnection.start() due to timing out.

Additional context

Related MatrixAI/Polykey-CLI#71
Related #487

Tasks

1. Investigate the cause of the failure.
2. Fix the bug.

[CMCDragonkai]

The punch protocol is executed regardless whether there is NAT or not. So you would want to establish whether your home node is receiving punch packets from the testnet node. You can do this by using wireshark or by logging out in the code on the punch message/receive message handler in Proxy.ts.

If that is true, then the error is somewhere else, maybe some logic problem or timeout issue. If that is not true, then the punch protocol is failing.

This may be related to the #474 nat testing failures since that was failing locally too.

[tegefaulkes]

Original context found here. #487 (comment)

[tegefaulkes]

Testing locally gives us the following error.

ErrorPolykeyRemote: Remote error from RPC call
  command       nodesHolePunchMessageSend
  nodeId        vi7dsecl9945pl22uff97tfee776auf3d3cjcfeimarihv1p4plqg
  host  127.0.0.1
  port  54143
  timestamp     Tue Oct 25 2022 13:46:11 GMT+1100 (Australian Eastern Daylight Time)
  cause: ErrorProxyConnectInvalidUrl: Invalid target host used for HTTP connect proxy
    exitCode    76
    timestamp   Tue Oct 25 2022 13:46:11 GMT+1100 (Australian Eastern Daylight Time)

The bug here is really weird.. just before the error in Proxy.establishConnectionReverse the value of proxyHost is 0.0.0.0. But just before calling Proxy.establishConnectionReverse in Proxy.connectReverse the value of proxyHost is 127.0.0.1 as expected... This is very odd.

Oh, it's called twice, the first time is correct, the 2nd one isnt.

[CMCDragonkai]

I think we've seen this before. There's always confusion between 0.0.0.0 and 127.0.0.1. The 0.0.0.0 should only be allowed when listening. It should never be allowed when connecting-to. Our validation routines should be checking to ensure this.

[CMCDragonkai]

Here's an example of a confusion I've seen before in the codebase:

One source of confusion is that we are using host in places that are both listening-to and connecting-to. When running a server that is listening to 0.0.0.0 and we ask the server what is it listening to, it will say 0.0.0.0. But you can't use this to connect. You have to swap it out for 127.0.0.1 instead (and you can only do this if you expect that 0.0.0.0 means you are on the same machine, if it is on a remote server, then that address is meaningless).

[tegefaulkes]

I've found the problem. When creating a NodeConnection we send hole punch request to the seed node with sendHolePunchMessage. with this we specify our own IP and port to for where to send the hole punch messages to. The problem here is. We don't know our host and port. we can't know this from the local perspective unless we ask someone else. So what we're doing is sending what we think our host and port is. In this case it's defaulted to 0.0.0.0 and whatever the proxy port was.

We need to fix this logic. What needs to happen is that the seed node fills in the details of the node making the hole punch request. This can be done in two ways. Our node an request our connection information from the seed node and fill in the details. Or the seed node when it receives the hole punch relay request can fill in the connection details of the node making the request.

I've been vaguely aware of this problem for a while now. I think I've mentioned it in the #365 issue.

[CMCDragonkai]

That hole punch messages is only relevant if we are connecting to another (third) node though.

I remember that in ICE, we do 3 things in parallel, direct connection, hole punch signalling, relay message.

Now since we are hitting the seed nodes, this is different from hitting random nodes.

I imagine that upon the initial network entry, it should not bother with trying to "hole punch signal", the seed nodes. It should only attempt direct connection.

We haven't really worked out how #365 nor #182 would work.

So at the network entry stage, there shouldn't be an ICE logic. Only direct connections should be necessary.

[CMCDragonkai]

with this we specify our own IP and port to for where to send the hole punch messages to. The problem here is. We don't know our host and port.

I don't understand this. The "hole punch message" is for signalling. The connecting node should not be putting in its own IP and port information because like you said, it wouldn't even know the right IP and port info.

[CMCDragonkai]

What needs to happen is that the seed node fills in the details of the node making the hole punch request. This can be done in two ways. Our node an request our connection information from the seed node and fill in the details. Or the seed node when it receives the hole punch relay request can fill in the connection details of the node making the request.

The seed node "fills in the details" when it receives a signalling message. It does this dynamically at the point of handling the service call.

[tegefaulkes]

Overall I think the process needs to be refactored. currently we ask the seed node to relay a hole punch message to our target. In this case we are providing all of the information here. What we need to do is ask the seed node to co-ordinate a connection between our self and our target node.

It would look something like this.

1. Node NA asks SA to co-ordinate connection to NB
2. SA responds OK, connect to NB at this address.
3. SA tells NB to connect to NA at this address.
4. Both NA and NB are attempting connections to each other on their respective public IP and port.

The key difference here is that SA is providing the connection information to both NA and NB. The limitation of this is that both NA and NB need to be connected to SA at the same time.

[CMCDragonkai]

As you're fixing this, can you change the name of this mechanism to be "signalling"? That is sendHolePunchMessage should be sendSignalingMessage.