Preserve meaningful last_error in LibevConnection close paths#700
Preserve meaningful last_error in LibevConnection close paths#700dkropachev wants to merge 1 commit intomasterfrom
Conversation
8543c1f to
6aabcee
Compare
LibevConnection.close() closes the socket immediately while watchers are stopped asynchronously in the next event loop iteration via _loop_will_run(). This creates a race window where handle_read() or handle_write() can operate on a closed socket fd, producing EBADF errors that surface as ConnectionShutdown. - Add is_closed/is_defunct guards in handle_read() and handle_write() error paths to silently exit during shutdown instead of calling defunct() with EBADF - Set last_error in close() when connected_event is not yet set to prevent factory() from returning a dead connection - Set last_error on server-initiated close (EOF) in handle_read() before calling close()
6aabcee to
96add52
Compare
| self.last_error = ConnectionShutdown( | ||
| "Connection to %s was closed by server" % self.endpoint) | ||
| self.close() |
There was a problem hiding this comment.
Why do you set it unconditionally here, but guard it with if not self.connected_event.is_set(): in the other place?
There was a problem hiding this comment.
in close, it guards against overwriting last_error when connection was closer before it was properly initialized, in handle_read it is just channeling error reason back to close, connection is already initialized at this point.
There was a problem hiding this comment.
Sorry, but I don't get it :(
connected_event can be set in 4 places (for libevreactor at least, I didn't loo at others):
defunctinconnection.py, which calls close before setting the event._handle_startup_responsein case ofReadyMessage- successfull connection creation._handle_auth_responsein case ofAuthSuccessMessage- successfull connection creation.closeinlibevreactor.py.
You said that you want to guard against overwriting error when connection was closed before properly initialized, but it looks to me like you are doing the opposite.
You are overwriting error only if event was not yet set. If it was not yet set then we didn't receive ReadyMessage or AuthSuccessMessage. We also can't be in defunct, because we are in if not self.is_defunct:, and also defunct calls close before setting event.
There was a problem hiding this comment.
@sylwiaszunejko since you approved the PR, maybe you understand this?
There was a problem hiding this comment.
Maybe I am wrong, but I understand it that way that if it was set by defunct or close we will not call handle_read anymore, so there is no other error that we would possible want to have here. In close if it was not yet set then we didn't receive ReadyMessage or AuthSuccessMessage, so the conn was not properly initialized as Dmitry said, or no?
I am now confused tbh
There was a problem hiding this comment.
Ok I think I get it. If the event is set in this check, it was set by either ReadyMessage (_handle_startup_response), or by AuthSuccessMessage (_handle_auth_response). In both of those cases the connection is properly initialized.
So the check looks correct - it will only set error if connection was not initialized yet.
I would love to better understand the paths that can trigger this, to verify that they don't set last_error already, but I won't hold this PR over that.
Summary
Improves error reporting in LibevConnection so that users see a clear
ConnectionShutdownmessage instead of a confusing[Errno 9] Bad file descriptorwhen connections close during node restarts.The libev reactor was already safe from the EBADF race —
defunct()returns early whenis_closedis true, and the single-threaded event loop +_loop_will_run()design prevents watchers from firing on closed fds. The actual problem was thatlast_errorwasn't set to a meaningful value in two close paths, so the stale EBADF string leaked up tofactory()and confused users.Changes
last_errorinclose()whenconnected_eventis not yet set, sofactory()reportsConnectionShutdowninstead of whatever stale error was on the connectionlast_erroron server-initiated close (EOF) inhandle_read()before callingclose()Test plan
tests/unit/io/test_libevreactor.pypassesRefs #614