epoll: edge-triggered is a contract, not an optimisation
Level-triggered epoll — the default — reports a descriptor as ready for as long as it is
ready. Call epoll_wait again without reading anything and you get the same descriptor
again. It is forgiving, and it is what you want unless you have a reason.
Edge-triggered mode reports only transitions. Data arrives, you get one notification. If
you read half of it and return to epoll_wait, you do not get another notification for
the remainder — no new data has arrived, so no edge has occurred. Your bytes sit in the socket
buffer indefinitely and the connection appears to hang.
The obligation
Under EPOLLET you must drain every ready descriptor until read returns
EAGAIN (or EWOULDBLOCK). Not until you have what you wanted — until the
kernel tells you there is nothing left. Which means the descriptor must be non-blocking, or the
draining loop blocks the whole event loop on the last read.
The same applies to writes: register for EPOLLOUT, write until EAGAIN,
and only then wait for the next edge.
What it buys
Fewer epoll_wait wakeups when you are already looping over a descriptor, and a
natural fit for thread-per-core designs where each descriptor belongs to exactly one thread and
repeated readiness reports are pure noise. Those are real gains at high connection counts.
They are not gains you will notice at a thousand connections, and the failure mode — a hang that only appears when a message happens to span two reads — is one of the more unpleasant bugs to chase. Use level-triggered until you have measured a reason not to.