<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/4.2.2">
</HEAD>
<BODY LINK="#0000ff">
El jue, 21-04-2016 a las 23:48 +0000, Kale, Rahul escribió:
<BLOCKQUOTE TYPE=CITE>
Hello,<BR>
</BLOCKQUOTE>
<BR>
Hi Rahul,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
Thanks for the quick turnaround to provide the fix.<BR>
</BLOCKQUOTE>
<BR>
Ok,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
However, I have a doubt about the code changes.<BR>
As you can see from my previous post, when I made local changes to the code<BR>
to see how many bytes were actually captured during the failed situation, the logs show<BR>
that instead of next 2 bytes, only one byte is captured from the wire. From your code changes <BR>
I understand that only zero byte captured case (EWOULDBLOCK) is accounted for.<BR>
</BLOCKQUOTE>
<BR>
Uhmn...certainly, you right...<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
I may be mistaken, but I think we need to handle the case more generically.<BR>
I see from the code that for the first 2 bytes, and later for the handling the 4 bytes of mask data,<BR>
you have already implemented correct buffering into 'pending_buf' for next attempt.<BR>
I think we will need to have similar handling for this case (as well as for<BR>
next 8 bytes of extended header for payload_size == 127 case).<BR>
<BR>
<BR>
<BR>
I am working on integrating your current code changes into our app to see<BR>
what would happen when the situation hits. It takes quite a while to reproduce<BR>
the issue. I will let you know what I observe.<BR>
</BLOCKQUOTE>
<BR>
Ok. I'll try to update regression tests to check these cases...<BR>
<BR>
Even though what you propose seems the right solution, it is a different case <BR>
to know that there are X pending bytes, that having broken headers (especially<BR>
when they are so small)....this is connected to your next comment:<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
In our application, I believe that the server side always sends the full header in one shot. <BR>
</BLOCKQUOTE>
<BR>
I highly doubt it (but I may be wrong :-). <BR>
<BR>
The websocket implementation that is sending the content<BR>
is sending the header in different steps ...and because of that, the kernel/os <BR>
may interrupt the userland code in each of those steps....<BR>
<BR>
It is nearly impossible for modern TCP stacks to present a, let's say, packet with<BR>
6 usable bytes, and just serve some of them to the application land...<BR>
<BR>
There are many obvious reasons, for example, the waste of time that causes<BR>
for the overall performance to force userland level to call several times to pick<BR>
just one or two bytes out of 6 bytes received...<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
However, the very nature of TCP is such that if a sender sends X bytes in one write/send call, <BR>
the receiver is not guaranteed to always get all X bytes in one read/recv call... <BR>
</BLOCKQUOTE>
<BR>
You are right, but that's for bigger sizes, not for the problem we are talking about. <BR>
<BR>
Given the costs of sending a TCP/IP packet with content, processing it, and the fact <BR>
that you have to avoid user land calling several times for what you have presumable <BR>
already received, it is clear that if the kernel has received a tcp segment with 8 usable<BR>
bytes, it will serve them all at once...and never by pieces...<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
even if X is very small. <BR>
</BLOCKQUOTE>
<BR>
No,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
The underlying OS networking system is free to break up the <BR>
streaming data at arbitrary boundaries. <BR>
</BLOCKQUOTE>
<BR>
It's free, certainly, but I as say, any modern TCP/IP stack will not do it because it is<BR>
inconvenient...for the practical reasons exposed,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
That is why the connection breaks very occasionally<BR>
and takes some time to reproduce.<BR>
</BLOCKQUOTE>
<BR>
Ok, I'll give a try to introduce more cases in the regression test so receiving just one byte<BR>
in the particular case we are talking is also supported,<BR>
<BR>
Best Regards,<BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
Regards,<BR>
<BR>
<BR>
<BR>
Rahul<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<B>From:</B> Francis Brosnan Blázquez [mailto:francis.brosnan@aspl.es] <BR>
<B>Sent:</B> Thursday, April 21, 2016 1:52 AM<BR>
<B>To:</B> Kale, Rahul<BR>
<B>Cc:</B> nopoll@lists.aspl.es<BR>
<B>Subject:</B> Fixed -- Re: [noPoll] Websocket disconnect issues<BR>
<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
Hello Rahul,<BR>
<BR>
I've updated noPoll sources to reproduce the problem you described (test_30)<BR>
and added needed updates to make it work (svn revision 262):<BR>
<BR>
<A HREF="https://github.com/ASPLes/nopoll">https://github.com/ASPLes/nopoll</A><BR>
<BR>
Check them to see if they fixes your issue,<BR>
Best Regards,<BR>
<BR>
<BR>
<BR>
<BR>
Hello Rahul,<BR>
<BR>
Thanks for reporting. I'm reviewing the issue. I'll let you know something,<BR>
<BR>
Best Regards,<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
Hello,<BR>
<BR>
<BR>
<BR>
I have some more debug information regarding this issue.<BR>
<BR>
<BR>
<BR>
To confirm the conjecture that the next two bytes may not always be immediately available,<BR>
<BR>
I made a local modification as shown below and ran the test again waiting for next failure. <BR>
<BR>
Instead of immediately aborting the connection when this situation occurs, I poll the connection<BR>
<BR>
for a short duration to see if the next two bytes appear subsequently, and then abort the connection anyway.<BR>
<BR>
<BR>
<BR>
} else if (msg->payload_size == 126) {<BR>
<BR>
/* get extended 2 bytes length as unsigned 16 bit<BR>
<BR>
unsigned integer */<BR>
<BR>
bytes = __nopoll_conn_receive (conn, buffer + 2, 2);<BR>
<BR>
if (bytes != 2) {<BR>
<BR>
int i;<BR>
<BR>
nopoll_log (conn->ctx, NOPOLL_LEVEL_CRITICAL, "Failed to get next 2 bytes to read header from the wire (read %d, %d), failed to received content, shutting down id=%d the connection", bytes, errno, conn->id);<BR>
<BR>
for (i = 0; i < 100; i++) {<BR>
<BR>
bytes = __nopoll_conn_receive (conn, buffer + 2, 2);<BR>
<BR>
if (bytes == 2) {<BR>
<BR>
nopoll_log (conn->ctx, NOPOLL_LEVEL_CRITICAL, "Finally got next 2 bytes to read header from the wire after %d tries, shutting down anyway id=%d the connection", i, conn->id);<BR>
<BR>
break;<BR>
<BR>
}<BR>
<BR>
nopoll_sleep(10); // 10 ms sleep<BR>
<BR>
}<BR>
<BR>
nopoll_msg_unref (msg);<BR>
<BR>
nopoll_conn_shutdown (conn);<BR>
<BR>
return NULL; <BR>
<BR>
} /* end if */<BR>
<BR>
<BR>
<BR>
The above is just a local change to confirm the root cause.<BR>
<BR>
<BR>
<BR>
I reproduced the disconnection bug again and from the logs, you can see that only one byte was available initially<BR>
<BR>
and after a very short duration we do get the remaining bytes.<BR>
<BR>
<BR>
<BR>
The relevant log lines look like:<BR>
<BR>
<BR>
<BR>
nopoll_conn.c:2798 Failed to get next 2 bytes to read header from the wire (read 1, 0), failed to received content, shutting down id=2 the connection <BR>
<BR>
nopoll_conn.c:2802 Finally got next 2 bytes to read header from the wire after 5 tries, shutting down anyway id=2 the connection <BR>
<BR>
<BR>
<BR>
I think, not getting next 2 (or 4 bytes) should not be treated as an error situation, but rather treated as<BR>
<BR>
full message not yet available just like when waiting for the first two bytes of the message. The call is supposed to be<BR>
<BR>
non-blocking so we of course cannot poll like the test code snippet above. <BR>
<BR>
<BR>
<BR>
Do let me know if you need any further info.<BR>
<BR>
This issue is very critical to move forward with our current application.<BR>
<BR>
<BR>
<BR>
Regards,<BR>
<BR>
<BR>
<BR>
Rahul<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<B>From:</B> <A HREF="mailto:nopoll-bounces@lists.aspl.es">nopoll-bounces@lists.aspl.es</A> [<A HREF="mailto:nopoll-bounces@lists.aspl.es">mailto:nopoll-bounces@lists.aspl.es</A>] <B>On Behalf Of </B>Kale, Rahul<BR>
<B>Sent:</B> Tuesday, April 19, 2016 6:33 PM<BR>
<B>To:</B> <A HREF="mailto:nopoll@lists.aspl.es">nopoll@lists.aspl.es</A><BR>
<B>Subject:</B> [noPoll] Websocket disconnect issues<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
Hello,<BR>
<BR>
<BR>
<BR>
I have been trying to track down a random websocket disconnection issue that we are facing.<BR>
<BR>
In our application, after some random number of hours of a good websocket connection that is <BR>
<BR>
transferring data back and forth, the connection suddenly breaks. We are using noPoll only<BR>
<BR>
for the client side of a WebSocket connection. The server side is Apache frontend with <BR>
<BR>
node app backend.<BR>
<BR>
<BR>
<BR>
After eliminating all other probable causes, I enabled noPoll debug logs and was finally able <BR>
<BR>
to catch the error which causes a disconnect: <BR>
<BR>
<BR>
<BR>
nopoll_conn.c:2797 Failed to get next 2 bytes to read header from the wire, failed to received content, shutting down id=2 the connection <BR>
<BR>
<BR>
<BR>
Reproducing this is difficult since it takes anything from a couple of hours to a couple of days<BR>
<BR>
but eventually it happens. My debugging efforts at narrowing this down is thus taking quite a while.<BR>
<BR>
<BR>
<BR>
We are using secure web sockets (wss://). <BR>
<BR>
<BR>
<BR>
Looking at the code, it seems that in the noPoll library, we expect the next two bytes of <BR>
<BR>
the websocket header to be always available if we read the first two bytes. <BR>
<BR>
I think that over TCP (and possibly even more so over TLS)<BR>
<BR>
this may not always be guaranteed. The socket is non-blocking and it may occasionally<BR>
<BR>
have a pattern that 4 bytes are split over two (or more) TCP segments.<BR>
<BR>
<BR>
<BR>
For reading first two bytes noPoll library seems to be buffering if data is not available.<BR>
<BR>
For correctness, should this be done for subsequent parts of the header too?<BR>
<BR>
Of course, I am not an expert here so could you analyze and let me know what is could<BR>
<BR>
be the root cause? What should be my next steps?<BR>
<BR>
<BR>
<BR>
I have already verified that this is not due to http(s) server error. I did a TCP dump<BR>
<BR>
and confirmed that when this happens, the first FIN is sent by the client (noPoll) to the http server.<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
Regards,<BR>
<BR>
<BR>
<BR>
Rahul<BR>
<BR>
<BR>
<BR>
Rahul Kale<BR>
<BR>
<BR>
<BR>
IP Video Systems<BR>
<BR>
Barco, Inc<BR>
<BR>
1287 Anvilwood Ave<BR>
<BR>
Sunnyvale, CA 94089<BR>
<BR>
<BR>
<BR>
Tel +1 408 400 4238<BR>
<BR>
<BR>
<BR>
This message is subject to the following terms and conditions: <A HREF="http://www.barco.com/en/maildisclaimer">MAIL DISCLAIMER</A> <BR>
<BR>
<BR>
This message is subject to the following terms and conditions: <A HREF="http://www.barco.com/en/maildisclaimer">MAIL DISCLAIMER</A><BR>
<BR>
<BR>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
_______________________________________________
noPoll mailing list
<A HREF="mailto:noPoll@lists.aspl.es">noPoll@lists.aspl.es</A>
<A HREF="http://lists.aspl.es/cgi-bin/mailman/listinfo/nopoll">http://lists.aspl.es/cgi-bin/mailman/listinfo/nopoll</A>
</PRE>
</TD>
</TR>
</TABLE>
<BR>
<BR>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
--
Francis Brosnan Blázquez - ASPL
<A HREF="http://www.asplhosting.com/">http://www.asplhosting.com/</A>
<A HREF="http://www.aspl.es/">http://www.aspl.es/</A>
<A HREF="https://twitter.com/aspl_es">https://twitter.com/aspl_es</A>
<A HREF="https://twitter.com/asplhosting">https://twitter.com/asplhosting</A>
<A HREF="https://twitter.com/francisbrosnanb">https://twitter.com/francisbrosnanb</A>
91 134 14 22 - 91 134 14 45 - 91 116 07 57
AVISO LEGAL
En virtud de lo dispuesto en la Ley Orgánica 15/1999, de 13 de
diciembre, de Protección de Datos de Carácter Personal, le informamos de
que sus datos de carácter personal, recogidos de fuentes accesibles al
público o datos que usted nos ha facilitado previamente, proceden de
bases de datos propiedad de Advanced Software Production Line, S.L.
(ASPL).
ASPL garantiza que los datos serán tratados con la finalidad de mantener
las oportunas relaciones comerciales o promocionales con usted o la
entidad que usted representa. No obstante, usted puede ejercitar sus
derechos de acceso, rectificación, cancelación y oposición dispuestos en
la mencionada Ley Orgánica, notificándolo por escrito a ASPL -
Protección Datos, C/Antonio Suárez 10 A-102, 28802, Alcalá de Henares
(Madrid).
</PRE>
</TD>
</TR>
</TABLE>
<BR>
<BR>
<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
This message is subject to the following terms and conditions: <A HREF="http://www.barco.com/en/maildisclaimer">MAIL DISCLAIMER</A><BR>
</BLOCKQUOTE>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
--
Francis Brosnan Blázquez - ASPL
<A HREF="http://www.asplhosting.com/">http://www.asplhosting.com/</A>
<A HREF="http://www.aspl.es/">http://www.aspl.es/</A>
<A HREF="https://twitter.com/aspl_es">https://twitter.com/aspl_es</A>
<A HREF="https://twitter.com/asplhosting">https://twitter.com/asplhosting</A>
<A HREF="https://twitter.com/francisbrosnanb">https://twitter.com/francisbrosnanb</A>
91 134 14 22 - 91 134 14 45 - 91 116 07 57
AVISO LEGAL
En virtud de lo dispuesto en la Ley Orgánica 15/1999, de 13 de
diciembre, de Protección de Datos de Carácter Personal, le informamos de
que sus datos de carácter personal, recogidos de fuentes accesibles al
público o datos que usted nos ha facilitado previamente, proceden de
bases de datos propiedad de Advanced Software Production Line, S.L.
(ASPL).
ASPL garantiza que los datos serán tratados con la finalidad de mantener
las oportunas relaciones comerciales o promocionales con usted o la
entidad que usted representa. No obstante, usted puede ejercitar sus
derechos de acceso, rectificación, cancelación y oposición dispuestos en
la mencionada Ley Orgánica, notificándolo por escrito a ASPL -
Protección Datos, C/Antonio Suárez 10 A-102, 28802, Alcalá de Henares
(Madrid).
</PRE>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>