[Vortex] Problem mixing wait_replies and vortex_channel_set_serialize()

[Philip Black-Knight]

Hi,
I'm trying to work though an issue I've found in the Vortex library. I'm using the votex-1.1.7.b4612.g4612 release on a 64bit Linux machine. I think I've tracked down how the issue is occurring, and I believe it would be easy to fix, but since I don't know the library well, I don't feel comfortable addressing the problem myself. The problem I am seeing is that when I call vortex_channel_set_serialize(channe, axl_true); for a particular channel, I never receive any frames. Looking through the logs, it appears that vortex believes that it hasn't received the first message on the channel, and so it is queuing the frames for later deliver.

I've dumped the TCP traffic, just to make sure there wasn't something wrong at that level, and all the messages and sequence numbers are correct, so I figured it must be something else. Additionally, if I leave the serialization as the default (i.e. do not call vortex_channel_set_serialize(channe, axl_true)), then my frame handler is called. However, I really want to take advantage the serialization provided by the library since that would make my code much simpler.

At any rate, I believe I've tracked down the problem. The first 2 messages I send on the channel  use vortex_channel_send_msg_and_wait() to receive the frames. Just before sending the third message, I make a call to vortex_set_channel_complete_flag(chan, axl_false) to disable frame joining. The first 2 messages (and replies) are MSG/RPY style communications and are relatively small. The third message is a MSG/ANS pattern message, where each ANS is potentially very large, so allowing vortex to join the frames is likely to exhaust memory.

If I'm reading the code right, it looks like when the first 2 replies are received, because I am using wait_replies, the code that updates the sequence numbers is not getting executed. (i.e. __vortex_channel_invoke_received_handler is not getting called). Because of this, the serialize_next_seqno is not updated when the first 2 replies are received, and vortex will wait forever for data it has already received.

It seems like the fix would be to update the serialize_next_seqno even when a wait_reply is handling the message, but I can see how this may cause other problems.

I'm hoping someone can weigh in and let me know if this seems to be the case? Or am I way off base. I don't think it would be too hard to adjust my code to not use wait_replies, but they are certainly convenient.

Thanks,
Phil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aspl.es/pipermail/vortex/attachments/20111014/13069130/attachment.htm>