[Vortex] Using vortex for IPC

[Mikhail Dmitrievsky]

Hello Francis!

Thanks a lot for your response! It was very helpful!

> > I'm trying to use vortex for exchanging large amount of data between
> > two processes on the same computer. I have found 2 problems.
> >
> >      1. Very low transfer speed (about 20-30 Mb/sec)
> >
> > After some changes in vortex I have speed about  100 Mb/sec, and it
> > looks much better than 30 Mb/sec.
> 
> Great job Mikhail. At this point, which are the changes introduced? You
> can post and describe them?

Looks like I was using vortex in wrong way. I was trying to extend window size, but vortex fails because of small buffer in VortexCtx (sequencer_send_buffer). That's why I made changes in vortex to make default buffer size much bigger - I replace values like 4096 and 4095 in source with bigger values. I know it was wrong, but it was a way to try to speed up vortex. Now I clearly understand that I need to do to make vortex working faster. May be my experience helps someone else:

1. I created frame size handler function:

int my_frame_size_maker (VortexChannel *channel, unsigned int next_seq_no, int message_size, unsigned int max_seq_no, axlPointer user_data) 
{
	int res = VORTEX_MIN (PTR_TO_INT(user_data), 
		    VORTEX_MIN (message_size, max_seq_no - next_seq_no + 1));
	return res;
};

Main idea of this function is to tell frame size as big as possible.

2. In listener and initiator I added these calls after I have connection created:

vortex_connection_set_next_frame_size_handler (connection, my_frame_size_maker, 													INT_TO_PTR(BIG_BUFFER_SIZE));
vortex_channel_set_window_size(channel, BIG_BUFFER_SIZE); // set bigger window size

After some tests I've got that the best buffer size is the value near
#define BIG_BUFFER_SIZE 61440
I made this value less than 64k to eliminate possibility of creating frame bigger than MAX_BUFFER_SIZE (64k).

> 
> > - sequencer_send_buffer and sequencer_send_buffer_size from VortexCtx
> > inits only once in vortex_sequencer_run function and after extending
> > channel window size sequencer_send_buffer seems to be too small.
> 
> This is not a bug. Take a look on vortex_sequencer_build_packet_to_send
> [2]. It dynamically increase the buffer size according to the value
> returned by vortex_channel_get_next_frame_size, which is, configurable
> via setting a handler that can return your desired value by
> vortex_connection_set_next_frame_size_handler.
> 

Looks like there is a bug in vortex_sequencer_build_packet_to_send buffer increasing algorithm, because it only doubles buffer. In my case I want buffer greater or equal 60k, but if we have initial value 4k, doubled value will be only 8k. I changed 

/* check if we have to realloc buffer */
if (size_to_copy + 100 > ctx->sequencer_send_buffer_size) {
	/* update buffer size */
	ctx->sequencer_send_buffer_size = (ctx->sequencer_send_buffer_size - 100) * 2 + 100;
	/* free current buffer */
	axl_free (ctx->sequencer_send_buffer);
	/* allocate new buffer */
	ctx->sequencer_send_buffer      = axl_new (char, ctx->sequencer_send_buffer_size);
} /* end if */

to 

/* check if we have to realloc buffer */
while (size_to_copy + 100 > ctx->sequencer_send_buffer_size) {
	/* update buffer size */
	ctx->sequencer_send_buffer_size = (ctx->sequencer_send_buffer_size - 100) * 2 + 100;
	/* free current buffer */
	axl_free (ctx->sequencer_send_buffer);
	/* allocate new buffer */
	ctx->sequencer_send_buffer      = axl_new (char, ctx->sequencer_send_buffer_size);
} /* end if */

and now it work perfectly!

> > Have anyone some ideas how to fix vortex for extending window size?
> >
> >      2. Transfer limit is 2Gb.
> >
> > As I understand, the problem is that vortex uses variables of "int"
> > type instead of "unsigned int" for saving sequence numbers.  Then the
> > transfer size becomes more that 2Gb, vortex tries to increase
> sequence
> > number and it becomes greater than INT_MAX, so  we get sequence
> number
> > value near -INT_MAX. After that vortex closes connection.
> 
> You are right, this is a bug. Reading the RFC it is clear that seq no
> values must use unsigned int (2^32 - 1). I'll take a look on this.
> 
> > To fix this problem I made a lot of changes in vortex sources (about
> > 30 places in 10 files), but I'm not sure that I did all necessary
> > changes.  If someone want to fix this bug in vortex, I can send my
> > changes to him.
> 
> Sure. You can always post your changes to the mailing list. I'll take a
> look on them ASAP.
> 

I think it will be better if I send them to you directly, because message will be ~700k, don't think this size applicable to mail list :)

WBR, Mikhail.