[Vortex] feeder-api question
Francis Brosnan Blazquez
francis at aspl.es
Mon Mar 21 10:17:43 CET 2011
> Hello
Hi Martin,
> Thanks, enabling fragmentation solved it.
>
> I understand the rationale for the sequencer to get optimal frames. I
> would like to offer some observation from vortex usage that you might
> find interesting. On the server side, when not using the feeder-api
> but just a series of ANS, if the data generating part is faster than
> vortex can send data, vortex will queue everything one throws at it
> (sometimes until the server gets killed due to excessive memory usage)
> and in such situations, the feeder-api offers a kind of throttling
> since it only requests frames when it needs them. I can understand
> that the rationale behind the feeder-api is not throttling of data
> generation, but I do think some sort of mechanism in a library like
> vortex for throttling data generation could be usefull in some
> circumstances.
That's right Martin. Feeder api main purpose is to avoid memory
retention, which has this collateral side effect..
> Again, I might have misunderstood how to use the library when it comes
> to throttling.
Don't thing so, but you have to separate two cases: throttling data
exchange (automatically done by vortex, using SEQ frames) and throttling
how fast the caller push data into vortex..where there is no easy
solution unless the caller do some work..
> As a side note, my first 'hackish' successful way of throttling was a
> small hack to vortex exposing the number of pending frames in the
> sequencer and comparing it to a threshold when generating new data to
> send. Works well with very little performance penalty, but I have a
> feeling it is not the right way to do it.
Ok, we have already implemented such mechanism (see [1]) but showed us
poor results especially in this cases (with big messages), because the
limit is done by counting messages not bytes...and counting bytes leads
to tricky situations (you must anticipate the right size so you won't
send any message)...
...so that's why we come to feeder api. Cheers!
[1] http://www.aspl.es/fact/files/af-arch/vortex-1.1/html/group__vortex__channel_g2bf0dd83e5296bc155f6b6a46033db2d.html#g2bf0dd83e5296bc155f6b6a46033db2d
> Thanks
> Martin
>
> 2011/3/18 Francis Brosnan Blazquez <francis at aspl.es>
>
> > Hello
>
> Hi Martin,
>
> > Thanks for the help. It could be that I don't fully
> understand how to
> > use the API.
> >
> > One thing that I misunderstood (fixing it does not solve my
> problem)
> > is that in the feed handler, the feed_op
> PAYLOAD_FEEDER_GET_SIZE is
> > implemented in file feeder to always return the entire file
> size but I
> > misunderstood the documentation and returned the number of
> bytes left
> > to feed.
>
>
> Ok, documentation is now updated with some additional
> comments.
>
> Currently PAYLOAD_FEEDER_GET_SIZE should return the entire
> message that
> was meant to be sent...however you can play with feeder by
> "increasing"
> dynamically this value (always increasing) if you want to send
> more
> content than the one initially expected...
>
> As conclusion, you can always return the entire message or a
> bigger
> value than the last one reported.
>
> > Inspecting debug output and doing some tests indicates the
> problem for
> > me is that in my original implementation I send ANS frames
> having size
> > of about 16k which on the client side becomes handle-frame
> events.
> > Using the feeder API the
> vortex_channel_send_ans_rpy_from_feeder
> > apparently send ANS frames but message size isn't 4k or 16k
> or similar
> > but the entire file size so on the client side I eventually
> get a
> > handle-frame event but it's so unexpectedly big the client
> runs into
> > trouble.
>
>
> Ok, two things. A VortexPayloadFeeder represents a single
> message, so,
> triggering a single operation with a feeder results into a
> single
> message delivered to the remote peer (that may or may not be
> fragmented,
> see next), which would be the same results as sending a single
> ANS with
> the entire message.
>
> In the other hand, you might want to check [1] to enable
> fragmentation
> (or disable complete frame delivery) so even if the message is
> fragmented, you get all pieces notified at the frame received
> as they
> come.
>
> [1]
> http://www.aspl.es/fact/files/af-arch/vortex-1.1/html/group__vortex__channel_gf352be87d3fc33fb5cf3eb4607fde99c.html#gf352be87d3fc33fb5cf3eb4607fde99c
>
> > So, perhaps I should call
> vortex_channel_send_ans_rpy_from_feeder
> > repeatedly with the feeder_op PAYLOAD_FEEDER_GET_SIZE
> returning about
> > 16k to obtain handle-frame event?
>
>
> You can do so, but you'll get poor results and eventually it
> will become
> more work, see [1] to get all pieces as they come.
>
> > If this is so, might I suggest a small improvment to the
> feeder API,
> > that there is a feeder-op PAYLOAD_FEEDER_GET_CHUNK_SIZE that
> makes the
> > feeder API send the entire file as a sequence of messages of
> size
> > returned by the feed-op?
>
>
> Ok, I see your proposal, but consider this. The payload feeder
> is
> designed in a way it get the optimal size from the feeder at
> the right
> moment. This collides with the chunk size concept mainly
> because the
> sequencer may truncate that value to the value that is already
> proposed
> by the feeder...
>
> Keep us updated Martin, cheers!
>
>
> > Thanks for all help.
> > /Martin
> >
> >
> >
> >
> >
> >
> >
> > 2011/3/18 Francis Brosnan Blázquez <francis at aspl.es>
> > > Hello
> >
> > Hi Martin,
> >
> > > Lately I've been experimenting with the feeder-api
> which
> > seems like
> > > the right way to go. However, I do have a problem
> I am
> > trying to
> > > understand.
> >
> >
> > Ok,
> >
> > > The scenario is roughly this:
> > >
> > > There is a file-server serving large media files
> (about 500
> > - 1500 mb
> > > each) and clients connect to this server
> requesting to
> > download an
> > > image file. Nothing fancy really.
> > >
> > > In the original implementation, the client
> connects, send a
> > msg and
> > > the server replies with a series of ANS. The
> client gets a
> > > handle-frame callback for each ANS and stores it
> to some
> > local
> > > storage.
> > >
> > > I just tried to do this the feeder-api way. It
> looks like
> > the server
> > > sends a series of ANS like it should but on the
> client side
> > I never
> > > get any handle-frame callbacks, instead the client
> just
> > grows in
> > > memory size so I assume it's queueing all recieved
> ANS.
> >
> >
> > ..so frames somehow aren't delivered..
> >
> > > I would appreciate some hints on ways forward to
> debug this.
> > I don't
> > > need desperately it but its the right way to do it
> so I want
> > to try to
> > > get it to work.
> >
> >
> > Ok, as a general hint I would try enabling debug
> (either
> > calling to
> > vortex_log_enable, or setting environment variable
> > VORTEX_DEBUG=1 ./your-app). This will give you lot
> of
> > output...
> >
> > You can also check [1] and [2] to see how those
> examples are
> > implemented
> > to get hints about what is failing..
> >
> > I thinking about ways to make frames to be not
> notified is to
> > not set a
> > frame received on the particular
> channel/profile/ctx, or to
> > send some
> > replies with some of them missing (ie, sending
> replies 2 and
> > 3, without
> > sending 1), and having vortex_channel_set_serialize
> > activated...
> >
> > > By the way, I don't use the supplied file-feeder,
> I wrote my
> > own
> > > because the source of the media files isn't really
> the file
> > system.
> >
> >
> > Nice Martin...
> >
> > > Regards and thanks for a great library.
> >
> >
> > Thanks!
> >
> > [1]
> >
> https://dolphin.aspl.es/svn/publico/af-arch/trunk/libvortex-1.1/test/vortex-file-transfer-client.c
> > [2]
> >
> https://dolphin.aspl.es/svn/publico/af-arch/trunk/libvortex-1.1/test/vortex-file-transfer-server.c
> >
> > > /Martin
> > >
> > > --
> > > Martin Eliasson
> > > +46 (0) 739 97 87 33
> > > http://asplunden.org
> >
> > > _______________________________________________
> > > Vortex mailing list
> > > Vortex at lists.aspl.es
> > >
> http://lists.aspl.es/cgi-bin/mailman/listinfo/vortex
> >
> >
> >
> >
> > --
> > Martin Eliasson
> > +46 (0) 739 97 87 33
> > http://asplunden.org
>
> --
> Francis Brosnan Blázquez <francis.brosnan at aspl.es>
> ASPL
> 91 134 14 22 - 91 134 14 45 - 91 116 07 57
>
> AVISO LEGAL
>
> Este mensaje se dirige exclusivamente a su destinatario. Los
> datos
> incluidos en el presente correo son confidenciales y sometidos
> a secreto
> profesional, se prohíbe divulgarlos, en virtud de las leyes
> vigentes. Si
> usted no lo es y lo ha recibido por error o tiene conocimiento
> del mismo
> por cualquier motivo, le rogamos que nos lo comunique por este
> medio y
> proceda a destruirlo o borrarlo.
>
> En virtud de lo dispuesto en la Ley Orgánica 15/1999, de 13 de
> diciembre, de Protección de Datos de Carácter Personal, le
> informamos de
> que sus datos de carácter personal, recogidos de fuentes
> accesibles al
> público o datos que usted nos ha facilitado previamente,
> proceden de
> bases de datos propiedad de Advanced Software Production Line,
> S.L.
> (ASPL). No obstante, usted puede ejercitar sus derechos de
> acceso,
> rectificación, cancelación y oposición dispuestos en la
> mencionada Ley
> Orgánica, notificándolo por escrito a:
> ASPL - Protección Datos, C/Antonio Suárez 10 A-102, 28802,
> Alcalá de
> Henares (Madrid).
>
>
>
>
> --
> Martin Eliasson
> +46 (0) 739 97 87 33
> http://asplunden.org
--
Francis Brosnan Blázquez <francis.brosnan at aspl.es>
ASPL
91 134 14 22 - 91 134 14 45 - 91 116 07 57
AVISO LEGAL
Este mensaje se dirige exclusivamente a su destinatario. Los datos
incluidos en el presente correo son confidenciales y sometidos a secreto
profesional, se prohíbe divulgarlos, en virtud de las leyes vigentes. Si
usted no lo es y lo ha recibido por error o tiene conocimiento del mismo
por cualquier motivo, le rogamos que nos lo comunique por este medio y
proceda a destruirlo o borrarlo.
En virtud de lo dispuesto en la Ley Orgánica 15/1999, de 13 de
diciembre, de Protección de Datos de Carácter Personal, le informamos de
que sus datos de carácter personal, recogidos de fuentes accesibles al
público o datos que usted nos ha facilitado previamente, proceden de
bases de datos propiedad de Advanced Software Production Line, S.L.
(ASPL). No obstante, usted puede ejercitar sus derechos de acceso,
rectificación, cancelación y oposición dispuestos en la mencionada Ley
Orgánica, notificándolo por escrito a:
ASPL - Protección Datos, C/Antonio Suárez 10 A-102, 28802, Alcalá de
Henares (Madrid).
More information about the Vortex
mailing list