[Vortex] Possible race condition between reader worker thread and TLS negotiation

[David Drusowitsch]

Hi Vortex Team,

we are currently using a relatively old vortex version 1.0.5.b2830.g2831 in our applications/services.
A few weeks ago we got a bug report, one of our services was acting weird, it got dis-functional or even worse it segfaults over the day.
After a lot of investigation work, we were able to reproduce it.

Most of the time it happens in the in the region of vortex reader.
The reader struggles after trying to dispatch a previously freed connection.  Classic invalid memory access.
Our service uses multiple threads to establish secure TLS connections. Due to stability and other paranoid reasons, this is done before every data transmission.
The more threads are doing communication work, the bigger the possibility to crash.


So this is my explanation:
The connection gets renewed during TLS negotiation. A new "object" would be allocated and the other one get marked as not connected and waiting for destruction (unref).
If I understood the process of the reader correctly, a new connection gets add to the watching queue and epoll is waiting till new data is available on a socket.

If the connection "object" exchange exactly happens during the run of the reader worker thread, it could happen that it frees the "object" to soon, resulting in strange behaviours I mentioned before.
Seems the watching of the former connection was not already finished and needs at least one run through of __vortex_reader_run.
The object got freed during __vortex_reader_build_set_to_watch but it's reference still remaining as an reference entry in the private area on epoll.
What would lead to an invalid memory access during vortex_io_waiting_invoke_dispatch in further consequence.

Attached you'll find a short patch, where we moved the finally destruction of the connection towards the place dispatching finished.
This fixed out issues.
Would be possible that this issue is also present in the last release of version 1.0 (I think 1.0.17),  would be cool if you could check on this.



Best Regards,
David




--

David Drusowitsch

Software Engineer

Lassallestrasse 7A / Unit 4 / 2 • 1020 Vienna • Austria

o: +43 508-100 | ddrusowitsch at barracuda.com<mailto:ddrusowitsch at barracuda.com>

Connect with us: barracuda.com/connect<https://www.barracuda.com/connect>

[cid:part2.03000505.09030708 at barracuda.com]
________________________________
Barracuda Networks AG
Vorsitzender des Aufsichtsrates/ Chairman of the supervisory board: Dr. Klaus Perktold
Vorstand/ Executive Board: Dr. Wieland Alge, Mag. Guenter Klausner
Sitz der Gesellschaft/ Registered office: 6020 Innsbruck, Austria
Handelsgericht Innsbruck Firmenbuch/ Registration Number: 184392s
UID-Nr/ VAT Number: ATU47509003

Zweigniederlassung Deutschland/ Office Germany: Radlkoferstr. 2, 81373 München
Handelsregister München / Registration Number: HRB 171749
UID-Nr/ VAT Number: DE237607533

===========================================================
Considering Office 365? Barracuda security and storage solutions can help. Learn more about Barracuda solutions for Office 365 at http://barracuda.com/office365. 

DISCLAIMER:
This e-mail and any attachments to it contain confidential and proprietary material of Barracuda, its affiliates or agents, and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient. If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.
===========================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aspl.es/pipermail/vortex/attachments/20161019/f5c5328c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dcjgbjfi.gif
Type: image/gif
Size: 6743 bytes
Desc: not available
URL: <http://lists.aspl.es/pipermail/vortex/attachments/20161019/f5c5328c/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vortex-phion-invalid-mem-access-fix.patch
Type: text/x-patch
Size: 3433 bytes
Desc: not available
URL: <http://lists.aspl.es/pipermail/vortex/attachments/20161019/f5c5328c/attachment-0001.bin>