Merged requested to merge branch/hpy-0.0.2-experiment into branch/hpy-0.0.2
The original problem is best described by this IRC log: https://libera.irclog.whitequark.org/hpy/2021-06-22
15:52 <ronan> antocuni: I don't know if that's the issue but the free_list logic doesn't seem threadsafe 17:03 <ronan> hmmm, self.free_list.append(index) fails in a way that corrupts the list?! 17:10 <ronan> and that happens because handles.close() is called with an RPython exception raised 17:42 <ronan> aaahh, the problem is that you can't call anything RPython, not even HPyTracker_Close or HPy_Close, with an exception raised 17:44 <ronan> but you need them for cleanup... 21:59 <antocuni> ronan: ouch, indeed 21:59 <antocuni> I think you got it 22:00 <antocuni> I also arrived to the conclusion that the free_list got corrupt, but didn't understand yet what caused it 22:00 <antocuni> so, I think that what happens is this: 22:01 <antocuni> 1. to raise an HPy exception we raise an RPython exception 22:02 <antocuni> 2. while we are still in C, we call HPy_Close, which causes self.free_list.append() in HandleManager 22:02 <antocuni> 3. if the free_list doesn't have enough room, it needs to reallocate. But since the RPython exception is set, the malloc think it fails 22:02 <antocuni> 4. and the funny thing happens 22:03 <antocuni> we need to find a fix and/or a workaround for this. I fear that mapping HPy exceptions to RPython exceptions is essential for performance 22:04 <antocuni> one possible workaround is to ensure that self.free_list has always as much capacity as self.handles_w, so that self.free_list.append does not causes any reallocation. But it looks more like a hack than a proper fix :( 22:05 <antocuni> arigato: ^^^, in case you have any idea 22:18 <ronan> no, it's dumber than that. At 3. it calls ll_list_resize() which merely increments the length field, but then it checks for an exception and sees the one from 1. so it exits from append() without setting the item
This MR fixes is by implementing
HandleManager.free_list with a custom class specifically designed to ensure it cannot raise on a push().
It is more a workaround than a proper fix though. For the proper fix we probably need to implement a
@cannot_raise RPython decorator which actually ensure that the marked functions don't raise.