The original problem is best described by this IRC log: https://libera.irclog.whitequark.org/hpy/2021-06-22
15:52 <ronan> antocuni: I don't know if that's the issue but the free_list logic doesn't seem threadsafe
17:03 <ronan> hmmm, self.free_list.append(index) fails in a way that corrupts the list?!
17:10 <ronan> and that happens because handles.close() is called with an RPython exception raised
17:42 <ronan> aaahh, the problem is that you can't call anything RPython, not even HPyTracker_Close or HPy_Close, with an exception raised
17:44 <ronan> but you need them for cleanup...
21:59 <antocuni> ronan: ouch, indeed
21:59 <antocuni> I think you got it
22:00 <antocuni> I also arrived to the conclusion that the free_list got corrupt, but didn't understand yet what caused it
22:00 <antocuni> so, I think that what happens is this:
22:01 <antocuni> 1. to raise an HPy exception we raise an RPython exception
22:02 <antocuni> 2. while we are still in C, we call HPy_Close, which causes self.free_list.append() in HandleManager
22:02 <antocuni> 3. if the free_list doesn't have enough room, it needs to reallocate. But since the RPython exception is set, the malloc think it fails
22:02 <antocuni> 4. and the funny thing happens
22:03 <antocuni> we need to find a fix and/or a workaround for this. I fear that mapping HPy exceptions to RPython exceptions is essential for performance
22:04 <antocuni> one possible workaround is to ensure that self.free_list has always as much capacity as self.handles_w, so that self.free_list.append does not causes any reallocation. But it looks more like a hack than a proper fix :(
22:05 <antocuni> arigato: ^^^, in case you have any idea
22:18 <ronan> no, it's dumber than that. At 3. it calls ll_list_resize() which merely increments the length field, but then it checks for an exception and sees the one from 1. so it exits from append() without setting the item
This MR fixes is by implementing HandleManager.free_list
with a custom class specifically designed to ensure it cannot raise on a push().
It is more a workaround than a proper fix though. For the proper fix we probably need to implement a @cannot_raise
RPython decorator which actually ensure that the marked functions don't raise.