EDIT: name changed from "Complex Pointer Calculation in Array Length (offsetOf)"
Seems to me, CFFI should be able to handle this case given ...:
int a[((((long) &((HeapTupleHeaderData *)0)->t_infomask2) - sizeof(uint32)) % 8)]
This is more a question I think of adding leniency to the parser (perhaps via a new keyword argument to cdef?) since CFFI can already handle this case using __dotdotdot__. From what I recall, the code in question is in CFFI proper rather than pycparser (shouldn't be hard to fix); I can propose a patch
This would help in exposing larger libraries with CFFI I think.
There's related issue (enhancement) I'd like to discuss as well (can file a separate feature request if desired). __dotdotdot__ doesn't work in the following case and I'm curious 1) why and 2) if we can fix it in at least some cases (a similar construct does work for structs, so it seems like we might be able to fix this in all cases):
typedef int jmp_buf[...]
This would help with something jmp_buf for example. Happy to take a crack at it if you think it's feasible and can point me in the right direction :)
Edited
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Child items
0
Show closed items
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Linked items
0
Link issues together to show that they're related.
Learn more.
int a[complicated_expression]; is not supported and I think it's correct. If you say int a[10]; in the cdef, then cffi will check that a is really an array of length 10. But if you say int a[...]; then it doesn't check the length. So I think it's potentially dangerous to add pseudo-support for int a[whatever]; and assume it's equivalent to int a[...]; because we can't make sense out of whatever.
typedef int jmp_buf[...]; seems to work already. Can you include a complete example where it doesn't work as expected?
I'm replacing them manually right now with __dotdotdot__ and it's working fine, but it's not long-term sustainable and certainly isn't efficient. It means that I have to wait for an error, identify the problematic expression manually and add it to my replacement function (not ideal). It also means I have to parse the code only to pass it back to CFFI to be parsed again.
I don't think it necessarily needs to be built into CFFI, but right now it's very difficult to extend the parsing without copying an uncomfortably large amount of code.
What if I exposed the parser to subclassing? It doesn't need to be public API. Something like ffibuilder = FFI(_parser=MyParser) and then add some error hooks into the CFFI parser. In particular, I'm currently using a subclass that needs a hook in _internal_parse here:
else: raise CDefError("unexpected <%s>: this construct is valid " "C but not valid in cdef()" % decl.__class__.__name__, decl)
(my code here catches inline function definitions and converts them into declarations, only tangentially related to the current issue). The thing is that _internal_parse is quite long and I don't like copying that much code... I guess I could catch the error in a subclass and do something that way, but it doesn't seem ideal either.
Another place that's a bit difficult to hook into right now is the underlying CParser from pycparsing (since it's fetched from a global cache). Right now I'm only in proof of concept stage and am just using mock to sub mine in, hehehe, but it'd be nice to be able to specify this on the main Parser class, something like MyParser.cparser_cls. (I need to do this currently to use the GnuCParser from pycparsingext, not sure if this will be necessary in the end, but it's been useful in getting going, again tangentially related to the current issue.)
In order to address the current issue, a hook would need to be added to Parser._parse_constant when no op is recognized.
Let me double check the typedef int blah[...] case again. I could have sworn it was throwing an error, but as you may be able to tell, I'm wading through a fair bit of complexity and might have gotten turned about.
In any, case I love CFFI, just hope we can expose the guts a bit more to make her more amenable to parsing large libraries (I know you've talked about this being a goal in the past).
I'm going to rename this issue to better reflect my goals here. Happy to bring it to the mailing list too, if that's preferable. Or just put up a small patch and you can see what you think.
Thanks!
(also my apologies for the duplicate issue; I was having internet troubles)
Dylan Youngchanged title from Complex Pointer Calculation in Array Length (offsetOf) to Add Hooks to Extend CFFI Parsing (to make it easier to handle large libraries)
changed title from Complex Pointer Calculation in Array Length (offsetOf) to Add Hooks to Extend CFFI Parsing (to make it easier to handle large libraries)
Verified the case of the cffi.VerificationError: type <int[...]> badly placed: the '...' array length can only be used on global arrays or on fields of structures coming from typedef int jmp_buf[...];.
Here's the full stack trace (I'm beginning to doubt this is a CFFI issue at all):
..."/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/setuptools_ext.py", line 142, in run base_class.run(self) File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/setuptools_ext.py", line 141, in run ext.sources[0] = make_mod(self.build_temp, pre_run) File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/setuptools_ext.py", line 127, in make_mod updated = recompiler.make_c_source(ffi, module_name, source, c_file) File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/recompiler.py", line 1424, in make_c_source verbose) File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/recompiler.py", line 1398, in _make_c_or_py_source recompiler.collect_type_table() File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/recompiler.py", line 186, in collect_type_table method(tp, self._typesdict[tp]) File "/usr/local/Cellar/python3/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/cffi/recompiler.py", line 1368, in _emit_bytecode_ArrayType str(tp).replace('/*...*/', '...'),))
To give you some context, I'm working on a project to expose more or less all of PostgreSQL to python (incrementally of course) and eventually spin a PyPy compatible procedural language for PostgreSQL with better support for PostgreSQL extensibility mechanisms than the current pl/python.
So far I've managed to expose the function manager and the server programming interface and it's been (relatively) straightforward. Thanks for the great tool!
I should have some code up in the next week or two.
In any, case I love CFFI, just hope we can expose the guts a bit more to make her more amenable to parsing large libraries (I know you've talked about this being a goal in the past).
I'm pretty sure I have talked about this not being a goal. My goal is rather that CFFI comes with parsing rules for cdef() that makes sense for CFFI only; then, if you want a non-manual way to expose some large library, the general idea is that you should use a different tool on top of CFFI to help produce the cdef(). It might sound like pointless work to parse C code, emit more C-ish code into a cdef(), and have CFFI re-parse that, but it's not really important IMHO.
Nothing in the modules cffi.cparser or cffi.model is supposed to be public or subclassable, and I don't plan to change that---I want to keep the option to make any change there. If you really want to go that route anyway, then yes, I'd probably accept changes that have no effect apart from making things easier to subclass, but be warned that there is guarantee that it won't change again in the next version.
ffibuilder.cdef(""" typedef int jmp_buf[...]; extern jmp_buf *PG_exception_stack;""")
Ah, I understood again what is going on. It works fine if you say typedef int jmp_buf[]; instead. This means that you're not interested in the length at all, so e.g. you won't be able to ask ffi.sizeof('jmp_buf') at runtime; if you use instead [...] then CFFI produces C code that will know this size.
The problem of [...] is that there is support for typedef int jmp_buf[...]; but no support for extern int (*PG)[...];, and for some reason the computed length of jmp_buf does not propagate to the global variable declaration.
That's fine with me. I think it'd be nice to eventually have a public API for this, but the internals of CFFI are changing fast enough that this isn't really feasible right now. In fact, it's changes within CFFI that worry me most with my current approach, as I'd have to duplicate some rather large functions. If hooks were present, breakage and necessary changes on upgrades would be easier to detect and implement.
Hmmm... maybe it wasn't you that mentioned CFFI being weak at handling large libraries, but I'm certain I've seem it in my ambling through the docs. In particular, I saw mention of developing another library to handle large C libraries (and this seems like a bad idea to me unless it's built on top of CFFI).
In my experience so far the issues aren't that severe, just need a way to handle a few corner cases in header files.
Note that I am still preprocessing using pcpp. I could potentially hook in there to do my extra parsing, but it's not a complete C parser, so the work is a bit harder there.