Tokenizer conformance to spec
Created originally on Bitbucket by felipeochoa (Felipe)
Not sure how I hadn't found this before, but the formula syntax is included in ISO/IEC-29500-1 §18.17, which can be found in the publicly available standards section of the ISO website. (Ctrl-F for ISO/IEC 29500-1
). Reading through the spec, I noticed a variety of issues with the tokenizer:
-
#GETTING_DATA
is not recognized as a valid error code - Scientific notation only accepts capitalized
E
(should accept both capital and lowercase) -
+
is not allowed as a prefix operator (though this one probably causes no harm) -
,
is always treated as a separator (should be an infix operator for reference unions) -
:
is treated as part of a range reference (should be an infix operator for reference ranges) -
whitespace
is not dealt with properly. As an infix operator, it intersects ranges. Otherwise it should be ignored according to:
In a formula, an arbitrary number of space characters (U+0020) can precede the first token or follow the final token. An arbitrary number of space characters can separate two adjacent tokens, except that no space characters shall separate a function-name from the left parenthesis (() that follows it. Such space characters have no effect on the semantics of a formula; however, such spaces shall be distinguished from the space operator