Tokenizer improvements
Created originally on Bitbucket by felipeochoa (Felipe)
Was already merged in Bitbucket before import, marked as merged by the import user
Tokenizer improvements
(Closes issue #582 (closed))
This PR addresses most of the issues from issue #582 (closed). Specifically, the following changes are in here:
-
#GETTING_DATA
is now recognized as a valid error code -
Scientific notation now accepts both capital and lowercase
E
-
Tokenizer.parse()
is renamed toTokenizer._parse()
and is now called automatically upon creating aTokenizer
instance. Client code can now simply do:tok = Tokenizer(formula) for token in tok.items: # ...
-
The
Tokenizer.parse_*
have similarly been renamed toTokenizer._parse_*
and are now private -
Updates
DefinedName
to use the new tokenizer pattern
Note: I wasn't able to get tox
tests running on my Windows machine. I tested under Python 3.4, which I think should be enough, given these changes are unlikely to introduce Py 2 vs. 3 issues or binary issues.