Tokenizer fails when encountering unexpected character
Created originally on Bitbucket by artifex_knowledge (Gericke Potgieter)
Environment: Python 3.5.2 Openpyxl version: 2.4.0
Issue: In some cases Tokenizer fails when encountering an unexpected character. It was found that the issue lies with the parsing of ranges in other worksheets. The naming convention for other worksheets in formulas are as follows:
For a single word wordsheet name: Worksheet!A1 For a multiple word worksheet name: 'This Worksheet'!A1
However, in some instances the formula may contain a single worksheet name as follows: 'Worksheet'!A1 when it forms part of a range e.g. Worksheet!A1:'Worksheet'!A15
This will lead to an error as the single quotes are not expected. The traceback is as follows:
#!python
Traceback: Traceback (most recent call last):
File "[project file]", line 1251, in test_all
formula_table = get_formulas(full_workbook, file_name)
File "[project file]", line 24, in get_formulas
base_tokens = returns_tokens(cell)
File "[project file]", line 105, in returns_tokens
tok = Tokenizer(t_cell.value)
File "/usr/local/lib/python3.5/dist-packages/openpyxl/formula/tokenizer.py", line 53, in __init__
self._parse()
File "/usr/local/lib/python3.5/dist-packages/openpyxl/formula/tokenizer.py", line 86, in _parse
self.offset += dispatcher[curr_char]()
File "/usr/local/lib/python3.5/dist-packages/openpyxl/formula/tokenizer.py", line 107, in _parse_string
self.assert_empty_token()
File "/usr/local/lib/python3.5/dist-packages/openpyxl/formula/tokenizer.py", line 298, in assert_empty_token
(self.offset, self.formula))
openpyxl.formula.tokenizer.TokenizerError: Unexpected character at position 94 in '=IF('Debt Sizing'!$E$15="Yes",ROUND('Debt Sizing'!H227,2)+IF(X104=tenor*2,1-SUM(Inputs!$W$111:'Inputs'!W111)-ROUND('Debt Sizing'!H227,2)),'Debt Sizing'!H227)'
Due to the sensitivity of the workbook I am unfortunately unable to include a sample.