Commit aaeefa3e authored by CharlieC's avatar CharlieC
Browse files

Merge

......@@ -64,6 +64,7 @@ Thanks to all those who participate in the project (in alphabetical order):
* Masato Yoshida
* Max Bolingbroke
* Nicholas Laver
* Nis Martensen
* Paul Joyce
* Paul Van Der Linden
* Philip Roche
......
3.0.7 (unreleased)
==================
Bugfixes
--------
* `#1588 <https://foss.heptapod.net/openpyxl/openpyxl/-/issues/1588>`
* `#1589 <https://foss.heptapod.net/openpyxl/openpyxl/-/issues/1589>`_ Exception raised when merging cells which do not have borders all the way round.
* `#1594 <https://foss.heptapod.net/openpyxl/openpyxl/-/issues/1594>`_ Python 2 print statement in the tutorial
Pull Requests
-------------
* `PR392 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/392/>`_ Add documentation on datetime handling
* `PR393 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/393/>`_ Drop dependency on jdcal
* `PR394 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/394/>`_ Datetime rounding
* `PR395 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/395/>`_ Unify handling of 1900 epoch
* `PR397 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/397/>`_ Add explicit support for reading datetime deltas
* `PR399 <https://foss.heptapod.net/openpyxl/openpyxl/-/merge_requests/399/>`_ Millisecond precision for datetimes
3.0.6 (2021-01-14)
==================
......
Dates and Times
===============
Dates and times can be stored in two distinct ways in XLSX files: as an
ISO 8601 formatted string or as a single number. `openpyxl` supports
both representations and translates between them and Python's datetime
module representations when reading from and writing to files. In either
representation, the maximum date and time precision in XLSX files is
millisecond precision.
XLSX files are not suitable for storing historic dates (before 1900),
due to bugs in Excel that cannot be fixed without causing backward
compatibility problems. To discourage users from trying anyway, Excel
deliberately refuses to recognize and display such dates. Consequently,
it is not advised to use `openpyxl` for such purposes either, especially
when exchanging files with others.
The date and time representations in Excel do not support timezones.
Timezone information attached to Python datetimes is therefore lost when
datetimes are stored in XLSX files.
Using the ISO 8601 format
-------------------------
To make `openpyxl` store dates and times in the ISO 8601 format on
writing your file, set the workbook's ``iso_dates`` flag to ``True``:
>>> import openpyxl
>>> wb = openpyxl.Workbook()
>>> wb.iso_dates = True
The benefit of using this format is that the meaning of the stored
information is not subject to interpretation, as it is with the single
number format [#f1]_.
The Office Open XML standard does not specify a supported subset of the
ISO 8601 duration format for representing time interval durations.
`openpyxl` therefore always uses the single number format for timedelta
values when writing them to file.
The 1900 and 1904 date systems
------------------------------
The 'date system' of an XLSX file determines how dates and times in the
single number representation are interpreted. XLSX files always use one
of two possible date systems:
* In the 1900 date system (the default), the reference date (with number 1) is 1900-01-01.
* In the 1904 date system, the reference date (with number 0) is 1904-01-01.
Complications arise not only from the different start numbers of the
reference dates, but also from the fact that the 1900 date system has a
built-in (but wrong) assumption that the year 1900 had been a leap year.
Excel deliberately refuses to recognize and display dates before the
reference date correctly, in order to discourage people from storing
historical data.
More information on this issue is available from Microsoft:
* https://docs.microsoft.com/en-us/office/troubleshoot/excel/1900-and-1904-date-system
* https://docs.microsoft.com/en-us/office/troubleshoot/excel/wrongly-assumes-1900-is-leap-year
In workbooks using the 1900 date system, `openpyxl` behaves the same as
Excel when translating between the worksheets' date/time numbers and
Python datetimes in January and February 1900. The only exception is 29
February 1900, which cannot be represented as a Python datetime object
since it is not a valid date.
You can get the date system of a workbook like this:
>>> import openpyxl
>>> wb = openpyxl.Workbook()
>>> if wb.epoch == openpyxl.utils.datetime.CALENDAR_WINDOWS_1900:
... print("This workbook is using the 1900 date system.")
...
This workbook is using the 1900 date system.
and set it like this:
>>> wb.epoch = openpyxl.utils.datetime.CALENDAR_MAC_1904
Handling timedelta values
-------------------------
Excel users can use number formats resembling ``[h]:mm:ss`` or
``[mm]:ss`` to display time interval durations, which `openpyxl`
considers to be equivalent to timedeltas in Python.
`openpyxl` recognizes these number formats when reading XLSX files and
returns datetime.timedelta values for the corresponding cells.
When writing timedelta values from worksheet cells to file, `openpyxl`
uses the ``[h]:mm:ss`` number format for these cells.
.. rubric:: Footnotes
.. [#f1] For example, the serial 1 in an Excel worksheet can be
interpreted as 00:00, as 24:00, as 1900-01-01, as 1440
(minutes), etc., depending solely on the formatting applied.
......@@ -237,6 +237,11 @@ Other topics
formula
.. toctree::
:maxdepth: 1
datetime
.. toctree::
:maxdepth: 1
......
......@@ -327,7 +327,7 @@ open an existing workbook::
>>> from openpyxl import load_workbook
>>> wb2 = load_workbook('test.xlsx')
>>> print wb2.sheetnames
>>> print(wb2.sheetnames)
['Sheet2', 'New Title', 'Sheet1']
This ends the tutorial for now, you can proceed to the :doc:`usage` section
......@@ -9,4 +9,4 @@ __author_email__ = "charlie.clark@clark-consulting.eu"
__license__ = "MIT"
__maintainer_email__ = "openpyxl-users@googlegroups.com"
__url__ = "https://openpyxl.readthedocs.io"
__version__ = "3.0.6"
__version__ = "3.0.7"
......@@ -3,7 +3,7 @@
from openpyxl.compat import safe_string
from openpyxl.xml.functions import Element, SubElement, whitespace, XML_NS, REL_NS
from openpyxl import LXML
from openpyxl.utils.datetime import to_excel, days_to_time
from openpyxl.utils.datetime import to_excel, days_to_time, to_ISO8601
from datetime import timedelta
......@@ -24,10 +24,8 @@ def _set_attributes(cell, styled=None):
value = cell._value
if cell.data_type == "d":
if cell.parent.parent.iso_dates:
if isinstance(value, timedelta):
value = days_to_time(value)
value = value.isoformat()
if cell.parent.parent.iso_dates and not isinstance(value, timedelta):
value = to_ISO8601(value)
else:
attrs['t'] = "n"
value = to_excel(value, cell.parent.parent.epoch)
......
......@@ -45,9 +45,6 @@ ILLEGAL_CHARACTERS_RE = re.compile(r'[\000-\010]|[\013-\014]|[\016-\037]')
ERROR_CODES = ('#NULL!', '#DIV/0!', '#VALUE!', '#REF!', '#NAME?', '#NUM!',
'#N/A')
ERROR_CODES = ERROR_CODES
TYPE_STRING = 's'
TYPE_FORMULA = 'f'
TYPE_NUMERIC = 'n'
......
......@@ -78,7 +78,7 @@ def test_write_cell(worksheet, write_cell_implementation, value, expected):
(datetime.time(14, 15, 25), False, """<c r="A1" t="n" s="1"><v>0.5940393518518519</v></c>"""),
(datetime.time(14, 15, 25), True, """<c r="A1" t="d" s="1"><v>14:15:25</v></c>"""),
(datetime.timedelta(1, 3, 15), False, """<c r="A1" t="n" s="1"><v>1.000034722395833</v></c>"""),
(datetime.timedelta(1, 3, 15), True, """<c r="A1" t="d" s="1"><v>00:00:03.000015</v></c>"""),
(datetime.timedelta(1, 3, 15), True, """<c r="A1" t="n" s="1"><v>1.000034722395833</v></c>"""),
]
)
def test_write_date(worksheet, write_cell_implementation, value, expected, iso_dates):
......
......@@ -3,11 +3,6 @@
import datetime
from openpyxl.compat import safe_string
from openpyxl.utils.datetime import (
CALENDAR_WINDOWS_1900,
to_ISO8601,
from_ISO8601,
)
from openpyxl.descriptors import (
String,
DateTime,
......@@ -34,7 +29,7 @@ class NestedDateTime(DateTime, NestedText):
tagname = "{%s}%s" % (namespace, tagname)
el = Element(tagname)
if value is not None:
el.text = to_ISO8601(value)
el.text = value.isoformat(timespec="seconds") + 'Z'
return el
......@@ -91,12 +86,12 @@ class DocumentProperties(Serialisable):
lastPrinted=None,
revision=None,
version=None,
created=datetime.datetime.now(),
created=datetime.datetime.utcnow(),
creator="openpyxl",
description=None,
identifier=None,
language=None,
modified=datetime.datetime.now(),
modified=datetime.datetime.utcnow(),
subject=None,
title=None,
):
......
......@@ -78,7 +78,7 @@ def test_from_tree(datadir, SampleProperties):
def test_qualified_datetime():
from ..core import QualifiedDateTime
dt = QualifiedDateTime()
tree = dt.to_tree("time", datetime.datetime(2015, 7, 20, 12, 30))
tree = dt.to_tree("time", datetime.datetime(2015, 7, 20, 12, 30, 00, 123456))
xml = tostring(tree)
expected = """
<time xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="dcterms:W3CDTF">
......@@ -100,7 +100,7 @@ def dcterms_prefix(request):
def test_qualified_datetime_ns(dcterms_prefix):
from ..core import QualifiedDateTime
dt = QualifiedDateTime()
tree = dt.to_tree("time", datetime.datetime(2015, 7, 20, 12, 30))
tree = dt.to_tree("time", datetime.datetime(2015, 7, 20, 12, 30, 00, 987654))
xml = tostring(tree) # serialise to make remove QName
tree = fromstring(xml)
xsi = tree.attrib["{%s}type" % XSI_NS]
......
......@@ -93,8 +93,9 @@ FORMAT_CURRENCY_EUR_SIMPLE = '[$EUR ]#,##0.00_-'
COLORS = r"\[(BLACK|BLUE|CYAN|GREEN|MAGENTA|RED|WHITE|YELLOW)\]"
LITERAL_GROUP = r'".*?"' # anything in quotes
LOCALE_GROUP = r'\[.+\]' # anything in square brackets, including colours
LOCALE_GROUP = r'\[(?!hh?\]|mm?\]|ss?\])[^\]]*\]' # anything in square brackets, except hours or minutes or seconds
STRIP_RE = re.compile(f"{LITERAL_GROUP}|{LOCALE_GROUP}")
TIMEDELTA_RE = re.compile(r'\[hh?\](:mm(:ss(\.0*)?)?)?|\[mm?\](:ss(\.0*)?)?|\[ss?\](\.0*)?', re.I)
# Spec 18.8.31 numFmts
......@@ -108,6 +109,13 @@ def is_date_format(fmt):
return re.search(r"[^\\][dmhysDMHYS]", fmt) is not None
def is_timedelta_format(fmt):
if fmt is None:
return False
fmt = fmt.split(";")[0] # only look at the first format
return TIMEDELTA_RE.search(fmt) is not None
def is_datetime(fmt):
"""
Return date, time or datetime
......
......@@ -25,6 +25,7 @@ from .numbers import (
BUILTIN_FORMATS_MAX_SIZE,
BUILTIN_FORMATS_REVERSE,
is_date_format,
is_timedelta_format,
builtin_format_code
)
from .named_styles import (
......@@ -156,6 +157,7 @@ class Stylesheet(Serialisable):
And index datetime formats
"""
date_formats = set()
timedelta_formats = set()
custom = self.custom_formats
formats = self.number_formats
for idx, style in enumerate(self.cell_styles):
......@@ -170,7 +172,11 @@ class Stylesheet(Serialisable):
if is_date_format(fmt):
# Create an index of which styles refer to datetimes
date_formats.add(idx)
if is_timedelta_format(fmt):
# Create an index of which styles refer to timedeltas
timedelta_formats.add(idx)
self.date_formats = date_formats
self.timedelta_formats = timedelta_formats
def to_tree(self, tagname=None, idx=None, namespace=None):
......@@ -204,6 +210,7 @@ def apply_stylesheet(archive, wb):
wb._cell_styles = stylesheet.cell_styles
wb._named_styles = stylesheet.named_styles
wb._date_formats = stylesheet.date_formats
wb._timedelta_formats = stylesheet.timedelta_formats
for ns in wb._named_styles:
ns.bind(wb)
......
......@@ -77,6 +77,16 @@ def test_strip_quotes(fmt, stripped):
(r"0_ ;[Red]\-0\ ", False),
(r"\Y000000", False),
(r'#,##0.0####" YMD"', False),
('[h]', True),
('[ss]', True),
('[s].000', True),
('[m]', True),
('[mm]', True),
('[Blue]\+[h]:mm;[Red]\-[h]:mm;[Green][h]:mm', True),
('[>=100][Magenta][s].00', True),
('[h]:mm;[=0]\-', True),
('[>=100][Magenta].00', False),
('[>=100][Magenta]General', False),
]
)
def test_is_date_format(format, result):
......@@ -84,6 +94,41 @@ def test_is_date_format(format, result):
assert is_date_format(format) is result
@pytest.mark.parametrize("format, result",
[
('m:ss', False),
('[h]', True),
('[hh]', True),
('[h]:mm:ss', True),
('[hh]:mm:ss', True),
('[h]:mm:ss.000', True),
('[hh]:mm:ss.0', True),
('[h]:mm', True),
('[hh]:mm', True),
('[m]:ss', True),
('[mm]:ss', True),
('[m]:ss.000', True),
('[mm]:ss.0', True),
('[s]', True),
('[ss]', True),
('[s].000', True),
('[ss].0', True),
('[m]', True),
('[mm]', True),
('h:mm', False),
('[Blue]\+[h]:mm;[Red]\-[h]:mm;[h]:mm', True),
('[Blue]\+[h]:mm;[Red]\-[h]:mm;[Green][h]:mm', True),
('[>=100][Magenta][s].00', True),
('[h]:mm;[=0]\-', True),
('[>=100][Magenta].00', False),
('[>=100][Magenta]General', False),
]
)
def test_is_timedelta_format(format, result):
from ..numbers import is_timedelta_format
assert is_timedelta_format(format) is result
@pytest.mark.parametrize("fmt, typ",
[
(FORMAT_DATE_DATETIME, "datetime"),
......
......@@ -5,35 +5,34 @@ from __future__ import division
# Python stdlib imports
import datetime
from datetime import timedelta, tzinfo
from datetime import timedelta, timezone
from math import isnan
import re
from jdcal import (
gcal2jd,
jd2gcal,
MJD_0
)
# constants
MAC_EPOCH = datetime.date(1904, 1, 1)
WINDOWS_EPOCH = datetime.date(1899, 12, 30)
CALENDAR_WINDOWS_1900 = sum(gcal2jd(WINDOWS_EPOCH.year, WINDOWS_EPOCH.month, WINDOWS_EPOCH.day))
CALENDAR_MAC_1904 = sum(gcal2jd(MAC_EPOCH.year, MAC_EPOCH.month, MAC_EPOCH.day))
MAC_EPOCH = datetime.datetime(1904, 1, 1)
WINDOWS_EPOCH = datetime.datetime(1899, 12, 30)
CALENDAR_WINDOWS_1900 = 2415018.5 # Julian date of WINDOWS_EPOCH
CALENDAR_MAC_1904 = 2416480.5 # Julian date of MAC_EPOCH
CALENDAR_WINDOWS_1900 = WINDOWS_EPOCH
CALENDAR_MAC_1904 = MAC_EPOCH
SECS_PER_DAY = 86400
EPOCH = datetime.datetime.utcfromtimestamp(0)
ISO_FORMAT = '%Y-%m-%dT%H:%M:%SZ'
ISO_REGEX = re.compile(r'''
(?P<date>(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2}))?T?
(?P<time>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})(.(?P<ms>\d{1,3}))?)?Z?''',
(?P<time>(?P<hour>\d{2}):(?P<minute>\d{2})(:(?P<second>\d{2})(?P<microsecond>\.\d{1,3})?)?)?Z?''',
re.VERBOSE)
ISO_DURATION = re.compile(r'PT((?P<hours>\d+)H)?((?P<minutes>\d+)M)?((?P<seconds>\d+(\.\d{1,3})?)S)?')
def to_ISO8601(dt):
"""Convert from a datetime to a timestamp string."""
return datetime.datetime.strftime(dt, ISO_FORMAT)
if hasattr(dt, "microsecond") and dt.microsecond:
return dt.isoformat(timespec="milliseconds")
return dt.isoformat()
def from_ISO8601(formatted_string):
......@@ -45,80 +44,85 @@ def from_ISO8601(formatted_string):
Times B.1.2 and B.2.2
Datetimes B.1.3 and B.2.3
There is no concept of timedeltas
There is no concept of timedeltas in the specification, but Excel
writes them (in strict OOXML mode), so these are also understood.
"""
if not formatted_string:
return None
match = ISO_REGEX.match(formatted_string)
if not match:
raise ValueError("Invalid datetime value {}".format(formatted_string))
parts = {k:int(v) for k, v in match.groupdict().items() if v is not None and v.isdigit()}
if 'year' not in parts:
dt = datetime.time(parts['hour'], parts['minute'], parts['second'])
elif 'hour' not in parts:
dt = datetime.date(parts['year'], parts['month'], parts['day'])
else:
dt = datetime.datetime(year=parts['year'], month=parts['month'],
day=parts['day'], hour=parts['hour'], minute=parts['minute'],
second=parts['second'])
if 'ms' in parts:
# add to seconds and then subtract to avoid padding issues
ms = parts['ms']
s = parts['second']
ms = (float(s + ms) - int(s)) * 1000
dt += timedelta(microseconds=ms)
return dt
def to_excel(dt, offset=CALENDAR_WINDOWS_1900):
if match and any(match.groups()):
parts = match.groupdict(0)
for key in ["year", "month", "day", "hour", "minute", "second"]:
if parts[key]:
parts[key] = int(parts[key])
if parts["microsecond"]:
parts["microsecond"] = int(float(parts['microsecond']) * 1_000_000)
if not parts["date"]:
dt = datetime.time(parts['hour'], parts['minute'], parts['second'], parts["microsecond"])
elif not parts["time"]:
dt = datetime.date(parts['year'], parts['month'], parts['day'])
else:
del parts["time"]
del parts["date"]
dt = datetime.datetime(**parts)
return dt
match = ISO_DURATION.match(formatted_string)
if match and any(match.groups()):
parts = match.groupdict(0)
for key, val in parts.items():
if val:
parts[key] = float(val)
return datetime.timedelta(**parts)
raise ValueError("Invalid datetime value {}".format(formatted_string))
def to_excel(dt, epoch=WINDOWS_EPOCH):
"""Convert Python datetime to Excel serial"""
if isinstance(dt, datetime.time):
return time_to_days(dt)
if isinstance(dt, datetime.timedelta):
return timedelta_to_days(dt)
if isnan(dt.year): # Pandas supports Not a Date
if isnan(dt.year): # Pandas supports Not a Date
return
jul = sum(gcal2jd(dt.year, dt.month, dt.day)) - offset
if jul <= 60 and offset == CALENDAR_WINDOWS_1900:
jul -= 1
if hasattr(dt, 'time'):
jul += time_to_days(dt)
return jul
if not hasattr(dt, "date"):
dt = datetime.datetime.combine(dt, datetime.time())
def from_excel(value, offset=CALENDAR_WINDOWS_1900):
if value is None:
return
if 1 < value < 60 and offset == CALENDAR_WINDOWS_1900:
value += 1
parts = list(jd2gcal(MJD_0, value + offset - MJD_0))
_, fraction = divmod(value, 1)
jumped = (parts[-1] == 0 and fraction > 0)
diff = datetime.timedelta(days=fraction)
if 0 < abs(value) < 1:
return days_to_time(diff)
if not jumped:
return datetime.datetime(*parts[:3]) + diff
else:
return datetime.datetime(*parts[:3] + [0])
# rebase on epoch and adjust for < 1900-03-01
days = (dt - epoch).days
if 0 < days <= 60 and epoch == WINDOWS_EPOCH:
days -= 1
return days + time_to_days(dt)
class GMT(tzinfo):
def utcoffset(self, dt):
return timedelta(0)
def from_excel(value, epoch=WINDOWS_EPOCH, timedelta=False):
"""Convert Excel serial to Python datetime"""
if value is None:
return
def dst(self, dt):
return timedelta(0)
if timedelta:
td = datetime.timedelta(days=value)
if td.microseconds:
# round to millisecond precision
td = datetime.timedelta(seconds=td.total_seconds() // 1,
microseconds=round(td.microseconds, -3))
return td
day, fraction = divmod(value, 1)
diff = datetime.timedelta(milliseconds=round(fraction * SECS_PER_DAY * 1000))
if 0 <= value < 1 and diff.days == 0:
return days_to_time(diff)
if 0 < value < 60 and epoch == WINDOWS_EPOCH:
day += 1
return epoch + datetime.timedelta(days=day) + diff
def tzname(self,dt):
return "GMT"
try:
from datetime import timezone
UTC = timezone(timedelta(0))
except ImportError:
# Python 2
UTC = GMT()
UTC = timezone(timedelta(0))
def time_to_days(value):
......@@ -135,12 +139,7 @@ def time_to_days(value):
def timedelta_to_days(value):
"""Convert a timedelta value to fractions of a day"""
if not hasattr(value, 'total_seconds'):
secs = (value.microseconds +
(value.seconds + value.days * SECS_PER_DAY) * 10**6) / 10**6
else:
secs = value.total_seconds()
return secs / SECS_PER_DAY
return value.total_seconds() / SECS_PER_DAY
def days_to_time(value):
......
......@@ -12,15 +12,27 @@ from datetime import (
import pytest
def test_to_iso():
@pytest.mark.parametrize("value, expected",
[
(datetime(2013, 7, 15, 6, 52, 33), "2013-07-15T06:52:33"),
(datetime(2013, 7, 15, 6, 52, 33, 123456), "2013-07-15T06:52:33.123"),
(date(2013, 7, 15), "2013-07-15"),
(time(0, 1, 42), "00:01:42"),
(time(0, 1, 42, 123456), "00:01:42.123"),
]
)
def test_to_iso(value, expected):
from ..datetime import to_ISO8601
assert to_ISO8601(datetime(2013, 7, 15, 6, 52, 33)) == "2013-07-15T06:52:33Z"
assert to_ISO8601(value) == expected