diff --git a/README-DO-NOT-EDIT b/README-DO-NOT-EDIT deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_UkVBRE1FLURPLU5PVC1FRElU..0000000000000000000000000000000000000000 --- a/README-DO-NOT-EDIT +++ /dev/null @@ -1,3 +0,0 @@ -The content of public is not written by hand! -It are generated via a Makefile. -The files in source/* are the old yatiblog sources. diff --git a/source/README b/source/README deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL1JFQURNRQ==..0000000000000000000000000000000000000000 --- a/source/README +++ /dev/null @@ -1,14 +0,0 @@ -You can get necessary software by doing: - - pip install yatiblog - -(tested with version 1.0) you can do: - - yatiblog -o .. - -you'll get html output in the parent directory. you need an account -Then you can check it in, login to pypy@pypy.org -and go to pypy.org/htdocs/ and type "hg pull -u". - -(Note: nowadays I think there is a script that automatically does -"hg pull -u" every half hour, so logging in is not needed.) diff --git a/source/_layouts/archive.genshi b/source/_layouts/archive.genshi deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL19sYXlvdXRzL2FyY2hpdmUuZ2Vuc2hp..0000000000000000000000000000000000000000 --- a/source/_layouts/archive.genshi +++ /dev/null @@ -1,34 +0,0 @@ ---- -layout: site -license: MIT ---- -<div xmlns:py="http://genshi.edgewall.org/"> -<?python - - MONTHS = [ - 'Zero Month', - 'January', 'February', 'March', 'April', 'May', 'June', - 'July', 'August', 'September', 'October', 'November', 'December' - ] - - items = [] - -?> -<div class="section-info" id="all-articles"><strong>Blog Posts Archive</strong></div> -<ul> - <li py:for="item in sorted([item for item in items if item.get('x-created') and not item.get('subtitle')], key=lambda x: x['x-created'], reverse=True)"> - <a id="article-${item['__name__']}" href="${item['__name__']}.html">${Markup(item.get('title') or item['__title__'])}</a> - — - <span py:with="created=item['x-created']">${created[9:11]} ${MONTHS[int(created[6:8])]} ${created[1:5]}</span> - </li> -</ul> - -<br /> - -<div class="section-info" id="all-articles"><strong>Other Articles</strong></div> -<ul> - <li py:for="item in sorted([item for item in items if not(item.get('x-created') and not item.get('subtitle'))], key=lambda x: x['__name__'])"> - <a id="article-${item['__name__']}" href="${item['__name__']}.html">${Markup(item.get('title') or item['__title__'])}</a> - </li> -</ul> -</div> diff --git a/source/_layouts/page.genshi b/source/_layouts/page.genshi deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL19sYXlvdXRzL3BhZ2UuZ2Vuc2hp..0000000000000000000000000000000000000000 --- a/source/_layouts/page.genshi +++ /dev/null @@ -1,13 +0,0 @@ ---- -layout: site -license: MIT ---- - -<div xmlns:py="http://genshi.edgewall.org/"> -<div id="main"> -<h1 class="title" py:if="defined('title')">${Markup(title)}</h1> -${Markup(content)} -</div> -<div id="sidebar"> -</div> -</div> diff --git a/source/_layouts/py3k.genshi b/source/_layouts/py3k.genshi deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL19sYXlvdXRzL3B5M2suZ2Vuc2hp..0000000000000000000000000000000000000000 --- a/source/_layouts/py3k.genshi +++ /dev/null @@ -1,13 +0,0 @@ ---- -layout: site -license: MIT ---- - -<div xmlns:py="http://genshi.edgewall.org/"> -<div id="main"> -<h1 class="title" py:if="defined('title')">${Markup(title)}</h1> -${Markup(content)} -</div> -<div id="sidebar"> -</div> -</div> diff --git a/source/_layouts/site.genshi b/source/_layouts/site.genshi deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL19sYXlvdXRzL3NpdGUuZ2Vuc2hp..0000000000000000000000000000000000000000 --- a/source/_layouts/site.genshi +++ /dev/null @@ -1,86 +0,0 @@ -<!DOCTYPE html> -<html xmlns:py="http://genshi.edgewall.org/"> -<?python - - section_data = { - 'code': [ - ('Home', 'index.html'), - ('What is PyPy?', 'features.html'), - ('Download', 'download.html'), - ('Compatibility', 'compat.html'), - ('Performance', 'performance.html'), - ('Dev Documentation', 'http://doc.pypy.org'), - ('Blog', 'http://morepypy.blogspot.com'), - ('People', 'people.html'), - ('Contact', 'contact.html'), - ], - } - - menu_class = {} - - for section in section_data.keys(): - if section == section_id: - menu_class[section] = {'class': 'selected'} - else: - menu_class[section] = {} - -?> -<head> - <title py:if="defined('title')">PyPy - ${title}</title> - <title py:if="not defined('title')">${site_title}</title> - <meta http-equiv="content-language" content="en" /> - <meta http-equiv="content-type" content="text/html; charset=utf-8" /> - <meta name="author" content="${site_author}" /> - <meta name="description" content="${site_description}" /> - <meta name="copyright" content="${site_license}" /> - <meta name="document-rating" content="general" /> - <link rel="stylesheet" type="text/css" media="screen" title="default" href="css/site.css" /> - <link rel="alternate" type="application/rss+xml" title="RSS Feed for PyPy" href="http://feeds.feedburner.com/PyPyStatusBlog" /> - <link rel="stylesheet" type="text/css" href="css/jquery-ui-1.8.14.custom.css"/> - <py:if test="defined('typekit')"> - <script type="text/javascript" src="https://use.typekit.com/hdt8sni.js"></script> - <script type="text/javascript">try{Typekit.load();}catch(e){}</script> - </py:if> - <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script> - <script type="text/javascript" src="js/jquery-ui-1.8.14.custom.min.js"/> - <script type="text/javascript" src="js/script2.js?bust=1"></script> -</head> -<body> -<script type="text/javascript"> - var _gaq = [['_setAccount', '${analytics_id}'], ['_trackPageview']]; - if (document.location.protocol !== 'file:') { - (function() { - var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; - ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; - (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(ga); - })(); - } -</script> - -<div id="body-outer"><div id="body-inner"><div id="body" class="clearfix"> - -<div id="header"> - <div id="menu-follow"> - <div><a href="https://bitbucket.org/pypy/pypy"><img src="https://www.selenic.com/hg-logo/logo-droplets-25.png" width="14px" height="14px"/></a></div> - <div><a href="http://feeds.feedburner.com/PyPyStatusBlog" title="Subscribe to the RSS Feed"><img src="https://feedburner.google.com/fb/lib/images/icons/feed-icon-12x12-orange.gif" alt="Subscribe to the RSS Feed" width="14px" height="14px" /></a></div> - </div> - - <div id="logo"><a href="http://pypy.org"><img src="image/pypy-logo.png" alt="PyPy" height="110px" /></a></div> - <hr class="clear-left" /> - <div id="menu-sub" py:with="section_info=section_data[section_id]; last_menu_idx=len(section_info)-1"> - <py:for each="idx, (menu_item, menu_url) in enumerate(section_info)"><a href="${menu_url}">${menu_item}</a> - <span py:if="idx != last_menu_idx and menu_item != 'Contact'" class="menu-sub-sep"> | </span> - <br py:if="menu_item == 'Contact'"/> - </py:for> - </div> - <hr class="clear" /> -</div> - -<div id="content" py:if="defined('content')"> -${Markup(content)} -</div> - -</div></div></div> - -</body> -</html> diff --git a/source/_layouts/sponsor.genshi b/source/_layouts/sponsor.genshi deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL19sYXlvdXRzL3Nwb25zb3IuZ2Vuc2hp..0000000000000000000000000000000000000000 --- a/source/_layouts/sponsor.genshi +++ /dev/null @@ -1,18 +0,0 @@ ---- -layout: site -license: MIT ---- - -<div xmlns:py="http://genshi.edgewall.org/"> -<div id="main"> -<script> - $(function() { - $("#progressbar").progressbar({ - value: 0 - }); - }); -</script> -<h1 class="title" py:if="defined('title')">${Markup(title)}</h1> -${Markup(content)} -</div> -</div> diff --git a/source/compat.txt b/source/compat.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2NvbXBhdC50eHQ=..0000000000000000000000000000000000000000 --- a/source/compat.txt +++ /dev/null @@ -1,151 +0,0 @@ ---- -layout: page -title: Python compatibility ---- - -PyPy implements the Python language version 2.7.13. It supports all of the core -language, passing Python test suite (with minor modifications that were -already accepted in the main python in newer versions). It supports most -of the commonly used Python `standard library modules`_; details below. - -PyPy3 implements the Python language version 3.6.9. It has been released, -but Python is a large language and it is quite possible that a few things are missing. - -.. class:: download_menu - - `List of installable top 1000 PyPI packages`_ - -PyPy has support for the `CPython C API`_, however there are constructs -that are `not compatible`. We strongly advise use of `CFFI`_ -instead. CFFI come builtin with PyPy. Many libraries will require -a bit of effort to work, but there are known success stories. Check out -PyPy blog for updates, as well as the `Compatibility Wiki`__. - -.. __: https://bitbucket.org/pypy/compatibility/wiki/Home - -C extensions need to be recompiled for PyPy in order to work. Depending on -your build system, it might work out of the box or will be slightly harder. - -Standard library modules supported by PyPy. Note that large parts of python -library are implemented in pure python, so they don't have to be listed -there. Please just check if it imports. If it imports, it should work. - -* ``__builtin__, __pypy__, _ast, _cffi_backend, _codecs, _collections, _continuation, _csv, _file, _hashlib, _io, _locale, _lsprof, _md5, _minimal_curses, _multibytecodec, _multiprocessing, _numpypy, _pickle_support, _pypyjson, _random, _rawffi, _sha, _socket, _sre, _ssl, _struct, _testing, _warnings, _weakref, array, binascii, bz2, cStringIO, cmath, cppyy, cpyext, crypt, errno, exceptions, fcntl, gc, imp, itertools, marshal, math, mmap, operator, parser, posix, pwd, pyexpat, pypyjit, select, signal, symbol, sys, termios, thread, time, token, unicodedata, zipimport, zlib`` - -Supported, but written in pure Python: - -* ``cPickle, ctypes, datetime, dbm, _functools, grp, readline, resource, sqlite3, syslog`` - -All modules that are pure python in CPython of course work. - -Python libraries known to work under PyPy (the list is not exhaustive). -A `fuller list`_ is available. - -* ctypes - -* django - -* sqlalchemy - -* flask - -* twisted - -* pylons - -* divmod's nevow - -* pyglet - -* Pillow (the PIL fork) - -* `lxml`_ - -* NumPy - -The main difference that is not going to be fixed is that PyPy does -not support refcounting semantics. The following code won't fill the -file immediately, but only after a certain period of time, when the GC -does a collection: - -.. syntax:: python - - open("filename", "w").write("stuff") - -The proper fix is - -.. syntax:: python - - f = open("filename", "w") - f.write("stuff") - f.close() - -or using the ``with`` keyword - -.. syntax:: python - - with open("filename", "w") as f: - f.write("stuff") - -The same problem---not closing your files---can also show up if your -program opens a large number of files without closing them explicitly. -In that case, you can easily hit the system limit on the number of file -descriptors that are allowed to be opened at the same time. - -Since release 5.4, PyPy can be run with the command-line option ``-X -track-resources`` (as in, ``pypy -X track-resources myprogram.py``). -This produces a ResourceWarning when the GC closes a non-closed file or -socket. The traceback for the place where the file or socket was -allocated is given as well, which aids finding places where ``close()`` -is missing. - -Similarly, remember that you must ``close()`` a non-exhausted -generator in order to have its pending ``finally`` or ``with`` -clauses executed immediately: - -.. syntax:: python - - def mygen(): - with foo: - yield 42 - - for x in mygen(): - if x == 42: - break # foo.__exit__ is not run immediately! - - # fixed version: - gen = mygen() - try: - for x in gen: - if x == 42: - break - finally: - gen.close() - -More generally, ``__del__()`` methods are not executed as predictively -as on CPython: they run "some time later" in PyPy (or not at all if -the program finishes running in the meantime). See `more details -here`_. - -Note that PyPy returns unused memory to the operating system if there -is a madvise() system call (at least Linux, OS X, BSD) or on Windows. It is -important to realize that you may not see this in ``top``. The unused -pages are marked with ``MADV_FREE``, which tells the system "if you -need more memory at some point, grab this page". As long as memory is -plentiful, the ``RES`` column in ``top`` might remains high. (Exceptions to -this rule are systems with no ``MADV_FREE``, where we use -``MADV_DONTNEED``, which forcefully lowers the ``RES``. This includes -Linux <= 4.4.) - -A more complete list of known differences is available at `our dev site`_. - -.. _`CPython C API`: http://docs.python.org/c-api/ -.. _`CFFI`: http://cffi.readthedocs.org/ -.. _`not compatible`: http://doc.pypy.org/en/latest/cpython_differences.html#c-api-differences -.. _`standard library modules`: http://docs.python.org/library/ -.. _`our dev site`: http://pypy.readthedocs.org/en/latest/cpython_differences.html -.. _`more details here`: http://pypy.readthedocs.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies -.. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home -.. _`lxml`: https://github.com/amauryfa/lxml/tree/cffi/ -.. _`List of installable top 1000 PyPI packages`: http://packages.pypy.org -.. _`fuller list`: http://packages.pypy.org diff --git a/source/contact.txt b/source/contact.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2NvbnRhY3QudHh0..0000000000000000000000000000000000000000 --- a/source/contact.txt +++ /dev/null @@ -1,24 +0,0 @@ ---- -layout: page -title: Contact ---- - -Contact -============================================================ - -* irc: **#pypy** on **irc.freenode.net** - -* mailing list: `pypy-dev at python.org`__ - -* for security related issues, non-public funding enquiries etc. please contact pypy-z@python.org - -* the bitbucket `bug tracker`_ (registration required to open new issues or to comment) - -* more on our `dev site`_. - -* code on `bitbucket`_. - -.. __: http://mail.python.org/mailman/listinfo/pypy-dev -.. _`bug tracker`: https://bitbucket.org/pypy/pypy/issues?status=new&status=open -.. _`dev site`: http://doc.pypy.org -.. _`bitbucket`: https://bitbucket.org/pypy/pypy/overview diff --git a/source/download.txt b/source/download.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2Rvd25sb2FkLnR4dA==..0000000000000000000000000000000000000000 --- a/source/download.txt +++ /dev/null @@ -1,499 +0,0 @@ ---- -layout: page -title: Download and install ---- - -Download -============================================================ - -.. class:: download_menu - - There are `nightly binary builds`_ available. Those builds are not always - as stable as the release, but they contain numerous bugfixes and - performance improvements. - -We provide binaries for x86, aarch64, ppc64 and s390x running on different operating systems such as -Linux, Mac OS X and Windows (`what's new in PyPy 7.3.0?`_): - -* the Python2.7 compatible release — **PyPy2.7 v7.3.0** - -* the Python3.6 compatible release — **PyPy3.6 v7.3.0** - -* the Python2.7 Software Transactional Memory special release — **PyPy-STM 2.5.1** (Linux x86-64 only) - -.. _what's new in PyPy 7.3.0?: http://doc.pypy.org/en/latest/release-v7.3.0.html - - -.. class:: download_menu - - * Download - - * `Default (with a JIT Compiler)`_ - * `Other versions`_ - - * `Installing`_ (optional) - * `Installing more modules`_ - * `Building from source`_ - * `Packaging`_ - * `Checksums`_ - -.. _`Default (with a JIT Compiler)`: - -"JIT Compiler" version -------------------------------- - -These binaries include a Just-in-Time compiler. They only work on -x86 CPUs that have the SSE2_ instruction set (most of -them do, nowadays), or on x86-64 CPUs. They also contain `stackless`_ -extensions, like `greenlets`_. - -Linux binaries and common distributions ---------------------------------------- - -Since version 7.3, the linux x86 binaries in the links below ship with versions -of OpenSSL, SQLite3, libffi, expat, and TCL/TK binary libraries linked in. This -make the binaries "portable" so that they should run on any current glibc-based -linux platform. The ideas were adopted from the `portable-pypy`_ package. - -This solution to the portability problem means that the versions of the -packaged libraries are frozen to the version shipped, so updating your system -libraries will not affect this installation of PyPy. Also see the note about -SSL certificates below. - -For aarch64, s390x, and ppc64, the binaries target a specific operating system. -These binaries are dynamically linked, and thus might not be usable due to the -sad story of linux binary compatibility. This means that **Linux binaries are -only usable on the distributions written next to them** unless you're ready to -hack your system by adding symlinks to the libraries it tries to open. There -are better solutions: - -* download PyPy from your release vendor (usually an outdated - version): `Ubuntu`_ (`PPA`_), `Debian`_, `Homebrew`_, MacPorts, - `Fedora`_, `Gentoo`_ and `Arch`_ are known to package PyPy, with various - degrees of being up-to-date. - -* use ``sudo snap install --classic <package>``, where ``<package>`` is - ``pypy`` or `pypy3``. Snap is a non-vendor specific package manager for - linux, and repackages the download tarballs below with the latest platform- - specific libraries (again, without changing ``libffi``). - -* `recompile the CFFI-based`_ TCL/TK, OpenSSL, or sqlite3 modules, using system - libraries and the scripts in ``pypy/lib_pypy``. This solution will not solve - compatibility issues with libffi, since that is baked into PyPy. - -* or translate_ your own PyPy. - -.. class:: download_menu - - SSL Certificates - - While the linux binaries ship an OpenSSL library, they do not ship a - certificate store for SSL certificates. If you wish to use SSL module, - you will need a valid certificate store. You can use the `certifi`_ package - and set ``SSL_CERT_FILE`` to ``certifi.where()`` or install your platform - certificates which should be discovered by the ``_ssl`` module. - - -.. _`Ubuntu`: http://packages.ubuntu.com/search?keywords=pypy&searchon=names -.. _`PPA`: https://launchpad.net/~pypy/+archive/ppa -.. _`Debian`: http://packages.debian.org/sid/pypy -.. _`Fedora`: http://fedoraproject.org/wiki/Features/PyPyStack -.. _`Gentoo`: http://packages.gentoo.org/package/dev-python/pypy -.. _`Homebrew`: https://github.com/Homebrew/homebrew-core/blob/master/Formula/pypy.rb -.. _`Arch`: https://wiki.archlinux.org/index.php/PyPy -.. _`portable-pypy`: https://github.com/squeaky-pl/portable-pypy#portable-pypy-distribution-for-linux -.. _`recompile the CFFI-based`: https://doc.pypy.org/en/latest/build.html#build-cffi-import-libraries-for-the-stdlib -.. _`certifi`: https://pypi.org/project/certifi/ - -.. _release: - -Python2.7 compatible PyPy 7.3.0 -------------------------------- - -.. class:: download_menu - -* `Linux x86 binary (32bit, built on CenOS6)`__ -* `Linux x86-64 binary (64bit, built on CentOS6)`__ -* `Mac OS X binary (64bit)`__ -* FreeBSD x86 and x86_64: see FreshPorts_ -* `Windows binary (32bit)`__ (you might need the VC runtime library - installer `vcredist.x86.exe`_.) -* `Linux aarch64 binary (64bit, built on Ubuntu 18.04)`__ (see ``[1]`` below) -* `PowerPC PPC64 Linux binary (64bit big-endian, Fedora 20)`__ (see ``[1]`` below) -* `PowerPC PPC64le Linux binary (64bit little-endian, Fedora 21)`__ (see ``[1]`` below) -* `s390x Linux binary (built on Redhat Linux 7.2)`__ (see ``[1]`` below) -* `Source (tar.bz2)`__; `Source (zip)`__. See below for more about the sources. -* `All our downloads,`__ including previous versions. We also have a - mirror_, but please use only if you have troubles accessing the links above - - -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-linux32.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-linux64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-osx64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-win32.zip -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-aarch64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-ppc64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-ppc64le.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-s390x.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.zip -.. _`vcredist.x86.exe`: https://www.microsoft.com/en-us/download/details.aspx?id=52685 -.. __: https://bitbucket.org/pypy/pypy/downloads -.. _mirror: http://buildbot.pypy.org/mirror/ -.. _FreshPorts: http://www.freshports.org/lang/pypy - - -Python 3.6 compatible PyPy3.6 v7.3.0 ------------------------------------- - -.. class:: download_menu - -* `Linux x86-64 binary (64bit, built on CentOS6)`__ -* `Linux x86 binary (32bit, built on CentOS6)`__ -* `Mac OS X binary (64bit)`__ (High Sierra >= 10.13, not for Sierra and below) -* `Windows binary (32bit)`__ (you might need the VC runtime library - installer `vcredist.x86.exe`_.) -* `Linux aarch64 binary (64bit, built on Ubuntu 18.04)`__ (see ``[1]`` below) -* `PowerPC PPC64 Linux binary (64bit big-endian, Fedora 20)`__ (see ``[1]`` below) -* `PowerPC PPC64le Linux binary (64bit little-endian, Fedora 21)`__ (see ``[1]`` below) -* `s390x Linux binary (built on Redhat Linux 7.2)`__ (see ``[1]`` below) -* `Source (tar.bz2)`__; `Source (zip)`__. See below for more about the sources. -* `All our downloads,`__ including previous versions. We also have a - mirror_, but please use only if you have troubles accessing the links above - -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-linux64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-linux32.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-osx64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-win32.zip -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-aarch64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-ppc64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-ppc64le.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-s390x.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.zip -.. __: https://bitbucket.org/pypy/pypy/downloads - - -If your CPU is really, really old, it may be a x86-32 without SSE2. -There is untested support for manually translating PyPy's JIT without -SSE2 (``--jit-backend=x86-without-sse2``) but note that your machine -is probably low-spec enough that running CPython on it is a better -idea in the first place. - -``[1]:`` stating it again: the Linux binaries are provided for the -distributions listed here. **If your distribution is not exactly this -one, it won't work,** you will probably see: ``pypy: error while loading shared -libraries: ...``. - -PyPy-STM 2.5.1 ------------------------------- - -This is a special version of PyPy! See the `Software Transactional -Memory`_ (STM) documentation. - -* `PyPy-STM Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 16.04)`__ - -.. _`Software Transactional Memory`: http://doc.pypy.org/en/latest/stm.html -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-stm-2.5.1-linux64.tar.bz2 - - -.. _`Other versions (without a JIT)`: - -Other versions -------------------------------- - -The other versions of PyPy are: - -* The most up-to-date `nightly binary builds`_ with a JIT, if the official - release is too old for what you want to do. There are versions for - different libc on this site too. - -* Reverse debugger: This version enables debugging your Python - programs by going forward and backward in time. See the `RevDB - documentation`__. - -.. __: https://foss.heptapod.net/pypy/revdb/ - -* Old-style sandboxing: A special safe version. - *This is NOT the version announced in-development during 2019!* - Read the docs about sandboxing_. - This version is **not supported** and not actively maintained. You - will likely have to fix some issues yourself, or checkout an old - version, or otherwise play around on your own. We provide this - documentation only for historical reasons. Please do not use in - production. For reference, there are some very old, unmaintained - binaries for Linux (32bit__, 64bit__). - -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux64.tar.bz2 -.. __: https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-sandbox-linux.tar.bz2 -.. _`sandbox docs`: http://doc.pypy.org/en/latest/sandbox.html - -.. _`nightly binary builds`: http://buildbot.pypy.org/nightly/trunk/ - -Installing ----------- - -All binary versions are packaged in a ``tar.bz2`` or ``zip`` file. When -uncompressed, they run in-place. You can uncompress them -either somewhere in your home directory or, say, in ``/opt``. -If you want, put a symlink from somewhere like -``/usr/local/bin/pypy`` to ``/path/to/pypy_expanded/bin/pypy``. Do -not move or copy the executable ``pypy`` outside the tree --- put -a symlink to it, otherwise it will not find its libraries. - - -Installing more modules -------------------------------- - -There are as yet few distribution-ready packages. -We recommend installing ``pip``, which is the standard package -manager of Python. It works like it does on CPython as explained in the -`installation documentation`_. - -If you use your distribution's PyPy package we recommend you install packages -into a virtualenv. If you try to build a module and the build process complains -about "missing Python.h", you may need to install the pypy-dev package. - -.. _installation documentation: http://doc.pypy.org/en/latest/install.html - -.. _translate: - -Building from source --------------------- - -(see more build instructions_) - - -1. Get the source code. The preferred way is to checkout the current - trunk using Mercurial_. The trunk usually works and is of course - more up-to-date:: - - hg clone https://foss.heptapod.net/pypy/pypy/ - - The trunk contains PyPy 2. For PyPy 3, switch to the correct branch:: - - # for PyPy 3: switch to the branch of PyPy that implements Python 3.6 - hg update py3.6 - - Alternatively, get one of the following smaller packages for the source at - the same revision as the above binaries: - - * `pypy2.7-v7.3.0-src.tar.bz2`__ (sources, PyPy 2 only) - * `pypy3.6-v7.3.0-src.tar.bz2`__ (sources, PyPy 3 only) - - .. __: https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-src.tar.bz2 - .. __: https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-src.tar.bz2 - - -2. Make sure you **installed the dependencies.** See the list here__. - - .. __: http://pypy.readthedocs.org/en/latest/build.html#install-build-time-dependencies - -3. Enter the ``goal`` directory:: - - cd pypy/pypy/goal - -4. Run the ``rpython`` script. Here are the common combinations - of options (works also with ``python`` instead of ``pypy``; - requires CPython 2.7 or PyPy 2, even to build PyPy 3):: - - pypy ../../rpython/bin/rpython -Ojit targetpypystandalone # get the JIT version - pypy ../../rpython/bin/rpython -O2 targetpypystandalone # get the no-jit version - pypy ../../rpython/bin/rpython -O2 --sandbox targetpypystandalone # get the sandbox version - -5. Enjoy Mandelbrot ``:-)`` It takes on the order of half an hour to - finish the translation, and about 3GB of RAM on a 32-bit system - and about 5GB on 64-bit systems. (Do not start a translation on a - machine with insufficient RAM! It will just swap forever. See - notes below in that case.) - -6. If you want to install this PyPy as root, please read the next section, - Packaging_. - -Notes: - -* It is recommended to use PyPy to do translations, instead of using CPython, - because it is twice as fast. You should just start by downloading an - official release of PyPy (with the JIT). If you really have to use CPython - then note that we are talking about CPython 2.7 here, not CPython 3.x. - (Older versions like 2.6 are out.) - -* On some 32-bit systems, the address space limit of 2 or 3 GB of RAM - can be an issue. More generally you may be just a little bit low of - RAM. First note that 2 GB is really not enough nowadays; on Windows - you first need to refer to the `Windows build instructions`_. More - precisely, translation on 32-bit takes at this point 2.7 GB if PyPy is - used and 2.9 GB if CPython is used. There are two workarounds: - - 1. use PyPy, not CPython. If you don't have any PyPy so far, not even - an older version, then you need to build one first, with some parts - removed. So, first translate with ``...rpython -Ojit - targetpypystandalone --withoutmod-micronumpy --withoutmod-cpyext``, - then copy ``pypy-c`` and ``libpypy_c.so`` somewhere else, and finally - call it with ``...pypy-c ../../rpython/bin/rpython -Ojit``. - - 2. if even using PyPy instead of CPython is not enough, try to tweak - some internal parameters. Example (slower but saves around 400MB):: - - PYPY_DONT_RUN_SUBPROCESS=1 PYPY_GC_MAX_DELTA=200MB pypy --jit loop_longevity=300 ../../rpython/bin/rpython -Ojit --source - # then read the next point about --source - -* You can run translations with ``--source``, which only builds the C - source files (and prints at the end where). Then you can ``cd`` there - and execute ``make``. This is another way to reduce memory usage. - Note that afterwards, you have to run manually ``pypy-c - .../pypy/tool/build_cffi_imports.py`` if you want to be able to import - the cffi-based modules. - -* Like other JITs, PyPy doesn't work out of the box on some Linux - distributions that trade full POSIX compliance for extra security - features. E.g. with PAX, you have to run PyPy with ``paxctl -cm``. - This also applies to translation (unless you use CPython to run the - translation and you specify ``--source``). - -.. _instructions: http://pypy.readthedocs.org/en/latest/build.html -.. _`x86 (IA-32)`: http://en.wikipedia.org/wiki/IA-32 -.. _`x86-64`: http://en.wikipedia.org/wiki/X86-64 -.. _SSE2: http://en.wikipedia.org/wiki/SSE2 -.. _`contact us`: contact.html -.. _`sandboxing`: features.html#sandboxing -.. _`stackless`: http://www.stackless.com/ -.. _`greenlets`: http://pypy.readthedocs.org/en/latest/stackless.html#greenlets -.. _`Windows build instructions`: http://doc.pypy.org/en/latest/windows.html#preparing-windows-for-the-large-build -.. _`shadow stack`: http://pypy.readthedocs.org/en/latest/config/translation.gcrootfinder.html -.. _Mercurial: https://www.mercurial-scm.org/ - -Packaging ---------- - -Once PyPy is translated from source the binary package similar to those -provided in the section `Default (with a JIT Compiler)`_ above could be -easily created with ``package.py`` script as following:: - - cd ./pypy/pypy/tool/release/ - python package.py --help #for information - python package.py --archive-name pypy-my-own-package-name - -It is recommended to use package.py because custom scripts will -invariably become out-of-date. If you want to write custom scripts -anyway, note an easy-to-miss point: some modules are written with CFFI, -and require some compilation. If you install PyPy as root without -pre-compiling them, normal users will get errors: - -* PyPy 2.5.1 or earlier: normal users would see permission errors. - Installers need to run ``pypy -c "import gdbm"`` and other similar - commands at install time; the exact list is in `package.py`_. Users - seeing a broken installation of PyPy can fix it after-the-fact if they - have sudo rights, by running once e.g. ``sudo pypy -c "import gdbm``. - -* PyPy 2.6 and later: anyone would get ``ImportError: no module named - _gdbm_cffi``. Installers need to run ``pypy _gdbm_build.py`` in the - ``lib_pypy`` directory during the installation process (plus others; - see the exact list in `package.py`_). Users seeing a broken - installation of PyPy can fix it after-the-fact, by running ``pypy - /path/to/lib_pypy/_gdbm_build.py``. This command produces a file - called ``_gdbm_cffi.pypy-41.so`` locally, which is a C extension - module for PyPy. You can move it at any place where modules are - normally found: e.g. in your project's main directory, or in a - directory that you add to the env var ``PYTHONPATH``. - -.. _`package.py`: https://bitbucket.org/pypy/pypy/src/default/pypy/tool/release/package.py - -Checksums ---------- - -Here are the checksums for each of the downloads of PyPy 7.3.0, 7.2.0, 7.1.1, 7.1.0 - -pypy2.7-7.3.0 sha256:: - - a3dd8d5e2a656849fa344dce4679d854a19bc4a096a0cf62b46a1be127a5d56c pypy2.7-v7.3.0-aarch64.tar.bz2 - eac1308b7d523003a5f6d20f58406d52ab14611bcec750122ae513a5a35110db pypy2.7-v7.3.0-linux32.tar.bz2 - f4950a54378ac637da2a6defa52d6ffed96af12fcd5d74e1182fb834883c9826 pypy2.7-v7.3.0-linux64.tar.bz2 - ca7b056b243a6221ad04fa7fc8696e36a2fb858396999dcaa31dbbae53c54474 pypy2.7-v7.3.0-osx64.tar.bz2 - 82e62869812aa2953a4f83e96c813cbc52973dfa5e42605e72b6610ac13f2481 pypy2.7-v7.3.0-ppc64.tar.bz2 - 592a6db77270b922ffa13cbeced9eabbc36c532ded9fc145f6a19073d3e78499 pypy2.7-v7.3.0-ppc64le.tar.bz2 - d254b82a00021339762198e41ba7f72316010d0f9bd4dcd7b0755185da9c005e pypy2.7-v7.3.0-s390x.tar.bz2 - b0b25c7f8938ab0fedd8dedf26b9e73c490913b002b484c1b2f19d5844a518de pypy2.7-v7.3.0-src.tar.bz2 - 42dc84a277e7a5e635fe39bbd745f06135902c229a257123332b7555800d915b pypy2.7-v7.3.0-src.zip - a9e3c5c983edba0313a41d3c1ab55b080816c4129e67a6c272c53b9dbcdd97ec pypy2.7-v7.3.0-win32.zip - -pypy3.6-7.3.0 sha256:: - - b900241bca7152254c107a632767f49edede99ca6360b9a064141267b47ef598 pypy3.6-v7.3.0-aarch64.tar.bz2 - 7045b295d38ba0b5ee65bd3f078ca249fcf1de73fedeaab2d6ad78de2eab0f0e pypy3.6-v7.3.0-linux32.tar.bz2 - d3d549e8f43de820ac3385b698b83fa59b4d7dd6cf3fe34c115f731e26ad8856 pypy3.6-v7.3.0-linux64.tar.bz2 - 87b2545dad75fe3027b4b2108aceb9fdadcdd24e61ae312ac48b449fdd452bf3 pypy3.6-v7.3.0-osx64.tar.bz2 - e2587e8da2abb12a86bf75941ce739124d2a1156367a9a3d729ac31d0841c300 pypy3.6-v7.3.0-ppc64.tar.bz2 - d6f3b701313df69483b43ebdd21b9652ae5e808b2eea5fbffe3b74b82d2e7433 pypy3.6-v7.3.0-ppc64le.tar.bz2 - 0fe2f7bbf42ea88b40954d7de773a43179a44f40656f2f58201524be70699544 pypy3.6-v7.3.0-s390x.tar.bz2 - 48d12c15fbcbcf4a32882a883195e1f922997cde78e7a16d4342b9b521eefcfa pypy3.6-v7.3.0-src.tar.bz2 - 8ae9efd0a2aadb19e892bbd07eca8ef51536296a3ef93964149aceba511e79ca pypy3.6-v7.3.0-src.zip - 30e6870c4f3d8ef91890a6556a98080758000ba7c207cccdd86a8f5d358998c1 pypy3.6-v7.3.0-win32.zip - -pypy2.7-7.2.0 sha256:: - - 57b0be053c6a5f069e23b843f38863cf7920f5eef7bc89f2e086e5c3a28a2ba9 pypy2.7-v7.2.0-aarch64.tar.bz2 - 76d666e5aee54b519d6ec1af4ef0cbdc85f7f9276dd554e97deb026adfd0c936 pypy2.7-v7.2.0-linux32.tar.bz2 - 05acf28e6a243026ecad933b9361d8f74b41f00818071b76b38c4694cc4c9599 pypy2.7-v7.2.0-linux64.tar.bz2 - 36aa2f2440e762333569118dd0b3d5371d575c40966effa194d116c5453ddb52 pypy2.7-v7.2.0-osx64.tar.bz2 - fb51150a4ce94b0ca8587899ba69c41fc58a6b35c5340ea6926376ecb9cfcac4 pypy2.7-v7.2.0-ppc64.tar.bz2 - 5c4224525657c29b815cb2c6b3f9bc5a267368cc6adf0fedb235a6052929f65f pypy2.7-v7.2.0-ppc64le.tar.bz2 - bb7ae585ecb4d904c890e28a2c5b6bd379f57cc3d9e38ff45597ff54fa935eaa pypy2.7-v7.2.0-s390x.tar.bz2 - 55cb7757784fbe3952102447f65b27d80e6c885a464a7af1a9ce264492439dcc pypy2.7-v7.2.0-src.tar.bz2 - 897038550614d558f9f6718409b107e27903ef2b2b57ec250939d1b1ebdf0aba pypy2.7-v7.2.0-src.zip - 956eeaaaac053e5d0917e77a3d2ad1933ab5561eb3e6e71235780b5aa5fd2bb7 pypy2.7-v7.2.0-win32.zip - -pypy2.7-7.1.1 sha256:: - - 41ca390a76ca0d47b8353a0d6a20d5aab5fad8b0bb647b960d8c33e873d18ef5 pypy2.7-v7.1.1-linux32.tar.bz2 - 73b09ef0860eb9ad7997af3030b22909806a273d90786d78420926df53279d66 pypy2.7-v7.1.1-linux64.tar.bz2 - 31a17294dec96c2191885c776b4ee02112957dc874f7ba03e570537a77b78c35 pypy2.7-v7.1.1-osx64.tar.bz2 - 1ef94c3a9c67c2335cee0b21753036b4696ed588b9d54b7b8036a6ae47f7001d pypy2.7-v7.1.1-s390x.tar.bz2 - 5f06bede6d71dce8dfbfe797aab26c8e35cb990e16b826914652dc093ad74451 pypy2.7-v7.1.1-src.tar.bz2 - d9b07a2954ad6dbde94feffd848311e2b5169563d33e3e9f17969579b01a4158 pypy2.7-v7.1.1-src.zip - 9c59226311f216a181e70ee7b5aa4d9665a15d00f24ae02acec9af7d96355f63 pypy2.7-v7.1.1-win32.zip - -pypy2.7-7.1.0 sha256:: - - 44ec91e8cb01caab289d8763c203f3aaf288d14325a6c42692bd1ac4e870d758 pypy2.7-v7.1.0-linux32.tar.bz2 - fef176a29a2ef068c00c8098e59dab935ca6e956f089672b3f7351da95a034f5 pypy2.7-v7.1.0-linux64.tar.bz2 - 8be43685ce718b0768387450fc6dc395d60809b778b6146c353ef67826022153 pypy2.7-v7.1.0-osx64.tar.bz2 - b065f55741bcb37863f1eca30ce91c9d79159371a6994100930cdc2ede3237bc pypy2.7-v7.1.0-s390x.tar.bz2 - b051a71ea5b4fa27d0a744b28e6054661adfce8904dcc82500716b5edff5ce4b pypy2.7-v7.1.0-src.tar.bz2 - e60ce30f9947844da43daaa7658adc0c05330681305225954114772f42df06ec pypy2.7-v7.1.0-src.zip - 76658c9ad679d562b8b6a09d006caa666406337b9834ff56db16980c5e549f20 pypy2.7-v7.1.0-win32.zip - -pypy3.6-7.2.0 sha256:: - - f82dc9dc6c692417ee9727f23beae75364a5757ebdc657a2a1d0010ac3ad17ab pypy3.6-v7.2.0-aarch64.tar.bz2 - 45e99de197cb3e974cfc8d45e0076ad2066852e61e56b3eafd1237efafd2c43e pypy3.6-v7.2.0-linux32.tar.bz2 - aa128e555ad0fe5c4c15104ae0903052bd232b6e3a73f5fe023d27b8fd0d6089 pypy3.6-v7.2.0-linux64.tar.bz2 - 836abb0ec303b90a684533711ed3b8269d3e8c64805b595e410920abdea678ac pypy3.6-v7.2.0-osx64.tar.bz2 - 14021d196e393b3a6d2395ab94ceec347753715e37223efe4c50b7c141b351a2 pypy3.6-v7.2.0-ppc64.tar.bz2 - 6aef73a3b68e9a6c062cadd83d3db16790960cf97401ca6f2aad2195e9b05c35 pypy3.6-v7.2.0-ppc64le.tar.bz2 - a11da8118064db102d159e9221319c428b298c4a87f26166fd6ae94be8d6ae0d pypy3.6-v7.2.0-s390x.tar.bz2 - 0d7c707df5041f1593fe82f29c40056c21e4d6cb66554bbd66769bd80bcbfafc pypy3.6-v7.2.0-src.tar.bz2 - 405ac35695dd374d5ea192cb44cb47231f9a65812cc7b6549df33df12ffe54db pypy3.6-v7.2.0-src.zip - c926f622bec24a8b348591d631717ace83b3a6c3c2dac02b157b622b97d1fc9c pypy3.6-v7.2.0-win32.zip - -pypy3.6-7.1.1 sha256:: - - cb11ef4b0df569c28390b1ee93029159e1b90bfbad98df6abd629d5203b2abd9 pypy3.6-v7.1.1-linux32.tar.bz2 - 8014f63b1a34b155548852c7bf73aab2d41ebddf2c8fb603dc9dd8509be93db0 pypy3.6-v7.1.1-linux64.tar.bz2 - a5c2f2bfa2b4a4d29e8a67baab95699b169054066df218a14f171bb84a6df0c0 pypy3.6-v7.1.1-osx64.tar.bz2 - 4a91bf2d9a142b6dbf82b5301cb510535ae9a54e1645546b2e0735a7b5ed85ba pypy3.6-v7.1.1-s390x.tar.bz2 - 6a3ef876e3691a54f4cff045028ec3be94ab9beb2e99f051b83175302c1899a8 pypy3.6-v7.1.1-src.tar.bz2 - 4a3ebeb767740f2dc0b886d02797d21d7d69f154cf951bb991c19bd485e6cae1 pypy3.6-v7.1.1-src.zip - 8b513b254de5f31890f5956569de9aec3a0a91d7aba72fc89d66901f4a8ccf49 pypy3.6-v7.1.1-win32.zip - -pypy 3.6-v7.1.0 sha256:: - - - 031bfac61210a6e161bace0691b854dc15d01b0e624dc0588c544ee5e1621a83 pypy3.6-v7.1.0-linux32.tar.bz2 - 270dd06633cf03337e6f815d7235e790e90dabba6f4b6345c9745121006925fc pypy3.6-v7.1.0-linux64.tar.bz2 - d46e005ba095cb4a7006079ffbf4fe63c18cf5e9d8ce9ce8383efc1a4863ab5b pypy3.6-v7.1.0-osx64.tar.bz2 - 243cd0cc188a94c1f064f402ae72b8ba4303eb3137eac53c53826472b8005098 pypy3.6-v7.1.0-s390x.tar.bz2 - faa81f469bb2a7cbd22c64f22d4b4ddc5a1f7c798d43b7919b629b932f9b1c6f pypy3.6-v7.1.0-src.tar.bz2 - 4858e7e8a0007bc3b381bd392208b28d30889a4e5a88a3c28e3d9dc4f25b654e pypy3.6-v7.1.0-src.zip - 77a0576a3d518210467f0df2d0d9a1892c664566dc02f25d974c2dbc6b4749e7 pypy3.6-v7.1.0-win32.zip - diff --git a/source/features.txt b/source/features.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2ZlYXR1cmVzLnR4dA==..0000000000000000000000000000000000000000 --- a/source/features.txt +++ /dev/null @@ -1,165 +0,0 @@ ---- -layout: page -title: What is PyPy? ---- - -What is PyPy? -=========================================================== - -PyPy is a replacement for CPython. It is built using the RPython -language that was co-developed with it. The main reason to use it -instead of CPython is speed: it runs generally faster (see next section). - -**PyPy** implements **Python 2.7.13 and 3.6.9**. -It supports all of the core language, passing the Python 2.7 test suite -and most of the 3.6 test suite (with minor modifications) It supports most of -the commonly used Python standard library modules. For known differences with -CPython, see our `compatibility`_ page. - -The following CPU architectures are supported and maintained: - -* `x86 (IA-32)`_ and `x86_64`_ -* `ARM`_ platforms (ARMv6 or ARMv7, with VFPv3) -* `AArch64`_ -* `PowerPC`_ 64bit both little and big endian -* `System Z (s390x)`_ - -PyPy's x86 version runs on several operating systems, such as Linux -(32/64 bits), Mac OS X (64 bits), Windows (32 bits), OpenBSD, FreeBSD. -All non-x86 versions are only supported on Linux. - -If you are interested in helping to move forward, see our `howtohelp`_ page. - -.. _`compatibility`: compat.html -.. _`x86 (IA-32)`: http://en.wikipedia.org/wiki/IA-32 -.. _`x86_64`: http://en.wikipedia.org/wiki/X86_64 -.. _`ARM`: http://en.wikipedia.org/wiki/ARM -.. _`AArch64`: http://en.wikipedia.org/wiki/AArch64 -.. _`PowerPC`: https://de.wikipedia.org/wiki/PowerPC -.. _`System Z (s390x)`: https://de.wikipedia.org/wiki/System/390 -.. _`howtohelp`: howtohelp.html - - - -The main features of PyPy: --------------------------- - -Speed ------ - -Our `main executable`_ comes with a Just-in-Time compiler. It is -`really fast`_ in running most benchmarks --- including very large and -complicated Python applications, not just 10-liners. - -There are two cases that you should be aware where PyPy will *not* be -able to speed up your code: - -* Short-running processes: if it doesn't run for at least a few seconds, - then the JIT compiler won't have enough time to warm up. - -* If all the time is spent in run-time libraries (i.e. in C functions), - and not actually running Python code, the JIT compiler will not help. - -So the case where PyPy works best is when executing long-running -programs where a significant fraction of the time is spent executing -Python code. This is the case covered by the majority of `our -benchmarks`_, but not all of them --- the goal of PyPy is to get speed -but still support (ideally) any Python program. - -.. _`main executable`: download.html#with-a-jit-compiler -.. _`really fast`: http://speed.pypy.org/ -.. _`our benchmarks`: http://speed.pypy.org/ - - -Memory usage --------------------------- - -Memory-hungry Python programs (several hundreds of MBs or more) might -end up taking less space than they do in CPython. It is not always -the case, though, as it depends on a lot of details. Also note that -the baseline is higher than CPython's. - - -Stackless --------------------------- - -Support for Stackless_ and greenlets are now integrated in the normal -PyPy. More detailed information is available here__. - -.. _Stackless: http://www.stackless.com/ -.. __: http://doc.pypy.org/en/latest/stackless.html - - -Other features ---------------------------------------- - -PyPy has many secondary features and semi-independent -projects. We will mention here: - -* **Other languages:** we also implemented other languages that makes - use of our RPython toolchain: Prolog_ (almost complete), as - well as Smalltalk_, JavaScript_, Io_, Scheme_ and Gameboy_. - - There is also a Ruby implementation called Topaz_ and a PHP implementation - called HippyVM_. - - -Sandboxing --------------------- - -PyPy's *sandboxing* is a working prototype for the idea of running untrusted -user programs. Unlike other sandboxing approaches for Python, PyPy's does not -try to limit language features considered "unsafe". Instead we replace all -calls to external libraries (C or platform) with a stub that communicates -with an external process handling the policy. - -+-----------------------------------------------------------------------------+ -| **Please be aware that it is a prototype only.** *It needs work to become | -| more complete, and you are welcome to help. In particular, almost none | -| of the extension modules work (not even* ``time`` *), and* ``pypy_interact``| -| *is merely a demo. Also, a more complete system would include a way | -| to do the same as* ``pypy_interact`` *from other languages than Python, | -| to embed a sandboxed interpreter inside programs written in other | -| languages.* | -+-----------------------------------------------------------------------------+ - -To run the sandboxed process, you need to get the full sources and -build ``pypy-sandbox`` from it (see `Building from source`_). These -instructions give you a ``pypy-c`` that you should rename to -``pypy-sandbox`` to avoid future confusion. Then run:: - - cd pypy/sandbox - pypy_interact.py path/to/pypy-sandbox - # don't confuse it with pypy/goal/pyinteractive.py! - -You get a fully sandboxed interpreter, in its own filesystem hierarchy -(try ``os.listdir('/')``). For example, you would run an untrusted -script as follows:: - - mkdir virtualtmp - cp untrusted.py virtualtmp/ - pypy_interact.py --tmp=virtualtmp pypy-sandbox /tmp/untrusted.py - -Note that the path ``/tmp/untrusted.py`` is a path inside the sandboxed -filesystem. You don't have to put ``untrusted.py`` in the real ``/tmp`` -directory at all. - -To read more about its features, try ``pypy_interact.py --help`` or go to -`our documentation site`_. - -.. _`Building from source`: download.html#building-from-source -.. _`our documentation site`: http://pypy.readthedocs.org/en/latest/sandbox.html - - - - -.. _`the cli-jit branch`: https://bitbucket.org/pypy/pypy/src/cli-jit -.. _`contact us`: contact.html -.. _Prolog: https://bitbucket.org/cfbolz/pyrolog/ -.. _Smalltalk: https://bitbucket.org/pypy/lang-smalltalk/ -.. _JavaScript: https://bitbucket.org/pypy/lang-js/ -.. _Io: https://bitbucket.org/pypy/lang-io/ -.. _Scheme: https://bitbucket.org/pypy/lang-scheme/ -.. _Gameboy: https://bitbucket.org/pypy/lang-gameboy/ -.. _Topaz: http://topazruby.com/ -.. _HippyVM: http://www.hippyvm.com/ diff --git a/source/howtohelp.txt b/source/howtohelp.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2hvd3RvaGVscC50eHQ=..0000000000000000000000000000000000000000 --- a/source/howtohelp.txt +++ /dev/null @@ -1,25 +0,0 @@ ---- -layout: page -title: How to help? ---- - -How to help PyPy development? -===================================== - -Here are some ideas to help PyPy development: - -* use pypy for your projects and provide detailed feedback_ - -* talk to us about how to support Python 3.x - -* write blog posts or tweets about your experiences - -* help porting to new platforms - -* contact_ us and get involved - -* donate_ some money to enable others to help - -.. _contact: contact.html -.. _feedback: contact.html -.. _`donate`: http://morepypy.blogspot.com/2010/11/speeding-up-pypy-by-donations.html diff --git a/source/index.txt b/source/index.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL2luZGV4LnR4dA==..0000000000000000000000000000000000000000 --- a/source/index.txt +++ /dev/null @@ -1,51 +0,0 @@ ---- -layout: page -title: Welcome to PyPy ---- - -PyPy is a `fast`_, `compliant`_ alternative implementation of the `Python`_ -language (2.7.13 and 3.6.9). It has several advantages and distinct features: - - * **Speed:** thanks to its Just-in-Time compiler, Python programs - often run `faster`_ on PyPy. `(What is a JIT compiler?)`_ - - "If you want your code to run faster, you should probably just use PyPy." - -- Guido van Rossum (creator of Python) - - * **Memory usage:** memory-hungry Python programs (several hundreds of - MBs or more) might end up taking `less space`_ than they do in CPython. - - * **Compatibility:** PyPy is `highly compatible`_ with existing python code. - It supports `cffi`_ and can run popular python libraries like `twisted`_ - and `django`_. - - * **Stackless:** PyPy comes by default with support for `stackless mode`_, - providing micro-threads for massive concurrency. - - * As well as other `features`_. - -.. class:: download - -`Download and try out PyPy`__ - -.. __: download.html - -Want to know more? A good place to start is our detailed `speed`_ and -`compatibility`_ reports! - -.. _`stackless mode`: features.html#stackless -.. _`Python`: http://python.org/ -.. _`fast`: http://speed.pypy.org/ -.. _`faster`: http://speed.pypy.org/ -.. _`(What is a JIT compiler?)`: http://en.wikipedia.org/wiki/Just-in-time_compilation -.. _`run untrusted code`: features.html#sandboxing -.. _`compliant`: compat.html -.. _`Python docs`: http://docs.python.org/2.7 -.. _`twisted`: https://twistedmatrix.com/ -.. _`django`: https://www.djangoproject.com/ -.. _`cffi`: https://cffi.readthedocs.org -.. _`features`: features.html -.. _`less space`: http://morepypy.blogspot.com/2009/10/gc-improvements.html -.. _`highly compatible`: compat.html -.. _`speed`: http://speed.pypy.org/ -.. _`compatibility`: compat.html diff --git a/source/numpydonate.txt b/source/numpydonate.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL251bXB5ZG9uYXRlLnR4dA==..0000000000000000000000000000000000000000 --- a/source/numpydonate.txt +++ /dev/null @@ -1,198 +0,0 @@ ---- -layout: page -title: Call for donations - PyPy to support Numpy! ---- - -UPDATE (Febuary 2016): - - *We have closed this campaign. We have achieved most of the goals - of the orignal work plan while raising most of the funds. Work - will continue to make Numpy and the rest of the numeric stack more - usable on PyPy.* - -UPDATE (May 2014): - - *Thanks to our donors, we have raised 80% of - the total so far. Work on this topic has been happening, and - continues to happen, within the budget --- even if not within the - timeline described below. We have simply not found enough time to - work on it as much as we wanted, and thus did not consume the money as - quickly as predicted. The ratio "progress / $ used" so far - corresponds roughly to what we expected. The document below is the - original call for proposal, and we still accept donations for this - topic. See the* `latest status reports`_ *on our blog for updates. - There is also an automatically generated* `coverage dashboard`_ *showing - what parts of NumPy are already usable.* - - -This is a proposal to provide a fully compatible working `NumPy`_ implementation -for PyPy. This has long be en a very commonly `requested feature`_ for PyPy -as well as a worthy goal given that PyPy performs extremely well on numeric -workloads. - -We already had some success providing a very basic `NumPy implementation`_, -however we believe that raising funds can significantly speed up its -development, by paying people to work full time on it. - -Below you'll find the `work plan`_ and the associated -fundraising targets we need to make things happen. Once we reach the -necessary target for each stage, we will start contracting -developers. Contracts and money are managed by the non-profit -`Software Freedom Conservancy`_ of which the PyPy project is a member. -The current elected representatives are Carl Friedrich Bolz, Holger -Krekel and Jacob Hallen and they will – in close collaboration with -Conservancy and the core developers – select the best developers for -implementing NumPy among well known PyPy contributors. - -Should we not receive enough donations to complete all stages by 1st March 2012 -at the latest, we will try our best to make PyPy support NumPy anyway. We -however reserve the right to shift any unused funds to other PyPy activities -when that date is reached. Of course, since the Conservancy is a -501(c)(3) charitable organization incorporated in NY, USA, all funds will, -regardless of their use, be spent in a way that benefits the general -public, the advancement of Open Source and Free Software, -and in particular the PyPy community and the PyPy codebase. - -**Note** For donations higher than $1,000, we can arrange for an invoice -and a different payment method to avoid the high Paypal fees. Please -contact pypy at sfconservancy.org if you want to know details on how -to donate via other means. - -What is NumPy? --------------- - -`NumPy`_ is a framework for doing numerical calculations in Python. It has -become the de-facto standard for doing any kinds of computations that involve -n-dimensional arrays. Please consult the NumPy website for more details. - -Why does NumPy on PyPy make sense? ------------------------------------ - -NumPy on PyPy makes sense for a couple of reasons: Firstly, it is by -far the most requested feature from PyPy. Secondly, PyPy `performs -well`_ on numerical loads already. Therefore bringing NumPy into the -equation is a reasonable next step - as it's a very convenient and popular tool -for doing this kind of work. The resulting implementation should move -Python in scientific world from being a merely "glue" language into -being the main implementation language for a lot of people in -the scientific/numeric worlds. This will benefit current users of NumPy as -well as people who so far have to cope with lower level languages like C -or Fortran for speed purposes. - -.. _`NumPy`: http://numpy.scipy.org/ -.. _`requested feature`: http://morepypy.blogspot.com/2011/06/report-back-from-our-survey.html -.. _`NumPy implementation`: http://morepypy.blogspot.com/2011/05/numpy-in-pypy-status-and-roadmap.html -.. _`performs well`: http://morepypy.blogspot.com/2011/07/realtime-image-processing-in-python.html -.. _`latest status reports`: http://morepypy.blogspot.com/search/label/numpy -.. _`coverage dashboard`: http://buildbot.pypy.org/numpy-status/latest.html - -Speed ------ - -The current implementation of NumPy on PyPy is reasonably fast - it ranges -from roughly the same speed to 2-5x faster for stacked large array operations -to 100-300x for accessing NumPy array elements one by one. The exact speed -depends very much how NumPy is used, but the target would be to be within an -order of magnitude from handwritten C. -To achieve this, we would -need to teach our JIT backends how to use modern vector instructions, like SSE -or AVX. Hence, we split the proposal into two parts, first part covers compatibility -with a reasonable approach to keeping current speed achievements, second part -is about teaching the JIT how to vectorize certain operations, which should bring -PyPy's NumPy as a very competitive tool compared to other available -solutions for numerical computations, like matlab or C++ array libraries. - -.. _`work plan`: - -Work plan -========= - -About estimates and costs -------------------------- - -For each step, we estimated the time that it would take to complete for an -experienced developer who is already familiar with the PyPy codebase. From -this number, the money is calculated considering an hourly rate of $60, and a -5% general donation which goes to the `Software Freedom Conservancy`_ itself, the non-profit -organization of which the PyPy project is a member and which manages all the -issues related to donations, payments, and tax-exempt status. - -.. _`Software Freedom Conservancy`: http://sfconservancy.org/ - -We split the proposal into two parts - we plan to implement them in the same -order, starting by the time we raise the corresponding funding targets: - -* **Compatibility**: - - This part covers the core NumPy Python API. We'll implement all NumPy APIs - that are officially documented and we'll pass all of NumPy's tests that - cover documented APIs and are not implementation details. - Specifically, we don't plan to: - - * implement NumPy's C API - - * implement other scientific libraries, like SciPy, matplotlib or biopython - - Estimated costs: USD$30,000. Estimated duration: 3 months. - -* **Speed**: - - This part will cover significant speed improvements in the JIT that would - make numeric computations faster. This includes, but is not necesarilly - limited to: - - * write a set of benchmarks covering various use cases - - * teaching the JIT backend (or multiple backends) how to deal with vector - operations, like SSE - - * experiments with automatic parallelization using multiple threads, akin - to numexpr - - * improving the JIT register allocator that will make a difference, especially - for tight loops - - As with all speed improvements, it's relatively hard to predict exactly - how it'll cope, however we expect the results to be within an order - of magnitude of handwritten C equivalent. - - Estimated costs: USD$30,000. Estimated duration: 3 months. - -Benefits of This Work to the Python Community and the General Public -==================================================================== - -Python has become one of the most popular dynamic programming languages in -the world. Web developers, educators, and scientific programmers alike -all value Python because Python code is often more readable and because -Python often increases programmer productivity. - -Traditionally, languages like Python ran more slowly than static, compiled -languages; Python developers chose to sacrifice execution speed for ease -of programming. The PyPy project created a substantially improved Python -language implementation, including a fast Just-in-time (JIT) compiler. -The increased execution speed that PyPy provides has attracted many users, -who now find their Python code runs up to four times faster under PyPy -than under the reference implementation written in C. - -Meanwhile, adoption of Python is already underway for those researchers -and developers who work specifically on computing that requires fast -numeric operations. Numpy support in PyPy will allow for Python's use by -those developers and researchers who want the ease of programming that -Python provides, the speed of PyPy, *and* the speedups for numerical work -that Numpy can provide. - -PyPy's developers make all PyPy software available to the public without -charge, under PyPy's Open Source copyright license, the permissive MIT -License. PyPy's license assures that PyPy is equally available to -everyone freely on terms that allow both non-commercial and commercial -activity. This license allows for academics, for-profit software -developers, volunteers and enthusiasts alike to collaborate together to -make a better Python implementation for everyone. - -NumPy support for PyPy will be licensed similarly, and therefore NumPy -support can directly help researchers and developers who seek to do -numeric computing but want an easier programming language to use than -Fortan or C, which is typically used for these applications. Being -licensed freely to the general public means that opportunities to use, -improve and learn about how NumPy works itself will be generally available -to everyone. diff --git a/source/people.txt b/source/people.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3Blb3BsZS50eHQ=..0000000000000000000000000000000000000000 --- a/source/people.txt +++ /dev/null @@ -1,150 +0,0 @@ ---- -layout: page -title: People of PyPy ---- - -Armin Rigo -========== - -.. image:: image/people/arigo.png - -Armin Rigo is a former researcher at the Heinrich-Heine Universitat -Düsseldorf (Germany). He studied Mathematics at the University -of Lausanne (Switzerland), obtained his Ph.D. in Logic and Set -Theory at the Free University of Brussels (Belgium) in 2002, and -worked at the University of Southampton (UK) until 2005. He is -the author of Psyco, the first just-in-time compiler for Python. -He is one of the founders and lead developers of the PyPy project -which began in 2003. He has taken part in all areas, from the Python -language definition to the RPython translation framework, -including the garbage collector and the tracing just-in-time -compiler. - -Maciej FijaÅ‚kowski -================== - -.. image:: image/people/fijal_thumb.png - -Maciej is a freelancer working mostly on PyPy for the past several years. -He's a core developer since 2006, working on all kinds of parts in -the entire codebase including JIT, GC and assembler backends. -Maciej has been going to many conferences, advertising PyPy to a broader -audience for the past several years, including a keynote at Pycon 2010. -He's also the main maintainer of -`jitviewer`_, a tool for analyzing performance of your python programs under -PyPy. - -.. _`jitviewer`: https://bitbucket.org/pypy/jitviewer - -Carl Friedrich Bolz -=================== - -.. image:: image/people/cfbolz.jpg - -Carl Friedrich is a core developer since 2005, currently doing his PhD at the -Heinrich-Heine Universität Düsseldorf (Germany). He has worked on most aspects -of PyPy, from the core interpreter to the GC to the JIT. He has published -several papers about the inner workings of PyPy, presenting them at various -scientific conferences. Carl Friedrich is also interested in other dynamic -language implementation and was the original author of the Prolog -implementation. - -Carl Friedrich likes science fiction novels and sometimes plays the bassoon. - - -Antonio Cuni -============ - -.. image:: image/people/antocuni.png - -Antonio Cuni loves skiing, mountains and programming languages. He studied -Computer Science at the University of Genova (Italy), and then at the same -university he obtained his Ph.D. in Computer Science in 2010, with a -dissertation about the PyPy CLI JIT backend. He has been a core PyPy -developer since 2006, working in various areas including the "object oriented -backends" for the CLI and JVM, the RPython translation framework, the Python -interpreter and the JIT compiler generator. Apart from PyPy, he is the author of -other popular tools such as ``pdb++``. - -Benjamin Peterson -================= - -Both a PyPy and CPython core developer, Benjamin knows way too much about the -nooks and cranies of the Python language. He is driven by a fascination with -interpreters and compilers of all shapes and sizes. Around the PyPy project, he -tries to be generally useful and has taken on major projects including rewriting -PyPy's Python compiler and porting PyPy to Python 2.7. - -Alex Gaynor -=========== - -.. image:: image/people/alex.jpg - -Alex is software engineer living in Washington, DC. He's been a PyPy developer -since 2010, and has worked on many parts of the codebase, including the JIT -compiler's optimizers, the RPython translation toolchain, and the Python -interpreter. In addition to his work on PyPy, Alex is also the creator of -Topaz, a Ruby VM built on RPython and a core developer of Django (a Python web -framework) and CPython, as well as a retired member of the board of directors -of the Python Software Foundation. - -HÃ¥kan Ardö -========== - -.. image:: image/people/hakanardo.jpg - -HÃ¥kan Ardö received his master of science degree in electrical -engineering from Lund University in 2002. He specialized in -VLSI-design and Image Processing. He worked as a software -engineer at Axis Communications 2002-2003 before doing his -PhD at the Centre for Mathematical Sciences of Lund University -2003-2009 in the Mathematical Imaging Group. His thesis work consisted -of designing image processing algorithms for traffic surveillance, -aiming for a system that automatically measures the safety of an -intersection or road segment. He is currently working part-time as a -postdoc at the Centre for Mathematical Sciences of Lund University -continuing this work and part-time as CTO with a spinoff company -Cognimatics. His contributions to PyPy started 2010 and consists of -the array module as well as work on the JIT compiler's trace optimizers. - -Holger Krekel -================== - -.. image:: image/people/holger1.jpg - -Holger Krekel is a founder of the PyPy project and has participated in -PyPy core developement for several years as well as maintained much of -its infrastructure. He also is the author of the popular `py.test`_ and -`tox`_ testing tools as well as execnet_, a library for easily deploying -different interacting Python interpreters side by side. He helped -manage multiple PyPy funding contracts through his company merlinux and is a -PyPy representative within the Software Freedom Conservancy (SFC). He -holds a summa cum laude degree in computer science with a thesis about -artificial intelligence applied to the game of Go. As of 2011 he is on -another sabbatical-ish leave, caring for his newborn son, travelling -and pondering what comes next. Other than that he continues to care -for testing and some PyPy co-ordination bits behind the scene. - -.. _`py.test`: http://pytest.org -.. _`tox`: http://codespeak.net/tox -.. _`execnet`: http://codespeak.net/execnet - -Samuele Pedroni -=============== - -Samuele Pedroni got involved with PyPy almost at its inception in the -spring of 2003. One of the design contributors to PyPy, his help has -ranged from infrastructure and processes, through building out -RPython... optimizing the Python interpreter, to compressing resume -data in the last incarnation of the JIT compiler. Tempted away into the -application side of the software equation, these days he contributes -some words and wisdom to PyPy's paper writing. - - -Many more people -================ - -PyPy is and has always been an effort of many volunteers. Consult the `LICENSE`_ -file for details. - -.. _`LICENSE`: https://bitbucket.org/pypy/pypy/src/tip/LICENSE diff --git a/source/performance.txt b/source/performance.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3BlcmZvcm1hbmNlLnR4dA==..0000000000000000000000000000000000000000 --- a/source/performance.txt +++ /dev/null @@ -1,364 +0,0 @@ ---- -layout: page -title: Performance ---- - -.. contents:: - :depth: 1 - -This document collects strategies, tactics and tricks for making your -code run faster under PyPy. Many of these are also useful hints for -stock Python and other languages. For contrast, we also describe some -CPython (stock Python) optimizations that are not needed in PyPy. - - -================= - -.. _profiler: -.. _profiling: - -Profiling: vmprof -================= - -As a general rule, when considering performance issues, follow these -three points: first *measure* them (it is counter-productive to fight -imaginary performance issues); then *profile* your code (it is useless -to optimize the wrong parts). Only optimize then. - -PyPy 2.6 introduced vmprof_, a very-low-overhead statistical profiler. -The standard, non-statistical ``cProfile`` is also supported, and can be -enabled without turning off the JIT. We do recommend vmprof anyway -because turning on cProfile can distort the result (sometimes massively, -though hopefully this should not be too common). - -.. _vmprof: https://vmprof.readthedocs.org/ - - -===================== - -Optimization strategy -===================== - -These suggestions apply to all computer languages. They're here as -reminders of things to try before any Python or PyPy-specific tweaking. - -Build a regression-test suite ------------------------------ - -Before you start tuning, build a regression-test suite for your code. -This front-loads a significant amount of work, but it means you can -try lots of optimizations without worrying so much about introducing -functional bugs. - -Measure, don't guess --------------------- - -Human beings are bad at guessing or intuiting where the hotspots in code are. -Measure, don't guess; use a profiler_ to pin down the 20% of the -code where the code is spending 80% of its time, then speed-tune that. - -Measuring will save you a lot of effort wasted on tuning parts of the code -that aren't actually bottlenecks. - -As you tune, re-profile frequently so you can see how the hottest spots -are shifting around. - -I/O-bound is different from compute-bound ------------------------------------------ - -Be aware of the difference between code that is compute-bound (slow -because it's doing a huge number of instructions) and code that is I/O -bound (slow because of disk or network delays). - -Expect to get most of your gains from optimizing compute-bound code. -It's usually (though not always) a sign that you're near the end of -worthwhile tuning when profiling_ shows that the bulk of the -application's time is spent on network and disk I/O. - -Tune your algorithms first --------------------------- - -Generally, when your code is doing things that are O(n**2) or larger -in the size of your data set, the cost of those operations is going -to swamp any small gains you can pick up with the tricks we describe -here. - -Tune your algorithms first. It's time to think about applying our -list of micro-tuning tips *after* you think you've optimized out -intrinsically expensive operations. - -That said, be prepared for the possibility that you will discover -better-hidden algorithmic problems as you micro-tune. Likely -you will go through this cycle more than once. - -Focus on tight loops --------------------- - -It's extremely common for high time costs to lurk within some -innocuous-looking code inside a tight loop - especially in code -that does something like a searching/matching/lookup operation -or any kind of graph traversal. - -Probably the most common kind of performance-killer in compute-bound -code is an O(n**2) operation that is disguised by being some sort of -O(n) lookup or match inside an O(n) loop. - -Another common time-sink is relatively expensive common-setup -operations that are performed inside tight loops but could be moved -to before they start. (For a representative case of this, see the -micro-tuning tip on regexp compilation.) - -Smaller is faster ------------------ - -Modern computers have multiple levels of memory caching, some directly -on the processor chip. Causing a cache miss at any level incurs a -performance penalty proportional to random-access time for the next -outward (and much slower) layer of cache. - -Accordingly, smaller is faster. Programs or routines with a small -enough working set to fit inside a fast cache will be as fast as -that cache is. To make your code fast, reduce the length of the -series of Python or JIT-compiler opcodes it generates by making -it simpler. - -The tradeoff here is that algorithmic tuning often trades time for -space - that is, it increases the size of an algorithm's working set -by including pre-computations or tables or reverse maps in order to -avoid O(n**2) operations. - -It's impossible to predict in advance where the sweet spot in that -tradeoff will be. You have to try different things and measure - -which takes us right back to "Measure, don't guess". And another -function of your regression test suite can be as a speed benchmark. - - -================= - -Micro-tuning tips -================= - -These are in no particular order. - -Keep it simple --------------- - -Simple is better than complex. The PyPy JIT is not very smart; the -simpler your code is the better it will run. Here again, though, you face -a tradeoff: you may need to pay with more algorithmic complexity in order -to avoid brute-force operations that are O(n**2) or worse. - -Write plain-vanilla code in plain-vanilla ways. The PyPy JIT has many -productions that optimize a common usage pattern against an uncommon -usage pattern. - -Global variables ----------------- - -In CPython, global variables and functions (including package imports) -are much more expensive to reference than locals; avoid them. (This -is also good modularity practice). - -The cost of CPython global references is high enough that, for example, if you -have code in a frequently-visited inner loop that uses int() a lot, it -may be worthwhile to create a local copy of the reference with "int = -int" in an enclosing block. - -However, this in *not* true in JITted PyPy code. The "int = int" hack -won't buy you performance, it's just an extra copy. The modularity -reason for avoiding globals are still valid. - -Regular expressions -------------------- - -Regular-expression compilation is expensive. If the regexp pattern in -a search, match, or replace operation is static (doesn't mutate at -runtime) refactor so it's only done once. - -If the regexp compilation is in a class method, consider doing it as -the initializer of a regexp-valued static (shared) class member and -using that class member in your operation. - -If the regexp compilation is in a free function, consider moving it -to module level and referencing the resulting regexp object -(but see the warning above about global variables). - -Old- vs. new-style classes --------------------------- - -New-style classes allow faster attribute access and take up less core -per instance than old-style classes. Much of this advantage may be -lost, however, if attribute names are not constant. For example: x.a -= y or even setattr(x, 'a', y) will be much faster than a dynamic -version: setattr(x, 'a' + some_variable, y). - -Classes that inherit from both new- and old-style classes are -*extremely* slow; avoid at all costs. - -In PyPy, isinstance() called against an old-style class was very slow -until 2.0. - -String concatenation is expensive ----------------------------------- - -In CPython, you may want to replace:: - - s = head + body + maybe + tail - -with the admittedly less readable:: - - s = "%(head)s%(body)s%(maybe)s%(tail)s" % locals() - -or even:: - - s = "{head}{body}{maybe}{tail}".format(**locals()) - -Both of the latter forms avoid multiple-allocation overhead. -But PyPy's JIT makes the overhead of intermediate concatenations -go away in linear code that keeps the number of concatenations -small, bound and constant. (And ``locals()`` is rather slow -with PyPy's JIT.) - -On the other hand, in code like this with a string-valued foo() function:: - - for x in mylist: - s += foo(x) - -the JIT cannot optimize out intermediate copies. This code is -actually quadratic in the total size of the mylist strings due to -repeated string copies of ever-larger prefix segments. (Such code -is always fine for bytearrays, because in this case ``+=`` is an -in-place operation.) - -This:: - - parts = [] - for x in mylist: - parts.append(foo(x)) - s = "".join(parts) - -can be much faster because all the string concatenation in the last -line creates exactly one new string object with one C-level copy -sequence (and list operations are relatively cheap). - -Frame introspection and tracing are slow ----------------------------------------- - -Certain function calls can disable PyPy's speed options over -stretches of surrounding code called "JIT scopes". - -A JIT like PyPy's works based on the assumption that the only thing -worth optimizing are loops that are executed often. Whenever the -interpreter enters a loop in the interpreted program, the JIT records -what the interpreter does, creating a trace. This trace is optimized, -compiled to machine code and executed when the loop is hit with the -conditions observed during tracing. This trace is one kind of JIT scope. - -Another kind of JIT scope that matters is a function, considered as -a unit for inlining. - -Note that a JIT scope is a run-time phenomenon, not a compile-time -one. It's not confined by source-code module boundaries. A library- -or foreign-module call in a frequently-called loop or inlined function -will be part of its JIT scope. - -locals(), globals(), sys._getframe(), sys.exc_info(), and sys.settrace -work in PyPy, but they incur a performance penalty that can be huge by -disabling the JIT over the enclosing JIT scope. - -*(Thanks Eric S. Raymond for the text above)* - - -======================= - -Insider's point of view -======================= - -This section describes performance issues from the point of view of -insiders of the project; it should be particularly interesting if you -plan to contribute in that area. - -One of the goals of the PyPy project is to provide a fast and compliant -python interpreter. Some of the ways we achieve this are by providing a -high-performance garbage collector (GC) and a high-performance -Just-in-Time compiler (JIT). Results of comparing PyPy and CPython can -be found on the `speed website`_. Those benchmarks are not a random -collection: they are a combination of real-world Python programs --- -benchmarks originally included with the (now dead) Unladen Swallow -project --- and benchmarks for which we found PyPy to be slow (and improved). -Consult the descriptions of each for details. - -The JIT, however, is not a magic bullet. There are several characteristics -that might surprise people who are not used to JITs in -general or to the PyPy JIT in particular. The JIT is generally good at -speeding up straight-forward Python code that spends a lot of time in the -bytecode dispatch loop, i.e., running actual Python code --- as opposed -to running things that only are invoked by Python code. Good -examples include numeric calculations or any kind of heavily -object-oriented program. Bad examples include doing computations with -large longs --- which is performed by unoptimizable support code. When the -JIT cannot help, PyPy is generally slower than CPython. - -More specifically, the JIT is known not to work on: - -* **Tests**: The ideal unit tests execute each piece of tested code - once. This leaves no time for the JIT to warm up. - -* **Really short-running scripts**: A rule of thumb is if something runs below - 0.2s the JIT has no chance, but it depends a lot on the program in question. - In general, make sure you warm up your program before running benchmarks, if - you're measuring something long-running like a server. The time required - to warm up the JIT varies; give it at least a couple of seconds. (PyPy's - JIT takes an especially long time to warm up.) - -* **Long-running runtime functions**: These are the functions provided - by the runtime of PyPy that do a significant amount of work. - PyPy's runtime is generally not as optimized as CPython's and we expect those - functions to take somewhere between the same time as CPython to twice as long. - This includes, for example, computing with longs, or sorting large lists. - A counterexample is regular expressions: although they take time, they - come with their own JIT. - -Unrelated things that we know PyPy to be slow at (note that we're probably -working on it): - -* **CPython C extension modules**: Any C extension module recompiled - with PyPy takes a very large hit in performance. PyPy supports C - extension modules solely to provide basic functionality. - If the extension module is for speedup purposes only, then it - makes no sense to use it with PyPy at the moment. Instead, remove it - and use a native Python implementation, which also allows opportunities - for JIT optimization. If the extension module is - both performance-critical and an interface to some C library, then it - might be worthwhile to consider rewriting it as a pure Python version - that uses CFFI_ for the interface. - -* **Missing RPython modules**: A few modules of the standard library - (like ``csv`` and ``cPickle``) are written in C in CPython, but written - natively in pure Python in PyPy. Sometimes the JIT is able to do a - good job on them, and sometimes not. In most cases (like ``csv`` and - ``cPickle``), we're slower than CPython, with the notable exception of - ``json`` and ``heapq``. - -* **Abuse of itertools**: The itertools module is often "abused" in the - sense that it is used for the wrong purposes. From our point of view, - itertools is great if you have iterations over millions of items, but - not for most other cases. It gives you 3 lines in functional style - that replace 10 lines of Python loops (longer but arguably much easier - to read). The pure Python version is generally not slower even on - CPython, and on PyPy it allows the JIT to work much better --- simple - Python code is fast. The same argument also applies to ``filter()``, - ``reduce()``, and to some extend ``map()`` (although the simple case - is JITted), and to all usages of the ``operator`` module we can think - of. - -* **Ctypes**: Ctypes is slower than on CPython. Consider CFFI_ instead, - which has special paths inside the JIT. - -We generally consider things that are slower on PyPy than CPython to be bugs -of PyPy. If you find some issue that is not documented here, -please report it to our `bug tracker`_ for investigation. - -.. _`bug tracker`: https://bitbucket.org/pypy/pypy/issues?status=new&status=open -.. _`speed website`: http://speed.pypy.org -.. _CFFI: http://cffi.readthedocs.org/ diff --git a/source/py3donate.txt b/source/py3donate.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3B5M2RvbmF0ZS50eHQ=..0000000000000000000000000000000000000000 --- a/source/py3donate.txt +++ /dev/null @@ -1,316 +0,0 @@ ---- -layout: page -title: Call for donations - PyPy to support Python3! ---- - -UPDATE (December 2017): - - PyPy3 supporting Python 3.5 has been released. - -UPDATE (August 2016): - - *We are soon releasing a beta supporting Python 3.3. For the next - full year, though, see* `PyPy gets funding from Mozilla for Python 3.5 - support`__. *Individual donations through here are still welcome, and - go towards the same goal, which is upgrading PyPy to support Python - 3.x (which really means 3.3/3.5 by now). Thanks to all our past - contributors! Your money has been put to good use so far.* - -.. __: https://morepypy.blogspot.ch/2016/08/pypy-gets-funding-from-mozilla-for.html - -UPDATE (February 2014): - - *Thanks to our donors, we have raised 45% of - the total so far. Work on this topic has been happening, and - continues to happen, within the budget --- even if not within the - timeline described below. We have simply not found enough time to - work on it as much as we wanted, and thus did not consume the money as - quickly as predicted. The ratio "progress / $ used" so far - corresponds roughly to what we expected. The document below is the - original call for proposal, and we still accept donations for this - topic.* - - -The release of Python 3 has been a major undertaking for the Python -community, both technically and socially. So far the PyPy interpreter -implements only version 2 of the Python language and is increasingly -used in production systems. It thus contributes to the general risk -of a long lasting Python community split where a lot of people -continue using Python 2 while others work with Python 3, making it -harder for everyone. - -The PyPy project is in a unique position in that it could support -Python 3 without having to discontinue supporting Python 2, with the possibility of reusing a large part of the code base and fully -reusing its unique translation and JIT-Compiler technologies. -However, it requires a lot of work, and it will take a long time -before we can complete a Python 3 port if we only wait for volunteer -work. Thus, we are asking the community to help with funding the -necessary work, to make it happen faster. Here_ is a more detailed view on how our proposed work benefits the Python community and the general public. - -Below you'll find the `planned stages of work`_ and the associated -fundraising targets we need to make things happen. Once we reach the -necessary target for each stage, we will start contracting -developers. Contracts and money are managed by the non-profit -`Software Freedom Conservancy`_ of which the PyPy project is a member. -The current elected representatives are Carl Friedrich Bolz, Holger -Krekel and Jacob Hallen and they will - in close collaboration with -Conservancy and the core developers - select the best developers for -the Python 3 porting job among well known PyPy contributors. - -If you want to see PyPy support Python 3 and Python 2, donate using buttons -on the side. - -Should we not receive enough donations to complete all stages by 1st March 2012 -at the latest, we will try our best to make PyPy support Python 3 anyway. We -however reserve the right to shift any unused funds to other PyPy activities -when that date is reached. Of course, since the Conservancy is a -501(c)(3) charitable organization incorporated in NY, USA, all funds will, -regardless of their use, be spent in a way that benefits the general -public, the advancement of Open Source and Free Software, -and in particular the PyPy community and the PyPy codebase. - -**Note** For donations higher than $1,000, we can arrange for an invoice -and a different payment method to avoid the high Paypal fees. Please -contact pypy at sfconservancy.org if you want to know details on how -to donate via other means. - -.. _`Software Freedom Conservancy`: http://sfconservancy.org/ -.. _`planned stages of work`: - -Planned stages of work -====================== - -The goal of this project is to write an interpreter that interprets -version 3 of the Python language. To be precise we would aim at having -a Python 3.2 interpreter together in the same codebase as the python 2.7 -one. - -At the end of the project, it will be possible to decide at -translation time whether to build an interpreter which supports Python -2.7 or Python 3.2 and both versions will be nightly tested and -available from nightly builds. - -The focus of this project is on compatibility, not performance. In -particular, it might be possible that the resulting Python 3 -interpreter will be slower than the Python 2 one. If needed, -optimizing and making it more JIT friendly will be the scope of a -separate project. Our existing JIT generation technology should apply -out of the box; this disclaimer is only about the extra performance we -could obtain by tweaking the Python 3 interpreter or writing specific -interpreter-guided optimizations in the JIT. - -About estimates and costs -------------------------- - -For each step, we estimated the time that it would take to complete for an -experienced developer who is already familiar with the PyPy codebase. From -this number, the money is calculated considering an hourly rate of $60, and a -5% general donation which goes to the `Software Freedom Conservancy`_ itself, the non-profit -association of which the PyPy project is a member and which manages all the -issues related to donations, taxes and payments. - -The estimated time to complete the whole project is about 10.5 person-months. - -For comparison, the python-3000 mailing list was created in March 2006; Python -3.0 was released in December 2008 and Python 3.1, the first release genuinely -suitable for production use (due to the abysmal I/O performance of 3.0) was -released in June 2009. During these 3.5 years, a lot of people contributed to -the development of Python 3, and while it is hard to turn these numbers into -precise person-years, it sounds reasonable to think that in total it took -several person-years. - -We have the advantage of targeting something that already exists without -having to define the destination as they go along, and also the internal -architecture of PyPy makes it easier to do the porting. - - -Step 1: core language ----------------------- - -In this step, we implement all the changes to the core language, -i.e. everything which is not in the extension modules. This includes, but it -is not necessarily limited to the following items, which are split into two -big areas: - -* **Sub-step 1.1**: string vs unicode and I/O: - - - adapt the existing testing infrastructure to support running Python 3 code - - - string vs bytes: the interpreter uses unicode strings everywhere. - - - the ``print`` function - - - ``open`` is now an alias for ``io.open``, removal of the old file type. - - - string formatting (for the part which is not already implemented in Python - 2.7) - - - the _io module (for the part which is not already implemented in Python - 2.7) - - - syntactic changes to make ``io.py`` importable (in particular: - ``metaclass=...`` in class declarations) - - - **Estimate cost**: $35,000 - -* **Sub-step 1.2**: other syntactic changes, builtin types and functions, - exceptions: - - - views and iterators instead of lists (e.g., ``dict.items()``, ``map``, - ``range`` & co.) - - - new rules for ordering comparisons - - - removal of old-style classes - - - int/long unification - - - function annotations - - - smaller syntax changes, such as keyword-only arguments, ``nonlocal``, - extended iterable unpacking, set literals, dict and set comprehension, etc. - - - changes to exceptions: ``__traceback__`` attribute, chained exceptions, - ``del e`` at the end of the except block, etc. - - - changes to builtins: ``super``, ``input``, ``next()``, etc. - - - improved ``with`` statement - - - **Estimate cost**: $28,000 - - -Note that the distinction between sub-steps 1.1 and 1.2 is blurry, and it might be -possible that during the development we will decide to move items between the -two sub-steps, as needed. - -For more information, look at the various "What's new" documents: - -- http://docs.python.org/py3k/whatsnew/3.0.html - -- http://docs.python.org/py3k/whatsnew/3.1.html - -- http://docs.python.org/py3k/whatsnew/3.2.html - -**Total estimate cost**: $63,000 - -.. Time estimate: 3.5 + 2.5 = 6 man/months - - -Step 2: extension modules --------------------------- - -In this step, we implement all the changes to the extension modules which are -written in C in CPython. This includes, but it is not necessarily limited to: - -- ``collections``, ``gzip``, ``bz2``, ``decimal``, ``itertools``, ``re``, - ``functools``, ``pickle``, ``_elementtree``, ``math``, etc. - -**Estimate cost**: this is hard to do at this point, we will be able to give a -more precise estimate as soon as Step 1 is completed. As a reference, it -should be possible to complete it with $37,000 - -.. Time estimate: 4 man/months - -Step 3: cpyext --------------- - -The ``cpyext`` module allows to load CPython C extensions in PyPy. Since the -C API changed a lot between Python 2.7 and Python 3.2, ``cpyext`` will not -work out of the box in the Python 3 PyPy interpreter. In this step, we will -adapt it to work with Python 3 as well. - -Note that, even for Python 2, ``cpyext`` is still in a beta state. In -particular, not all extension modules compile and load correctly. As a -consequence, the same will be true for Python 3 as well. As a general rule, -we expect that if a Python 2 module works with ``cpyext``, the corresponding -Python 3 version will also work when this step is completed, although the -details might vary depending on the exact C extension module. - -**Estimate cost**: $5,000 - -.. Time estimate: 0.5 man/month - -.. _`here`: - -Benefits of This Work to the Python Community and the General Public -==================================================================== - -Python has become one of the most popular dynamic programming languages in -the world. Web developers, educators, and scientific programmers alike -all value Python because Python code is often more readable and because -Python often increases programmer productivity. - -Traditionally, languages like Python ran more slowly than static, compiled -languages; Python developers chose to sacrifice execution speed for ease -of programming. The PyPy project created a substantially improved Python -language implementation, including a fast Just-in-time (JIT) compiler. -The increased execution speed that PyPy provides has attracted many users, -who now find their Python code runs up to four times faster under PyPy -than under the reference implementation written in C. Some programs that -are particularly JIT-friendly see even greater speedups. See our speed__ -website. - -.. __: http://speed.pypy.org - -Meanwhile, the Python community is undergoing significant change, with the -introduction of a new 'version 3' of the Python language (Python 3). -Python 3 breaks some backwards compatibility with 2.x series, so -programmers who seek to use Python 3 must port old code. For example, -programmers who needed old libraries might want to use Python 2 only and -programmers wanting to use new language features would like to use Python -3, but would not be able to use libraries that written for Python 2.x -without substantial rewrites. This issue could lead to a dangerous -community split: programmers who needed old libraries might only use -Python 2 and programmers seeking new language features would use Python 3, -but would not be able to use libraries written for Python 2.x. - -PyPy currently supports only Python 2.7. While PyPy supports only version -2 of the Python language, PyPy users have an incentive to avoid Python 3. -Python programmers must chose between the language features in Python 3, -and the substantial performance benefits offered by PyPy. - -To address this issue, the PyPy team proposes to implement Python 3 on -PyPy. With such improvements to PyPy, PyPy can support the entire -Python-using community and hopefully help to prevent any community split. -PyPy support of Python 3 would also bring the excellent code execution -performance of PyPy to those who wish to migrate to Python 3. - -Moreover, by ensuring the latest version of the Python language -specification (Py3k) works properly and fully on PyPy, the maximal benefit -of collaboration, learning, and software improvement can happen in the -entire Python language community. - -A broad community of developers support and develop the PyPy project, -many of whom work as volunteers. The Py3k grant should help with turning -some attention towards implementing Python 3. This will not hinder other -directions in which PyPy is going like improving performance. The goal -of the PyPy community is to support both Python 2 and Python 3 for the -foreseeable future. - -PyPy's developers make all PyPy software available to the public without -charge, under PyPy's Open Source copyright license, the permissive MIT -License. PyPy's license assures that PyPy is equally available to -everyone freely on terms that allow both non-commercial and commercial -activity. This license allows for academics, for-profit software -developers, volunteers and enthusiasts alike to collaborate together to -make a better Python implementation for everyone. - -Finally, tracing JITs and other programming language execution technology -used in PyPy are of current and particular interest in computer science -research. PyPy helps cross-pollinate knowledge between academic computer -science and industrial use of Python, since PyPy can function well both as -a research tool and real-world Python programming language implementation. -Continued support and evolution of PyPy in any direction, such as support -for Py3k, increases the features available from PyPy and such improvements -are expected to spark even more general interest in PyPy itself and the -Python programming language generally. - -References on Educational Uses of Python: - - http://wiki.python.org/moin/SchoolsUsingPython - - http://www.python.org/doc/essays/cp4e.html - -References for PyPy and its benefits: - - http://pypy.org/ - - http://pypy.org/features.html - - http://pypy.org/compat.html diff --git a/source/speed.txt b/source/speed.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3NwZWVkLnR4dA==..0000000000000000000000000000000000000000 --- a/source/speed.txt +++ /dev/null @@ -1,1 +0,0 @@ - diff --git a/source/sponsor.txt b/source/sponsor.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3Nwb25zb3IudHh0..0000000000000000000000000000000000000000 --- a/source/sponsor.txt +++ /dev/null @@ -1,39 +0,0 @@ ---- -layout: sponsor -title: Sponsor PyPy ---- - -PyPy crowdfunding program -------------------------- - -As an Open Source project, PyPy is mostly an effort of volunteers. However, -we've decided to run a `pilot program`_ of `crowdfunding`_ to implement NumPy. - -Our goal is to raise 20000 Euros by the 1st of October 2011. That should allow -us to finish implementing NumPy by the end of 2011. - -.. raw:: html - - <div id="progressbar"> - <div class="progresstext"> - 0 of 20000 (0%) - </div> - </div> - <form action="https://www.paypal.com/cgi-bin/webscr" method="post"> - Sponsor numpy implementation in PyPy by the following amount:<br/> - <input type="text" name="amount"></input> - <input type="hidden" name="item_name" value="Sponsor NumPy on PyPy"/> - <input type="hidden" name="quantity" value="1"/> - <select name="currency_code"> - <option value="USD" selected="1">USD</option> - <option value="EUR">EUR</option> - <option value="GBP">GBP</option> - </select> - <input type="hidden" name="cmd" value=" _s-xclick"/> - <input type="hidden" name="hosted_button_id" value="QVB9EDQXBUPJY"> - <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_paynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!"> - <img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1"> - </form> - -.. _`pilot program`: xxxbloglink -.. _`crowdfunding`: http://en.wikipedia.org/wiki/Crowdfunding diff --git a/source/success.txt b/source/success.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3N1Y2Nlc3MudHh0..0000000000000000000000000000000000000000 --- a/source/success.txt +++ /dev/null @@ -1,29 +0,0 @@ ---- -layout: page -title: Success stories ---- - -PyPy has been successfully used in production by various people. Here are -few excepts of feedback we received: - -LWN short experiment --------------------- - -LWN has a tool they use for parsing "git log"'s output in order to do analysis -on the contributors to the Linux kernel. Simply `by switching to PyPy`_ they -were able to get a 3x speedup, from 63 seconds to 21 seconds, compared to -CPython. - -"In other ways, PyPy is ready for prime time; it implements the (Python 2.x) language faithfully, and it is fast." - -.. _`by switching to PyPy`: http://lwn.net/Articles/442268/ - -MyHDL ------ - -MyHDL got a speed up by `6-12x compared to CPython`_, on a range of benchmarks, -putting it in the same league as other HDL implementations. - -"By simply changing the Python interpreter, MyHDL is playing in the same league as Verilog and VHDL simulators. This is a remarkable achievement, given that Python's power stays completely available. There is no reason anymore to avoid MyHDL because of performance concerns." - -.. _`6-12x compared to CPython`: http://www.myhdl.org/doku.php/performance diff --git a/source/tmdonate.txt b/source/tmdonate.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3RtZG9uYXRlLnR4dA==..0000000000000000000000000000000000000000 --- a/source/tmdonate.txt +++ /dev/null @@ -1,406 +0,0 @@ ---- -layout: page -title: Call for donations - Transactional Memory / Automatic Mutual Exclusion in PyPy ---- - -================================================= -Transactional Memory / Automatic Mutual Exclusion -================================================= - -.. class:: download_menu - - **UPDATE (April 2014): this is the old Call for Donations about - Transactional Memory**, kept around for historical purposes. A `new Call - for Donations`__ is available. - -.. __: tmdonate2.html - - -Introduction -============ - -In the presence of today's machines with multiple processors, Python -progress is lagging behind: on any CPU-constrained program, developers -have a difficult choice to make. They can use in-process solutions that -do not offer multi-CPU usage. In this respect, the natural choice -nowadays is to use Twisted or other event-based paradigms, or systems -that hide events in the control flow, like Stackless; or alternatively, -they can use the existing ``threading`` module, with its associated GIL -and the complexities of real multi-threaded programming (locks, -deadlocks, races, etc.), which make this solution less attractive. The -big alternative is for them to rely on one of various multi-process -solutions that are outside the scope of the core language; all of them -in some way or another are hacks that require extra knowledge and time -to use and that have an impact on the structure of the whole program. - -This proposal is about researching and implementing Transactional Memory -in PyPy. This is a technique that recently came to the front of the -multi-core scene. It promises to offer multi-core CPU usage without -requiring to fall back to the multi-process solutions described above, -and also without using the ``threading`` module --- just as a small, -local extension of the programming language that would be used only in -the core of the event loops. - -(Jump directly to `What Python interface will I use?`_ for practical -details.) - - -In more details -=============== - -This is a call for financial help in researching and implementing a -version of PyPy able to use multiple processors in a single process. -This will give a GIL-less Python, i.e. a Python that runs without the -infamous Global Interpreter Lock. - -The basic ideas to do it have been discussed on pypy-dev `[1]`__ `[2]`__ -and on a blog post `[3]`__. -The goal is to adapt Transactional Memory --- currently only available -as software, but `soon available as hardware`_ --- to the task of running -sections of Python code in parallel on multiple processors, while giving -the programmer the illusion of serial execution. It is called below -"PyPy-TM". - -.. __: http://mail.python.org/pipermail/pypy-dev/2011-August/008153.html -.. __: http://mail.python.org/pipermail/pypy-dev/2012-January/009034.html -.. __: http://morepypy.blogspot.com/2012/01/transactional-memory-ii.html - -The main developer will be Armin Rigo. -This is a "researchy" goal in the sense of us not being quite sure of -the performance of the result. We currently estimate the one-year -performance goal at 2x-to-5x slower than regular PyPy in fully serial -applications. We feel confident that it can work, though, in the -following sense: the performance of PyPy-TM running suited applications -should scale linearly or close-to-linearly with the number of processors. -This means that you just need a machine with at least 4 or 8 processors, -which is already common today and will be even more so in one or two -years. - -You will find below a sketch of the `work plan`_. If more money than -requested is collected, then the excess will be entered into the general -PyPy pot, used for example to finance sprint travel costs to students. - -**Note** For donations higher than $1,000, we can arrange for an invoice -and a different payment method to avoid the high Paypal fees. Please -contact pypy at sfconservancy.org if you want to know details on how -to donate via other means. - - -What is the Global Interpreter Lock? ------------------------------------- - -The GIL, or Global Interpreter Lock, is a single lock in both CPython -and (so far) PyPy, that all threads must acquire in order to execute -Python bytecodes. This means that so far, in Python, even when using -threads we do not gain any benefit in term of multicore performance. - - -What is Transactional Memory? ------------------------------ - -`Transactional Memory`_ --- TM --- is a technique imported from databases: every -time we want to do a change to the processors' main memory, we do it in -a "transaction". Multiple transactions can be executed in parallel by -multiple cores. When a transaction is complete, we try to commit it. -This might either succeed, or (if another transaction committed -incompatible changes) fail. If it fails, which is hopefully rare, we -need to restart the transaction from scratch. - -.. _`Transactional Memory`: http://en.wikipedia.org/wiki/Transactional_memory - - -Why hasn't the idea been implemented for CPython already? ---------------------------------------------------------- - -Because of the additional complexity required, and mostly, because of -performance issues. There have been some experiments already with -CPython on *Hardware* Transactional Memory: - -* `Riley and Zilles (2006)`__ -* `Tabba (2010)`__ - -.. __: http://sabi.net/nriley/pubs/dls6-riley.pdf -.. __: http://www.cs.auckland.ac.nz/~fuad/parpycan.pdf - -The motivation for using PyPy instead of CPython is the extra -flexibility of the general approach, as well as the ability to utilize -the JIT in order to remove part of the overhead that comes with -*Software* Transactional Memory. - - -A GIL-less Python is impossible. --------------------------------- - -This is a classic criticism of research-oriented projects. We believe -that the `work plan`_ below can make a serious impact on considering -possible a GIL-less Python. We believe we can do it, but at the -very least, even if this work generates a negative result, the negative -result will document the challenges faced should someone else want to -reattempt the idea in the future. - -Nevertheless other projects, such as Psyco (also by Armin) and PyPy -itself, were called impossible before they were attempted, and have -hitherto been very successful. - - -Why do it with PyPy instead of CPython? ---------------------------------------- - -Because PyPy is designed to be open to this kind of research. This will -require no work in the Python interpreter part of PyPy, and instead we -can focus on e.g. the concurrent garbage collection and the JIT issues. -`Riley and Zilles`__ have also experimented with Hardware Transactional -Memory using PyPy. By contrast, for example, CPython is stuck with -reference counting, which is known to be an issue for Transactional -Memory (`Tabba`__ proposes to give up and use Boehm_, which is a bad -idea in our experience, particularly because of scalability issues). - -.. __: http://sabi.net/nriley/pubs/dls6-riley.pdf -.. __: http://www.cs.auckland.ac.nz/~fuad/parpycan.pdf -.. _Boehm: http://www.hpl.hp.com/personal/Hans_Boehm/gc/ - - -What Python interface will I use? ---------------------------------- - -Previous attempts on Hardware -Transactional Memory focused on parallelizing existing programs written -using the ``thread`` or ``threading`` modules. However, as argued -here__, this may not be the most practical way to achieve real -multithreading; it seems that better alternatives would offer good -scalability too. Notably, TM could benefit any event-based system that -is written to dispatch events serially (Twisted-based, most GUI toolkit, -Stackless, gevent, and so on). The events would internally be processed -in parallel, while maintaining the illusion of serial execution, with -all the corresponding benefits of safety. This should be possible with minimal -changes to the event dispatchers. This approach has been described by the -`Automatic Mutual Exclusion`_ work at Microsoft Research, but not been -implemented anywhere (to the best of our knowledge). - -.. _`Automatic Mutual Exclusion`: http://research.microsoft.com/en-us/projects/ame/default.aspx - -Note that, yes, this gives you both sides of the coin: you keep using -your non-thread-based program (without worrying about locks and their -drawbacks like deadlocks, races, and friends), *and* your programs -benefit from all your cores. - -.. __: http://mail.python.org/pipermail/pypy-dev/2012-January/009044.html - -In more details, a low-level built-in module will provide the basics to -start transactions in parallel; but this module will be only used -internally in a tweaked version of, say, a Twisted reactor. Using this -reactor will be enough for your existing Twisted-based programs to -actually run on multiple cores. You --- as a developer of the -Twisted-based program --- have only to care about improving the -parallelizability of your program (e.g. by splitting time-consuming -transactions into several parts; the exact rules will be published in -detail once they are known). But the point is that your program is -always correct. - -See some `concrete example`__ of the API. - -.. __: https://bitbucket.org/pypy/pypy/raw/stm-gc/lib_pypy/transaction.py - - -Speed ------ - -We estimate the one-year performance target to be 2x-to-5x slower than -the performance of the regular PyPy in fully serial applications. (Of -course, the regular PyPy will not disappear; for the foreseeable future, -PyPy-TM will be an alternative executable.) - -The performance of PyPy-TM running suited applications should scale -linearly or close-to-linearly with the number of processor. This means -that in order to see the benefits, you just need a machine with at least -4 or 8 processors --- which is already common today and will be even -more so in one or two years. - - -.. _`soon available as hardware`: - -Hardware Transactional Memory ------------------------------ - -The performance of PyPy-TM running on Hardware Transactional Memory is -unknown so far, but should ideally be close to the performance of a -regular PyPy. - -In more details: This proposal is for work based entirely on *Software* -Transactional Memory. However, in the future the ideas and most of the -code should map directly to Hardware Transactional Memory (HTM). We -expect HTM to reduce a lot the cost of some of the issues that we face, -but not completely remove it. For example, `AMD's old proposal`_ was -that there would be two ways to emit memory-referencing -instructions: one that goes via the HTM mechanisms, and one (the regular -one) which doesn't. Choosing the best one on a -case-by-case basis in the JIT makes a difference in -performance (although of course not as great as with Software -Transactional Memory). - -`Intel's current proposal`_ on Haswell_ processors does not have this -distinction, which means transactions are likely to quickly overflow the -internal buffers. As a result, the first generation HTM-capable -processors may not be suited for the approach described here. But -this depends on details like the capacity of the hardware buffers that -are still secret at this point. - -(Note also that HTM does not solve some of the issues for implementing -Transactional Memory in CPython, notably the issue with reference -counting. We will have to wait for a real CPython experiment before -we can settle this question. Also, this would "just" remove the GIL -but not offer a multi-core version of non-thread-based programs.) - -.. _`AMD's old proposal`: http://developer.amd.com/tools/ASF/Pages/default.aspx -.. _`Intel's current proposal`: http://software.intel.com/en-us/avx/ -.. _Haswell: http://en.wikipedia.org/wiki/Haswell_%28microarchitecture%29 - - -Alternatives ------------- - -PyPy-TM will be slower than judicious usage of existing alternatives, -based on multiple processes that communicate with each other in one way -or another. The counter-argument is that TM is not only a cleaner -solution: there are cases in which it is not doable to organize (or -retrofit) an existing program into the particular format needed for the -alternatives. In particular, small quickly-written programs don't need -the additional baggage of cross-process communication, and large -programs can sometimes be almost impossible to turn into multi-process -versions. By contrast, we believe that TM can fit naturally into most -programs, because it only requires local changes to some dispatcher; the -rest of the program should work without changes. - - -More readings -------------- - -* `Original blog post`__ -* pypy-dev mails `[1]`__ `[2]`__ -* `All our blog posts about stm`__ - -.. __: http://morepypy.blogspot.com/2011/08/we-need-software-transactional-memory.html -.. __: http://mail.python.org/pipermail/pypy-dev/2011-August/008153.html -.. __: http://mail.python.org/pipermail/pypy-dev/2012-January/009034.html -.. __: http://morepypy.blogspot.com/search/label/stm - - -Work plan -========= - -This is an very rough estimate of the amount of work it would take to -complete the steps for an experienced developer who is already familiar -with the PyPy codebase. As this is a research proposal, we cannot -guarantee the time estimates here, but we do agree to report regularly to -the community, so our progress can be followed publicly. - -Paid work will be at $60/hour, but at least one developer who will work on -the project --- Armin Rigo --- has committed to 2 hours -of volunteer work per paid hour (so -the total amount of money that we ask is divided by three). A 5% general -donation will go to the `Software Freedom Conservancy`_ itself, the -non-profit organization of which the PyPy project is a member and which -manages all the issues related to donations, payments, and tax-exempt -status. - -.. _`Software Freedom Conservancy`: http://sfconservancy.org/ -.. _rstm: http://www.cs.rochester.edu/research/synchronization/rstm/ - - -* **STM Library**: - - This part covers adapting an existing STM library for PyPy. It is - already mostly done (based on rstm_), but additional tweaks may be - required. - -* **Basic tweaks of the translation process**: - - This part covers tweaks needed during the translation process in - order to generate an STM-aware version of the RPython programs, - including PyPy itself. It is partly done, but not finished. - Estimate: 1 month. - -* **Garbage collection**: - - We need a different garbage collector that is able to cope at least - with concurrent allocations. From there, improving the situation is - its own open-ended subproject: we can add for example various kinds of - parallel collection, synchronized or unsynchronized root tracing, - etc. Estimate for the basic part: 2 months. Estimate for the rest: - 4 extra months. - -* **User interface**: - - This is the problem of designing and implementing some interface or - interfaces for the Python programmer. We put it in its own category - because of its "end-user" importance. Estimate: 2 months. - -* **JIT integration**: - - The above would give us a (probably very slow) version of PyPy-TM. - This final part is to integrate it with the JIT compiler generator. - The main issue we foresee is integration with the new GC, detecting - with object flags or JIT optimizations which objects need - transactional memory status or not. We think that with enough such - optimizations we can seriously lower the overhead of PyPy-TM, maybe - down to 2x slower than a regular PyPy or better. Estimate: unknown; - at least 4 months. - -* **Longer term**: - - In the longer term, we might need to refine the TM processing done - above, for example to better support I/O (e.g. we can queue the writes - done to a log file) or to add some special fine-grained support - (e.g. two transactions that each do ``samelist.append()`` do not need - to conflict in the simple case). This part is not included - in the estimates. - - Note: by default, any I/O can be done, but turns the transaction - "inevitable". An inevitable transaction must not abort, so it must be - the next one to commit. This introduces delays at the end of the other - CPUs' transactions. - -Total: 5 months for the initial version; at least 8 additional months -for the fast version. We will go with a total estimate of 15 months, -corresponding to USD$151200. The amount sought by this fundraising -campaign, considering the 2 volunteer hours per paid hour is thus USD$50400. - - -Benefits of This Work to the Python Community and the General Public -==================================================================== - -Python has become one of the most popular dynamic programming languages in -the world. Web developers, educators, and scientific programmers alike -all value Python because Python code is often more readable and because -Python often increases programmer productivity. - -Traditionally, languages like Python ran more slowly than static, compiled -languages; Python developers chose to sacrifice execution speed for ease -of programming. The PyPy project created a substantially improved Python -language implementation, including a fast Just-in-time (JIT) compiler. -The increased execution speed that PyPy provides has attracted many users, -who now find their Python code runs up to four times faster under PyPy -than under the reference implementation written in C. - -However, in the presence of today's machines with multiple processors, -Python progress lags behind. The issue has been described in the -introduction: developers that really need to use multiple CPUs are -constrained to select and use one of the multi-process solutions that -are all in some way or another hacks requiring extra knowledge and -efforts to use. The focus of the work described in this proposal is to -offer an alternative in the core of the Python language --- an -alternative that can naturally integrate with the rest of the program. -This alternative will be implemented in PyPy. - -PyPy's developers make all PyPy software available to the public without -charge, under PyPy's Open Source copyright license, the permissive MIT -License. PyPy's license assures that PyPy is equally available to -everyone freely on terms that allow both non-commercial and commercial -activity. This license allows for academics, for-profit software -developers, volunteers and enthusiasts alike to collaborate together to -make a better Python implementation for everyone. - -PyPy-TM will be available under the same license. Being licensed freely -to the general public means that opportunities to use, improve and learn -about how Transactional Memory works itself will be generally available -to everyone. diff --git a/source/tmdonate2.txt b/source/tmdonate2.txt deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3RtZG9uYXRlMi50eHQ=..0000000000000000000000000000000000000000 --- a/source/tmdonate2.txt +++ /dev/null @@ -1,500 +0,0 @@ ---- -layout: page -title: 2nd Call for donations - Transactional Memory in PyPy ---- - -============================== -Transactional Memory, 2nd Call -============================== - -UPDATE (September 2016): - - **This call for donation is closed. Thank you everybody for - contributing!** - - *We have actually more money in the pot that we can use in the near - future. As it turns out, STM is a hard researchy topic. Remi Meier - is still actively working on this topic as part of his Ph.D. thesis. - Armin Rigo thinks more fundamental work is needed, which will - eventually lead us to a next generation attempt.* - -======================== - -.. raw:: html - - <br><br> - - -.. class:: download_menu - - * `Preamble`_ - - * `Introduction`_ - - * `In more detail`_ - - - `What is the Global Interpreter Lock?`_ - - `What is Transactional Memory?`_ - - `Hardware vs Software Transactional Memory`_ - - `Why do TM with PyPy instead of CPython?`_ - - `Alternatives`_ - - `Platforms other than the x86-64 Linux`_ - - `More readings`_ - - * `Work plan and funding details`_ - - - `Goal 1`_ - - `Goal 2`_ - - `Goal 3`_ - - `Funding`_ - - * `Benefits of This Work to the Python Community and the General Public`_ - - -Preamble -======== - -This is the second call for donations on the topic of Transactional -Memory (*TM*) in PyPy, a way to run CPU-hungry Python programs in -multithreaded mode. It is a follow-up on our `first call for -donations`_ from two years ago. At that time, we suggested a -single-threaded slow-down of somewhere between 2x and 5x. The aim that -seems now within reach is rather closer to 1.25x, i.e. running only 25% -slower than the regular PyPy. - -We achieved – or overachieved – most goals laid out in the first call by -a large margin, while at the same time raising only about half the -money. The result of this first step is `described in the documentation -of PyPy`__. It is a PyPy without the GIL. In the best (artificial) -examples, it runs only 30% slower than a regular PyPy with the JIT. - -The present proposal is about development of the second half: first, -fixing the various missing low-level optimizations (aiming for this -25%-30% figure, but for most cases rather than only special examples). -Then it will most -importantly focus on developing the Python-facing interface. This -includes both internal things (e.g. do dictionaries need to be more -TM-friendly in general?) as well as directly visible things (e.g. some -profiler-like interface to explore common conflicts in a program). -Finally, the third part is -exploring and tweaking some existing libraries to improve -their TM-friendliness (e.g. Twisted and Stackless). - -See also the `update on HTM`_ below. - -.. _`first call for donations`: http://pypy.org/tmdonate.html -.. __: https://pypy.readthedocs.org/en/latest/stm.html - - - -Introduction -============ - -In the presence of today's machines with multiple processors, Python -progress is lagging behind: on any CPU-constrained program, developers -have a difficult choice to make. They can use in-process solutions that -do not offer multi-CPU usage. In this respect, the natural choice -nowadays is to use Twisted or other event-based paradigms, or systems -that hide events in the control flow, like Stackless; or alternatively, -they can use the existing ``threading`` module, with its associated GIL -and the complexities of real multi-threaded programming (locks, -deadlocks, races, etc.), which make this solution less attractive. The -most attractive alternative for most developers is to rely on one of various multi-process -solutions that are outside the scope of the core Python language. All of them require a -major restructuring of the program and often need extreme care and extra -knowledge to use them. - -We propose an implemention of -Transactional Memory in PyPy. This is a technique that recently came to -the forefront of the multi-core scene. It promises to offer multi-core CPU -usage in a single process. -In particular, by modifying the core of the event systems -mentioned above, we will enable the use of multiple cores, without the -user needing to use explicitly the ``threading`` module. - -The first proposal was launched near the start of 2012 and has covered -much of the fundamental research, up to the point of getting a first -version of PyPy working in a very roughly reasonable state (after -collecting about USD$27'000, which is little more than half of the money -that was sought; hence the present second call for donations). - -We now propose fixing the remaining issues to obtaining a -really good GIL-free PyPy (described in `goal 1`_ below). We -will then focus on the various new features needed to actually use multiple -cores without explicitly using multithreading (`goal 2`_ below), up to -and including adapting some existing framework libraries, for -example Twisted, Tornado, Stackless, or gevent (`goal 3`_ below). - - - -In more detail -============== - -This is a call for financial help in implementing a version of PyPy able -to use multiple processors in a single process, called PyPy-TM; and -developing the APIs and libraries needed as well as enhancing commonly -available frameworks to use the new feature. The developers will be -Armin Rigo and Remi Meier and possibly others. - -We currently estimate the final performance goal to be a slow-down of -25% to 40% from the current non-TM PyPy; i.e. running a fully serial application would take between -1.25 and 1.40x the time it takes in a regular PyPy. This goal has -been reached already in some cases, but we need to make this result more -broadly applicable. We feel confident that we can reach this goal more -generally: the performance of PyPy-TM running any suitable -application should scale linearly or close-to-linearly with the number -of processors. This means that starting with two cores, such -applications should perform better than a non-TM PyPy. (All numbers -presented here are comparing different versions of PyPy which all have -the JIT enabled. A "suitable application" is one without many conflicts; -see `goal 2`_.) - -You will find below a sketch of the `work plan`_. We start with a Q&A. - - -What is the Global Interpreter Lock? ------------------------------------- - -The GIL, or Global Interpreter Lock, is a single lock in both CPython -and the regular PyPy. Every thread must acquire it in order to execute -Python bytecodes. This means that both with CPython and with the -regular PyPy, Python programs do not gain any benefit in term of -multicore performance even if they are using threads. - - -What is Transactional Memory? ------------------------------ - -`Transactional Memory`_ (*TM*) is a technique imported from -databases: every time we want to do a change to the processors' main -memory, we do it in a "transaction". Multiple transactions can be -executed in parallel by multiple cores. When a transaction is complete, -we try to commit it. This might either succeed, or (if another -transaction committed incompatible changes) fail. If it fails, which is -hopefully rare, we need to restart the transaction from scratch. - -Transactional Memory research has progressed a lot since two years ago, -notably with the introduction of Intel's Haswell_ processors, which -offer Hardware Transactional Memory (HTM). We discuss below why we -think HTM is, so far, still not suitable for our goals. - -.. _`Transactional Memory`: http://en.wikipedia.org/wiki/Transactional_memory -.. _Haswell: http://en.wikipedia.org/wiki/Haswell_%28microarchitecture%29 - - -.. _`update on HTM`: - -Hardware vs Software Transactional Memory ------------------------------------------ - -The idea of Transactional Memory was recently made popular by Intel's -Haswell_ processor (released in 2013). We could replace most of the -Software Transactional Memory (STM) library currently used inside PyPy -with a much smaller Hardware Transactional Memory (HTM) library based on -hardware features and running on Haswell-generation processors. This -has been attempted by Remi Meier recently. However, it seems that it -fails to scale as we would expect it to: the current generation of HTM -processors is limited to run small-scale transactions. Even the default -transaction size used in PyPy-STM is often too much for HTM; and -reducing this size increases overhead without completely solving the -problem. Based on this experience, it seems safe to say that right now -HTM-enabled processors lack the support that we need. - -Future processors might improve on various aspects. We are particularly -interested in `Virtualizing Transactional Memory`_, a 2005 paper that -describes the limits that we're running into and how to solve them more -generally. A CPU with support for the virtual memory described in this -paper would certainly be better for running PyPy-HTM. - -Another issue in HTM is sub-cache-line false conflicts (conflicts caused by two -independent objects that happens to live in the same cache line, which -is usually 64 bytes). This is in contrast with the current PyPy-STM, -which doesn't have false conflicts of this kind at all and might thus be -ultimately better for very-long-running transactions. We are not aware of -published research discussing issues of sub-cache-line false conflicts. - -Note that right now PyPy-STM has false conflicts within the same object, -e.g. within a list or a dictionary; but we can easily do something -about it (see `goal 2_`). Also, it might be possible in PyPy-HTM to -arrange objects in memory ahead of time so that such conflicts are very -rare; but we will never get a rate of exactly 0%, which might be -required for very-long-running transactions. - -.. _`Virtualizing Transactional Memory`: http://pages.cs.wisc.edu/~isca2005/papers/08A-02.PDF - - -Why do TM with PyPy instead of CPython? ---------------------------------------- - -While there have been early experiments on Hardware Transactional Memory -with CPython (`Riley and Zilles (2006)`__, `Tabba (2010)`__), there has -been none in the past few years. To the best of our knowledge, -the closest is an attempt using `Haswell on the -Ruby interpreter`__. None of these attempts tries to do the same using -Software Transactional Memory. We would nowadays consider it possible -to adapt our stmgc-c7 library for CPython, but it would be a lot of -work, starting from changing the reference-counting garbage collection scheme. PyPy is -better designed to be open to this kind of research. - -However, the best argument from an objective point of view is probably -that PyPy has already implemented a Just-in-Time compiler. It is thus -starting from a better position in terms of performance, particularly -for the long-running kind of programs that we target here. - -.. __: http://sabi.net/nriley/pubs/dls6-riley.pdf -.. __: http://www.cs.auckland.ac.nz/~fuad/parpycan.pdf -.. __: http://researcher.watson.ibm.com/researcher/files/jp-ODAIRA/PPoPP2014_RubyGILHTM.pdf - - -Alternatives ------------- - -PyPy-TM will be slower than judicious usage of existing alternatives, -based on multiple processes that communicate with each other in one way -or another. The counter-argument is that TM is not only a cleaner -solution: there are cases in which it is not really possible to organize (or -retrofit) an existing program into the particular format needed for the -alternatives. In particular, small quickly-written programs don't need -the additional baggage of cross-process communication; and large -programs can sometimes be almost impossible to turn into multi-process -versions. By contrast, we believe that TM can fit naturally into most -programs, because it only requires local changes to some dispatcher; the -rest of the program should work without changes. - - -Platforms other than the x86-64 Linux -------------------------------------- - -The current solution depends on having a -huge address space available. Porting to any 32-bit -architecture would quickly run into the limitation of a 2GB or 4GB of address space. -The way TM works right now would further divide this -limit by N+1, where N is the number of segments. It might be possible -to create partially different memory views for multiple threads that -each access the same range of addresses; but this would likely require -changes inside the OS. We didn't investigate so far. - -The current 64-bit version relies -heavily on Linux- and clang-only features. We believe it is a suitable -restriction: a lot of multi- and many-core servers commonly available -are nowadays x86-64 machines running Linux. Nevertheless, non-Linux -solutions appear to be possible as well. OS X (and likely the various -BSDs) seems to handle ``mmap()`` better than Linux does, and can remap -individual pages of an existing mapping to various pages without hitting -a limit of 65536 like Linux. Windows might also have a solution, although we -didn't measure yet; but first we would need a 64-bit Windows PyPy, which has -not seen much active support. - -We will likely explore the OS X path (as well as the Windows path if Win64 -support grows in PyPy), but this is not part of this current -donation proposal. - -It might be possible to adapt the work done on x86-64 to the 64-bit -ARMv8 as well. We have not investigated this so far. - - -More readings -------------- - -See the `STM page of our documentation`__ and `our blog posts about STM`__. - -.. __: http://pypy.readthedocs.org/en/latest/stm.html -.. __: http://morepypy.blogspot.com/search/label/stm - - - -.. _`work plan`: - -Work plan and funding details -============================= - -This is an very rough estimate of the amount of work it would take to -complete the steps for an experienced developer who is already familiar -with the PyPy codebase. As before, we cannot guarantee the time -estimates here, but we do agree to report regularly to the community, so -our progress can be followed publicly. We currently expect the duration -of the whole project to be up to two years starting from April 2014. - -Paid work will be at $60/hour, but at least one developer who will work -on the project – Armin Rigo – has committed to one hour of volunteer -work per paid hour; and another developer – Remi Meier – is a Ph.D. student -and gets paid from another source already. The total amount of money -that we ask below corresponds roughly to one half-time job. - -A 10% general donation will go to the `Software Freedom -Conservancy`_ itself, the non-profit organization of which the PyPy -project is a member and which manages all the issues related to -donations, payments, and tax-exempt status. -An extra fraction of the money collected will be entered into the -general PyPy pot, used for example to finance sprint travel costs to -students. This fraction is 10% maximum, unless more money than -requested is collected, in which case the whole excess will go to -the general PyPy pot. - -**Note** For donations higher than $1,000, we can arrange for an invoice -and a different payment method to avoid the high Paypal fees. Please -contact pypy at sfconservancy.org if you want to know details on how -to donate via other means. - -.. _`Software Freedom Conservancy`: http://sfconservancy.org/ - - -Goal 1 ------- - -The PyPy-TM that we have in the end of March 2014 is good enough in -some cases to run existing multithreaded code without a GIL, but not in -all of them. There are a number of caveats for the user and missing -optimizations. The goal #1 is to improve this case and address -the caveats. The current status is written down `in the docs`__ and -will evolve over time. - -.. __: http://pypy.readthedocs.org/en/latest/stm.html - -For future reference, at the end of March the main identified issues -are: - -* There are still a number of bugs. - -* The JIT warm-up time is abysmal. - -* The GC is missing a number of optimizations that are present in - a regular PyPy. - -* Destructors are not supported (``__del__()`` methods). - -* The STM bookkeeping logic could see more improvements. - -* Forking the process is slow. - -* We don't foresee particularly high conflict rates in regular - multithreaded programs, but this assertion needs to be checked - and possibly some heuristics improved. - -Fixing all these issues is required before we can confidently say that -PyPy-TM is an out-of-the-box replacement of a regular PyPy which gives -speed-ups over the regular PyPy independently of the Python program it -runs, as long as it is using at least two threads. - - -Goal 2 ------- - -This goal contains the various new features needed to use multiple cores -without explicitly using multithreading; in other words, the new APIs -and libraries accessible from Python programs that want to make use of -this benefit. - -This goal requires good support for very-long-running transactions, -started with the ``with atomic`` construct documented here__. This -approach hides the notion of threads from the end programmer, including -all the hard multithreading-related issues. This is not the first -alternative approach to explicit threads; for example, OpenMP_ is one. -However, it is one of the first ones which does not require the code to -be organized in a particular fashion. Instead, it works on any Python -program which has got latent, imperfect parallelism. Ideally, it only -requires that the end programmer identifies where this parallelism is -likely to be found, and communicates it to the system, using some -lightweight library on top of ``with atomic``. - -However, this introduces new issues. The main one is that by forcing -transactions to be longer, "conflicts" will become more common, up to -the point of partially or completely offsetting the benefit of using -PyPy-TM in the first place. - -So the programmer using PyPy-TM needs a way to get -feedback about what conflicts we get in these long-running transactions, -and where they are produced. A first step will be to implement getting -"tracebacks" that point to the places where the most time is lost. This -could be later integrated into some "profiler"-like variant where we can -navigate the conflicts, either in a live program or based on data logs. - -Some of these conflicts can be solved by improving PyPy-TM directly. -The system works on the granularity of objects and doesn't generate -false conflicts, but some conflicts may be regarded as "false" anyway: -these involve most importantly the built-in dictionary type, for which -we would like accesses and writes using independent keys to be truly -independent. Other built-in data structures have a similar issue, like -lists: ideally, writes to different indexes should not cause conflicts; -but more generally, we would need a mechanism, possibly under the -control of the application, to do things like append an item to a list -in a "delayed" manner, to avoid conflicts. - -.. __: http://pypy.readthedocs.org/en/latest/stm.html -.. _OpenMP: http://en.wikipedia.org/wiki/OpenMP - -Similarly, we might need a way to delay some I/O: doing it only at the -end of the transaction rather than immediately, in order to prevent the -whole transaction from turning inevitable. - -The goal 2 is thus the development of tools to inspect and fix the -causes of conflicts, as well as fixing the ones that are apparent inside -PyPy-TM directly. - - -Goal 3 ------- - -The third goal is to look at some existing event-based frameworks (for -example Twisted, Tornado, Stackless, gevent, ...) and attempt to make -them use threads and atomic sections internally. We would appreciate -help and feedback from people more involved in these frameworks, of -course. - -The idea is to apply the techniques described in the `goal 2`_ until we -get a version of framework X which can transparently parallelize the -dispatching and execution of multiple events. This might require some slight -reorganization of the core in order to split the I/O and the actual -logic into separate transactions. - - -Funding -------- - -We forecast that goal 1 and a good chunk of goal 2 should be reached in -around 6 months of work. The remaining parts of goal 2 as well as goal -3 are likely to be more open-ended jobs. We will go with a total -estimate of two years in order to get a final, well-tested PyPy-STM with -stable performance. The amount sought by this fundraising campaign is -USD$80'000, corresponding to one half-time job for 16 months (1200 hours -at $60/hour plus 10% overhead). - - -Benefits of This Work to the Python Community and the General Public -==================================================================== - -Python has become one of the most popular dynamic programming languages in -the world. Web developers, educators, and scientific programmers alike -all value Python because Python code is often more readable and because -Python often increases programmer productivity. - -Traditionally, languages like Python ran more slowly than static, compiled -languages; Python developers chose to sacrifice execution speed for ease -of programming. The PyPy project created a substantially improved Python -language implementation, including a fast Just-in-time (JIT) compiler. -The increased execution speed that PyPy provides has attracted many users, -who now find their Python code runs between 2 and 50 times faster under PyPy -than under the reference implementation written in C. - -However, in the presence of today's machines with multiple processors, -Python progress lags behind. The issue has been described in the -introduction: developers that really need to use multiple CPUs are -constrained to select and use one of the multi-process solutions that -are all in some way or another hacks requiring extra knowledge and -efforts to use. The focus of the work described in this proposal is to -offer an alternative in the core of the Python language — an -alternative that can naturally integrate with the rest of the program. -This alternative is implemented in PyPy. - -PyPy's developers make all PyPy software available to the public without -charge, under PyPy's Open Source copyright license, the permissive MIT -License. PyPy's license assures that PyPy is equally available to -everyone freely on terms that allow both non-commercial and commercial -activity. This license allows for academics, for-profit software -developers, volunteers and enthusiasts alike to collaborate together to -make a better Python implementation for everyone. - -PyPy-TM is and continues to be available under the same license. Being -licensed freely to the general public means that opportunities to use, -improve and learn about how Transactional Memory works itself will be -generally available to everyone. diff --git a/source/yatiblog.conf b/source/yatiblog.conf deleted file mode 100644 index 69d5538d4b81fb4d16bd023e04c14adc199ea75a_c291cmNlL3lhdGlibG9nLmNvbmY=..0000000000000000000000000000000000000000 --- a/source/yatiblog.conf +++ /dev/null @@ -1,12 +0,0 @@ -site_author: PyPy Team -site_description: PyPy -site_license: MIT -site_title: PyPy -site_url: http://pypy.org - -analytics_id: UA-7778406-3 -section_id: code -typekit: disabled - -index_pages: -- archive.html: archive.genshi