Commit 0961d51e authored by Samuele Pedroni's avatar Samuele Pedroni
Browse files

porting doc updates from 0.8 branch to trunk. copying back.

parent 00a5aac9f83b
.. _`demo/`: ../../demo
.. _`lib-python/`: ../../lib-python
.. _`lib-python/2.4.1/dis.py`: ../../lib-python/2.4.1/dis.py
.. _`annotation/`:
.. _`pypy/annotation`: ../../pypy/annotation
.. _`annotation/binaryop.py`: ../../pypy/annotation/binaryop.py
.. _`pypy/annotation/model.py`: ../../pypy/annotation/model.py
.. _`doc/`: ../../pypy/doc
.. _`doc/revreport/`: ../../pypy/doc/revreport
.. _`interpreter/`:
.. _`pypy/interpreter`: ../../pypy/interpreter
.. _`pypy/interpreter/argument.py`: ../../pypy/interpreter/argument.py
.. _`interpreter/astcompiler/`:
.. _`pypy/interpreter/astcompiler`: ../../pypy/interpreter/astcompiler
.. _`pypy/interpreter/function.py`: ../../pypy/interpreter/function.py
.. _`interpreter/gateway.py`:
.. _`pypy/interpreter/gateway.py`: ../../pypy/interpreter/gateway.py
.. _`pypy/interpreter/generator.py`: ../../pypy/interpreter/generator.py
.. _`pypy/interpreter/mixedmodule.py`: ../../pypy/interpreter/mixedmodule.py
.. _`pypy/interpreter/nestedscope.py`: ../../pypy/interpreter/nestedscope.py
.. _`pypy/interpreter/pyopcode.py`: ../../pypy/interpreter/pyopcode.py
.. _`pypy/interpreter/pyparser`: ../../pypy/interpreter/pyparser
.. _`pypy/interpreter/typedef.py`: ../../pypy/interpreter/typedef.py
.. _`lib/`:
.. _`pypy/lib/`: ../../pypy/lib
.. _`lib/test2/`:
.. _`pypy/lib/test2`: ../../pypy/lib/test2
.. _`module/`:
.. _`pypy/module`: ../../pypy/module
.. _`module/__builtin__/`: ../../pypy/module/__builtin__
.. _`pypy/module/__builtin__/__init__.py`: ../../pypy/module/__builtin__/__init__.py
.. _`module/_sre/`: ../../pypy/module/_sre
.. _`module/recparser/`: ../../pypy/module/recparser
.. _`module/sys/`: ../../pypy/module/sys
.. _`objspace/`:
.. _`pypy/objspace`: ../../pypy/objspace
.. _`objspace/flow/`: ../../pypy/objspace/flow
.. _`objspace/std/`:
.. _`pypy/objspace/std`: ../../pypy/objspace/std
.. _`objspace/thunk.py`: ../../pypy/objspace/thunk.py
.. _`objspace/trace.py`:
.. _`pypy/objspace/trace.py`: ../../pypy/objspace/trace.py
.. _`pypy/rpython`:
.. _`rpython/`: ../../pypy/rpython
.. _`pypy/rpython/extfunctable.py`: ../../pypy/rpython/extfunctable.py
.. _`pypy/rpython/lltypesystem/lltype.py`:
.. _`rpython/lltypesystem/lltype.py`: ../../pypy/rpython/lltypesystem/lltype.py
.. _`rpython/memory/`: ../../pypy/rpython/memory
.. _`pypy/rpython/memory/gc.py`: ../../pypy/rpython/memory/gc.py
.. _`pypy/rpython/memory/lladdress.py`: ../../pypy/rpython/memory/lladdress.py
.. _`pypy/rpython/memory/simulator.py`: ../../pypy/rpython/memory/simulator.py
.. _`pypy/rpython/memory/support.py`: ../../pypy/rpython/memory/support.py
.. _`pypy/rpython/module/`: ../../pypy/rpython/module
.. _`pypy/rpython/module/ll_os.py`: ../../pypy/rpython/module/ll_os.py
.. _`pypy/rpython/module/test`: ../../pypy/rpython/module/test
.. _`pypy/rpython/objectmodel.py`: ../../pypy/rpython/objectmodel.py
.. _`rpython/rint.py`: ../../pypy/rpython/rint.py
.. _`rpython/rlist.py`: ../../pypy/rpython/rlist.py
.. _`rpython/rmodel.py`: ../../pypy/rpython/rmodel.py
.. _`pypy/rpython/rtyper.py`: ../../pypy/rpython/rtyper.py
.. _`pypy/rpython/test/test_llinterp.py`: ../../pypy/rpython/test/test_llinterp.py
.. _`pypy/test_all.py`: ../../pypy/test_all.py
.. _`tool/`: ../../pypy/tool
.. _`tool/algo/`: ../../pypy/tool/algo
.. _`tool/pytest/`: ../../pypy/tool/pytest
.. _`tool/tb_server/`: ../../pypy/tool/tb_server
.. _`pypy/translator`:
.. _`translator/`: ../../pypy/translator
.. _`pypy/translator/annrpython.py`: ../../pypy/translator/annrpython.py
.. _`translator/c/`: ../../pypy/translator/c
.. _`pypy/translator/c/extfunc.py`: ../../pypy/translator/c/extfunc.py
.. _`pypy/translator/c/src/`: ../../pypy/translator/c/src
.. _`pypy/translator/c/src/ll_os.h`: ../../pypy/translator/c/src/ll_os.h
.. _`pypy/translator/c/test/test_extfunc.py`: ../../pypy/translator/c/test/test_extfunc.py
.. _`translator/goal/`: ../../pypy/translator/goal
.. _`pypy/translator/goal/targetnopstandalone.py`: ../../pypy/translator/goal/targetnopstandalone.py
.. _`translator/llvm/`: ../../pypy/translator/llvm
.. _`translator/tool/`: ../../pypy/translator/tool
\ No newline at end of file
==================================================
PyPy - Architecture Overview
==================================================
.. contents::
.. sectnum::
This document gives an overview of the goals and architecture of PyPy.
See also `getting started`_ for a practical introduction.
Mission statement
====================
PyPy is an implementation of the Python_ programming language written in
Python itself, flexible and easy to experiment with. Our long-term goals are
to target a large variety of platforms, small and large, by providing a
compiler toolsuite that can produce custom Python versions. Platform, memory
and threading models are to become aspects of the translation process - as
opposed to encoding low level details into the language implementation itself.
Eventually, dynamic optimization techniques - implemented as another
translation aspect - should become robust against language changes.
PyPy - an implementation of Python in Python
============================================
It has become a tradition in the development of computer languages to
implement each language in itself. This serves many purposes. By doing so,
you demonstrate the versatility of the language and its applicability for
large projects. Writing compilers and interpreters are among the most
complex endeavours in software development.
An important aspect of implementing Python in Python is the high level of
abstraction and compactness of the language. This allows an implementation
that is, in some respects, easier to understand and play with than the one
written in C (referred to throughout the PyPy documentation and source as
"CPython").
Another central idea in PyPy is building the implementation in the form
of a number of independent modules with clearly defined and well tested API's.
This eases reuse and allows experimenting with multiple implementations
of specific features.
Later in the project we will introduce optimizations, following the ideas
of Psyco_ that should make PyPy run Python programs faster than CPython,
and extensions, following the ideas of Stackless_ and others, that will
increase the expressive power available to python programmers.
.. _Python: http://www.python.org/doc/current/ref/ref.html
.. _Psyco: http://psyco.sourceforge.net
.. _Stackless: http://stackless.com
Higher level picture
====================
As you would expect from a project implemented using ideas from the world
of `Extreme Programming`_, the architecture of PyPy has evolved over time
and continues to evolve. Nevertheless, the high level architecture is now
clear. There are two independent basic subsystems: `the Standard
Interpreter`_ and `the Translation Process`_.
.. _`Extreme Programming`: http://www.extremeprogramming.com/
.. _`standard interpreter`:
The Standard Interpreter
------------------------
The *standard interpreter* is the subsystem implementing the Python language.
It is divided into two components:
- the `bytecode interpreter`_ which is responsible for interpreting
code objects and implementing bytecodes,
- the `standard object space`_ which implements creating, accessing and
modifying application level objects.
Note that the *standard interpreter* can run fine on top of CPython if one
is willing to pay the performance penalty for double-interpretation.
The Translation Process
-----------------------
The *translation process* aims at producing a different (low-level)
representation of our standard interpreter. The *translation process*
is done in four steps:
- producing a *flow graph* representation of the standard interpreter.
A combination of the `bytecode interpreter`_ and a *flow object space*
performs *abstract interpretation* to record the flow of objects
and execution throughout a python program into such a *flow graph*;
- the *annotator* which performs type inference on the flow graph;
- the *typer* which, based on the type annotations, turns the flow graph
into one using only low-level, C-like operations;
- the *code generator* which translates the resulting flow graph into
another language, currently C or LLVM_.
See below for the `translation process in more details`_.
.. _`bytecode interpreter`:
The Bytecode Interpreter
=========================
The *plain bytecode interpreter* handles python code objects.
The interpreter can build code objects from Python sources,
when needed, by invoking a bytecode compiler. Code objects
are a nicely preprocessed, structured representation of source code, and
their main content is *bytecode*. We use the same compact bytecode format
as CPython 2.4.
Our bytecode compiler is implemented as a chain of flexible passes
(tokenizer, lexer, parser, abstract syntax tree builder, bytecode
generator). The latter passes are based on the ``compiler`` package
from the standard library of CPython, with various improvements and
bug fixes. The bytecode compiler (living under
`interpreter/astcompiler/`_) is now integrated and is translated with
the rest of PyPy.
In addition to storing bytecode, code objects also know
how to create a *frame* object which has the responsibility to
*interpret* a code object's bytecode. Each bytecode is implemented by a
python function, which, in turn, delegates operations on
application-level objects to an object space. This interpretation and
delegation is the core of the bytecode interpreter.
This part is implemented in the `interpreter/`_ directory. People familiar
with the CPython implementation of the above concepts will easily recognize
them there. The major differences are the overall usage of the `Object Space`_
indirection to perform operations on objects, and the organization of the
built-in modules (described `here`_).
.. _`here`: coding-guide.html#modules
.. _`objectspace`:
.. _`Object Space`:
The Object Space
================
The object space creates all objects and knows how to perform operations
on the objects. You may think of an object space as being a library
offering a fixed API, a set of *operations*, with implementations that
correspond to the known semantics of Python objects. An example of an
operation is *add*: add's implementations are, for example, responsible
for performing numeric addition when add works on numbers, concatenation
when add works on built-in sequences.
All object-space operations take and return `application-level`_ objects.
There are only a few, very simple, object-space operations which allow the
bytecode interpreter to gain some knowledge about the value of an
application-level object.
The most important one is ``is_true()``, which returns a boolean
interpreter-level value. This is necessary to implement, for example,
if-statements (or rather, to be pedantic, to implement the
conditional-branching bytecodes into which if-statements get compiled).
We currently have four working object spaces which can be plugged into
the bytecode interpreter:
.. _`standard object space`:
- The *Standard Object Space* is a complete implementation
of the various built-in types and objects of Python. The Standard Object
Space, together with the bytecode interpreter, is the foundation of our Python
implementation. Internally, it is a set of `interpreter-level`_ classes
implementing the various `application-level`_ objects -- integers, strings,
lists, types, etc. To draw a comparison with CPython, the Standard Object
Space provides the equivalent of the C structures ``PyIntObject``,
``PyListObject``, etc.
- the *Trace Object Space* wraps e.g. the standard
object space in order to trace the execution of bytecodes,
frames and object space operations.
- the *Thunk Object Space* wraps another object space (e.g. the standard
one) and adds two capabilities: lazily computed objects (computed only when
an operation is performed on them), and "become", which completely and
globally replaces one object with another.
- the *Flow Object Space* transforms a Python program into a
flow-graph representation, by recording all operations that the bytecode
interpreter would like to perform when it is shown the given Python
program. This technique is explained `later in this document`_.
For a description of the object spaces, please see the
`objspace document`_. The sources of PyPy contain the various object spaces
in the directory `objspace/`_.
.. _`objspace document`: objspace.html
.. _`application-level`:
.. _`interpreter-level`:
Application-level and interpreter-level execution and objects
=============================================================
Since Python is used for implementing all of our code base, there is a
crucial distinction to be aware of: that between *interpreter-level* objects and
*application-level* objects. The latter are the ones that you deal with
when you write normal python programs. Interpreter-level code, however,
cannot invoke operations nor access attributes from application-level
objects. You will immediately recognize any interpreter level code in
PyPy, because half the variable and object names start with a ``w_``, which
indicates that they are `wrapped`_ application-level values.
Let's show the difference with a simple example. To sum the contents of
two variables ``a`` and ``b``, one would write the simple application-level
``a+b`` -- in contrast, the equivalent interpreter-level code is
``space.add(w_a, w_b)``, where ``space`` is an instance of an object space,
and ``w_a`` and ``w_b`` are typical names for the wrapped versions of the
two variables.
It helps to remember how CPython deals with the same issue: interpreter
level code, in CPython, is written in C and thus typical code for the
addition is ``PyNumber_Add(p_a, p_b)`` where ``p_a`` and ``p_b`` are C
variables of type ``PyObject*``. This is conceptually similar to how we write
our interpreter-level code in Python.
Moreover, in PyPy we have to make a sharp distinction between
interpreter- and application-level *exceptions*: application exceptions
are always contained inside an instance of ``OperationError``. This
makes it easy to distinguish failures (or bugs) in our interpreter-level code
from failures appearing in a python application level program that we are
interpreting.
.. _`app-preferable`:
Application level is often preferable
-------------------------------------
Application-level code is substantially higher-level, and therefore
correspondingly easier to write and debug. For example, suppose we want
to implement the ``update`` method of dict objects. Programming at
application level, we can write an obvious, simple implementation, one
that looks like an **executable definition** of ``update``, for
example::
def update(self, other):
for k in other.keys():
self[k] = other[k]
If we had to code only at interpreter level, we would have to code
something much lower-level and involved, say something like::
def update(space, w_self, w_other):
w_keys = space.call_method(w_other, 'keys')
w_iter = space.iter(w_keys)
while True:
try:
w_key = space.next(w_iter)
except OperationError, e:
if not e.match(space, space.w_StopIteration):
raise # re-raise other app-level exceptions
break
w_value = space.getitem(w_other, w_key)
space.setitem(w_self, w_key, w_value)
This interpreter-level implementation looks much more similar to the C
source code. It is still more readable than its C counterpart because
it doesn't contain memory management details and can use Python's native
exception mechanism.
In any case, it should be obvious that the application-level implementation
is definitely more readable, more elegant and more maintainable than the
interpreter-level one.
In fact, in almost all parts of PyPy, you find application level code in
the middle of interpreter-level code. Apart from some bootstrapping
problems (application level functions need a certain initialization
level of the object space before they can be executed), application
level code is usually preferable. We have an abstraction (called the
'Gateway') which allows the caller of a function to remain ignorant of
whether a particular function is implemented at application or
interpreter level.
.. _`wrapped`:
Wrapping
---------
The ``w_`` prefixes so lavishly used in the previous example indicate,
by PyPy coding convention, that we are dealing with *wrapped* (or *boxed*) objects,
that is, interpreter-level objects which the object space constructs
to implement corresponding application-level objects. Each object
space supplies ``wrap``, ``unwrap``, ``int_w``, ``interpclass_w``,
etc. operations that move between the two levels for objects of simple
built-in types; each object space also implements other Python types
with suitable interpreter-level classes with some amount of internal
structure.
For example, an application-level Python ``list``
is implemented by the `standard object space`_ as an
instance of ``W_ListObject``, which has an instance attribute
``wrappeditems`` (an interpreter-level list which contains the
application-level list's items as wrapped objects).
The rules are described in more details `in the coding guide`_.
.. _`in the coding guide`: coding-guide.html#wrapping-rules
.. _`translation process in more details`:
.. _`later in this document`:
.. _`initialization time`:
RPython, the Flow Object Space and translation
==============================================
One of PyPy's now achieved objectives is to enable translation of our
bytecode interpreter and standard object space into a lower-level language.
In order for our translation and type inference mechanisms to work
effectively, we need to restrict the dynamism of our interpreter-level
Python code at some point. However, in the start-up phase, we are
completely free to use all kinds of powerful python constructs, including
metaclasses and execution of dynamically constructed strings. However,
when the initialization phase (mainly, the function
``objspace.initialize()``) finishes, all code objects involved need to
adhere to a more static subset of Python:
Restricted Python, also known as `RPython`_.
The Flow Object Space then, with the help of our bytecode interpreter,
works through those initialized RPython code objects. The result of
this `abstract interpretation`_ is a flow graph: yet another
representation of a python program, but one which is suitable for
applying translation and type inference techniques. The nodes of the
graph are basic blocks consisting of Object Space operations, flowing
of values, and an exitswitch to one, two or multiple links which connect
each basic block to other basic blocks.
The flow graphs are fed as input into the Annotator. The Annotator,
given entry point types, infers the types of values that flow through
the program variables. This is the core of the definition of `RPython`_:
RPython code is restricted in such a way that the
Annotator is able to infer consistent types. How much
dynamism we allow in RPython depends on, and is restricted by, the Flow
Object Space and the Annotator implementation. The more we can improve
this translation phase, the more dynamism we can allow. In some cases,
however, it is more feasible and practical to just get rid
of some of the dynamism we use in our interpreter level code. It is
mainly because of this trade-off situation that the definition of
RPython has shifted over time. Although the Annotator is
pretty stable now and able to process the whole of PyPy, the RPython
definition will probably continue to shift marginally as we improve it.
The actual low-level code (and, in fact, also other high-level code) is
emitted by "visiting" the type-annotated flow graph. Currently we have
a C-producing backend, and an LLVM_-producing backend. The former also
accepts non-annotated or partially-annotated graphs, which allow us to
test it on a larger class of programs than what the Annotator can (or
ever will) fully process.
The newest piece of this puzzle is the
*Typer*, which inputs the high-level types inferred by the Annotator and
uses them to modify the flow graph in-place to replace its operations with
low-level ones, directly manipulating C-like values and data structures.
The complete translation process is described in more details in the
`translation document`_. There is a graph_ that gives an overview of the
whole process.
Status of the implementation (Oct 2005)
==========================================
With the pypy-0.8.0 release we have integrated our AST compiler with
the rest of PyPy. The compiler gets translated with the rest to a
static self-contained version of our `standard interpreter`_. Like
with 0.7.0 this version is `very compliant`_ to CPython 2.4.1 but you
cannot run many existing programs on it yet because we are
still missing a number of C-modules like socket or support for process
creation.
The self-contained PyPy version (single-threaded and using the
`Boehm-Demers-Weiser garbage collector`_) now runs around 10-20 times
slower than CPython, i.e. around 10 times faster than 0.7.0.
This is the result of optimizing, adding short
cuts for some common paths in our interpreter and adding relatively
straightforward optimization transforms to our tool chain, like inlining
paired with simple escape analysis to remove unnecessary heap allocations.
We still have some way to go, and we still expect most of our speed
will come from our Just-In-Time compiler work, which we have barely started
at the moment.
With the 0.8.0 release the "thunk" object space can also be translated
(see `getting started`_), obtaining a self-contained version of PyPy
with its features (and some speed degradation), show-casing at a small
scale how our whole tool-chain supports flexibility from the interpreter
written in Python to the resulting self-contained executable.
Our rather complete and Python 2.4-compliant interpreter consists
of about 30'000-50'000 lines of code (depending on the way you
count code borrowed and adapted from other sources), with
another 14'000 lines of unit tests. If we include the tools,
the parts related to code analysis and generation, and the
standard library modules ported from C, PyPy is now 138'000
lines of code and 32'000 lines of tests. Refer to
the `statistics web page`_ for more detailed information.
.. _`statistics web page`: http://codespeak.net/~hpk/pypy-stat/
.. _`very compliant`: http://codespeak.net/~hpk/pypy-testresult/
.. _`Boehm-Demers-Weiser garbage collector`: http://www.hpl.hp.com/personal/Hans_Boehm/gc/
.. _`RPython`: coding-guide.html#rpython
.. _`abstract interpretation`: theory.html#abstract-interpretation
.. _`translation document`: translation.html
.. _LLVM: http://llvm.cs.uiuc.edu/
.. _graph: image/translation.pdf
.. _`getting started`: getting-started.html
.. include:: _ref.txt
Contributors to PyPy
====================
Here is a list of developers who have committed to the PyPy source
code base, ordered by number of commits (which is certainly not a very
appropriate measure but it's something)::
Armin Rigo
Samuele Pedroni
Christian Tismer
Holger Krekel
Michael Hudson
Carl Friedrich Bolz
Eric van Riet Paap
Anders Chrigstrom
Richard Emslie
Anders Lehmann
Seo Sanghyeon
Alex Martelli
Ludovic Aubry
Adrien Di Mascio
Stefan Schwarzer
Tomek Meka
Patrick Maupin
Bob Ippolito
Jacob Hallen
Laura Creighton
Marius Gedminas
Niklaus Haldimann
Amaury Forgeot d Arc
Guido van Rossum
Stephan Diehl
Dinu Gherman
Guenter Jantzen
Rocco Moretti
Boris Feigin
Olivier Dormond
Valentino Volonghi
Brian Dorsey
Jonathan David Riehl
Andreas Friedge
Jens-Uwe Mager
Bert Freudenberg
Alan McIntyre
Anders Qvist
Lutz Paelike
Jacek Generowicz
Andrew Thompson
Ben Young
Alexander Schremmer
Michael Chermside
==========================
Frequently Asked Questions
==========================
.. contents::
General
============================================================
Do I have to rewrite my programs in RPython?
--------------------------------------------
No. PyPy always runs your code in its own interpreter, which is a full
and compliant Python 2.4 interpreter. RPython_ is only the language in
which parts of PyPy itself are written.
I am getting strange errors while playing with PyPy, what should I do?
----------------------------------------------------------------------
It seems that a lot of strange, unexplainable problems can be magically
solved by removing all the \*.pyc files from the PyPy source
tree. Another thing you can do is removing the pypy/_cache
completely. If the error is persistent and still annoys you after this
treatment please send us a bug report (or even better, a fix :-)
Using the PyPy translator
============================================================
How do I compile PyPy?
----------------------
See the `getting-started`_ guide. Note that at the moment this produces
an executable that contains a lot of things that are hard-coded for your
particular system (including paths and other stuff), so it's not
suitable for being installed or redistributed.
How do I compile my own programs?
---------------------------------
Start from the example of
`pypy/translator/goal/targetnopstandalone.py`_, which you compile by
typing::
python translate_pypy.py targetnopstandalone
You can have a look at intermediate C source code, which is (at the
moment) put in ``/tmp/usession-*/testing_1/testing_1.c``. Of course,
all the function and stuff indirectly used by your ``entry_point()``
function has to be RPython_.
Compiling to other languages
============================================================
Couldn't we simply take a Python syntax tree and turn it into Lisp?
-------------------------------------------------------------------
It's not necessarily nonsense, but it's not really The PyPy Way. It's
pretty hard, without some kind of type inference, to translate, say this
Python::
a + b
into anything significantly more efficient than this Common Lisp::
(py:add a b)
And making type inference possible is what RPython is all about.
You could make ``#'py:add`` a generic function and see if a given CLOS
implementation is fast enough to give a useful speed (but I think the
coercion rules would probably drive you insane first). -- mwh
Speed
============================================================
How fast is PyPy?
-----------------
As of August 2005, PyPy was successfully translated to C. The version
of PyPy that still runs on top of CPython is slower by a factor of 2000
compared to CPython. The first translated version was roughly 300 times
slower than CPython, and current versions (as of the 0.8.0 release) about
10-20 times slower than CPython. On the other hand, the really interesting
question is: Why is PyPy so slow?
.. _whysoslow:
Why is PyPy so slow?
--------------------
Our translation process does not try to optimize the produced code very
much. So far the project has been focused on getting a well tested very
compliant self-contained static Python implementat