README.rst 41.6 KB
Newer Older
1 2
How can I run Magic Lantern in QEMU?
====================================
alex@thinkpad's avatar
alex@thinkpad committed
3 4 5

This guide shows you how to emulate Magic Lantern (or plain Canon firmware) in QEMU.

6 7 8 9 10 11 12 13
.. class:: align-center

|pic1| |pic2|

.. |pic1| image:: doc/img/qemu-60D.png
    :width: 30 %
.. |pic2| image:: doc/img/qemu-M2-hello.jpg
    :width: 32.3 %
alex@thinkpad's avatar
alex@thinkpad committed
14

15 16
.. contents::

alex@thinkpad's avatar
alex@thinkpad committed
17 18 19 20 21
Current state
-------------

What works:

22
- Canon GUI with menu navigation — most DIGIC 3, 4 and 5 models
23 24
- Limited support for DIGIC 2, 6 and 7 models
- Limited support for some PowerShot models (including recent EOS M models)
alex@thinkpad's avatar
alex@thinkpad committed
25 26 27
- Limited support for secondary DryOS cores (such as Eeko or 5D4 AE processor)
- File I/O works on most models (both SD and CF); might be unreliable
- Bootloader emulation works on all supported models (from DIGIC 2 to DIGIC 7)
28
- Loading AUTOEXEC.BIN / DISKBOOT.BIN from the virtual SD/CF card (from DIGIC 2 to DIGIC 7)
alex@thinkpad's avatar
alex@thinkpad committed
29 30 31
- Unmodified autoexec.bin works on many single-core camera models
  (and, with major limitations, on dual-core models)
- ML modules and Lua scripts (within the limitations of the emulation)
alex@thinkpad's avatar
alex@thinkpad committed
32
- DryOS/VxWorks timer (heartbeat) and task switching (all supported models)
33
- UART emulation (DryOS shell aka Dry-shell or DrySh on DIGIC 4, 5, 6 and 7 models)
34
- Deterministic execution with the ``-icount`` option (except CF PIO)
35
- Cache hacks are emulated to some extent, but not exactly accurate
alex@thinkpad's avatar
alex@thinkpad committed
36 37 38 39 40 41 42 43 44
- EDMAC memcpy, including geometry parameters (matches the hardware closely, but not perfectly)
- Debugging with GDB:

  - assembly level for Canon code
  - source level for ML code (if compiled with -ggdb3 or similar)
  - ML stubs can be loaded as debugging symbols for Canon code (todo: also import from IDA or other systems)
  - predefined GDB scripts (log calls to DebugMsg, task_create, register_interrupt and a few others)
  - front-ends tested: cgdb (``splitgdb.sh``), DDD, gdbgui

45 46
  |

alex@thinkpad's avatar
alex@thinkpad committed
47 48 49 50 51 52 53
- Debug messages at QEMU console:

  - Use qprintf / qprint / qprintn / qdisas for printing to QEMU console
  - Compile Magic Lantern with ``CONFIG_QEMU=y``
  - By default (``CONFIG_QEMU=n``), the debug messages are not compiled
    (therefore not increasing the size of the executable that runs on the camera)

54 55
  |

alex@thinkpad's avatar
alex@thinkpad committed
56 57 58
- Log various actions of the guest operating system (Canon firmware, ML):

  - execution trace: ``-d exec,nochain -singlestep``
alex@thinkpad's avatar
alex@thinkpad committed
59 60
  - I/O trace: ``-d io``, ``-d io,int``
  - log hardware devices: ``-d mpu/sflash/sdcf/uart/int``
alex@thinkpad's avatar
alex@thinkpad committed
61 62
  - log all debug messages from Canon: ``-d debugmsg``
  - log all memory accesses: ``-d rom/ram/romr/ramw/etc``
63
  - log all function calls: ``-d calls``
alex@thinkpad's avatar
alex@thinkpad committed
64
  - log all DryOS/VxWorks task switches: ``-d tasks``
alex@thinkpad's avatar
alex@thinkpad committed
65 66 67 68
  - track all function calls to provide a stack trace: ``-d callstack``
  - export all called functions to IDC script: ``-d idc``
  - identify memory blocks copied from ROM to RAM: ``-d romcpy``
  - check for memory errors (a la valgrind): ``-d memchk``
alex@thinkpad's avatar
alex@thinkpad committed
69
  - list all available items: ``-d help`` 
alex@thinkpad's avatar
alex@thinkpad committed
70 71 72 73

What does not work (yet):

- LiveView (WIP, very hard);
74
- Still photo capture (WIP — the capture process itself works);
alex@thinkpad's avatar
alex@thinkpad committed
75 76 77 78
- Image review (WIP);
- Dual core emulation aka IPC (WIP);
- Touch screen (TODO);
- Flash reprogramming (TODO, low priority);
alex@thinkpad's avatar
alex@thinkpad committed
79
- Most hardware devices (audio chip, ADTG, FPGAs, JPCORE, image processing engine...);
80
- MPU properties are working to some extent, but have major limitations and bugs;
alex@thinkpad's avatar
alex@thinkpad committed
81 82 83 84
- Lens communication (done via MPU); initial lens info is replayed on startup on some models, but that's pretty much it;
- Cache behavior is not emulated (very hard; feel free to point us to code that can be reused);
- Native Windows build (QEMU can be compiled on Windows => contribution welcome).

85 86
Common issues and workarounds
`````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
87

88
- Camera was not shut down cleanly — Skipping module loading
alex@thinkpad's avatar
alex@thinkpad committed
89 90

  - closing QEMU window does not perform a clean shutdown
91
  - ``Machine -> Power Down`` — see `Shutdown and reboot`_ for more info
92
  - quicker: press ``C`` to `"open" the card door`__ => also clean shutdown.
alex@thinkpad's avatar
alex@thinkpad committed
93

94
__ `Opening the card door`_
alex@thinkpad's avatar
alex@thinkpad committed
95

96 97
.. _netcat-issue:

98 99 100 101 102 103 104 105 106 107 108 109
- Netcat issues when interacting with ``qemu.monitor``

  There are many versions of netcat around.
  Newer variants of openbsd netcat
  (`since 1.111, Mar 2013 <https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/nc/netcat.c.diff?r1=1.110&r2=1.111&f=h>`_)
  require ``-N``, but older versions do not have ``-N`` and will print an error if you attempt to use it. GNU netcat or other variants were not tested.
  
  TLDR: try openbsd netcat. If you get an error after copy/pasting some ``nc`` command from this guide, simply remove the ``-N``.
  
  Look in ``run_tests.sh`` for a slightly more portable workaround.
  

alex@thinkpad's avatar
alex@thinkpad committed
110 111 112
Installation
------------

113
It is recommended to install from the `qemu <https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/qemu/contrib/qemu>`_
alex@thinkpad's avatar
alex@thinkpad committed
114 115 116 117
branch, as it will contain the latest developments. The installation will be performed
outside the main magic-lantern directory, therefore you will be able to emulate any ML branch
without additional gymnastics (you will **not** have to merge ``qemu`` into your working branch or worry about it).

118 119 120 121
1. Clone the ``magic-lantern`` repository if you haven't already:

   .. code:: shell

122
     hg clone https://foss.heptapod.net/magic-lantern/magic-lantern
123 124
     cd magic-lantern

alex@thinkpad's avatar
alex@thinkpad committed
125 126
   |

127 128
2. Run ``install.sh`` from the ``qemu`` branch:

alex@thinkpad's avatar
alex@thinkpad committed
129
   .. code:: shell
130

alex@thinkpad's avatar
alex@thinkpad committed
131 132 133
     /path/to/magic-lantern$  hg update qemu -C
     /path/to/magic-lantern$  cd contrib/qemu
     /path/to/magic-lantern/contrib/qemu$  ./install.sh
134

135 136 137 138 139
   By default, QEMU will be installed in the ``qemu-eos`` directory,
   near the ``magic-lantern`` one, at the same level - e.g. ``/path/to/qemu-eos``.
   Please refer to `HACKING.rst <HACKING.rst#rst-header-how-is-this-code-organazized>`__
   for details on the directory structure and how to change the installation directory.

alex@thinkpad's avatar
alex@thinkpad committed
140 141
   |

142 143
3. Follow the instructions; you will have to supply your ROM files and compile QEMU:

alex@thinkpad's avatar
alex@thinkpad committed
144
   .. code:: shell
145

alex@thinkpad's avatar
alex@thinkpad committed
146
     # replace camera model (60D) with yours
147 148 149 150 151
     /path/to/qemu-eos$  cp /path/to/sdcard/ML/LOGS/ROM*.BIN 60D/
     /path/to/qemu-eos$  cd qemu-2.5.0
     /path/to/qemu-eos/qemu-2.5.0$  ../configure_eos.sh
     /path/to/qemu-eos/qemu-2.5.0$  make -j2
     /path/to/qemu-eos/qemu-2.5.0$  cd ..
152

153 154 155 156 157 158
   Some recent camera models also use a serial flash. To list them, run this command:

   .. code:: shell
   
     cat qemu-2.5.0/hw/eos/model_list.c | grep -B 10 serial_flash_size | grep -E "\.(serial_flash_size|name) "

alex@thinkpad's avatar
alex@thinkpad committed
159
   or just watch out for an error about missing ``SFDATA.BIN`` when trying the next step.
160 161

   If your camera requires a serial flash, compile the 
162
   `sf_dump module <https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/unified/modules/sf_dump>`_
163
   and run it on your camera to get this file.
164

165
   |
alex@thinkpad's avatar
alex@thinkpad committed
166

167
4. Test your installation.
alex@thinkpad's avatar
alex@thinkpad committed
168

alex@thinkpad's avatar
alex@thinkpad committed
169
   The pre-installed SD/CF images come with a small autoexec.bin
alex@thinkpad's avatar
alex@thinkpad committed
170 171 172
   (the `portable display test <http://www.magiclantern.fm/forum/index.php?topic=14732.0>`_)
   that works on all supported EOS models. It does not work on PowerShot models
   (including recent EOS M models based on PowerShot firmware).
173

alex@thinkpad's avatar
alex@thinkpad committed
174
   .. code:: shell
175
 
alex@thinkpad's avatar
alex@thinkpad committed
176
     # all EOS models should run this without any trickery
177
     /path/to/qemu-eos$  ./run_canon_fw.sh 60D,firmware="boot=1"
178

alex@thinkpad's avatar
alex@thinkpad committed
179 180
   |

181 182
5. Compile and run Magic Lantern

alex@thinkpad's avatar
alex@thinkpad committed
183
   .. code:: shell
184

alex@thinkpad's avatar
alex@thinkpad committed
185
     # from the QEMU directory
186
     /path/to/qemu-eos$  make -C ../magic-lantern 60D_install_qemu
187
     
alex@thinkpad's avatar
alex@thinkpad committed
188
     # some models will work only with this:
189
     /path/to/qemu-eos$  ./run_canon_fw.sh 60D,firmware="boot=1"
190

alex@thinkpad's avatar
alex@thinkpad committed
191
     # some models require running under GDB (they won't boot the GUI otherwise)
192
     /path/to/qemu-eos$  ./run_canon_fw.sh EOSM,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
193

alex@thinkpad's avatar
alex@thinkpad committed
194 195
   |

196
For reference, you may also look at `our test suite <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console>`_,
alex@thinkpad's avatar
alex@thinkpad committed
197 198 199
where QEMU is installed from scratch every time the tests are run.
These logs can be very useful for troubleshooting.

alex@thinkpad's avatar
alex@thinkpad committed
200 201
While we don't provide a native Windows build yet,
it is possible to install QEMU and ML development tools
202
`under the Windows 10 Linux Subsystem (WSL) <http://www.magiclantern.fm/forum/index.php?topic=20214.0>`_.
alex@thinkpad's avatar
alex@thinkpad committed
203 204 205 206

Running Canon firmware
----------------------

207
From the ``qemu-eos`` directory, use the ``run_canon_fw.sh`` script and make sure
alex@thinkpad's avatar
alex@thinkpad committed
208 209 210 211
the `boot flag <http://magiclantern.wikia.com/wiki/Bootflags>`_ is disabled:

.. code:: shell

212
  # from the qemu-eos directory
alex@thinkpad's avatar
alex@thinkpad committed
213
  ./run_canon_fw.sh 60D,firmware="boot=0"
alex@thinkpad's avatar
alex@thinkpad committed
214

215
Some models may need additional patches to run — these are stored under ``CAM/patches.gdb``.
216
To emulate these models, you will also need ``arm-none-eabi-gdb`` or ``gdb-multiarch``:
alex@thinkpad's avatar
alex@thinkpad committed
217 218 219

.. code:: shell

220
  ./run_canon_fw.sh EOSM,firmware="boot=0" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
221

222 223 224
You'll probably want to `see a few internals`__ as well. To get started, try these:

__ `Tracing guest events (execution, I/O, debug messages, RAM, function calls...)`_
alex@thinkpad's avatar
alex@thinkpad committed
225 226 227

.. code:: shell

alex@thinkpad's avatar
alex@thinkpad committed
228 229 230 231 232
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,tasks
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io,int
  ./run_canon_fw.sh 60D,firmware="boot=0" -d help
alex@thinkpad's avatar
alex@thinkpad committed
233

234 235 236 237 238 239 240
Or you may want to `run the firmware under GDB`__ and log additional functions:

__ `Debugging with GDB`_

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -s -S & arm-none-eabi-gdb -x 60D/debugmsg.gdb
241
  ./run_canon_fw.sh 60D,firmware="boot=0" -s -S & gdb-multiarch -x 60D/debugmsg.gdb
242

alex@thinkpad's avatar
alex@thinkpad committed
243 244 245 246
Running Magic Lantern
---------------------

As you already know, Magic Lantern runs from the SD or CF card. For emulation,
247
we provide two card images (``sd.img`` and ``cf.img``) which you can mount on your operating system
alex@thinkpad's avatar
alex@thinkpad committed
248 249 250 251 252 253
and copy files on them. If these images use a FAT filesystem (they do, by default), we prefer 
`mtools <https://www.gnu.org/software/mtools/>`_ for automated tasks
(such as copying files to/from the card images without mounting them).

To install Magic Lantern to the virtual card, you may:

254
- mount the card image (``sd.img`` or ``cf.img``) as ``/whatever/EOS_DIGITAL``,
alex@thinkpad's avatar
alex@thinkpad committed
255 256 257 258 259 260 261 262 263 264 265
  then run ``make install`` from your platform directory:

  .. code:: shell

    # from the magic-lantern directory
    cd platform/60D.111
    make clean; make
    # make sure your virtual card is mounted (this step is operating system specific)
    make install
    # make sure your virtual card is no longer mounted

alex@thinkpad's avatar
alex@thinkpad committed
266 267
  |

268 269 270
- use ``make install_qemu`` from your platform directory, or ``make CAM_install_qemu`` 
  or ``make CAM.FW_install_qemu`` from your ML root directory
  (requires mtools, but you do not have to mount your card images;
271
  works since `27f4105 <https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/5437eeb81f6759e42c7c1b194b2fdacb6916be28>`_):
alex@thinkpad's avatar
alex@thinkpad committed
272

273
  .. code:: shell
alex@thinkpad's avatar
alex@thinkpad committed
274 275 276 277 278 279

    # from the magic-lantern directory
    cd platform/60D.111
    make clean; make
    make install_qemu

280 281 282 283 284 285
  .. code:: shell

    # from the magic-lantern directory
    make 60D_clean
    make 60D_install_qemu

286
  They also work from the ``qemu-eos`` directory:
287 288 289

  .. code:: shell

290
    # from the qemu-eos directory
291 292
    make -C ../magic-lantern/platform/60D.111 clean
    make -C ../magic-lantern/platform/60D.111 install_qemu
293 294 295

  .. code:: shell

296
    # from the qemu-eos directory
297 298 299
    make -C ../magic-lantern 5D3.113_clean
    make -C ../magic-lantern 5D3.113_install_qemu

300
  Please note: ``make install_qemu`` is a recent addition and may not be available in all branches.
301
  In this case, you may either use the first method, or sync with the "unified" branch (``hg merge unified``),
302
  or manually import changeset `d5ad86f <https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/6313c5fa52c38104fa8f2bb10a1ab8ae6e54e9bb>`_.
303
  Unfortunately, these rules won't work from ``Makefile.user``.
alex@thinkpad's avatar
alex@thinkpad committed
304

alex@thinkpad's avatar
alex@thinkpad committed
305 306
The included card images are already bootable for EOS firmwares (but not for PowerShots).

307
After you have copied Magic Lantern to the card, you may run it from the ``qemu-eos`` directory
308
(near the ``magic-lantern`` one, at the same level):
alex@thinkpad's avatar
alex@thinkpad committed
309 310 311

.. code:: shell

312
  # from the qemu-eos directory
alex@thinkpad's avatar
alex@thinkpad committed
313 314 315
  ./run_canon_fw.sh 60D,firmware="boot=1"
  
  # or, if your camera requires patches.gdb:
316
  ./run_canon_fw.sh EOSM,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
317

alex@thinkpad's avatar
alex@thinkpad committed
318 319 320 321 322 323 324 325 326 327 328 329 330 331 332
Navigating menus
````````````````

Press ``F1`` to show the available keys (they are model-specific), or just read them
from the emulation log. Some keys will only send "press" events, while others
will send "press and release" events (this is also model-specific
and printed on the console).

Scrollwheels, if turned very quickly, may send a single event that includes
more than one click. This complicates scrollwheel handling code on ML side
and often leads to subtle issues. Currently, this behavior is not emulated.

Shutdown and reboot
```````````````````

333
By default, closing the QEMU window is equivalent to unplugging the power cord
alex@thinkpad's avatar
alex@thinkpad committed
334 335 336
(if your camera is on external power source). This appears to be the default
with other operating systems as well, so we did not change it.

337 338
Please note: closing QEMU window is **not** equivalent to taking the battery out —
see `Opening the battery door`_ for details.
alex@thinkpad's avatar
alex@thinkpad committed
339 340 341 342 343 344 345 346 347 348 349 350

Shutting down
'''''''''''''

To perform a clean shutdown (similar to powering down the camera from the main switch),
you may:

- select ``Machine -> Power Down`` from the menu (QEMU window)
- send the ``system_powerdown`` command to QEMU monitor:

.. code:: shell

351
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
352 353 354 355

Internally, Canon code refers to this kind of shutdown as ``SHUTDOWN_REQUEST``
(watch their debug messages with ``-d debugmsg``).

alex@thinkpad's avatar
alex@thinkpad committed
356 357 358 359 360 361 362
Caveat: the ``system_powerdown`` event does not guarantee a shutdown will actually occur
(either because the firmware handles this event in a different way, or if the execution
gets stuck for any reason). As a workaround, you could issue the ``quit`` command
after some timeout, if QEMU is still running:

.. code:: shell

363
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
364
  sleep 2
365 366
  if nc -N -U qemu.monitor < /dev/null > /dev/null 2>&1; then
    echo "quit" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
367 368
  fi

alex@thinkpad's avatar
alex@thinkpad committed
369 370 371 372 373 374 375 376 377 378 379 380 381
Opening the card door
'''''''''''''''''''''

Opening the SD/CF card door is a clean (non-rushed) way to shut down Canon firmware (``SHUTDOWN_REQUEST``).
To emulate this kind of shutdown, press ``C`` to simulate opening the card door,
then wait for a few seconds for QEMU to shutdown.

Opening the card door and closing it back quickly enough may result
in shutdown being canceled. Closing the card door is not implemented,
therefore this behavior is not emulated yet.

Opening the battery door
''''''''''''''''''''''''
alex@thinkpad's avatar
alex@thinkpad committed
382

alex@thinkpad's avatar
alex@thinkpad committed
383 384 385 386 387 388 389 390 391 392 393 394
Opening the battery door is interpreted by Canon firmware as an emergency shutdown (``PROP_ABORT``),
but it's still a clean(ish) shutdown. To emulate this kind of shutdown, press ``B``,
then close the QEMU window manually (or send the ``quit`` command to QEMU monitor).
Currently we do not know how to trigger or recognize a hardware shutdown event.

Rebooting
'''''''''

The camera can be rebooted from software by triggering ``PROP_REBOOT``.
Canon firmware handles it as a regular shutdown (``SHUTDOWN_REQUEST``),
followed by a regular boot. In QEMU, triggering ``PROP_REBOOT`` from software
will perform a clean shutdown (rebooting is not implemented).
alex@thinkpad's avatar
alex@thinkpad committed
395 396

Running ML Lua scripts
alex@thinkpad's avatar
alex@thinkpad committed
397
``````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
398

399
- Install ML on the virtual SD card:
alex@thinkpad's avatar
alex@thinkpad committed
400 401 402

  .. code:: shell

403
    # from the qemu-eos directory
404 405
    wget http://builds.magiclantern.fm/jenkins/job/lua_fix/431/artifact/platform/60D.111/magiclantern-lua_fix.2017Dec23.60D111.zip
    unzip magiclantern-lua_fix.2017Dec23.60D111.zip -d ml-tmp
alex@thinkpad's avatar
alex@thinkpad committed
406 407 408
    ./mtools_copy_ml.sh ml-tmp
    rm -rf ml-tmp/

alex@thinkpad's avatar
alex@thinkpad committed
409 410
  |

411
- Run QEMU:
alex@thinkpad's avatar
alex@thinkpad committed
412 413 414 415

  .. code:: shell

    ./run_canon_fw.sh 60D,firmware="boot=1"
alex@thinkpad's avatar
alex@thinkpad committed
416

alex@thinkpad's avatar
alex@thinkpad committed
417 418
  |

alex@thinkpad's avatar
alex@thinkpad committed
419
- enable the Lua module
alex@thinkpad's avatar
alex@thinkpad committed
420
- reboot the virtual camera cleanly (menu: Machine -> Power Down, then start it again)
alex@thinkpad's avatar
alex@thinkpad committed
421 422
- run the Hello World script

423 424
Note: api_test.lua does not run completely, but `can be patched <http://www.magiclantern.fm/forum/index.php?topic=2864.msg195347#msg195347>`_.

alex@thinkpad's avatar
alex@thinkpad committed
425 426 427
TODO: make api_test.lua run, fix bugs, polish the guide.

Using multiple firmware versions
alex@thinkpad's avatar
alex@thinkpad committed
428
````````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
429 430 431 432 433 434 435

In most cases, Magic Lantern only supports one firmware version, to keep things simple.
However, there may be good reasons to support two firmware versions
(for example, on the 5D Mark III, there are valid reasons to choose
both `1.1.3 <http://www.magiclantern.fm/forum/index.php?topic=14704.0>`_
and `1.2.3 <http://www.magiclantern.fm/forum/index.php?topic=11017.0>`_)
or you may want to test both versions when porting Magic Lantern
436
`to a newer Canon firmware <http://www.magiclantern.fm/forum/index.php?topic=19417.0>`_.
alex@thinkpad's avatar
alex@thinkpad committed
437 438 439 440 441 442 443

The invocation looks like this (notice the ``113``):

.. code:: shell

  ./run_canon_fw.sh 5D3,firmware="113;boot=0" -s -S & arm-none-eabi-gdb -x 5D3/debugmsg.gdb

444
And the directory layout should be like this::
alex@thinkpad's avatar
alex@thinkpad committed
445

446 447 448 449 450 451 452
  /path/to/qemu-eos/5D3/113/ROM0.BIN
  /path/to/qemu-eos/5D3/113/ROM1.BIN
  /path/to/qemu-eos/5D3/123/ROM0.BIN
  /path/to/qemu-eos/5D3/123/ROM1.BIN
  /path/to/qemu-eos/5D3/113/ROM0.BIN
  /path/to/qemu-eos/5D3/debugmsg.gdb  # common to both versions
  /path/to/qemu-eos/5D3/patches.gdb   # common to both versions
alex@thinkpad's avatar
alex@thinkpad committed
453

454
Compare this to a camera model where only one firmware version is supported::
alex@thinkpad's avatar
alex@thinkpad committed
455

456 457 458
  /path/to/qemu-eos/60D/ROM0.BIN
  /path/to/qemu-eos/60D/ROM1.BIN
  /path/to/qemu-eos/5D3/patches.gdb
459

460
You may also store ``debugmsg.gdb`` and ``patches.gdb`` under the firmware version subdirectory if you prefer, but other QEMU-related scripts won't pick them up.
alex@thinkpad's avatar
alex@thinkpad committed
461

462 463
Creating custom SD/CF card images
`````````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
464

465 466
You may want to run the emulation using the same card contents as on the physical camera
(for example, for troubleshooting). The recommended way is to create an image
alex@thinkpad's avatar
alex@thinkpad committed
467 468 469 470 471 472 473 474
of your SD (or CF) card (`tutorial for Windows and Mac <https://thepihut.com/blogs/raspberry-pi-tutorials/17789160-backing-up-and-restoring-your-raspberry-pis-sd-card>`_):

.. code:: shell

  dd if=/dev/your-sd-card of=sd.img bs=1M

and run QEMU from the resulting ``sd.img`` (or ``cf.img``).

475 476 477 478 479 480
Note: you need to copy the raw contents of the entire card, not just one partition. For example:

.. code:: shell

  dd if=/dev/mmcblk0 of=sd.img    # not mmcblk0p1

alex@thinkpad's avatar
alex@thinkpad committed
481 482 483 484 485 486 487 488 489 490 491 492 493 494
Tip: to save disk space, format the SD card first, copy the desired files,
then create the image as `qcow2 <https://rwmj.wordpress.com/2010/05/18/tip-compress-raw-disk-images-using-qcow2/>`_:

.. code:: shell

  qemu-img convert -f raw -O qcow2 /dev/mmcblk0 sd.qcow2

then update ``run_canon_fw.sh`` to use the new image::

  -drive if=sd,format=qcow2,file=sd.qcow2

That way, a 64GB card with Magic Lantern installed, after formatting in camera
with the option to keep ML on the card, will only take up about 15MB as a ``qcow2`` image.

495 496
Running from the physical SD/CF card
````````````````````````````````````
497
You may also start QEMU from the same card you use in the physical camera — 
498 499 500 501 502 503 504
this might be useful for troubleshooting issues with Magic Lantern, 
hopefully including certain card compatibility issues.

It is **strongly** recommended to avoid this procedure, unless absolutely needed.
Please consider creating an image of your SD (or CF) card instead, as described above.

If you still wish to run the emulation directly from a physical SD/CF card:
alex@thinkpad's avatar
alex@thinkpad committed
505

alex@thinkpad's avatar
alex@thinkpad committed
506
- See `QEMU docs (Using host drives) <https://qemu.weilnetz.de/doc/5.2/system/images.html#using-host-drives>`_
507
  and make sure you understand `these warnings <https://wiki.archlinux.org/index.php/QEMU#Using_any_real_partition_as_the_single_primary_partition_of_a_hard_disk_image>`_
alex@thinkpad's avatar
alex@thinkpad committed
508 509 510 511 512 513 514 515 516 517
- Replace ``file=sd.img`` (or ``file=cf.img``) with ``file=/dev/your-sd-card`` in ``run_canon_fw.sh``
- Configure the appropriate permissions and run the emulation as usual.

**Warning: Canon firmware WILL write to the SD/CF card during startup, 
and might even attempt to format it if the filesystem is not recognized,
or because of emulation bugs. 
Be very careful not to give it write access to your physical hard-disk!!!**

Note: the ROM files will not be loaded from the SD/CF card.

518 519 520 521 522 523 524 525 526 527 528 529 530
Image capture emulation
```````````````````````

WIP, still pretty rough.

To capture a full-res image (aka FRSP) using a CR2 as reference data for the virtual sensor:

.. code:: shell

    make -C ../magic-lantern/minimal/qemu-frsp MODEL=5D3 CONFIG_QEMU=y clean install_qemu
    env QEMU_EOS_VRAM_PH_QR_RAW='/path/to/IMG_1234.CR2' ./run_canon_fw.sh 5D3,firmware="boot=1"


alex@thinkpad's avatar
alex@thinkpad committed
531 532 533 534 535 536 537
Automation
----------

QEMU monitor
````````````

By default, the QEMU monitor console is available by default as a UNIX socket.
alex@thinkpad's avatar
alex@thinkpad committed
538 539 540 541 542 543
That means, during emulation you can interact with it using netcat:

- interactive console

  .. code:: shell

544
    nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
545

alex@thinkpad's avatar
alex@thinkpad committed
546 547
  |

alex@thinkpad's avatar
alex@thinkpad committed
548
- one-liner commands, usable from scripts:
alex@thinkpad's avatar
alex@thinkpad committed
549 550 551

  .. code:: shell

552
    echo "log io" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
553

alex@thinkpad's avatar
alex@thinkpad committed
554 555
  |

alex@thinkpad's avatar
alex@thinkpad committed
556
- check whether QEMU monitor is active:
alex@thinkpad's avatar
alex@thinkpad committed
557 558 559

  .. code:: shell

560
    if nc -N -U qemu.monitor < /dev/null > /dev/null 2>&1; then
alex@thinkpad's avatar
alex@thinkpad committed
561 562
      ...
    fi
alex@thinkpad's avatar
alex@thinkpad committed
563

alex@thinkpad's avatar
alex@thinkpad committed
564 565
  |

alex@thinkpad's avatar
alex@thinkpad committed
566 567
You can redirect the monitor console to stdio with... ``-monitor stdio``.

568 569 570
If you have trouble running the above ``nc`` commands, don't forget to check this `common netcat issue`__.

__ `netcat-issue`_
571

alex@thinkpad's avatar
alex@thinkpad committed
572 573 574
Taking screenshots
``````````````````

575 576 577 578 579 580
The easiest way is to use the ``screendump`` command from QEMU monitor:

.. code:: shell

  echo "screendump snap.ppm" | nc -N -U qemu.monitor

alex@thinkpad's avatar
alex@thinkpad committed
581
In the following example, we'll redirect the monitor to stdio
582
and take a screenshot after 10 seconds:
alex@thinkpad's avatar
alex@thinkpad committed
583 584 585 586 587 588

.. code:: shell

  ( 
    sleep 10
    echo screendump snap.ppm
alex@thinkpad's avatar
alex@thinkpad committed
589
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
590 591 592 593 594 595 596 597 598 599 600 601 602
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=0' \
        -monitor stdio
  )

Another option is to use the VNC interface:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' \
        -vnc :1234 &
  sleep 10
  vncdotool -s :1234 capture snap.png
603
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
604 605 606 607

Sending keystrokes
``````````````````

608
From QEMU monitor::
alex@thinkpad's avatar
alex@thinkpad committed
609 610 611 612 613 614 615 616

  (qemu) help
  sendkey keys [hold_ms] -- send keys to the VM (e.g. 'sendkey ctrl-alt-f1', default hold time=100 ms)

.. code:: shell

  ( 
    sleep 10
alex@thinkpad's avatar
alex@thinkpad committed
617
    echo sendkey m
alex@thinkpad's avatar
alex@thinkpad committed
618 619
    sleep 1
    echo screendump menu.ppm
alex@thinkpad's avatar
alex@thinkpad committed
620
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
621 622 623 624 625
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=0' \
        -monitor stdio
  )

alex@thinkpad's avatar
alex@thinkpad committed
626 627 628 629 630 631 632
Or, if QEMU runs as a background process:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' &
  
  sleep 10
633
  echo "sendkey m" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
634
  sleep 1
635 636
  echo "screendump menu.ppm" | nc -N -U qemu.monitor
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
637

alex@thinkpad's avatar
alex@thinkpad committed
638 639 640 641 642
From VNC:

.. code:: shell

  vncdotool -h | grep key
643 644 645
  key KEY               send KEY to server, alphanumeric or keysym: ctrl-c, del
  keyup KEY             send KEY released
  keydown KEY           send KEY pressed
alex@thinkpad's avatar
alex@thinkpad committed
646 647 648 649 650 651 652 653

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' \
        -vnc :1234 &
  sleep 10
  vncdotool -s :1234 key m
  sleep 1
654
  vncdotool -s :1234 capture menu.png
655
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676

Running multiple ML builds from a single command
````````````````````````````````````````````````

You may run ML builds from multiple models, unattended,
with the ``run_ml_all_cams.sh`` script:

.. code:: shell

  env ML_PLATFORMS="500D.111/ 60D.111/" \
      TIMEOUT=10 \
      SCREENSHOT=1 \
      ./run_ml_all_cams.sh

Internally, this is how the emulator is invoked:

.. code:: shell

  ( 
    sleep 10
    echo screendump 60D.111.ppm
alex@thinkpad's avatar
alex@thinkpad committed
677
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
678 679
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=1' \
680
        -display none -monitor stdio
alex@thinkpad's avatar
alex@thinkpad committed
681 682 683 684 685
  ) &> 60D.111.log


This script is very customizable (see the source code for available options).

686
**More examples:**
alex@thinkpad's avatar
alex@thinkpad committed
687

688 689 690
- `EOSM2 hello world <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/18/console>`_
- running ML from the dm-spy-experiments branch in the emulator (`QEMU-dm-spy <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-dm-spy/65/consoleFull>`_)
- running the FA_CaptureTestImage test based on the minimal ML target (`QEMU-FA_CaptureTestImage <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-FA_CaptureTestImage>`_)
alex@thinkpad's avatar
alex@thinkpad committed
691

692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716
Parallel execution
``````````````````

On modern machines, you will get significant speed gains by running multiple instances of QEMU in parallel.
This is tricky and not automated. You need to be careful with the following global resources:

- SD and CF images (``sd.img`` and ``cf.img``):

  If all your parallel instances require the same initial SD/CF card contents,
  and you do not need to inspect the changes to SD/CF after the experiment,
  you may use these files as read-only shared resources with the help of QEMU's
  `temporary snapshot <https://wiki.qemu.org/Documentation/CreateSnapshot#Temporary_snapshots>`_ feature
  (simply add ``-snapshot`` to your command line). This will discard any changes to ``sd.img`` and ``cf.img``.
  `Implementation details <https://lists.gnu.org/archive/html/qemu-devel/2008-09/msg00712.html>`_.

  Otherwise, you could allocate different SD/CF images for each instance, but it's up to you to modify the scripts to handle that.

- QEMU monitor socket (``qemu.monitor``):

  Set the ``QEMU_JOB_ID`` environment variable; it will be used as suffix for ``qemu.monitor``.
  
  Example: ``QEMU_JOB_ID=1 ./run_canon_fw.sh 5D3`` will use ``qemu.monitor1`` for monitor commands.

- GDB port (with ``-s -S``, this port is 1234):

717
  Set QEMU_JOB_ID to a small positive integer, for example:
718 719 720

  .. code:: shell

721 722 723 724 725 726 727
    export QEMU_JOB_ID=1
  
  Then you'll be able to do this:

  .. code:: shell

    QEMU_MONITOR=qemu.monitor$QEMU_JOB_ID
alex@thinkpad's avatar
alex@thinkpad committed
728
    GDB_PORT=$((1234+QEMU_JOB_ID))
729
    ./run_canon_fw.sh EOSM2,firmware="boot=0" -S -gdb tcp::$GDB_PORT &
730 731 732 733
    arm-none-eabi-gdb -ex "set \$TCP_PORT=$GDB_PORT" -x EOSM2/patches.gdb -ex quit &
    
    # interact with monitor commands
    sleep 5
734
    echo "sendkey m" | nc -N -U $QEMU_MONITOR
735 736 737
    sleep 1

    # quit when finished
738 739 740
    echo "quit" | nc -N -U $QEMU_MONITOR

  |
741 742 743 744 745 746 747

- VNC display

  Same as above:

  .. code:: shell

748
    QEMU_MONITOR=qemu.monitor$QEMU_JOB_ID
749
    VNC_DISP=":$((12345+QEMU_JOB_ID))"
750
    ./run_canon_fw.sh 5D3,firmware="boot=0" -vnc $VNC_DISP &
751 752 753
    
    # interact with vncdotool
    sleep 5
754
    vncdotool -s $VNC_DISP key m
755 756 757
    sleep 1
    
    # quit when finished
758 759 760
    echo "quit" | nc -N -U $QEMU_MONITOR

  |
761 762 763

- any temporary files you may want to use

764 765
  Try to achieve the same thing without a temporary file (pipes, process substitution, command-line options).
  If that doesn't work, consider using ``mktemp`` rather than hardcoding a filename.
766 767 768 769 770

- any other global resources (you'll have to figure them out on your own).

TODO: can this be automated somehow with containers?

alex@thinkpad's avatar
alex@thinkpad committed
771 772 773
Debugging
---------

alex@thinkpad's avatar
alex@thinkpad committed
774 775 776 777 778 779 780 781 782 783 784 785 786 787
From the QEMU monitor
`````````````````````

.. code:: shell

  echo "help" | ./run_canon_fw.sh 60D -monitor stdio |& grep dump
  ...
  pmemsave addr size file -- save to disk physical memory dump starting at 'addr' of size 'size'
  xp /fmt addr -- physical memory dump starting at 'addr'

Using ``qprintf`` and friends
`````````````````````````````

The QEMU debugging API
788
(`qemu-util.h <https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/qemu/src/qemu-util.h>`_, included by default by ``dryos.h``)
alex@thinkpad's avatar
alex@thinkpad committed
789 790 791 792 793 794 795 796 797 798 799 800
exposes the following functions to be used in Magic Lantern code:

:qprintf: heavyweight, similar to printf; requires vsnprintf from Canon code
:qprint: lightweight, inline, similar to puts, without newline
:qprintn: lightweight, prints a 32-bit integer
:qdisas: lightweight, tells QEMU to disassemble one ARM or Thumb instruction at the given address

These functions will print to QEMU console whenever ML (or a subset of it)
is compiled with ``CONFIG_QEMU=y``. They won't get compiled in regular builds
(``CONFIG_QEMU=n`` is the default), therefore they won't increase the executable size.
For this reason, feel free to use them *anywhere*.

801 802
You may use the debugging API for either the entire ML, or just for a subset of it —
e.g. the source file(s) you are currently editing, or only some modules.
alex@thinkpad's avatar
alex@thinkpad committed
803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819
The lightweight functions can also be used in very early boot code,
where you can't call vsnprintf or you may not even have a stack properly set up.

CONFIG_QEMU
```````````

:``CONFIG_QEMU=n``: (default):

- regular build
- the executable works in QEMU (within the limitations of the emulation)
- no guest debugging code (no additional debugging facilities)

:``CONFIG_QEMU=y``: (optional, on the command line or in ``Makefile.user``):

- debug build for QEMU only
- does **not** run on the camera (!)
- enables ``qprintf`` and friends to print to the QEMU console
820
- enables unlimited number of ROM patches — useful for 
821
  `dm-spy-experiments <http://www.magiclantern.fm/forum/index.php?topic=2388.0>`_
alex@thinkpad's avatar
alex@thinkpad committed
822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840
  (in QEMU you can simply write to ROM as if it were RAM)
- may enable workarounds for models or features that are not emulated very well

Example:

.. code:: shell

    cd platform/550D.109
    make clean; make                            # regular build
    make clean; make CONFIG_QEMU=y              # debug build for QEMU
    make clean; make install_qemu               # build and install a regular build to the QEMU SD/CF image
    make clean; make install_qemu CONFIG_QEMU=y # build and install a QEMU build to the QEMU SD/CF image

It works for modules as well:

.. code:: shell

    cd modules/lua
    # add some qprintf call in lua_init for testing
841 842 843 844 845
    make clean; make                            # regular build
    make clean; make CONFIG_QEMU=y              # debug build for QEMU
    make clean; make install_qemu               # build and install a regular build to the QEMU SD/CF image
    make clean; make install_qemu CONFIG_QEMU=y # build and install a QEMU build to the QEMU SD/CF image
 
alex@thinkpad's avatar
alex@thinkpad committed
846 847 848 849 850 851 852 853 854
Tracing guest events (execution, I/O, debug messages, RAM, function calls...)
`````````````````````````````````````````````````````````````````````````````

Execution trace:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d exec,nochain -singlestep

855
I/O trace (precise):
alex@thinkpad's avatar
alex@thinkpad committed
856 857 858 859 860

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d io

861
I/O trace (quick):
alex@thinkpad's avatar
alex@thinkpad committed
862 863 864

.. code:: shell

865
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io_quick
alex@thinkpad's avatar
alex@thinkpad committed
866 867 868 869 870

I/O trace with interrupts (precise):

.. code:: shell

871
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io,int
alex@thinkpad's avatar
alex@thinkpad committed
872

873
I/O trace with Canon debug messages (precise):
alex@thinkpad's avatar
alex@thinkpad committed
874 875 876 877 878 879 880 881 882 883 884

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io

Canon debug messages and task switches:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,tasks

885
Memory access trace (ROM reads, RAM writes) — very verbose:
alex@thinkpad's avatar
alex@thinkpad committed
886 887 888 889 890

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d romr,ramw

891
Call/return trace, redirected to a log file:
alex@thinkpad's avatar
alex@thinkpad committed
892 893 894

.. code:: shell

895
  ./run_canon_fw.sh 60D,firmware="boot=0" -d calls &> calls.log
alex@thinkpad's avatar
alex@thinkpad committed
896 897 898 899 900 901 902 903

Tip: set your editor to highlight the log file as if it were Python code.
You'll get collapse markers for free :)

Also with debug messages, I/O events and interrupts, redirected to file

.. code:: shell

904
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,calls,io,int &> full.log
alex@thinkpad's avatar
alex@thinkpad committed
905 906 907 908 909

Filter the logs with grep:

.. code:: shell

910
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io |& grep --text -C 5 "\[Display\]"
alex@thinkpad's avatar
alex@thinkpad committed
911 912 913 914 915

Only log autoexec.bin activity (skip logging the bootloader):

.. code:: shell

916
  ./run_canon_fw.sh 60D,firmware="boot=1" -d exec,io,int,autoexec
alex@thinkpad's avatar
alex@thinkpad committed
917 918 919 920 921 922


Execution trace incomplete? PC values from MMIO logs not correct?
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

That's because QEMU compiles multiple guest instructions into a single TranslationBlock,
alex@thinkpad's avatar
alex@thinkpad committed
923
for faster execution. In this mode, ``-d exec`` will print guest instructions as they are compiled
alex@thinkpad's avatar
alex@thinkpad committed
924 925 926
(for example, if you have a tight loop, only the first pass will be printed).

To log every single guest instruction, as executed, and get exact PC values
alex@thinkpad's avatar
alex@thinkpad committed
927 928
in execution traces and other logs, you need to use ``-d nochain -singlestep`` 
(for example: ``-d exec,nochain -singlestep``) —
929
`source <http://qemu-discuss.nongnu.narkive.com/f8A4tqdT/singlestepping-target-assembly-instructions>`_.
alex@thinkpad's avatar
alex@thinkpad committed
930

931
Please note: ``-d io`` implies ``-d nochain -singlestep`` by default, to avoid this issue. Should you want to disable this,
alex@thinkpad's avatar
alex@thinkpad committed
932
to get faster emulation at the expense of incorrect PC values, use ``-d io_quick``.
933

alex@thinkpad's avatar
alex@thinkpad committed
934
Additionally, ``-d nochain`` implies ``-singlestep``, unlike in vanilla QEMU.
935

alex@thinkpad's avatar
alex@thinkpad committed
936 937 938
Debugging with GDB
``````````````````

alex@thinkpad's avatar
alex@thinkpad committed
939 940 941
.. code:: shell

    ./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
alex@thinkpad's avatar
alex@thinkpad committed
942

943 944 945
Logging hooks
'''''''''''''

946 947 948 949 950 951 952
Probably the most powerful feature of GDB is its scripting engine —
in many cases it's a lot faster than manually stepping over assembly code.
We may use it for tracing various function calls in the firmware, to understand what they do,
how to call them and so on. At any code address from the disassembly, we may set a breakpoint
and print some more info (such as function name, arguments, register values,
call location, DryOS task name and so on).

953
Predefined logging hook example (this goes into ``CAM/debugmsg.gdb``)::
954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969

  b *0x8580
  take_semaphore_log

Custom logging hook (with colors)::

  b *0xE0008DA6
  commands
    silent
    print_current_location
    KRED
    printf "dryos_panic(%x, %x)\n", $r0, $r1
    KRESET
    c
  end

970
Look in `debug-logging.gdb <https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/qemu/contrib/qemu/scripts/debug-logging.gdb>`_
alex@thinkpad's avatar
alex@thinkpad committed
971
for common firmware functions you may want to log, and in ``*/debugmsg.gdb`` for usage examples.
972

alex@thinkpad's avatar
alex@thinkpad committed
973
You may also use `dprintf <https://sourceware.org/gdb/onlinedocs/gdb/Dynamic-printf.html>`_ if you prefer::
974 975 976

  dprintf *0x8b10, "[ %s:%08X ] task_create(%s, prio=%x, stack=%x, entry=%x, arg=%x)\n", CURRENT_TASK_NAME, $lr-4, $r0, $r1, $r2, $r3, *(int*)$sp

977 978
but the output won't be as pretty.

979 980 981 982
To log the return value of any function, use ``log_result`` in the definition of a logging hook.

For quick experiments, you may find ``generic_log`` and ``generic_log_with_result`` handy.

983 984 985 986 987 988 989 990 991 992 993 994 995
To get help on all of these user-defined commands at the GDB prompt:

  (gdb) help user-defined
  ...
  printf_log -- Log calls to plain printf
  register_cmd_log -- Log named functions registered to the DryOS shell
  xxd -- Memory dump formatted with xxd
  ...

  (gdb) help xxd
  Memory dump formatted with xxd. Temporary file: dump.tmp.
  Syntax: xxd startaddr [size]

996 997 998 999
Debugging symbols
'''''''''''''''''

There are no debugging symbols in Canon firmware, but you can import
1000
some of them from Magic Lantern (see comments in ``debugmsg.gdb``). Typically, you want to use one of these
1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013
`elf <https://jvns.ca/blog/2014/09/06/how-to-read-an-executable/>`_ files
from the platform directory:

- ``magiclantern`` if debugging regular Magic Lantern code (without modules)
- ``autoexec`` if debugging ML bootloader code — the `reboot shim <http://magiclantern.wikia.com/wiki/Autoboot>`_ (``reboot.c`` and related)
- ``stubs.o`` if debugging Canon firmware (add your symbols to ``stubs.S`` and recompile.)

For some reason, ML stubs are not recognized as functions in GDB;
to set breakpoints on function names, you need this trick::

  b *&task_create
  task_create_log

1014 1015 1016
Printing call stack from GDB
''''''''''''''''''''''''''''

1017 1018
The ``callstack`` option from QEMU (``eos/dbi/logging.c``) can be very useful to find where a function was called from.
This works even when gdb's ``backtrace`` command cannot figure it out from the stack contents, does not require any debugging symbols,
1019
but you need to run the emulation with instrumentation enabled: ``-d callstack``.
1020 1021 1022

Then, in GDB, use ``print_current_location_with_callstack`` to see the call stack for the current DryOS task.

1023 1024 1025
Tip: the callstack feature is enabled by default in ``assert_log``.

**Example for 80D:**
1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061

The following goes into ``80D/debugmsg.gdb`` (modify the existing entry):

.. code::

  b *0xFE237C9E
  commands
    silent
    print_current_location_with_callstack
    printf "Memory region: start=%08X end=%08X flags=%08X\n", $r0, $r1, $r2
    c
  end

Start the emulation with:

.. code:: shell

  ./run_canon_fw.sh 80D,firmware="boot=0" -d callstack -s -S & arm-none-eabi-gdb -x 80D/debugmsg.gdb

Output:

.. code::

  ...
  Current stack: [2e9118-2e8118] sp=2e90c0                                         at [init:fe237c9e:fe238001]
  0xFE0D3385(0, fe0d3385, 19980218, 19980218)                                      at [init:8000173d:2e9108] (pc:sp)
   0xFE237E2D(0, feff65ab "Initialize SerialIO", 2e0f04, 44f4)                     at [init:fe0d403d:2e90f0] (pc:sp)
    0xFE237F93(18, 203a0, 0, 44f4)                                                 at [init:fe237e55:2e90d8] (pc:sp)
     0xFE237C9F(fe000000, ffffffff, 8, 5)                                          at [init:fe237ffd:2e90c0] (pc:sp)
  [        init:fe237ffd ] Memory region: start=FE000000 end=FFFFFFFF flags=00000008
  ...

The above shows the callers for the function being analyzed,
with 4 arguments (no attempts are made to guess the actual number of arguments)
and the locations for each call. You may examine these addresses in your disassembler.

1062 1063
GDB scripting docs
''''''''''''''''''
1064 1065 1066 1067 1068

- `Sequences <https://sourceware.org/gdb/onlinedocs/gdb/Sequences.html>`_ (command files, define, if, while, printf)
- `Convenience variables <https://sourceware.org/gdb/current/onlinedocs/gdb/Convenience-Vars.html>`_
- `GDB user manual <https://sourceware.org/gdb/current/onlinedocs/gdb/index.html>`_.

1069 1070
More examples
'''''''''''''
1071 1072 1073 1074

- `750D serial flash dumper <http://www.magiclantern.fm/forum/index.php?topic=17627.msg195357#msg195357>`_ (figuring out the parameters of an unknown function)
- `EOS M2 <http://www.magiclantern.fm/forum/index.php?topic=15895.msg186173#msg186173>`_ (examples with various GDB GUI front-ends):

alex@thinkpad's avatar
alex@thinkpad committed
1075 1076
|

1077 1078 1079 1080
.. image:: doc/img/ddd.png
   :scale: 50 %
   :align: center

alex@thinkpad's avatar
alex@thinkpad committed
1081

alex@thinkpad's avatar
alex@thinkpad committed
1082 1083 1084 1085 1086
Instrumentation
---------------

`TODO (see QEMU forum thread) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg184125#msg184125>`_

alex@thinkpad's avatar
alex@thinkpad committed
1087 1088
The instrumentation framework provides the following features:

alex@thinkpad's avatar
alex@thinkpad committed
1089 1090
- log all debug messages from Canon: ``-d debugmsg``
- log all memory accesses: ``-d rom/ram/romr/ramw/etc``
1091
- log all function calls: ``-d calls``
alex@thinkpad's avatar
alex@thinkpad committed
1092 1093 1094 1095 1096
- log all DryOS/VxWorks task switches: ``-d tasks``
- track all function calls to provide a stack trace: ``-d callstack``
- export all called functions to IDC script: ``-d idc``
- identify memory blocks copied from ROM to RAM: ``-d romcpy``
- check for memory errors (a la valgrind): ``-d memchk``
alex@thinkpad's avatar
alex@thinkpad committed
1097 1098 1099 1100

Debugging symbols from ML can be made available to instrumentation routines from environment variables (see ``export_ml_syms.sh``).

The address of DebugMsg is exported by ``run_canon_fw.sh`` (extracted from the GDB script, where it's commented out for speed reasons).
1101

1102 1103
Development and reverse engineering guide
-----------------------------------------
alex@thinkpad's avatar
alex@thinkpad committed
1104

1105
For the following topics, please see `HACKING.rst <HACKING.rst>`_:
alex@thinkpad's avatar
alex@thinkpad committed
1106

1107 1108 1109
- Finding your way around the source code;
- Adding support for emulating a new camera model;
- EOS (and PowerShot) firmware internals and reverse engineering tips.
alex@thinkpad's avatar
alex@thinkpad committed
1110 1111 1112 1113 1114

History
-------

:2008: `SD1100 boot (CHDK) <http://chdk.wikia.com/wiki/GPL_Qemu>`_
1115
:2009: `5D2/7D boot (Trammell Hudson) <http://magiclantern.wikia.com/wiki/Emulation>`_
alex@thinkpad's avatar
alex@thinkpad committed
1116 1117 1118 1119 1120
:2012: `TriX_EOS (g3gg0) <http://www.magiclantern.fm/forum/index.php?topic=2882.0>`_
:2013: `Initial Hello World <http://www.magiclantern.fm/forum/index.php?topic=2864.msg26022#msg26022>`_
:2013: `g3gg0 ports TriX changes to QEMU <http://www.magiclantern.fm/forum/index.php?topic=2864.msg29748#msg29748>`_
:2013: `Antony Pavlov submits initial DIGIC support to QEMU mainline <https://lists.gnu.org/archive/html/qemu-devel/2013-08/msg04509.html>`_
:2013: `Nikon Hacker is light years ahead us <http://www.magiclantern.fm/forum/index.php?topic=8823.0>`_ (we are not competing; it was just a fun notice that motivated us)
1121
:2014: `DryOS task scheduler running! <http://www.magiclantern.fm/forum/index.php?topic=2864.msg117430#msg117430>`_ (also ML menu and modules, but with massive hacks — emulating only a very small part of Canon firmware)