README.rst 41.6 KB
Newer Older
1
2
How can I run Magic Lantern in QEMU?
====================================
alex@thinkpad's avatar
alex@thinkpad committed
3
4
5

This guide shows you how to emulate Magic Lantern (or plain Canon firmware) in QEMU.

6
7
8
9
10
11
12
13
.. class:: align-center

|pic1| |pic2|

.. |pic1| image:: doc/img/qemu-60D.png
    :width: 30 %
.. |pic2| image:: doc/img/qemu-M2-hello.jpg
    :width: 32.3 %
alex@thinkpad's avatar
alex@thinkpad committed
14

15
16
.. contents::

alex@thinkpad's avatar
alex@thinkpad committed
17
18
19
20
21
Current state
-------------

What works:

22
- Canon GUI with menu navigation — most DIGIC 3, 4 and 5 models
23
24
- Limited support for DIGIC 2, 6 and 7 models
- Limited support for some PowerShot models (including recent EOS M models)
alex@thinkpad's avatar
alex@thinkpad committed
25
26
27
- Limited support for secondary DryOS cores (such as Eeko or 5D4 AE processor)
- File I/O works on most models (both SD and CF); might be unreliable
- Bootloader emulation works on all supported models (from DIGIC 2 to DIGIC 7)
28
- Loading AUTOEXEC.BIN / DISKBOOT.BIN from the virtual SD/CF card (from DIGIC 2 to DIGIC 7)
alex@thinkpad's avatar
alex@thinkpad committed
29
30
31
- Unmodified autoexec.bin works on many single-core camera models
  (and, with major limitations, on dual-core models)
- ML modules and Lua scripts (within the limitations of the emulation)
alex@thinkpad's avatar
alex@thinkpad committed
32
- DryOS/VxWorks timer (heartbeat) and task switching (all supported models)
33
- UART emulation (DryOS shell aka Dry-shell or DrySh on DIGIC 4, 5, 6 and 7 models)
34
- Deterministic execution with the ``-icount`` option (except CF PIO)
35
- Cache hacks are emulated to some extent, but not exactly accurate
alex@thinkpad's avatar
alex@thinkpad committed
36
37
38
39
40
41
42
43
44
- EDMAC memcpy, including geometry parameters (matches the hardware closely, but not perfectly)
- Debugging with GDB:

  - assembly level for Canon code
  - source level for ML code (if compiled with -ggdb3 or similar)
  - ML stubs can be loaded as debugging symbols for Canon code (todo: also import from IDA or other systems)
  - predefined GDB scripts (log calls to DebugMsg, task_create, register_interrupt and a few others)
  - front-ends tested: cgdb (``splitgdb.sh``), DDD, gdbgui

45
46
  |

alex@thinkpad's avatar
alex@thinkpad committed
47
48
49
50
51
52
53
- Debug messages at QEMU console:

  - Use qprintf / qprint / qprintn / qdisas for printing to QEMU console
  - Compile Magic Lantern with ``CONFIG_QEMU=y``
  - By default (``CONFIG_QEMU=n``), the debug messages are not compiled
    (therefore not increasing the size of the executable that runs on the camera)

54
55
  |

alex@thinkpad's avatar
alex@thinkpad committed
56
57
58
- Log various actions of the guest operating system (Canon firmware, ML):

  - execution trace: ``-d exec,nochain -singlestep``
alex@thinkpad's avatar
alex@thinkpad committed
59
60
  - I/O trace: ``-d io``, ``-d io,int``
  - log hardware devices: ``-d mpu/sflash/sdcf/uart/int``
alex@thinkpad's avatar
alex@thinkpad committed
61
62
  - log all debug messages from Canon: ``-d debugmsg``
  - log all memory accesses: ``-d rom/ram/romr/ramw/etc``
63
  - log all function calls: ``-d calls``
alex@thinkpad's avatar
alex@thinkpad committed
64
  - log all DryOS/VxWorks task switches: ``-d tasks``
alex@thinkpad's avatar
alex@thinkpad committed
65
66
67
68
  - track all function calls to provide a stack trace: ``-d callstack``
  - export all called functions to IDC script: ``-d idc``
  - identify memory blocks copied from ROM to RAM: ``-d romcpy``
  - check for memory errors (a la valgrind): ``-d memchk``
alex@thinkpad's avatar
alex@thinkpad committed
69
  - list all available items: ``-d help`` 
alex@thinkpad's avatar
alex@thinkpad committed
70
71
72
73

What does not work (yet):

- LiveView (WIP, very hard);
74
- Still photo capture (WIP — the capture process itself works);
alex@thinkpad's avatar
alex@thinkpad committed
75
76
77
78
- Image review (WIP);
- Dual core emulation aka IPC (WIP);
- Touch screen (TODO);
- Flash reprogramming (TODO, low priority);
alex@thinkpad's avatar
alex@thinkpad committed
79
- Most hardware devices (audio chip, ADTG, FPGAs, JPCORE, image processing engine...);
80
- MPU properties are working to some extent, but have major limitations and bugs;
alex@thinkpad's avatar
alex@thinkpad committed
81
82
83
84
- Lens communication (done via MPU); initial lens info is replayed on startup on some models, but that's pretty much it;
- Cache behavior is not emulated (very hard; feel free to point us to code that can be reused);
- Native Windows build (QEMU can be compiled on Windows => contribution welcome).

85
86
Common issues and workarounds
`````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
87

88
- Camera was not shut down cleanly — Skipping module loading
alex@thinkpad's avatar
alex@thinkpad committed
89
90

  - closing QEMU window does not perform a clean shutdown
91
  - ``Machine -> Power Down`` — see `Shutdown and reboot`_ for more info
92
  - quicker: press ``C`` to `"open" the card door`__ => also clean shutdown.
alex@thinkpad's avatar
alex@thinkpad committed
93

94
__ `Opening the card door`_
alex@thinkpad's avatar
alex@thinkpad committed
95

96
97
.. _netcat-issue:

98
99
100
101
102
103
104
105
106
107
108
109
- Netcat issues when interacting with ``qemu.monitor``

  There are many versions of netcat around.
  Newer variants of openbsd netcat
  (`since 1.111, Mar 2013 <https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/nc/netcat.c.diff?r1=1.110&r2=1.111&f=h>`_)
  require ``-N``, but older versions do not have ``-N`` and will print an error if you attempt to use it. GNU netcat or other variants were not tested.
  
  TLDR: try openbsd netcat. If you get an error after copy/pasting some ``nc`` command from this guide, simply remove the ``-N``.
  
  Look in ``run_tests.sh`` for a slightly more portable workaround.
  

alex@thinkpad's avatar
alex@thinkpad committed
110
111
112
Installation
------------

113
It is recommended to install from the `qemu <https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/qemu/contrib/qemu>`_
alex@thinkpad's avatar
alex@thinkpad committed
114
115
116
117
branch, as it will contain the latest developments. The installation will be performed
outside the main magic-lantern directory, therefore you will be able to emulate any ML branch
without additional gymnastics (you will **not** have to merge ``qemu`` into your working branch or worry about it).

118
119
120
121
1. Clone the ``magic-lantern`` repository if you haven't already:

   .. code:: shell

122
     hg clone https://foss.heptapod.net/magic-lantern/magic-lantern
123
124
     cd magic-lantern

alex@thinkpad's avatar
alex@thinkpad committed
125
126
   |

127
128
2. Run ``install.sh`` from the ``qemu`` branch:

alex@thinkpad's avatar
alex@thinkpad committed
129
   .. code:: shell
130

alex@thinkpad's avatar
alex@thinkpad committed
131
132
133
     /path/to/magic-lantern$  hg update qemu -C
     /path/to/magic-lantern$  cd contrib/qemu
     /path/to/magic-lantern/contrib/qemu$  ./install.sh
134

135
136
137
138
139
   By default, QEMU will be installed in the ``qemu-eos`` directory,
   near the ``magic-lantern`` one, at the same level - e.g. ``/path/to/qemu-eos``.
   Please refer to `HACKING.rst <HACKING.rst#rst-header-how-is-this-code-organazized>`__
   for details on the directory structure and how to change the installation directory.

alex@thinkpad's avatar
alex@thinkpad committed
140
141
   |

142
143
3. Follow the instructions; you will have to supply your ROM files and compile QEMU:

alex@thinkpad's avatar
alex@thinkpad committed
144
   .. code:: shell
145

alex@thinkpad's avatar
alex@thinkpad committed
146
     # replace camera model (60D) with yours
147
148
149
150
151
     /path/to/qemu-eos$  cp /path/to/sdcard/ML/LOGS/ROM*.BIN 60D/
     /path/to/qemu-eos$  cd qemu-2.5.0
     /path/to/qemu-eos/qemu-2.5.0$  ../configure_eos.sh
     /path/to/qemu-eos/qemu-2.5.0$  make -j2
     /path/to/qemu-eos/qemu-2.5.0$  cd ..
152

153
154
155
156
157
158
   Some recent camera models also use a serial flash. To list them, run this command:

   .. code:: shell
   
     cat qemu-2.5.0/hw/eos/model_list.c | grep -B 10 serial_flash_size | grep -E "\.(serial_flash_size|name) "

alex@thinkpad's avatar
alex@thinkpad committed
159
   or just watch out for an error about missing ``SFDATA.BIN`` when trying the next step.
160
161

   If your camera requires a serial flash, compile the 
162
   `sf_dump module <https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/unified/modules/sf_dump>`_
163
   and run it on your camera to get this file.
164

165
   |
alex@thinkpad's avatar
alex@thinkpad committed
166

167
4. Test your installation.
alex@thinkpad's avatar
alex@thinkpad committed
168

alex@thinkpad's avatar
alex@thinkpad committed
169
   The pre-installed SD/CF images come with a small autoexec.bin
alex@thinkpad's avatar
alex@thinkpad committed
170
171
172
   (the `portable display test <http://www.magiclantern.fm/forum/index.php?topic=14732.0>`_)
   that works on all supported EOS models. It does not work on PowerShot models
   (including recent EOS M models based on PowerShot firmware).
173

alex@thinkpad's avatar
alex@thinkpad committed
174
   .. code:: shell
175
 
alex@thinkpad's avatar
alex@thinkpad committed
176
     # all EOS models should run this without any trickery
177
     /path/to/qemu-eos$  ./run_canon_fw.sh 60D,firmware="boot=1"
178

alex@thinkpad's avatar
alex@thinkpad committed
179
180
   |

181
182
5. Compile and run Magic Lantern

alex@thinkpad's avatar
alex@thinkpad committed
183
   .. code:: shell
184

alex@thinkpad's avatar
alex@thinkpad committed
185
     # from the QEMU directory
186
     /path/to/qemu-eos$  make -C ../magic-lantern 60D_install_qemu
187
     
alex@thinkpad's avatar
alex@thinkpad committed
188
     # some models will work only with this:
189
     /path/to/qemu-eos$  ./run_canon_fw.sh 60D,firmware="boot=1"
190

alex@thinkpad's avatar
alex@thinkpad committed
191
     # some models require running under GDB (they won't boot the GUI otherwise)
192
     /path/to/qemu-eos$  ./run_canon_fw.sh EOSM,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
193

alex@thinkpad's avatar
alex@thinkpad committed
194
195
   |

196
For reference, you may also look at `our test suite <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console>`_,
alex@thinkpad's avatar
alex@thinkpad committed
197
198
199
where QEMU is installed from scratch every time the tests are run.
These logs can be very useful for troubleshooting.

alex@thinkpad's avatar
alex@thinkpad committed
200
201
While we don't provide a native Windows build yet,
it is possible to install QEMU and ML development tools
202
`under the Windows 10 Linux Subsystem (WSL) <http://www.magiclantern.fm/forum/index.php?topic=20214.0>`_.
alex@thinkpad's avatar
alex@thinkpad committed
203
204
205
206

Running Canon firmware
----------------------

207
From the ``qemu-eos`` directory, use the ``run_canon_fw.sh`` script and make sure
alex@thinkpad's avatar
alex@thinkpad committed
208
209
210
211
the `boot flag <http://magiclantern.wikia.com/wiki/Bootflags>`_ is disabled:

.. code:: shell

212
  # from the qemu-eos directory
alex@thinkpad's avatar
alex@thinkpad committed
213
  ./run_canon_fw.sh 60D,firmware="boot=0"
alex@thinkpad's avatar
alex@thinkpad committed
214

215
Some models may need additional patches to run — these are stored under ``CAM/patches.gdb``.
216
To emulate these models, you will also need ``arm-none-eabi-gdb`` or ``gdb-multiarch``:
alex@thinkpad's avatar
alex@thinkpad committed
217
218
219

.. code:: shell

220
  ./run_canon_fw.sh EOSM,firmware="boot=0" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
221

222
223
224
You'll probably want to `see a few internals`__ as well. To get started, try these:

__ `Tracing guest events (execution, I/O, debug messages, RAM, function calls...)`_
alex@thinkpad's avatar
alex@thinkpad committed
225
226
227

.. code:: shell

alex@thinkpad's avatar
alex@thinkpad committed
228
229
230
231
232
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,tasks
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io,int
  ./run_canon_fw.sh 60D,firmware="boot=0" -d help
alex@thinkpad's avatar
alex@thinkpad committed
233

234
235
236
237
238
239
240
Or you may want to `run the firmware under GDB`__ and log additional functions:

__ `Debugging with GDB`_

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -s -S & arm-none-eabi-gdb -x 60D/debugmsg.gdb
241
  ./run_canon_fw.sh 60D,firmware="boot=0" -s -S & gdb-multiarch -x 60D/debugmsg.gdb
242

alex@thinkpad's avatar
alex@thinkpad committed
243
244
245
246
Running Magic Lantern
---------------------

As you already know, Magic Lantern runs from the SD or CF card. For emulation,
247
we provide two card images (``sd.img`` and ``cf.img``) which you can mount on your operating system
alex@thinkpad's avatar
alex@thinkpad committed
248
249
250
251
252
253
and copy files on them. If these images use a FAT filesystem (they do, by default), we prefer 
`mtools <https://www.gnu.org/software/mtools/>`_ for automated tasks
(such as copying files to/from the card images without mounting them).

To install Magic Lantern to the virtual card, you may:

254
- mount the card image (``sd.img`` or ``cf.img``) as ``/whatever/EOS_DIGITAL``,
alex@thinkpad's avatar
alex@thinkpad committed
255
256
257
258
259
260
261
262
263
264
265
  then run ``make install`` from your platform directory:

  .. code:: shell

    # from the magic-lantern directory
    cd platform/60D.111
    make clean; make
    # make sure your virtual card is mounted (this step is operating system specific)
    make install
    # make sure your virtual card is no longer mounted

alex@thinkpad's avatar
alex@thinkpad committed
266
267
  |

268
269
270
- use ``make install_qemu`` from your platform directory, or ``make CAM_install_qemu`` 
  or ``make CAM.FW_install_qemu`` from your ML root directory
  (requires mtools, but you do not have to mount your card images;
271
  works since `27f4105 <https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/5437eeb81f6759e42c7c1b194b2fdacb6916be28>`_):
alex@thinkpad's avatar
alex@thinkpad committed
272

273
  .. code:: shell
alex@thinkpad's avatar
alex@thinkpad committed
274
275
276
277
278
279

    # from the magic-lantern directory
    cd platform/60D.111
    make clean; make
    make install_qemu

280
281
282
283
284
285
  .. code:: shell

    # from the magic-lantern directory
    make 60D_clean
    make 60D_install_qemu

286
  They also work from the ``qemu-eos`` directory:
287
288
289

  .. code:: shell

290
    # from the qemu-eos directory
291
292
    make -C ../magic-lantern/platform/60D.111 clean
    make -C ../magic-lantern/platform/60D.111 install_qemu
293
294
295

  .. code:: shell

296
    # from the qemu-eos directory
297
298
299
    make -C ../magic-lantern 5D3.113_clean
    make -C ../magic-lantern 5D3.113_install_qemu

300
  Please note: ``make install_qemu`` is a recent addition and may not be available in all branches.
301
  In this case, you may either use the first method, or sync with the "unified" branch (``hg merge unified``),
302
  or manually import changeset `d5ad86f <https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/6313c5fa52c38104fa8f2bb10a1ab8ae6e54e9bb>`_.
303
  Unfortunately, these rules won't work from ``Makefile.user``.
alex@thinkpad's avatar
alex@thinkpad committed
304

alex@thinkpad's avatar
alex@thinkpad committed
305
306
The included card images are already bootable for EOS firmwares (but not for PowerShots).

307
After you have copied Magic Lantern to the card, you may run it from the ``qemu-eos`` directory
308
(near the ``magic-lantern`` one, at the same level):
alex@thinkpad's avatar
alex@thinkpad committed
309
310
311

.. code:: shell

312
  # from the qemu-eos directory
alex@thinkpad's avatar
alex@thinkpad committed
313
314
315
  ./run_canon_fw.sh 60D,firmware="boot=1"
  
  # or, if your camera requires patches.gdb:
316
  ./run_canon_fw.sh EOSM,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb -ex quit
alex@thinkpad's avatar
alex@thinkpad committed
317

alex@thinkpad's avatar
alex@thinkpad committed
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
Navigating menus
````````````````

Press ``F1`` to show the available keys (they are model-specific), or just read them
from the emulation log. Some keys will only send "press" events, while others
will send "press and release" events (this is also model-specific
and printed on the console).

Scrollwheels, if turned very quickly, may send a single event that includes
more than one click. This complicates scrollwheel handling code on ML side
and often leads to subtle issues. Currently, this behavior is not emulated.

Shutdown and reboot
```````````````````

333
By default, closing the QEMU window is equivalent to unplugging the power cord
alex@thinkpad's avatar
alex@thinkpad committed
334
335
336
(if your camera is on external power source). This appears to be the default
with other operating systems as well, so we did not change it.

337
338
Please note: closing QEMU window is **not** equivalent to taking the battery out —
see `Opening the battery door`_ for details.
alex@thinkpad's avatar
alex@thinkpad committed
339
340
341
342
343
344
345
346
347
348
349
350

Shutting down
'''''''''''''

To perform a clean shutdown (similar to powering down the camera from the main switch),
you may:

- select ``Machine -> Power Down`` from the menu (QEMU window)
- send the ``system_powerdown`` command to QEMU monitor:

.. code:: shell

351
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
352
353
354
355

Internally, Canon code refers to this kind of shutdown as ``SHUTDOWN_REQUEST``
(watch their debug messages with ``-d debugmsg``).

alex@thinkpad's avatar
alex@thinkpad committed
356
357
358
359
360
361
362
Caveat: the ``system_powerdown`` event does not guarantee a shutdown will actually occur
(either because the firmware handles this event in a different way, or if the execution
gets stuck for any reason). As a workaround, you could issue the ``quit`` command
after some timeout, if QEMU is still running:

.. code:: shell

363
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
364
  sleep 2
365
366
  if nc -N -U qemu.monitor < /dev/null > /dev/null 2>&1; then
    echo "quit" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
367
368
  fi

alex@thinkpad's avatar
alex@thinkpad committed
369
370
371
372
373
374
375
376
377
378
379
380
381
Opening the card door
'''''''''''''''''''''

Opening the SD/CF card door is a clean (non-rushed) way to shut down Canon firmware (``SHUTDOWN_REQUEST``).
To emulate this kind of shutdown, press ``C`` to simulate opening the card door,
then wait for a few seconds for QEMU to shutdown.

Opening the card door and closing it back quickly enough may result
in shutdown being canceled. Closing the card door is not implemented,
therefore this behavior is not emulated yet.

Opening the battery door
''''''''''''''''''''''''
alex@thinkpad's avatar
alex@thinkpad committed
382

alex@thinkpad's avatar
alex@thinkpad committed
383
384
385
386
387
388
389
390
391
392
393
394
Opening the battery door is interpreted by Canon firmware as an emergency shutdown (``PROP_ABORT``),
but it's still a clean(ish) shutdown. To emulate this kind of shutdown, press ``B``,
then close the QEMU window manually (or send the ``quit`` command to QEMU monitor).
Currently we do not know how to trigger or recognize a hardware shutdown event.

Rebooting
'''''''''

The camera can be rebooted from software by triggering ``PROP_REBOOT``.
Canon firmware handles it as a regular shutdown (``SHUTDOWN_REQUEST``),
followed by a regular boot. In QEMU, triggering ``PROP_REBOOT`` from software
will perform a clean shutdown (rebooting is not implemented).
alex@thinkpad's avatar
alex@thinkpad committed
395
396

Running ML Lua scripts
alex@thinkpad's avatar
alex@thinkpad committed
397
``````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
398

399
- Install ML on the virtual SD card:
alex@thinkpad's avatar
alex@thinkpad committed
400
401
402

  .. code:: shell

403
    # from the qemu-eos directory
404
405
    wget http://builds.magiclantern.fm/jenkins/job/lua_fix/431/artifact/platform/60D.111/magiclantern-lua_fix.2017Dec23.60D111.zip
    unzip magiclantern-lua_fix.2017Dec23.60D111.zip -d ml-tmp
alex@thinkpad's avatar
alex@thinkpad committed
406
407
408
    ./mtools_copy_ml.sh ml-tmp
    rm -rf ml-tmp/

alex@thinkpad's avatar
alex@thinkpad committed
409
410
  |

411
- Run QEMU:
alex@thinkpad's avatar
alex@thinkpad committed
412
413
414
415

  .. code:: shell

    ./run_canon_fw.sh 60D,firmware="boot=1"
alex@thinkpad's avatar
alex@thinkpad committed
416

alex@thinkpad's avatar
alex@thinkpad committed
417
418
  |

alex@thinkpad's avatar
alex@thinkpad committed
419
- enable the Lua module
alex@thinkpad's avatar
alex@thinkpad committed
420
- reboot the virtual camera cleanly (menu: Machine -> Power Down, then start it again)
alex@thinkpad's avatar
alex@thinkpad committed
421
422
- run the Hello World script

423
424
Note: api_test.lua does not run completely, but `can be patched <http://www.magiclantern.fm/forum/index.php?topic=2864.msg195347#msg195347>`_.

alex@thinkpad's avatar
alex@thinkpad committed
425
426
427
TODO: make api_test.lua run, fix bugs, polish the guide.

Using multiple firmware versions
alex@thinkpad's avatar
alex@thinkpad committed
428
````````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
429
430
431
432
433
434
435

In most cases, Magic Lantern only supports one firmware version, to keep things simple.
However, there may be good reasons to support two firmware versions
(for example, on the 5D Mark III, there are valid reasons to choose
both `1.1.3 <http://www.magiclantern.fm/forum/index.php?topic=14704.0>`_
and `1.2.3 <http://www.magiclantern.fm/forum/index.php?topic=11017.0>`_)
or you may want to test both versions when porting Magic Lantern
436
`to a newer Canon firmware <http://www.magiclantern.fm/forum/index.php?topic=19417.0>`_.
alex@thinkpad's avatar
alex@thinkpad committed
437
438
439
440
441
442
443

The invocation looks like this (notice the ``113``):

.. code:: shell

  ./run_canon_fw.sh 5D3,firmware="113;boot=0" -s -S & arm-none-eabi-gdb -x 5D3/debugmsg.gdb

444
And the directory layout should be like this::
alex@thinkpad's avatar
alex@thinkpad committed
445

446
447
448
449
450
451
452
  /path/to/qemu-eos/5D3/113/ROM0.BIN
  /path/to/qemu-eos/5D3/113/ROM1.BIN
  /path/to/qemu-eos/5D3/123/ROM0.BIN
  /path/to/qemu-eos/5D3/123/ROM1.BIN
  /path/to/qemu-eos/5D3/113/ROM0.BIN
  /path/to/qemu-eos/5D3/debugmsg.gdb  # common to both versions
  /path/to/qemu-eos/5D3/patches.gdb   # common to both versions
alex@thinkpad's avatar
alex@thinkpad committed
453

454
Compare this to a camera model where only one firmware version is supported::
alex@thinkpad's avatar
alex@thinkpad committed
455

456
457
458
  /path/to/qemu-eos/60D/ROM0.BIN
  /path/to/qemu-eos/60D/ROM1.BIN
  /path/to/qemu-eos/5D3/patches.gdb
459

460
You may also store ``debugmsg.gdb`` and ``patches.gdb`` under the firmware version subdirectory if you prefer, but other QEMU-related scripts won't pick them up.
alex@thinkpad's avatar
alex@thinkpad committed
461

462
463
Creating custom SD/CF card images
`````````````````````````````````
alex@thinkpad's avatar
alex@thinkpad committed
464

465
466
You may want to run the emulation using the same card contents as on the physical camera
(for example, for troubleshooting). The recommended way is to create an image
alex@thinkpad's avatar
alex@thinkpad committed
467
468
469
470
471
472
473
474
of your SD (or CF) card (`tutorial for Windows and Mac <https://thepihut.com/blogs/raspberry-pi-tutorials/17789160-backing-up-and-restoring-your-raspberry-pis-sd-card>`_):

.. code:: shell

  dd if=/dev/your-sd-card of=sd.img bs=1M

and run QEMU from the resulting ``sd.img`` (or ``cf.img``).

475
476
477
478
479
480
Note: you need to copy the raw contents of the entire card, not just one partition. For example:

.. code:: shell

  dd if=/dev/mmcblk0 of=sd.img    # not mmcblk0p1

alex@thinkpad's avatar
alex@thinkpad committed
481
482
483
484
485
486
487
488
489
490
491
492
493
494
Tip: to save disk space, format the SD card first, copy the desired files,
then create the image as `qcow2 <https://rwmj.wordpress.com/2010/05/18/tip-compress-raw-disk-images-using-qcow2/>`_:

.. code:: shell

  qemu-img convert -f raw -O qcow2 /dev/mmcblk0 sd.qcow2

then update ``run_canon_fw.sh`` to use the new image::

  -drive if=sd,format=qcow2,file=sd.qcow2

That way, a 64GB card with Magic Lantern installed, after formatting in camera
with the option to keep ML on the card, will only take up about 15MB as a ``qcow2`` image.

495
496
Running from the physical SD/CF card
````````````````````````````````````
497
You may also start QEMU from the same card you use in the physical camera — 
498
499
500
501
502
503
504
this might be useful for troubleshooting issues with Magic Lantern, 
hopefully including certain card compatibility issues.

It is **strongly** recommended to avoid this procedure, unless absolutely needed.
Please consider creating an image of your SD (or CF) card instead, as described above.

If you still wish to run the emulation directly from a physical SD/CF card:
alex@thinkpad's avatar
alex@thinkpad committed
505

alex@thinkpad's avatar
alex@thinkpad committed
506
- See `QEMU docs (Using host drives) <https://qemu.weilnetz.de/doc/5.2/system/images.html#using-host-drives>`_
507
  and make sure you understand `these warnings <https://wiki.archlinux.org/index.php/QEMU#Using_any_real_partition_as_the_single_primary_partition_of_a_hard_disk_image>`_
alex@thinkpad's avatar
alex@thinkpad committed
508
509
510
511
512
513
514
515
516
517
- Replace ``file=sd.img`` (or ``file=cf.img``) with ``file=/dev/your-sd-card`` in ``run_canon_fw.sh``
- Configure the appropriate permissions and run the emulation as usual.

**Warning: Canon firmware WILL write to the SD/CF card during startup, 
and might even attempt to format it if the filesystem is not recognized,
or because of emulation bugs. 
Be very careful not to give it write access to your physical hard-disk!!!**

Note: the ROM files will not be loaded from the SD/CF card.

518
519
520
521
522
523
524
525
526
527
528
529
530
Image capture emulation
```````````````````````

WIP, still pretty rough.

To capture a full-res image (aka FRSP) using a CR2 as reference data for the virtual sensor:

.. code:: shell

    make -C ../magic-lantern/minimal/qemu-frsp MODEL=5D3 CONFIG_QEMU=y clean install_qemu
    env QEMU_EOS_VRAM_PH_QR_RAW='/path/to/IMG_1234.CR2' ./run_canon_fw.sh 5D3,firmware="boot=1"


alex@thinkpad's avatar
alex@thinkpad committed
531
532
533
534
535
536
537
Automation
----------

QEMU monitor
````````````

By default, the QEMU monitor console is available by default as a UNIX socket.
alex@thinkpad's avatar
alex@thinkpad committed
538
539
540
541
542
543
That means, during emulation you can interact with it using netcat:

- interactive console

  .. code:: shell

544
    nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
545

alex@thinkpad's avatar
alex@thinkpad committed
546
547
  |

alex@thinkpad's avatar
alex@thinkpad committed
548
- one-liner commands, usable from scripts:
alex@thinkpad's avatar
alex@thinkpad committed
549
550
551

  .. code:: shell

552
    echo "log io" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
553

alex@thinkpad's avatar
alex@thinkpad committed
554
555
  |

alex@thinkpad's avatar
alex@thinkpad committed
556
- check whether QEMU monitor is active:
alex@thinkpad's avatar
alex@thinkpad committed
557
558
559

  .. code:: shell

560
    if nc -N -U qemu.monitor < /dev/null > /dev/null 2>&1; then
alex@thinkpad's avatar
alex@thinkpad committed
561
562
      ...
    fi
alex@thinkpad's avatar
alex@thinkpad committed
563

alex@thinkpad's avatar
alex@thinkpad committed
564
565
  |

alex@thinkpad's avatar
alex@thinkpad committed
566
567
You can redirect the monitor console to stdio with... ``-monitor stdio``.

568
569
570
If you have trouble running the above ``nc`` commands, don't forget to check this `common netcat issue`__.

__ `netcat-issue`_
571

alex@thinkpad's avatar
alex@thinkpad committed
572
573
574
Taking screenshots
``````````````````

575
576
577
578
579
580
The easiest way is to use the ``screendump`` command from QEMU monitor:

.. code:: shell

  echo "screendump snap.ppm" | nc -N -U qemu.monitor

alex@thinkpad's avatar
alex@thinkpad committed
581
In the following example, we'll redirect the monitor to stdio
582
and take a screenshot after 10 seconds:
alex@thinkpad's avatar
alex@thinkpad committed
583
584
585
586
587
588

.. code:: shell

  ( 
    sleep 10
    echo screendump snap.ppm
alex@thinkpad's avatar
alex@thinkpad committed
589
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
590
591
592
593
594
595
596
597
598
599
600
601
602
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=0' \
        -monitor stdio
  )

Another option is to use the VNC interface:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' \
        -vnc :1234 &
  sleep 10
  vncdotool -s :1234 capture snap.png
603
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
604
605
606
607

Sending keystrokes
``````````````````

608
From QEMU monitor::
alex@thinkpad's avatar
alex@thinkpad committed
609
610
611
612
613
614
615
616

  (qemu) help
  sendkey keys [hold_ms] -- send keys to the VM (e.g. 'sendkey ctrl-alt-f1', default hold time=100 ms)

.. code:: shell

  ( 
    sleep 10
alex@thinkpad's avatar
alex@thinkpad committed
617
    echo sendkey m
alex@thinkpad's avatar
alex@thinkpad committed
618
619
    sleep 1
    echo screendump menu.ppm
alex@thinkpad's avatar
alex@thinkpad committed
620
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
621
622
623
624
625
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=0' \
        -monitor stdio
  )

alex@thinkpad's avatar
alex@thinkpad committed
626
627
628
629
630
631
632
Or, if QEMU runs as a background process:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' &
  
  sleep 10
633
  echo "sendkey m" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
634
  sleep 1
635
636
  echo "screendump menu.ppm" | nc -N -U qemu.monitor
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
637

alex@thinkpad's avatar
alex@thinkpad committed
638
639
640
641
642
From VNC:

.. code:: shell

  vncdotool -h | grep key
643
644
645
  key KEY               send KEY to server, alphanumeric or keysym: ctrl-c, del
  keyup KEY             send KEY released
  keydown KEY           send KEY pressed
alex@thinkpad's avatar
alex@thinkpad committed
646
647
648
649
650
651
652
653

.. code:: shell

  ./run_canon_fw.sh 60D,firmware='boot=0' \
        -vnc :1234 &
  sleep 10
  vncdotool -s :1234 key m
  sleep 1
654
  vncdotool -s :1234 capture menu.png
655
  echo "system_powerdown" | nc -N -U qemu.monitor
alex@thinkpad's avatar
alex@thinkpad committed
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676

Running multiple ML builds from a single command
````````````````````````````````````````````````

You may run ML builds from multiple models, unattended,
with the ``run_ml_all_cams.sh`` script:

.. code:: shell

  env ML_PLATFORMS="500D.111/ 60D.111/" \
      TIMEOUT=10 \
      SCREENSHOT=1 \
      ./run_ml_all_cams.sh

Internally, this is how the emulator is invoked:

.. code:: shell

  ( 
    sleep 10
    echo screendump 60D.111.ppm
alex@thinkpad's avatar
alex@thinkpad committed
677
    echo system_powerdown
alex@thinkpad's avatar
alex@thinkpad committed
678
679
  ) | (
    ./run_canon_fw.sh 60D,firmware='boot=1' \
680
        -display none -monitor stdio
alex@thinkpad's avatar
alex@thinkpad committed
681
682
683
684
685
  ) &> 60D.111.log


This script is very customizable (see the source code for available options).

686
**More examples:**
alex@thinkpad's avatar
alex@thinkpad committed
687

688
689
690
- `EOSM2 hello world <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/18/console>`_
- running ML from the dm-spy-experiments branch in the emulator (`QEMU-dm-spy <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-dm-spy/65/consoleFull>`_)
- running the FA_CaptureTestImage test based on the minimal ML target (`QEMU-FA_CaptureTestImage <http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-FA_CaptureTestImage>`_)
alex@thinkpad's avatar
alex@thinkpad committed
691

692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
Parallel execution
``````````````````

On modern machines, you will get significant speed gains by running multiple instances of QEMU in parallel.
This is tricky and not automated. You need to be careful with the following global resources:

- SD and CF images (``sd.img`` and ``cf.img``):

  If all your parallel instances require the same initial SD/CF card contents,
  and you do not need to inspect the changes to SD/CF after the experiment,
  you may use these files as read-only shared resources with the help of QEMU's
  `temporary snapshot <https://wiki.qemu.org/Documentation/CreateSnapshot#Temporary_snapshots>`_ feature
  (simply add ``-snapshot`` to your command line). This will discard any changes to ``sd.img`` and ``cf.img``.
  `Implementation details <https://lists.gnu.org/archive/html/qemu-devel/2008-09/msg00712.html>`_.

  Otherwise, you could allocate different SD/CF images for each instance, but it's up to you to modify the scripts to handle that.

- QEMU monitor socket (``qemu.monitor``):

  Set the ``QEMU_JOB_ID`` environment variable; it will be used as suffix for ``qemu.monitor``.
  
  Example: ``QEMU_JOB_ID=1 ./run_canon_fw.sh 5D3`` will use ``qemu.monitor1`` for monitor commands.

- GDB port (with ``-s -S``, this port is 1234):

717
  Set QEMU_JOB_ID to a small positive integer, for example:
718
719
720

  .. code:: shell

721
722
723
724
725
726
727
    export QEMU_JOB_ID=1
  
  Then you'll be able to do this:

  .. code:: shell

    QEMU_MONITOR=qemu.monitor$QEMU_JOB_ID
alex@thinkpad's avatar
alex@thinkpad committed
728
    GDB_PORT=$((1234+QEMU_JOB_ID))
729
    ./run_canon_fw.sh EOSM2,firmware="boot=0" -S -gdb tcp::$GDB_PORT &
730
731
732
733
    arm-none-eabi-gdb -ex "set \$TCP_PORT=$GDB_PORT" -x EOSM2/patches.gdb -ex quit &
    
    # interact with monitor commands
    sleep 5
734
    echo "sendkey m" | nc -N -U $QEMU_MONITOR
735
736
737
    sleep 1

    # quit when finished
738
739
740
    echo "quit" | nc -N -U $QEMU_MONITOR

  |
741
742
743
744
745
746
747

- VNC display

  Same as above:

  .. code:: shell

748
    QEMU_MONITOR=qemu.monitor$QEMU_JOB_ID
749
    VNC_DISP=":$((12345+QEMU_JOB_ID))"
750
    ./run_canon_fw.sh 5D3,firmware="boot=0" -vnc $VNC_DISP &
751
752
753
    
    # interact with vncdotool
    sleep 5
754
    vncdotool -s $VNC_DISP key m
755
756
757
    sleep 1
    
    # quit when finished
758
759
760
    echo "quit" | nc -N -U $QEMU_MONITOR

  |
761
762
763

- any temporary files you may want to use

764
765
  Try to achieve the same thing without a temporary file (pipes, process substitution, command-line options).
  If that doesn't work, consider using ``mktemp`` rather than hardcoding a filename.
766
767
768
769
770

- any other global resources (you'll have to figure them out on your own).

TODO: can this be automated somehow with containers?

alex@thinkpad's avatar
alex@thinkpad committed
771
772
773
Debugging
---------

alex@thinkpad's avatar
alex@thinkpad committed
774
775
776
777
778
779
780
781
782
783
784
785
786
787
From the QEMU monitor
`````````````````````

.. code:: shell

  echo "help" | ./run_canon_fw.sh 60D -monitor stdio |& grep dump
  ...
  pmemsave addr size file -- save to disk physical memory dump starting at 'addr' of size 'size'
  xp /fmt addr -- physical memory dump starting at 'addr'

Using ``qprintf`` and friends
`````````````````````````````

The QEMU debugging API
788
(`qemu-util.h <https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/qemu/src/qemu-util.h>`_, included by default by ``dryos.h``)
alex@thinkpad's avatar
alex@thinkpad committed
789
790
791
792
793
794
795
796
797
798
799
800
exposes the following functions to be used in Magic Lantern code:

:qprintf: heavyweight, similar to printf; requires vsnprintf from Canon code
:qprint: lightweight, inline, similar to puts, without newline
:qprintn: lightweight, prints a 32-bit integer
:qdisas: lightweight, tells QEMU to disassemble one ARM or Thumb instruction at the given address

These functions will print to QEMU console whenever ML (or a subset of it)
is compiled with ``CONFIG_QEMU=y``. They won't get compiled in regular builds
(``CONFIG_QEMU=n`` is the default), therefore they won't increase the executable size.
For this reason, feel free to use them *anywhere*.

801
802
You may use the debugging API for either the entire ML, or just for a subset of it —
e.g. the source file(s) you are currently editing, or only some modules.
alex@thinkpad's avatar
alex@thinkpad committed
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
The lightweight functions can also be used in very early boot code,
where you can't call vsnprintf or you may not even have a stack properly set up.

CONFIG_QEMU
```````````

:``CONFIG_QEMU=n``: (default):

- regular build
- the executable works in QEMU (within the limitations of the emulation)
- no guest debugging code (no additional debugging facilities)

:``CONFIG_QEMU=y``: (optional, on the command line or in ``Makefile.user``):

- debug build for QEMU only
- does **not** run on the camera (!)
- enables ``qprintf`` and friends to print to the QEMU console
820
- enables unlimited number of ROM patches — useful for 
821
  `dm-spy-experiments <http://www.magiclantern.fm/forum/index.php?topic=2388.0>`_
alex@thinkpad's avatar
alex@thinkpad committed
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
  (in QEMU you can simply write to ROM as if it were RAM)
- may enable workarounds for models or features that are not emulated very well

Example:

.. code:: shell

    cd platform/550D.109
    make clean; make                            # regular build
    make clean; make CONFIG_QEMU=y              # debug build for QEMU
    make clean; make install_qemu               # build and install a regular build to the QEMU SD/CF image
    make clean; make install_qemu CONFIG_QEMU=y # build and install a QEMU build to the QEMU SD/CF image

It works for modules as well:

.. code:: shell

    cd modules/lua
    # add some qprintf call in lua_init for testing
841
842
843
844
845
    make clean; make                            # regular build
    make clean; make CONFIG_QEMU=y              # debug build for QEMU
    make clean; make install_qemu               # build and install a regular build to the QEMU SD/CF image
    make clean; make install_qemu CONFIG_QEMU=y # build and install a QEMU build to the QEMU SD/CF image
 
alex@thinkpad's avatar
alex@thinkpad committed
846
847
848
849
850
851
852
853
854
Tracing guest events (execution, I/O, debug messages, RAM, function calls...)
`````````````````````````````````````````````````````````````````````````````

Execution trace:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d exec,nochain -singlestep

855
I/O trace (precise):
alex@thinkpad's avatar
alex@thinkpad committed
856
857
858
859
860

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d io

861
I/O trace (quick):
alex@thinkpad's avatar
alex@thinkpad committed
862
863
864

.. code:: shell

865
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io_quick
alex@thinkpad's avatar
alex@thinkpad committed
866
867
868
869
870

I/O trace with interrupts (precise):

.. code:: shell

871
  ./run_canon_fw.sh 60D,firmware="boot=0" -d io,int
alex@thinkpad's avatar
alex@thinkpad committed
872

873
I/O trace with Canon debug messages (precise):
alex@thinkpad's avatar
alex@thinkpad committed
874
875
876
877
878
879
880
881
882
883
884

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io

Canon debug messages and task switches:

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,tasks

885
Memory access trace (ROM reads, RAM writes) — very verbose:
alex@thinkpad's avatar
alex@thinkpad committed
886
887
888
889
890

.. code:: shell

  ./run_canon_fw.sh 60D,firmware="boot=0" -d romr,ramw

891
Call/return trace, redirected to a log file:
alex@thinkpad's avatar
alex@thinkpad committed
892
893
894

.. code:: shell

895
  ./run_canon_fw.sh 60D,firmware="boot=0" -d calls &> calls.log
alex@thinkpad's avatar
alex@thinkpad committed
896
897
898
899
900
901
902
903

Tip: set your editor to highlight the log file as if it were Python code.
You'll get collapse markers for free :)

Also with debug messages, I/O events and interrupts, redirected to file

.. code:: shell

904
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,calls,io,int &> full.log
alex@thinkpad's avatar
alex@thinkpad committed
905
906
907
908
909

Filter the logs with grep:

.. code:: shell

910
  ./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg,io |& grep --text -C 5 "\[Display\]"
alex@thinkpad's avatar
alex@thinkpad committed
911
912
913
914
915

Only log autoexec.bin activity (skip logging the bootloader):

.. code:: shell

916
  ./run_canon_fw.sh 60D,firmware="boot=1" -d exec,io,int,autoexec
alex@thinkpad's avatar
alex@thinkpad committed
917
918
919
920
921
922


Execution trace incomplete? PC values from MMIO logs not correct?
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

That's because QEMU compiles multiple guest instructions into a single TranslationBlock,
alex@thinkpad's avatar
alex@thinkpad committed
923
for faster execution. In this mode, ``-d exec`` will print guest instructions as they are compiled
alex@thinkpad's avatar
alex@thinkpad committed
924
925
926
(for example, if you have a tight loop, only the first pass will be printed).

To log every single guest instruction, as executed, and get exact PC values
alex@thinkpad's avatar
alex@thinkpad committed
927
928
in execution traces and other logs, you need to use ``-d nochain -singlestep`` 
(for example: ``-d exec,nochain -singlestep``) —
929
`source <http://qemu-discuss.nongnu.narkive.com/f8A4tqdT/singlestepping-target-assembly-instructions>`_.
alex@thinkpad's avatar
alex@thinkpad committed
930

931
Please note: ``-d io`` implies ``-d nochain -singlestep`` by default, to avoid this issue. Should you want to disable this,
alex@thinkpad's avatar
alex@thinkpad committed
932
to get faster emulation at the expense of incorrect PC values, use ``-d io_quick``.
933

alex@thinkpad's avatar
alex@thinkpad committed
934
Additionally, ``-d nochain`` implies ``-singlestep``, unlike in vanilla QEMU.
935

alex@thinkpad's avatar
alex@thinkpad committed
936
937
938
Debugging with GDB
``````````````````

alex@thinkpad's avatar
alex@thinkpad committed
939
940
941
.. code:: shell

    ./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
alex@thinkpad's avatar
alex@thinkpad committed
942

943
944
945
Logging hooks
'''''''''''''

946
947
948
949
950
951
952
Probably the most powerful feature of GDB is its scripting engine —
in many cases it's a lot faster than manually stepping over assembly code.
We may use it for tracing various function calls in the firmware, to understand what they do,
how to call them and so on. At any code address from the disassembly, we may set a breakpoint
and print some more info (such as function name, arguments, register values,
call location, DryOS task name and so on).

953
Predefined logging hook example (this goes into ``CAM/debugmsg.gdb``)::
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969

  b *0x8580
  take_semaphore_log

Custom logging hook (with colors)::

  b *0xE0008DA6
  commands
    silent
    print_current_location
    KRED
    printf "dryos_panic(%x, %x)\n", $r0, $r1
    KRESET
    c
  end

970
Look in `debug-logging.gdb <https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/qemu/contrib/qemu/scripts/debug-logging.gdb>`_
alex@thinkpad's avatar
alex@thinkpad committed
971
for common firmware functions you may want to log, and in ``*/debugmsg.gdb`` for usage examples.
972

alex@thinkpad's avatar
alex@thinkpad committed
973
You may also use `dprintf <https://sourceware.org/gdb/onlinedocs/gdb/Dynamic-printf.html>`_ if you prefer::
974
975
976

  dprintf *0x8b10, "[ %s:%08X ] task_create(%s, prio=%x, stack=%x, entry=%x, arg=%x)\n", CURRENT_TASK_NAME, $lr-4, $r0, $r1, $r2, $r3, *(int*)$sp

977
978
but the output won't be as pretty.

979
980
981
982
To log the return value of any function, use ``log_result`` in the definition of a logging hook.

For quick experiments, you may find ``generic_log`` and ``generic_log_with_result`` handy.

983
984
985
986
987
988
989
990
991
992
993
994
995
To get help on all of these user-defined commands at the GDB prompt:

  (gdb) help user-defined
  ...
  printf_log -- Log calls to plain printf
  register_cmd_log -- Log named functions registered to the DryOS shell
  xxd -- Memory dump formatted with xxd
  ...

  (gdb) help xxd
  Memory dump formatted with xxd. Temporary file: dump.tmp.
  Syntax: xxd startaddr [size]

996
997
998
999
Debugging symbols
'''''''''''''''''

There are no debugging symbols in Canon firmware, but you can import
1000
some of them from Magic Lantern (see comments in ``debugmsg.gdb``). Typically, you want to use one of these
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
`elf <https://jvns.ca/blog/2014/09/06/how-to-read-an-executable/>`_ files
from the platform directory:

- ``magiclantern`` if debugging regular Magic Lantern code (without modules)
- ``autoexec`` if debugging ML bootloader code — the `reboot shim <http://magiclantern.wikia.com/wiki/Autoboot>`_ (``reboot.c`` and related)
- ``stubs.o`` if debugging Canon firmware (add your symbols to ``stubs.S`` and recompile.)

For some reason, ML stubs are not recognized as functions in GDB;
to set breakpoints on function names, you need this trick::

  b *&task_create
  task_create_log

1014
1015
1016
Printing call stack from GDB
''''''''''''''''''''''''''''

1017
1018
The ``callstack`` option from QEMU (``eos/dbi/logging.c``) can be very useful to find where a function was called from.
This works even when gdb's ``backtrace`` command cannot figure it out from the stack contents, does not require any debugging symbols,
1019
but you need to run the emulation with instrumentation enabled: ``-d callstack``.
1020
1021
1022

Then, in GDB, use ``print_current_location_with_callstack`` to see the call stack for the current DryOS task.

1023
1024
1025
Tip: the callstack feature is enabled by default in ``assert_log``.

**Example for 80D:**
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061

The following goes into ``80D/debugmsg.gdb`` (modify the existing entry):

.. code::

  b *0xFE237C9E
  commands
    silent
    print_current_location_with_callstack
    printf "Memory region: start=%08X end=%08X flags=%08X\n", $r0, $r1, $r2
    c
  end

Start the emulation with:

.. code:: shell

  ./run_canon_fw.sh 80D,firmware="boot=0" -d callstack -s -S & arm-none-eabi-gdb -x 80D/debugmsg.gdb

Output:

.. code::

  ...
  Current stack: [2e9118-2e8118] sp=2e90c0                                         at [init:fe237c9e:fe238001]
  0xFE0D3385(0, fe0d3385, 19980218, 19980218)                                      at [init:8000173d:2e9108] (pc:sp)
   0xFE237E2D(0, feff65ab "Initialize SerialIO", 2e0f04, 44f4)                     at [init:fe0d403d:2e90f0] (pc:sp)
    0xFE237F93(18, 203a0, 0, 44f4)                                                 at [init:fe237e55:2e90d8] (pc:sp)
     0xFE237C9F(fe000000, ffffffff, 8, 5)                                          at [init:fe237ffd:2e90c0] (pc:sp)
  [        init:fe237ffd ] Memory region: start=FE000000 end=FFFFFFFF flags=00000008
  ...

The above shows the callers for the function being analyzed,
with 4 arguments (no attempts are made to guess the actual number of arguments)
and the locations for each call. You may examine these addresses in your disassembler.

1062
1063
GDB scripting docs
''''''''''''''''''
1064
1065
1066
1067
1068

- `Sequences <https://sourceware.org/gdb/onlinedocs/gdb/Sequences.html>`_ (command files, define, if, while, printf)
- `Convenience variables <https://sourceware.org/gdb/current/onlinedocs/gdb/Convenience-Vars.html>`_
- `GDB user manual <https://sourceware.org/gdb/current/onlinedocs/gdb/index.html>`_.

1069
1070
More examples
'''''''''''''
1071
1072
1073
1074

- `750D serial flash dumper <http://www.magiclantern.fm/forum/index.php?topic=17627.msg195357#msg195357>`_ (figuring out the parameters of an unknown function)
- `EOS M2 <http://www.magiclantern.fm/forum/index.php?topic=15895.msg186173#msg186173>`_ (examples with various GDB GUI front-ends):

alex@thinkpad's avatar
alex@thinkpad committed
1075
1076
|

1077
1078
1079
1080
.. image:: doc/img/ddd.png
   :scale: 50 %
   :align: center

alex@thinkpad's avatar
alex@thinkpad committed
1081

alex@thinkpad's avatar
alex@thinkpad committed
1082
1083
1084
1085
1086
Instrumentation
---------------

`TODO (see QEMU forum thread) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg184125#msg184125>`_

alex@thinkpad's avatar
alex@thinkpad committed
1087
1088
The instrumentation framework provides the following features:

alex@thinkpad's avatar
alex@thinkpad committed
1089
1090
- log all debug messages from Canon: ``-d debugmsg``
- log all memory accesses: ``-d rom/ram/romr/ramw/etc``
1091
- log all function calls: ``-d calls``
alex@thinkpad's avatar
alex@thinkpad committed
1092
1093
1094
1095
1096
- log all DryOS/VxWorks task switches: ``-d tasks``
- track all function calls to provide a stack trace: ``-d callstack``
- export all called functions to IDC script: ``-d idc``
- identify memory blocks copied from ROM to RAM: ``-d romcpy``
- check for memory errors (a la valgrind): ``-d memchk``
alex@thinkpad's avatar
alex@thinkpad committed
1097
1098
1099
1100

Debugging symbols from ML can be made available to instrumentation routines from environment variables (see ``export_ml_syms.sh``).

The address of DebugMsg is exported by ``run_canon_fw.sh`` (extracted from the GDB script, where it's commented out for speed reasons).
1101

1102
1103
Development and reverse engineering guide
-----------------------------------------
alex@thinkpad's avatar
alex@thinkpad committed
1104

1105
For the following topics, please see `HACKING.rst <HACKING.rst>`_:
alex@thinkpad's avatar
alex@thinkpad committed
1106

1107
1108
1109
- Finding your way around the source code;
- Adding support for emulating a new camera model;
- EOS (and PowerShot) firmware internals and reverse engineering tips.
alex@thinkpad's avatar
alex@thinkpad committed
1110
1111
1112
1113
1114

History
-------

:2008: `SD1100 boot (CHDK) <http://chdk.wikia.com/wiki/GPL_Qemu>`_
1115
:2009: `5D2/7D boot (Trammell Hudson) <http://magiclantern.wikia.com/wiki/Emulation>`_
alex@thinkpad's avatar
alex@thinkpad committed
1116
1117
1118
1119
1120
:2012: `TriX_EOS (g3gg0) <http://www.magiclantern.fm/forum/index.php?topic=2882.0>`_
:2013: `Initial Hello World <http://www.magiclantern.fm/forum/index.php?topic=2864.msg26022#msg26022>`_
:2013: `g3gg0 ports TriX changes to QEMU <http://www.magiclantern.fm/forum/index.php?topic=2864.msg29748#msg29748>`_
:2013: `Antony Pavlov submits initial DIGIC support to QEMU mainline <https://lists.gnu.org/archive/html/qemu-devel/2013-08/msg04509.html>`_
:2013: `Nikon Hacker is light years ahead us <http://www.magiclantern.fm/forum/index.php?topic=8823.0>`_ (we are not competing; it was just a fun notice that motivated us)
1121
:2014: `DryOS task scheduler running! <http://www.magiclantern.fm/forum/index.php?topic=2864.msg117430#msg117430>`_ (also ML menu and modules, but with massive hacks — emulating only a very small part of Canon firmware)
alex@thinkpad's avatar
alex@thinkpad committed
1122
1123
1124
1125
1126
:2015: `Portable display test and Linux PoC working! <http://www.magiclantern.fm/forum/index.php?topic=2864.msg144760#msg144760>`_
:2015: `Canon GUI boots on 60D! <http://www.magiclantern.fm/forum/index.php?topic=2864.msg148240#msg148240>`_ (no menus yet, but most Canon tasks are working!)
:2015: `100D emulation, serial flash and GDB scripts from nkls <http://www.magiclantern.fm/forum/index.php?topic=2864.msg153064#msg153064>`_
:2016: `More EOS models boot Canon GUI (no menus yet) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg168603#msg168603>`_
:2016: `Low-level button codes and GUI modes understood <http://www.magiclantern.fm/forum/index.php?topic=2864.msg169517#msg169517>`_
alex@thinkpad's avatar
alex@thinkpad committed
1127
:2016: `Users start wondering why the heck are we wasting our time on this <http://www.magiclantern.fm/forum/index.php?topic=2864.msg169970#msg169970>`_
alex@thinkpad's avatar
alex@thinkpad committed
1128
1129
1130
:2016: `Leegong from Nikon Hacker starts documenting MPU messages <http://www.magiclantern.fm/forum/index.php?topic=17596.msg171304#msg171304>`_
:2017: `500D menu navigation! (Greg) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg179867#msg179867>`_
:2017: `nkls solves an important issue that was very hard to track down! <http://www.magiclantern.fm/forum/index.php?topic=2864.msg183311#msg183311>`_
alex@thinkpad's avatar
alex@thinkpad committed
1131
:2017: `Menu navigation works on most DIGIC 4 and 5 models <http://www.magiclantern.fm/forum/index.php?topic=2864.msg181786#msg181786>`_
alex@thinkpad's avatar
alex@thinkpad committed
1132
1133
1134
:2017:  Working on `Mac (dfort) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg184981#msg184981>`_ 
        and `Windows 10 / Linux subsystem (g3gg0) <http://www.magiclantern.fm/forum/index.php?topic=20214.0>`_
:2017: `EOS M2 porting walkthrough <http://www.magiclantern.fm/forum/index.php?topic=15895.msg185103#msg185103>`_
alex@thinkpad's avatar
alex@thinkpad committed
1135
:2017: `Automated tests for ML builds in QEMU <http://www.magiclantern.fm/forum/index.php?topic=20560>`_
alex@thinkpad's avatar
alex@thinkpad committed
1136
1137
1138
:2017: `RTC emulation, many patches no longer needed <http://www.magiclantern.fm/forum/index.php?topic=2864.msg190823#msg190823>`_ (g3gg0)
:2017: `Major progress with DIGIC 6 emulation <http://www.magiclantern.fm/forum/index.php?topic=17360.msg194898#msg194898>`_
:2017: `Menu navigation works on 20 EOS models! (DIGIC 3-5) <http://www.magiclantern.fm/forum/index.php?topic=2864.msg195117#msg195117>`_
1139
:2018: `Initial support for DIGIC 7 EOS models (77D, 200D, 6D Mark II) <https://www.magiclantern.fm/forum/index.php?topic=19737.msg200799#msg200799>`_
alex@thinkpad's avatar
alex@thinkpad committed
1140
1141
1142
1143



Happy hacking!