System-on-Chip
(SoC) Design
EE382M.20,
Fall 2018
QEMU/SystemC Tutorial
Notes:
•
This
is a tutorial related to the class project.
•
Please
use the discussion board on Piazza for Q&A.
•
Please
check relevant web pages.
The
goal of this tutorial is to:
•
Give
an introduction to the QEMU/SystemC virtual platform
simulation environment.
This tutorial includes
the following:
•
A
simulation model of the Zynq development/prototyping
platform.
•
A
SystemC model of a simple hardware module mapped to
the Zynq's FPGA fabric.
•
An
application example running on the ARM that calls and interfaces with the
hardware.
We use the free QEMU
simulation model provided by Xilinx for their Zynq
platform for virtual prototyping of our SoC. More information
about Xilinx's QEMU setup is available through their wiki page and support forums.
We
will be using a special setup of QEMU provided by Xilinx that integrates with a
SystemC co-simulation environment. For QEMU/SystemC co-simulation, a separate SystemC
simulator instance is launched in parallel to the QEMU simulation, where the
two simulators communicate through special co-simulation ports that model the
available bus and other interfaces between the Zynq’s
programmable ARM subsystem and the FPGA fabric. On the SystemC
side, these interfaces are represented as standard TLM2.0 bus models and
sockets that other SystemC modules representing
blocks in the programmable logic can be connected and interfaced to in order to
construct a complete SoC virtual platform simulation.
More information about the QEMU/SystemC integration
from Xilinx is available in their
wiki page.
The following example
demonstrates a QEMU/SystemC simulation of a Zynq platform that includes a simple hardware module
implemented in the FPGA fabric, where the application running on the ARM
accesses the external hardware through memory-mapped I/O or a Linux kernel
module.
(a)
Environment
Setup
We
will be using the QEMU simulator that comes with the PetaLinux
tools provided by Xilinx. Log into the ECE-LRC servers (see instructions) and setup the
PetaLinux environment:
% module load xilinx/2017.4
% source /usr/local/packages/xilinx_2017.4/petalinux/2017.4/settings.[c]sh /usr/local/packages/xilinx_2017.4/petalinux/2017.4
% umask 022
Next,
either work with an existing or create a new PetaLinux
project:
% petalinux-create -t
project –n Lab2 -s
/home/projects/gerstl/ee382m/avnet-digilent-zedboard-v2017.4-final.bsp
% cd Lab2
Note
that PetaLinux creates a temporary directory under /tmp to work with. Since /tmp is not shared across
machines, you need to stay on the machine that you created the original project
on. If you want to switch machines, you need to create a new project (please
clean up any leftover old project directories under /tmp then – PetaLinux uses a lot of temporary disk space and /tmp fills up pretty fast
otherwise).
At
this point, you should be able to boot the pre-built image that comes with PetaLinux in their QEMU:
% petalinux-boot –-qemu –-prebuilt 3
You should then see
Linux booting in the terminal. At the prompt, login as 'root' (password 'root')
and you will have access to a bare-bone, minimal Linux installation (busybox) running on the emulated ARM platform. Note that to
end the simulation, you will have to terminate the QEMU simulation using Ctrl-A
X (Ctrl-C will not work).
(b)
Linux
Image
The
pre-built PetaLinux image boots into Linux using a
RAM disk as root filesystem, where any modifications will not be permanent and
space is limited. Furthermore, we need to be able to customize the image and
optionally boot into the same Ubuntu installation that is on the physical
boards. For that, we need to configure PetaLinux to
boot from an SD card image instead:
% petalinux-config
Select Image Packaging Configuration->Root
filesystem type->SD Card
Unfortunately, the
QEMU installation that comes with PetaLinux does not
fully emulate the SD card hardware, so we need to patch the Linux kernel to
bypass some of the hardware checks:
% cp /home/projects/gerstl/ee382m/linux-xlnx-QEMU-SD.patch
project-spec/meta-user/recipes-kernel/linux/linux-xlnx
% vi
project-spec/meta-user/recipes-kernel/linux/linux-xlnx_%.bbappend
Replace SRC_URI += "file://bsp.cfg"
with SRC_URI +=
"file://bsp.cfg file://linux-xlnx-QEMU-SD.patch"
% petalinux-build –c kernel
With this, you can
use a pre-built SD card image that we created to boot into a minimal busybox Linux installation that includes everything needed
to run YOLO/Darknet for Lab 2:
% cp /home/projects/gerstl/ee382m/images/zed_EE382M_busybox_2017.img images/linux
% petalinux-boot --qemu
--kernel --qemu-args "-drive
file=images/linux/zed_EE382M_busybox_2017.img,if=sd,format=raw,index=0 -boot
mode=5"
You can optionally
also create your own customized Linux image. If you do, make sure to include
the necessary libraries (libstdc++) needed to run
YOLO/Darknet:
% petalinux-config
–c rootfs
Select Filesystem Package -> misc
-> gcc-runtime -> libstdc++
Select Filesystem Package -> misc
-> gdb -> gdbserver
Select apps -> peekpoke
Select other optional
packages as desired…
% petalinux-build –c rootfs
Create the SD card
image containing two partitions (for /boot and the root filesystem) and copy
the files on there and boot the image:
% guestfish -N
bootroot:vfat:ext4:200M -m /dev/sda1 upload images/linux/zImage /zImage : upload images/linux/vmlinux /vmlinux : upload images/linux/system.dtb /system.dtb : exit
% guestfish -a test1.img run : upload images/linux/rootfs.ext4 /dev/sda2
% guestfish -a test1.img run : e2fsck /dev/sda2 correct:true
% guestfish -a test1.img run : resize2fs /dev/sda2
% mv test1.img images/linux/zed_EE382M_busybox_2017.img
% petalinux-boot --qemu
--kernel --qemu-args "-drive
file=images/linux/zed_EE382M_busybox_2017.img,if=sd,format=raw,index=0 -boot
mode=5"
Alternatively, you
can boot the standard Ubuntu image that is running on your physical boards. You
will have a full Linux environment, but boot times will be much slower. For
Ubuntu to work, we first need to re-configure the kernel to include the
required SquashFS compressed file system:
% cp /home/projects/gerstl/ee382m/linux-xlnx-ubuntu.cfg
project-spec/meta-user/recipes-kernel/linux/linux-xlnx
% vi
project-spec/meta-user/recipes-kernel/linux/linux-xlnx_%.bbappend
Replace SRC_URI += "file://bsp.cfg
file://linux-xlnx-QEMU-SD.patch"
with SRC_URI += "file://bsp.cfg
file://linux-xlnx-ubuntu.cfg file://linux-xlnx-QEMU-SD.patch"
% petalinux-build –c kernel
Then, copy the Ubuntu image and boot it:
% cp
/home/projects/gerstl/ee382m/images/zed_16G_UBUNTU.18.04_MASTER_CLASS.img
images/linux
% petalinux-boot --qemu
--kernel --qemu-args "-drive
file=images/linux/zed_16G_UBUNTU.18.04_MASTER_CLASS.img,if=sd,format=raw,index=0
-boot mode=5 -nographic"
(c)
SystemC Co-Simulation
As described by
Xilinx for their QEMU/SystemC co-simulation setup, to make the external SystemC interfaces accessible in the QEMU simulation, we
first need to update the Linux device tree:
% wget
https://raw.githubusercontent.com/Xilinx/qemu-devicetrees/master/zynq-pl-remoteport.dtsi
-O
components/plnx_workspace/device-tree/device-tree-generation/zynq-pl-remoteport.dtsi
% vi
project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
Add /include/ "zynq-pl-remoteport.dtsi" as the last, i.e. below
all other includes
% petalinux-build -c device-tree
Then launch the QEMU
simulation with extra co-simulation arguments including a pointer to a
directory used for exchange of information between the two simulators. You can
again boot either the busybox or Ubuntu image. Using
the busybox image:
% mkdir ../qemu-tmp
% petalinux-boot --qemu
--kernel --qemu-args "-drive
file=images/linux/zed_EE382M_busybox_2017.img,if=sd,format=raw,index=0 -boot
mode=5 -machine-path ../qemu-tmp
-icount 1 -sync-quantum 10000"
The icount and sync quantum parameters allow you to play with
the tradeoff between simulation speed and accuracy.
QEMU will now start
but immediately hang waiting for a connection from the SystemC
simulator. To setup the SystemC simulation, we need
to include a special co-simulation
library provided by Xilinx. For this example and tutorial, we will be using
a small SystemC demo platform provided by Xilinx that
is built on top of this library. Download and compile the Xilinx SystemC co-simulation demo:
% cd ..
% git clone https://github.com/Xilinx/systemctlm-cosim-demo.git
% cd systemctlm-cosim-demo
% git submodule update --init
libsystemctlm-soc
% vi Makefile
Change SYSTEMC ?= /usr/local/systemc-2.3.1/
To
SYSTEMC
?= /usr/local/packages/systemc-2.3.1/
% make
Then, launch the SystemC side of the co-simulation:
% env LD_LIBRARY_PATH=/usr/local/packages/systemc-2.3.1/lib-linux64/ ./zynq_demo unix:../qemu-tmp/qemu-rport-_cosim@0 10000
The SystemC simulator will start and in turn wait to try to
establish the connection to the QEMU simulator. Once the connection is made,
both simulators will proceed, and the QEMU side should continue to boot Linux
as normal (but slower). Note that the last argument is the sync quantum, which
should match the value passed to QEMU.
This SystemC demo example includes a simple debug device (see debugdev.h/.cc) attached to the
main system bus (see the top-level zync_demo.cc SystemC module,
which stitches everything together). The debug device is accessible on the
following bus addresses:
0x40000000
– Read SystemC time (in seconds) / Write debug
message and measure time
0x40000004 – Write output ASCII character
to terminal
0x40000008 – Write to stop the simulation
via an exit(1) call
0x4000000c – Read/Write status of
interrupt line (high/low, i.e. 1/0)
0x40000010 – Read SystemC
clock() count
You can use the peek and poke commands from within
the busybox Linux shell to test reading/writing
from/to these addresses.
(d)
Application
software
We demonstrate
application development and HW/SW interfacing using a simple software
application running on the virtual platform that interfaces with the debug
hardware device modeled in SystemC. To compile and
link applications for the board, we need to use the ARM cross-compiler tool
chain provided by Xilinx together with their overall tool environment, i.e.
first make sure that the PetaLinux environment is
setup correctly (see step (a) above):
% module load xilinx/2017.4
% source /usr/local/packages/xilinx_2017.4/petalinux/2017.4/settings.[c]sh /usr/local/packages/xilinx_2017.4/petalinux/2017.4
Then unpack the
application example (QEMU_SystemC_app.tar.gz)
and cross-compile the executable (example) for the ARM using the provided Makefile:
% tar xzf /home/projects/gerstl/ee382m/QEMU_SystemC_app.tar.gz
% cd application
% make
Next, we need to copy
the cross-compiled application into the simulated platform. The virtual
platform simulator includes emulated network access, so the easiest way to do
this is by copying the executable into the simulated platform over the network.
From within the simulation execute (you can use either scp or wget):
# scp
<user>@<server>.ece.utexas.edu:<path>/application/example
.
Finally, run the
application example from there:
# ./example [<val>]
The example (see example.c) demonstrates use of
memory-mapped I/O under Linux to access the debug device registers and either
read the SystemC time or write to the debug port
similar to the peek and poke commands above. See
the included README.application for more details.
(e)
Interrupt-driven
application using a Linux kernel module
The application
package also includes a version of the example code that uses a kernel module (fpga_drv.c) to implement an
interrupt-based device driver for all accesses to the FPGA hardware. For the
kernel module to work, we first need to tell the Linux kernel about the
existence of the external hardware device. Go into your PetaLinux
project directory from steps (a)-(c), and either copy the
‘components/plnx_workspace/device-tree/device-tree-generation/zynq-fpga.dtsi’
included with the application package into your project directory, or just
unpack the application package right into the PetaLinux
project tree:
% cd Lab2
% tar xzf /home/projects/gerstl/ee382m/zynq-fpga-example.tar.gz
--strip-components=1
Update the Linux device tree to include the
additional ‘fpga’ device:
% vi project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
Add /include/ "zynq-fpga.dtsi"
% petalinux-build -c device-tree
Finally, (re)start
the QEMU+SystemC simulation of our (virtual)
platform.
Then cross-compile
the executable (example) and the kernel
module (fpga_drv.ko) of the interrupt-driven application for the
ARM:
% cd application.irq
% make
Copy both files into
the virtual platform as shown above. From within the simulation, load the
kernel module, check that the device driver is properly installed and
registered for GIC interrupt 61, and look at the output reported by the driver
under its /proc/fpga entry:
# insmod
fpga_drv.ko
# lsmod
# cat /proc/interrupts
# cat /proc/fpga
Finally, run the
application example:
# ./example [<val>]
In addition to the
output from before, you should see messages from the 'fpga_drv'
about I/O accesses, including handling of incoming interrupts for
synchronization with the hardware.
In the following, we
describe various options that are available for debugging both the SystemC side of the virtual platform as well as the
application running on the simulated ARM.
(a)
SystemC debugging
By
default, the SystemC/C++ model of the virtual
platform is already compiled with debug information (-g option) enabled.
However, you may want to also disable compile optimizations (remove the
–O2 switch in the Makefile and recompile). The
‘zynq_demo’ executable can then be
executed in your favorite debugger, such as GDB:
% LD_LIBRARY_PATH=/usr/local/packages/systemc-2.3.1/lib-linux64/
gdb
zynq_demo
Or, if you prefer a graphical
debugging environment (using the DDD graphical GDB frontend):
% LD_LIBRARY_PATH=/usr/local/packages/systemc-2.3.1/lib-linux64/
ddd
zynq_demo
Finally, inside the
debugger, launch the ‘zync_demo’ program
with the correct command line arguments:
(gdb) run unix:../qemu-tmp/qemu-rport-_cosim@0 10000
From there on, you
can debug the virtual platform’s SystemC/C++
executable using standard means, e.g., Ctrl-C will interrupt the running
program and bring you back to the (gdb) prompt.
(b)
Application
debugging
The
Linux installation running on our platform includes a GDB server,
which can be used to attach an external GDB instance and remotely debug an application
running on top of it. In order to do that, first make sure that your
application is compiled with debug information (-g switch enabled). In case of
the given application example using the provided Makefiles:
% cd application[.irq]
% make debug
Then, (re)start the virtual platform and forward the debug port in
the platform to any available port on the local simulation host:
% cd ..
% petalinux-boot --qemu --kernel --qemu-args
"...
–netdev user,id=gdbsrv,hostfwd=tcp::<port>-:1234"
Copy your application
into the platform and start it there under the control of the GDB server:
# gdbserver
HOST:1234 example
Finally, open another
shell on the host and launch an ARM cross-debugger instance pointing it to the
same application binary (with compiled-in debug information):
% arm-linux-gnueabihf-gdb application[.irq]/example
Inside GDB, connect
to the (forwarded) remote server and run the application under control of the
cross-debugger:
(gdb)
target remote localhost:<port>
(gdb)
b main
(gdb)
c
(c)
Kernel
debugging
If
you want to venture into debugging the Linux kernel, including any applications
it launches, QEMU supports the capability to attach a remote debugger to the
simulator itself:
% petalinux-boot --qemu --kernel --qemu-args
"... –gdb tcp::<port> -S"
Note that with the
–S option, QEMU will not boot any code until a remote debugger is
attached. If you want to be able to use source information, make sure whatever
code you want to debug is compiled with –g and point the cross-debugger
to it, e.g.:
% arm-linux-gnueabihf-gdb application[.irq]/example
Finally, connect the
cross-debugger to QEMU and start execution:
(gdb)
target remote localhost:<port>
(gdb)
b main
(gdb)
c
You can then use the
virtual platform normally, while being able to control the emulated code from
the remote debugger (e.g. Ctrl-C will stop the simulated ARM and bring it back
under GDB control). Setting a breakpoint as shown above will trigger when the
application you pointed the debugger to is launched inside the platform (# ./example) and it hits the
given source line.