Source level debugger?

Requests for features
Post Reply
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Source level debugger?

Post by siggi »

While doing some frustrating "printf"-debugging with my ZX81 to find errors in a FAT16 filesystem package, I thought it would be nice to debug on source level and to single-step through C statements.

Since this debugger should also help to debug hardware-dependent programs, it must run on the target (not on an or together with an emulator).
Since this debugger should work on the target to debug a program on the target, it's size (and thus its features) may not be too big:
I think, that the functions
- STEP
- STEP-INTO
- STEP-OUT
- STEP-OVER
and to dump memory

would help a lot.

This debugger should run at various memory addresses on the target, depending on the memory space not used by the program under test.

Any chance for that?

Siggi

Addendum: some links to compilers with source level debugger:

http://sdcc.sourceforge.net/

http://www.htsoft.com/products/compilers/Z80c.php
Last edited by siggi on Sun Feb 10, 2008 11:44 am, edited 1 time in total.
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

Do you think the "debug" library can help ? ;)

http://www.z88dk.org/wiki/doku.php/library:debug
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

Yes, if the debugger has got the control. My idea was something like that (addresses used fit to my ZX81):

The single-step debugger is a small program e. g. at 8192.
The program to be tested is compiled with DEBUG option 8192 and runs e. g. at 32768.

Then the compiler adds a

call 8192 (location)

into the generated code at "locations" (e. g. line numbers), that are passed to the debugger. The "locations" I can find in the source file (e. g. line number).
The called debugger then could show (on ZX81: after having switched to SLOW mode / lowres display ;-)
"Hi, welcome at location ..."
Then I could use the debug library to examine the real code at "location (or elsewhere) and data and parameters and look into the source file, what should happen there.

A STEP command would return to the application, which calls 8192 again at the next "location" (e. g. after the next source line)- > debugger runs again.
A GO command would also return, but the later called debugger would not enter command mode, but return immediately (so the application runs nearly in real time)
The debugger could also hold a list of "breakpoints" (several "locations"), at which, when execetuded, the debugger would go to command mode for user intervention.

So I could STEP through my application line by line or set breakpoints anywhere (at line level) to enter the debugger when the application is there.

Should be doable????

Siggi
Last edited by siggi on Wed Jan 30, 2008 3:56 pm, edited 1 time in total.
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Post by dom »

Gunther wrote one call Intuition for the z88: http://www.worldofspectrum.org/z88forever/z88wbrev.htm

I've never used it, but by all accounts it was pretty good. I might have the source code kicking around with the native z80asm source code, failing that he may well allow distribution if we ask.
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

As I understood this is a debugger on assembler level. But my thoughts were to have one at C level. Thus a STEP is not the execution of one machine opcode, but the execution of a complete C line (or parts of it).

And the Z80 is "emulated" with this tool. So time critical subroutines (as I have in my program to be debugged: access to MMC cards using my mmc-interface requires real-time operation) will probably fail :(

Siggi
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

I've been thinking at it for a while.
The biggest problems I see are:
- having a source level debugger probably means to have some of the source text lines embedded in the resulting code; this costs a lot in terms of memory.. otherwise could a simple line number marker be enough ?
- the resulting code HAS TO be different when running in the debugging mode: this could be a problem when you're trying to spot some odd behaviour.
- if someone would ever want to add such functionality, a serious definition of the "overall requirements" is a must, before starting developement.
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

stefano wrote:I've been thinking at it for a while.
The biggest problems I see are:
- having a source level debugger probably means to have some of the source text lines embedded in the resulting code; this costs a lot in terms of memory.. otherwise could a simple line number marker be enough ?
- the resulting code HAS TO be different when running in the debugging mode: this could be a problem when you're trying to spot some odd behaviour.
All the source level debuggers I used during the last years (running on UNIX or the PC to debug 16 or 32 bit targets) allowed to set breakpoints in the code to be debugged by double-clicking on a line in the C source file. So a STEP from line to line is sufficient.

All programs compiled for debugging contained additional symbolic information (not the source code itself, but references to it) necessary to debug them. So we compiled a "debug image" for debugging, which often was too big to run on the real target. Then we had to use special "debug targets" with more RAM. When the program was bug-free :lol: , it was compiled as "release image" to run on the normal target (and sometimes came back for further debugging :) )
- if someone would ever want to add such functionality, a serious definition of the "overall requirements" is a must, before starting developement.
Here is the first attempt of requirement specs:

1) The "debug kernel" running on the target must be small: < 2/4/8K (so probably needs to be written is assembler). It should be run at locations, where the target has free ram (so maybe needs to be assembled for each location or needs to be relocated)

2) It should support a list of e. g. 10 BREAKPOINTS

3) Commands for the debug kernel are:
- SET/CLEAR (maybe also ACTIVATE/DEACTIVATE) a breakpoint at address XX: put XX into/out of the list of BREAKPOINTS where the program should be stopped
- STEP: go to next line at same program level
- INTO: go one level down (e. g. step into a function and stop at next line there)
-OUT: leave current level (e. g. within a function) and stop at next line at level above (after the function call). So the rest of the function runs in real time (GO-Mode), until the upper level (detected by evaluation of the stack pointer???) is reached.
-GO: run nearly in real-time until end of program or an active breakpoint is reached.

4) More commands to show variables or disassemble memory would make sense

When a breakpoint or the next step (at next line) is reached: show current address XX (maybe the 16-bit call-address on the stack, when the debugger is called by additional code in the program to be debugged, or maybe a far-address, passed as parameter from the debugged application) and wait for the next command.

The source-level debuggers I know have a data link (ethernet, serial or parallel port) between the target and the host debugger (having the user interface). Using that they fetch the address XX from the target to display the location, where the target stopped, in the source file on the host.

We don't have that link, so we must use the displayed XX address to find the location, where we are, in the source file. Maybe a tool on the host may do that: convert address XX (maybe using additional information from the MAP file) to name the source file and line number of the location (there are text editors, which can be started with parameters "filename", "line" and "column" to open a source file and set the cursor to line/column).

And we need a tool for the opposite direction: convert a line-numer within a source file to a breakpoint-address XX to be entered into the debugger-kernel on the target (SET BREAKPOINT XX)

Furthermore the compiler has to to add code tho the debug image, to call the debug-kernel and to tell him the address (maybe the real return address on the stack is sufficient, or a far-adress is passed as parameter).

Some examples, where this call should be inserted in the code: [xx] indicates such a location:

[XX] if (expression) a(); else b():

[XX] statement;

[XX] if (expression)
[XX] a();
else
[XX] b();

An easy rule would be to add those calls after each ";" in the source file. But then we might have several STEPS within one line, which might be too complicated.

Since we cannot add more ram to run a "debug image" on our real targets (my ZX81), it might be necessary to compile only a part of the source (containing the bug :) ) for debugging (to save ram).
So the debug-option should be turned on/off by compiler options within the source files.

What do you think about that?

Siggi
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

Hello!

Great initiative to discus debugging. Its the thing that really makes one compiler useful over another!

My idea would be to implement this as an embedded debugger.

First thing I'd better admit it's because "my" target (the Rabbit in case any1 missed it ;-) is equipped with a serial port.

The embedded debugger relies heavily on the host machine to do the crunching of symbols etc. This makes sense since we have like
5-6 orders of magnitude (100.000 - 1000.000) times worth of CPU-power here than on the target machine. With me so far?

The target machine would be equipped with a serial or otherwise connection to the host. (virtual in the case of an emulator or if one
really wants to go wild with one of those ZX81 boards I built at twelwe, equip it with a simple debug port, a few 74HCXX and away you go)

On the target one also need a very small monitor, this could be a binary protocol containing commands to read and modify memory, registers and I/O and also to start execution at a given point. One also needs to decide about a good breakpoint mnemonic, such as one of the rst XXh commands.

This monitor can be implemented with a fixed-size for all programs at around 1k of mem (if you need to make it larger you weren't with the gang in the happy 80's ;-)

One such protocol is the NoICE protocol, (check out http://www.noicedebugger.com/)

There are of course other options, we could cook one ourselves its really simple.

So for the fun part. We need those clever brains here that designed (or at least understand) the compiler to come in and do some
work (translator and compilers are beyond me, I never took that fancy course in university ;-)

What is needed is a software module that will take input and translate it to monitor commands.

So when we say "break my_function" or similar, the host should talk to the target and instruct it to set the breakpoint at the machine-code
level to that point.

Then we might want to inspect the value of variable "x". The debugger would then make an inquiry to the register or mem address that contains that value, translate it in someway if its a float, for instance. In case of a struct, we might have to do several readings towards the target.

The nicest part here is that the code being debugged is also the code being run as the final result, thus we eliminate some of those frustrating "Heisenbugs" that seem to dissapear when not debugging and vice versa.

Anyway, the point is: try to keep debugging on the host side, since it tends to chew up big resources in order to be useful!

A serial port or a debug port in the emulator is a small price to pay for this feature and it would also make the compiler-core gang happier to get more elbow room for the debugger

Sincerly /Erik

P.S. I didn't find this forum until today so now I can avoid blowing the email-list with out-of-sequence stuff
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

The good message is: all ZX81 and similars already have a "serial" port, that could be used: their CASSETTE interface!

Some bytes of m/c would be necessary to send/receive a byte. And using the "supertape" (or similar) technology, it would be fast enough for debugging.

We would only need an external amplifier to make the signals "pc compatible" ;-)

Siggi

PS: My Zeddy has in addition to that 2 real RS232-ports, most ZX96 have one ...
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

A few things come to mind!

It might not be sufficient to set the breakpoint at a certain line number since C allows you to put several statements on one line
so if someone does something silly like this:

printf("One statement"); for (i=0;i<10;i++) { printf ("several prints\n"); }

This line would require three breakpoints to cover all cases

C also allows one statement to spread over several lines:

some_function_with_lots_of_params
(
a,
b,
c,
d
);

C is not basic after all.


This might not be a big problem, a larger problem is the peep-hole optimizer.

All statements translate to a series of assembler lines, maybe something like this

ld hl,my_string
call printf

ld a,0
loop385:
ld hl,more_text
call printf

The peephole optimizer might change the order of things if it thinks it is more efficient, so we might run into trouble
when trying to set the breakpoint at a certain (assembler) line since it is not clear where one statement begins and one
other ends, they can be intermixed.

One could disable the peep-hole optimizer. Ok, this brings back the Heisenbugs and makes the code larger
but that could be an ok simplification for a first attempt.

So without the peep-hole opt. we need to be able to modify the compiler to answer us where in the assembler code we should
set the breakpoint for it to break at a certain statement in the C-code.

We need both the hex address of the machinecode breakpoint, but also it is useful to get the line and file of the assembler output!!

Then it is useful to examine the values of variables.

Here we might use an idea I been working with. Suppose we could make the compiler generate "as-if" code, which will aid us
in the debugging

If we want to access the value of the variable 'abc' in the current scope, we might compile the following code:

/** This code belongs to the actual program */
int abc=10;
abc += 10;
abc = 2*abc;

/** This code is on-the-fly compiled when the debugger (running on the host machine) gets the command to
display the value of variable abc
*/
{
/* I have introduced the non-ansi operator "typeof" which returns the type of variable a, say 'int' in this case. */

typeof(abc) * virtual_store=0; /** We never want to run this on target since we don't know whats at this address */
*virtual_store=abc; /** This code might get more complex if we try to read structs strings etc... */
}




This might compile into, (assuming the value of abc is held in the top of the stack at this moment)

pop de ; this value we fetch from target, modifying emulator stackpointer
push hl ; only put on emulator stack
ld hl,0 ; load to emul hl
ld (hl),e ; load to emulator memory
inc hl ;
ld (hl),d ; "
pop hl ; this is not neccessary, but we don't "know" that since we trust the compiler
push de ; "

Without knowing how its done, the debugger will now always have the 'abc' variables 16-bit value in emulator memory at address 0 and 1
(big endian?)


This code needn't be run on the target, my idea was to use an emulator for this, everytime we need to access a register value or a
place in memory we should use the serial protocol for this.

In this example most stuff can be accessed just by obtaining a register dump and then we need to make some stack operations as well.

stack operations can be performed virtually except for the pop de, which would need a target access at that memory.
The emulated stack needs to remember its changes and also never modify the targets stack, since all of these instructions are "make-belive"
and only aiming at lazily obtaining values of variables.


Pthew, My thoughts so far, what do you think, can this work???
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

mimarob wrote:It might not be sufficient to set the breakpoint at a certain line number since C allows you to put several statements on one line
so if someone does something silly like this:

printf("One statement"); for (i=0;i<10;i++) { printf ("several prints\n"); }
As written before: debuggers I know allow only breakpoints at line level. If someone wants to write ugly C programs, the then has less chances to debug them easily ... :)
mimarob wrote:This line would require three breakpoints to cover all cases

C also allows one statement to spread over several lines:

some_function_with_lots_of_params
(
a,
b,
c,
d
);
Other source level debuggers allow only to set breakpoint at lines, which corresponds to executable code. So "clicks" at lines containing declarations or comments result in setting a breakpoint at the next following line with executable code.

mimarob wrote:This might not be a big problem, a larger problem is the peep-hole optimizer.

All statements translate to a series of assembler lines, maybe something like this

ld hl,my_string
call printf

ld a,0
loop385:
ld hl,more_text
call printf

The peephole optimizer might change the order of things if it thinks it is more efficient, so we might run into trouble
when trying to set the breakpoint at a certain (assembler) line since it is not clear where one statement begins and one
other ends, they can be intermixed.

One could disable the peep-hole optimizer. Ok, this brings back the Heisenbugs and makes the code larger
but that could be an ok simplification for a first attempt.
So the optimizer has to learn to optimize only code between breakpoints, not across them ...
mimarob wrote:Then it is useful to examine the values of variables.

Here we might use an idea I been working with. Suppose we could make the compiler generate "as-if" code, which will aid us
in the debugging

(..)

This code needn't be run on the target, my idea was to use an emulator for this, everytime we need to access a register value or a
place in memory we should use the serial protocol for this.
I dont like emulation in that case, because the reality happens on the targent, not in the emulating PC!
If the NMI-Routine running on the target destroys my variables or registers, it happens only on the target ....
If an "escpaped" pointer runs througth the memory and writes there, the memory on the target is overwritten, not in the PC...

I want to test the real hardware with all components, which are not all emulated by the PC!

So the information needed should always be fetched from the target. There is the reality!

Siggi
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

Hello again! Thank you for your comments!

The first and second sections about ugly C-code I totally agree.

The third section about the optimizer, I am not even sure that it works the way I described, but ok, it should not be difficult to let the optimizer only optimize inside a statement (maybe thats what it does already?). Also it should be possible to obtain statement limits, the compiler wont care about the structure of the code so in the silly case of two statements on one line it will simply return the same line number twice.

On the forth section, I better rephrase myself, I was merely speculating on _how_ to get the debugger to fetch the value of variable X. We probably need to save a ton of debugging information from the compiler, symbol tables, adresses etc.
That ton would be great if we could have on the host machine instead of the target, right?
As long as the variables are stored in memory, it should be easy to get them using the symboltable files .map and .sym that the compiler already has! Another story is when variables are stored on the stack or as a register, it takes a whole lot of clever to find out that we should get, say, the forth element of the stack in order to get the value of pointer P.


As for the tedious debugging of NMI routines, I agree, it only happens on the target, but maybe an assembler debugger is what we need in that case anyway.

An assembler debugger is much easier to write than a C-level one!
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

mimarob wrote:Another story is when variables are stored on the stack or as a register, it takes a whole lot of clever to find out that we should get, say, the forth element of the stack in order to get the value of pointer P.
Indeed, this is a little bit more complex. But if the user wants to know these values on the stack: why not? Transfer them (e.g. 128 byte) from the stack via the serial port to the pc to be interpreted there. The user then would have to wait some 100 milliseconds more ... :)
mimarob wrote:As for the tedious debugging of NMI routines, I agree, it only happens on the target, but maybe an assembler debugger is what we need in that case anyway.
My thoughts were to debug a foreground program, which is in background affected by an interrupt-routine. To debug the interrupt-routine itsef: if it's written in C, it could also be debugged at source level (if it does not block/disable the serial transmission during interrupt).

Siggi
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

hmm to get this going so people can test it, contribute and enjoy it, we really need some common platform, not everyone is gifted
with a rabbit or a Zeddy.

Is there any good emulated platform that would allow me to use a "serial port" and program the target, something simple.. I looked into Mame before but its rather huge...

/Erik
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

Hey, hey.. don't go too far !
Let's first focus on the current kit release...
siggi
Well known member
Posts: 344
Joined: Thu Jul 26, 2007 9:06 am

Post by siggi »

I think, this feature request should be moved to the feature request list for Z88DK V 2.8 ;-)

Siggi
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

Hello stef, I was not targetting this release, I'm just restless since I'm happy with the rabbit stuff done sofar, all things on my todo list is to long to fit into this release anyway...

I also found a great platform for the debugger hacking...

The test/ directory with the "machine" simulator has a stdin/stdout which can be used for playing with some initial stuff. Since it is rude to
hijack stdin/stdout for debugging, it can later be complemented with a debug port.

I'll dust of my expect script hacking and maybe I'll kludge something together without touching the compiler at all!
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Post by stefano »

Really ? I thought the compiler _had_to_ be touched, but I think I see our point :)

I tried to do something similar to support the multi-dim array style syntax, then since it was unsuccesful I tried to hack the compiler, and got some first weak result, but I broke it :(
mimarob
Member
Posts: 26
Joined: Tue Feb 26, 2008 12:46 pm

Post by mimarob »

Hello again!

My quick-hack has now extended over 2.5 months but finally I think I can show some useful results!

I made some code which demostrates remote debugging on the z88dk/test/machine platform.

I made alterations to 10-12 different files and added a few of my own!

It can currently only set one breakpoint and cannot read/alter variables!

Look at the screen dump: http://www.algonet.se/~mimarob/z80dbg.jpg

Worth integrating towards main trunk?

/Erik
Post Reply