Issues when migrating back to Classic from newlib

Requests for features
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Issues when migrating back to Classic from newlib

Post by jorgegv »

Hi,

I'd like to share some issues I have faced when I started migrating my library from SDCC+NEWLIB to SDCC+Classic. I have found workarounds for all but the last one, but I thought it would be good to share them here and get feedback. Also, if some of them are wrong, please tell me so :-)

So here are my findings:
  • SCCZ80 apparently supports only C89/C90, not C99. This means that designated initializers for const static unions or structs are not supported. The following compiles OK with SDCC but does not compile with SCCZ80:

    Code: Select all

    struct data {
        char a;
        int b;
    };
    
    struct data instance = { .b = 16, .a = 'A' };
    
    void main(void) {
    }
    
  • This also means that you cannot statically initialize any fields in a union except the first one (as per C89). For the moment this forces me to use SDCC (which is no problem, since I prefer it).
  • BeepFX routines (`bit_beepfx` and pals) are available in NEWLIB but not in Classic
  • Current SP1 header file (sp1.h) for Classic does not compile with SDCC, but if it is replaced with the NEWLIB one it compiles OK (using Classic). It seems some `__FASTCALL__` decorators should be replaced with `__z88dk_fastcall` ones
  • M_BEGIN_ISR and M_END_ISR macros in Classic's im2.h seem not compatible with SDCC, they use 'asm("...")' syntax. If it is replaced by '__asm__("...")' they work OK.
Finally: I have a custom mmap.inc file which was shamelessly copied and modified from stock NEWLIB configuration. But I have not found the equivalent file in Classic, so I'm stuck at this point. Any help with this, please?

Thanks in advance, and as I said, any feedback is welcome.

J.
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

Ah, designated initialisers - I had an issue open https://github.com/z88dk/z88dk/issues/1144 and then I closed it in a fit of grumpiness last year.

Thanks for the heads up on the other things, I'll see what I can do with beepfx and the others I'll fix up this weekend.

The classic memory maps start here: https://github.com/z88dk/z88dk/blob/mas ... ection.asm
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Hi @dom, would you accept Pull Requests over Z88DK? I'd be happy to help you with this.

Regarding the memory map, here is my second batch of Classic vs. Newlib issues: :-)
  • Issue 1: When using standard memory map (no custom __MMAP pragma), when I use

    Code: Select all

    #pragma output CLIB_FOPEN_MAX         = -1      // no FILE* list
    
    ...I get the following error:

    Code: Select all

    Bulding main.bin....
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/crt/classic/crt_section.asm:109: error: integer range: -10
      ^---- __sgoioblk: defs CLIB_FOPEN_MAX * 10 ;stdio control block
          ^---- __sgoioblk:defs CLIB_FOPEN_MAX*10
    make[2]: *** [Makefile-48:42: main.bin] Error 1
    make[1]: *** [Makefile-48:27: build] Error 2
    make: *** [Makefile:66: build48] Error 2
    
  • Issue 2: standard memory map for Classic (crt_section.asm) starts with this:

    Code: Select all

    ; Include the default memory map. You can override this
    
    IF __MMAP == -1
        ; The user has supplied a memory map.
        INCLUDE     "./mmap.inc"
    ELSE
        ; Include the standard memory map
        INCLUDE     "crt/classic/crt_section_standard.asm"
    ENDIF
    
    So it seems reasonable that if I copy crt_section_standard.asm to mmap.inc (with no modifications) and define pragma __MMAP to -1, nothing should change and everything should build exactly as before, since the files are the same...

    ...but instead I get the following errors (you can ignore the FOPEN_MAX one, it's the same as in issue 1 above):

    Code: Select all

    /home/jorgegv/src/spectrum/z88dk/lib/../lib/crt/classic/crt_section.asm:109: error: integer range: -10
      ^---- __sgoioblk: defs CLIB_FOPEN_MAX * 10 ;stdio control block
          ^---- __sgoioblk:defs CLIB_FOPEN_MAX*10
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:19: error: undefined symbol: CRT_ORG_BANK_0
      ^---- CRT_ORG_BANK_0
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:20: error: undefined symbol: CRT_ORG_BANK_0
      ^---- __BANK_0_END_tail-CRT_ORG_BANK_0
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:24: error: undefined symbol: CRT_ORG_BANK_1
      ^---- CRT_ORG_BANK_1
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:25: error: undefined symbol: CRT_ORG_BANK_1
      ^---- __BANK_1_END_tail-CRT_ORG_BANK_1
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:29: error: undefined symbol: CRT_ORG_BANK_2
      ^---- CRT_ORG_BANK_2
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:30: error: undefined symbol: CRT_ORG_BANK_2
      ^---- __BANK_2_END_tail-CRT_ORG_BANK_2
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:34: error: undefined symbol: CRT_ORG_BANK_3
      ^---- CRT_ORG_BANK_3
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:35: error: undefined symbol: CRT_ORG_BANK_3
      ^---- __BANK_3_END_tail-CRT_ORG_BANK_3
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:39: error: undefined symbol: CRT_ORG_BANK_4
      ^---- CRT_ORG_BANK_4
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:40: error: undefined symbol: CRT_ORG_BANK_4
      ^---- __BANK_4_END_tail-CRT_ORG_BANK_4
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:44: error: undefined symbol: CRT_ORG_BANK_5
      ^---- CRT_ORG_BANK_5
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:45: error: undefined symbol: CRT_ORG_BANK_5
      ^---- __BANK_5_END_tail-CRT_ORG_BANK_5
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:49: error: undefined symbol: CRT_ORG_BANK_6
      ^---- CRT_ORG_BANK_6
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:50: error: undefined symbol: CRT_ORG_BANK_6
      ^---- __BANK_6_END_tail-CRT_ORG_BANK_6
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:54: error: undefined symbol: CRT_ORG_BANK_7
      ^---- CRT_ORG_BANK_7
    /home/jorgegv/src/spectrum/z88dk/lib/../lib/target/zx/classic/zx_tapeloader.asm:55: error: undefined symbol: CRT_ORG_BANK_7
      ^---- __BANK_7_END_tail-CRT_ORG_BANK_7
    make[2]: *** [Makefile-48:42: main.bin] Error 1
    make[1]: *** [Makefile-48:27: build] Error 2
    make: *** [Makefile:66: build48] Error 2
    
Any hints about these?
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Update: I have solved issue 2 by adding the following to the top of my mmap.inc:

Code: Select all

defc CRT_ORG_BANK_0 = 0xc000
defc CRT_ORG_BANK_1 = 0xc000
defc CRT_ORG_BANK_2 = 0xc000
defc CRT_ORG_BANK_3 = 0xc000
defc CRT_ORG_BANK_4 = 0xc000
defc CRT_ORG_BANK_5 = 0xc000
defc CRT_ORG_BANK_6 = 0xc000
defc CRT_ORG_BANK_7 = 0xc000
So it seems that those values are defined in NEWLIB but not in Classic, and they should (I think).
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Update 2: Issue 1 can be solved by changing lines 53 and 106 in crt_section.asm from

Code: Select all

IF CRT_ENABLE_STDIO = 1
to

Code: Select all

IF CRT_ENABLE_STDIO = 1 && CLIB_FOPEN_MAX > 0
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

Of course, PRs are always welcome!

FOPEN_MAX change is sensible - I think my original thought process was to supply CLIB_FOPEN_MAX=0 rather than -1

Re the banking problem, if you supply a custom memory map, then it's assumed that you're also going to set up the memory maps for the banks. If you're not using the banks then pass -pragma-define:CRT_DISABLELOADER=1 to disable the loader.
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

I forgot to mention, I should have sorted out the header file issues last night and also added beepfx into the library
derekfountain
Member
Posts: 121
Joined: Mon Mar 26, 2018 1:49 pm

Re: Issues when migrating back to Classic from newlib

Post by derekfountain »

Good work @jorgegv! I'm planning on using Classiclib with my next project so I'll need to do some of this migration groundwork as well. If you end up doing it all for me I'll be very grateful! :D
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

dom wrote: Sun Feb 20, 2022 11:35 am I forgot to mention, I should have sorted out the header file issues last night and also added beepfx into the library
Just discovered it for the PR, good work and thanks so much! I'm just rebuilding Z88DK right now to test the changes with my code.
derekfountain wrote: Sun Feb 20, 2022 12:12 pm Good work @jorgegv! I'm planning on using Classiclib with my next project so I'll need to do some of this migration groundwork as well. If you end up doing it all for me I'll be very grateful! :D
We'll, I'm on it and I intend to go to the end of it, so feel free to send me a beer! :D

Just for the record, my main motivation for the migration (besides the recent forum postings about the future of newlib), has been the discovery of the automatic banking code. This will help me very much to focus on my library code - I spent about 6 months in getting my own banking routines working with my library!
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

dom wrote: Sun Feb 20, 2022 11:34 am Re the banking problem, if you supply a custom memory map, then it's assumed that you're also going to set up the memory maps for the banks. If you're not using the banks then pass -pragma-define:CRT_DISABLELOADER=1 to disable the loader.
Mmm I disagree over this. I had a custom memory map in NEWLIB and I didn't have to define the CRT_ORG_BANK_x constants. In fact, those constants are defined unconditionally for NEWLIB in libsrc/_DEVELOPMENT/target/zx/zx_rules.inc.

I guess similar definitions should be done for Classic in some of the CRT init files. I'll post a PR for this :D
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

OK, I see, the definitions I'm searching for are in lib/target/zx/classic/spec_crt0.asm. It is indeed as you indicated, dom: the definitions are enclosed with an "if __MMAP != -1" clause.

Point is, in NEWLIB they are defined unconditionally. So what should the correct approach? Define unconditionally or only if __MMAP != -1?

I see that for the ZXN target the definitions are done unconditionally... See lib/target/zxn/classic/memory_map.asm

My opinion is that since they are only definitions (they do not occupy space) and they are always the same (banking on the ZX is always done at 0xC000), they should be defined unconditionally.
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

I think there are arguments for and against.

Logically if you are overwriting the memory map you expect to have complete control. However, defining them unconditionally doesn't prevent that since mmap.inc is read before them and if someone wants complete control they could be defined in there.
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

A new issue:

In newlib I was initializing my heap manually with:

Code: Select all

unsigned char *_malloc_heap;
void init_memory(void) {
    _malloc_heap = MALLOC_HEAP_START;
    heap_init( MALLOC_HEAP_START, MALLOC_HEAP_SIZE );
}
Apparently heap_init function is not available for Classic, and the similar HeapCreate function does not seem to work in the same way...

Missing definitions or missing functions in Classic?
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Sorry, just saw https://github.com/z88dk/z88dk/wiki/Classic-allocation, so don't spend any time on my previous question until I read this :-) Sorry for the quick post-before-rtfm :D
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Another one: when using input.h, in NEWLIB you can directly use key scancodes, they are #defined and can be directly used if your keys are known at compile time (i.e. QAOP-SP).

But in Classic, if you want to get the scancode which corresponds to a given keypress, you need to use the function in_LookupKey. This means code is being generated for getting a value which is static and could be just specified at compile time (code adds no value here and occupies memory); and moreover, it brings in a keycode->ASCII mapping table (in_keytranstbl) which is 160 bytes long. It is defined in libsrc/target/zx/input/in_keytranstbl.asm.

I believe it would be very convenient to port the scancode #defines (or equivalent ones) from NEWLIB to Classic for those cases when the scancodes are known at compile time.
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Hi @dom, I see you added the changes after reworking the PR... Next time you need to rework something feel free to ask me to do It if you want, I'll be very pleased to collaborate :D
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

No worries, and thank you - it was such a little change that I didn't see much point in asking, but I will in future!
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Mmmm, some problem with that last input.h change? After git pull'ing, this fails:

Code: Select all

// zcc +zx -vn -compiler=sdcc input.c

#include <input.h>

struct in_UDK keys;

void main(void) {
}
These are the errors:

Code: Select all

$ zcc +zx -vn -compiler=sdcc input.c
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:61: error 1: Syntax error, declaration ignored at '__LIB__'
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:61: syntax error: token -> 'in_JoyFuller' ; column 40
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:62: error 1: Syntax error, declaration ignored at '__LIB__'
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:62: syntax error: token -> 'in_JoyKempston' ; column 42
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:63: error 1: Syntax error, declaration ignored at '__LIB__'
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:63: syntax error: token -> 'in_JoySinclair1' ; column 43
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:64: error 1: Syntax error, declaration ignored at '__LIB__'
(...etc)
If compiled with SCCZ80 instead of SDCC it also gives errors:

Code: Select all

$ zcc +zx -vn input.c
/home/jorgegv/src/spectrum/z88dk/lib/config//../..//include/arch/zx/input.h:73:55: fatal error: Cannot parse function argument 'uchar' - unknown type, but not K&R syntax
Compilation aborted
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

After running the compilation with strace, I see that the "#include <input.h>" is going to file "include/arch/zx/input.h", instead of going to "include/input.h", as it should. Compilation fails, because the arch file is quite barebones and is missing lots of definitions which should come from the upper input.h (which includes "arch/zx/input.h")

It seems that ZCC is setting the include paths for the compilers so that "include/arch/zx" is searched before "include"...

Should it be this way...? I have my doubts...
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

I cheated and just renamed the arch/zx file. It's a constant battle with priority of paths - arch/zx is only there so that I could move sp1 from the top level into the arch/zx directory.

Given that I did try this out last night does make wonder what target I used!
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Hey Dom, everything works fine after the new change, thanks.

...And here goes a new one:

I get a linker error like this:

Code: Select all

...
Compiling engine/src/debug.c ...
...
Bulding main.bin....
engine/src/debug.c:923: error: undefined symbol: _sp1_UpdateNow
  ^---- _sp1_UpdateNow
engine/src/debug.c:923: error: undefined symbol: _debug_flush
  ^---- _sp1_UpdateNow
make[2]: *** [Makefile-48:42: main.bin] Error 1
The linker error is for both symbols because they end up defined at the same address (I've seen this during my debugging sessions). The relevant code in debug.c is this:

Code: Select all

...
void debug_flush(void) {
    sp1_UpdateNow();
}
...
(since the only thing debug_flush() does is to call sp1_UpdateNow(), it makes sense to optimize and not generate a full function for it, and just do a direct call to the latter).

I initially thought about some problem with the sections (maybe the SP1 section was wrong for the sp1 function asm file?) but I see that the section for all of SP1 functions is code_sprite_sp1, and that this section is included in the main memory map for Classic. Also, the generated code for debug.c comes in section code_compiler, which is the standard one and is also included in the Classic memory map (of course!).

So I'm a but of stuck now, I don't know what else to look for...

Also: where are the C-compatible public exports for the SP1 functions(the ones starting with _sp1*)? I only can find the NEWLIB ones under libsrc/_DEVELOPMENT...

Doing a grep for _sp1 in the repo and removing everything related to doc, _DEVELOPMENT, etc. reveals no Spectrum-related symbols starting with _sp1...
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

Reproducible:

Code: Select all

$ cat sp1.c
// zcc +zx -vn -compiler=sdcc sp1.c
#include <arch/zx/sprites/sp1.h>

void main(void) {
    sp1_UpdateNow();
}
$ zcc +zx -vn -compiler=sdcc sp1.c
sp1.c:375: error: undefined symbol: _sp1_UpdateNow
  ^---- _sp1_UpdateNow
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Issues when migrating back to Classic from newlib

Post by jorgegv »

I'm thinking that may be the proper way to treat all these issues is to open GitHub issues for them? I'm not sure all these investigations are interesting to everyone....
derekfountain
Member
Posts: 121
Joined: Mon Mar 26, 2018 1:49 pm

Re: Issues when migrating back to Classic from newlib

Post by derekfountain »

jorgegv wrote: Tue Feb 22, 2022 5:54 pm I'm thinking that may be the proper way to treat all these issues is to open GitHub issues for them? I'm not sure all these investigations are interesting to everyone....
I think it's a useful exercise in what's being checked and what's being fixed. But maybe my interest is exceptional. :)
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Issues when migrating back to Classic from newlib

Post by dom »

Sometimes it becomes obvious that my attention span is shorter than it should be, see: viewtopic.php?p=19466#p19466

It looks like about a year ago I was updating the classic sp1 to use the newlib sp1 and solve the sdcc compatibility issue. However, it does look like I got a little distracted by splib2 instead and I can't find any trace of the sp1 work.

Fancy giving it a go?

I usually start by just exporting the labels to sdcc using this script:

Code: Select all

for file in `ls */c/sccz80/*.asm`; do
modulename=`basename $file | sed s,.asm,,g`
cat >> $file << EOD

; SDCC bridge for Classic
IF __CLASSIC
PUBLIC _${modulename}
defc _${modulename} = $modulename
ENDIF

EOD
done
And then creating a makefile that just globs the newlib directories and builds (pretty much any makefile should be a rough guide how to do this)

I'd just do this now myself, but sp1 is a little more complicated since it uses ix internally which will need to be saved or sdcc will go slightly wrong and there's a couple of issues I want to get to the bottom of.
Post Reply