Documentation on adding new software to Z88DK

Other misc things
Post Reply
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Documentation on adding new software to Z88DK

Post by jorgegv »

Hi people,

After my fights with migrating SP1 back to Classic and integrating Arkos with Z88DK, I have written down some notes that may help other contributors (or myself!) in future integrations. They can be found here: https://github.com/z88dk/z88dk/wiki/Adding-new-software , and any feedback/fixes are welcome.

Thanks so much Dom for your support, I couldn't have done it without you.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Re: Documentation on adding new software to Z88DK

Post by Timmy »

I've been reading the several pages linked to it, and I think you pointed out the problem pretty clearly: if you need to write a library for everything, you need to write all versions of it.

Classic, new IX, new IY, and then they each have different callees and different register usage. Then you need to know where to put each of the file, and which files need to be updated.

What I believe is important, is that for the future, we are (hopefully) going to go back to classic again.

Probably important is also that location relocatable code is preferable above fixed blobs, for a library, of course.

I also haven't figured out how to do stuff like extern, or storage for extra variables.

So I guess I will probably still end up writing test versions and then let others figure out how to incorporate them. :)

And of course I also missed some examples, but perhaps that's fine.
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Documentation on adding new software to Z88DK

Post by jorgegv »

Timmy wrote: Thu Mar 23, 2023 11:08 pm I've been reading the several pages linked to it, and I think you pointed out the problem pretty clearly: if you need to write a library for everything, you need to write all versions of it.

Classic, new IX, new IY, and then they each have different callees and different register usage.
Thanks Timmy for your proofreading, I think your feedback starts an interesting debate.

Yes, actually it's not that you need to write different versions of your code for each of the C libs, but instead you can write (or use) a generic implementation, and yes: you must write the C stubs for all versions. It's quite tedious and a bit error prone, you are mostly copying/pasting the same short code snippets wityh slight fixes, over and over again.

With Arkos there were just 5 functions, and still it took me about a morning to get them all sorted out and tested. With SP1 there were like 30 functions, and the adaptation of the C stubs back to Classic was a real PITA (and of course because it was my first dive into the Newlib sources :) )
Timmy wrote: Thu Mar 23, 2023 11:08 pm What I believe is important, is that for the future, we are (hopefully) going to go back to classic again.
This is a good point: are we on a path to actually deprecate Newlib? I find it somehow inconsistent that we are suppossedly comming back to Classic but then the Arkos sources had to go under libsrc/_DEVELOPMENT and the Classic makefiles adjusted for that...

Also, do SP1 sources really need to go in directory libsrc/_DEVELOPMENT/temp ? Are they "temporary"? Shouldn't the be moved to a more sensible place, say libsrc/_DEVELOPMENT/sp1? Shouldn't instead all sources under libsrc/_DEVELOPMENT be just moved under libsrc/ (in the proper directories) for a single, unified source tree for all C libs?

Please @Dom, don't take these questions as a criticism, it's just that the path mess in Newlib sources permanently triggers my OCD =D I'm honestly asking for debate here. Would some source-relocating Pull Requests be accepted?
Timmy wrote: Thu Mar 23, 2023 11:08 pm Probably important is also that location relocatable code is preferable above fixed blobs, for a library, of course.
This is also a worthy point, I'll add it to the general guidelines wiki page.
Timmy wrote: Thu Mar 23, 2023 11:08 pm I also haven't figured out how to do stuff like extern, or storage for extra variables.
You mean for extra storage used internally by the library? I did it for Arkos (the ROM player uses some RAM) by just adding some reservation in the BSS section:

Code: Select all

(...)
;; wherever you need to use the extra data area, declare it
EXTERN my_extra_lib_vars
(...)
;; ...and somewhere else
section bss
PUBLIC my_extra_lib_vars
my_extra_lib_vars:
	ds 100	;; reserve 100 bytes in BSS
(...)
Timmy wrote: Thu Mar 23, 2023 11:08 pm And of course I also missed some examples, but perhaps that's fine.
Mmm... I think I can add a small example with the Arkos integration, now that I have it still fresh in my mind...

Thanks again for your time Timmy.

Let's talk! =D
stefano
Well known member
Posts: 2137
Joined: Mon Jul 16, 2007 7:39 pm

Re: Documentation on adding new software to Z88DK

Post by stefano »

>This is a good point: are we on a path to actually deprecate Newlib?

The reason for having the two libraries is historical, Alvin introduced plenty of innovations to the libraries, such as the CALLEE trick, aiming to squeeze out as much speed and size reduction as possible, and he broadly succeeded.
His project was way more ambitious, though, he thought to introduce a standard model to provide interchangeable block and file drivers which would boost the opportunity of code reusing (e.g. a FAT file access driver could have been made generic, interchangeable with different file systems, and attached to hw specific block drivers, etc..). The original idea was to add a profiler to customize your library specifying the required "building blocks" for stdin, stdout, etc..

Dom is a fantastic team leader, he's able to encourage everybody and always let us plenty of freedom to propose and introduce changes and innovations. Still, he is patient enough to put his own hands in the code and steer the project when/where appropriate.. to me this is one of the key reasons for z88dk succeeding, together with willing to have fun and a reasonable risk oriented approach.
This approach permitted, in example, to add sdcc (furthermore, without dumping sccz80).

Thus, Newlib won't probably be 'deprecated' (not soon). By the way, it is not yet complete, by the means of the original ideas and covers a limited number of targets... and Dom already brought a good part of it outside the "DEVELOPMENT" tree and made it the standard code for both the libraries and both the compilers.
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Documentation on adding new software to Z88DK

Post by jorgegv »

Oh, that's a nice post about the historical context, I very much appreciate It. Now I have a clearer picture about posible contributions and the general philosophy of z88dk development.

Thanks again.
J.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Re: Documentation on adding new software to Z88DK

Post by Timmy »

We've had a similar discussion a year ago at: viewtopic.php?t=11638
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Documentation on adding new software to Z88DK

Post by jorgegv »

Yes I was involved in that one. In fact, that conversation was what induced me to backport SP1 newlib sources to Classic =D.
Timmy
Well known member
Posts: 392
Joined: Sat Mar 10, 2012 4:18 pm

Re: Documentation on adding new software to Z88DK

Post by Timmy »

jorgegv wrote: Fri Mar 24, 2023 3:56 pm Yes I was involved in that one. In fact, that conversation was what induced me to backport SP1 newlib sources to Classic =D.
I don't understand. Are you saying that SP1 was in the newlib and not in classic? I never realised this then. :)
User avatar
jorgegv
Well known member
Posts: 287
Joined: Wed Nov 18, 2020 5:08 pm

Re: Documentation on adding new software to Z88DK

Post by jorgegv »

About a year ago, SP1 worked with Classic, but only with SCCZ80 (no SDCC) and with sources in Classic tree. Also SP1 worked with Newlib both with SCCZ80 and SDCC, but used some new fixed sources under _DEVELOPMENT. And I wanted SP1 to work with Classic (for easier banking support) + SDCC (because of some C syntax features that SCCZ80 does not understand).

So @Dom proposed me to backport the Newlib SP1 sources to Classic, so that only the fixed set of SP1 sources would be used for both Classic and Newlib, and with both compilers.

So the old sources got removed, the Newlib SP1 prevailed, and now everything SP1 compiles and runs in both compilers and all lib flavors.
User avatar
dom
Well known member
Posts: 2076
Joined: Sun Jul 15, 2007 10:01 pm

Re: Documentation on adding new software to Z88DK

Post by dom »

jorgegv wrote: Fri Mar 24, 2023 9:00 amThis is a good point: are we on a path to actually deprecate Newlib? I find it somehow inconsistent that we are suppossedly comming back to Classic but then the Arkos sources had to go under libsrc/_DEVELOPMENT and the Classic makefiles adjusted for that..

Also, do SP1 sources really need to go in directory libsrc/_DEVELOPMENT/temp ? Are they "temporary"? Shouldn't the be moved to a more sensible place, say libsrc/_DEVELOPMENT/sp1? Shouldn't instead all sources under libsrc/_DEVELOPMENT be just moved under libsrc/ (in the proper directories) for a single, unified source tree for all C libs?
There are projects out there that use newlib so we need to keep them working, as mentioned previously I'm not planning on adding any new targets to newlib.

The thoughts I've had are to flatten the directory structure (so moving for example _DEVELOPMENT/ctype up to libsrc/ctype) since most of the code is now shared. There's several GitHub issues about this so it's definitely on the radar - it's just not a fun job and there's a fair bit of preliminary work needed before we can start to create a flatter structure.

Off the top of my head some of the preliminary work is:

- _DEVELOPMENT/target is really crt0 files, I made space for them to be located in lib/[target]/newlib and lib/crt/newlib/
- rename libsrc/_DEVELOPMENT to "newlib" with the contents of _DEVELOPMENT/arch (code not shared with classic) _DEVELOPMENT/stdio
- moving any "shared with classic" code in _DEVELOPMENT/arch into the appropriate place in libsrc/target
- moving libsrc/float into libsrc/math/float
- Making libsrc/Makefile build the newlib libraries (and update the newlib .lst files to temporarily have _DEVELOPMENT in them until the directories move up)
- ....

It's a lot of painful rebuilding and testing and tbh, I'm not sure I've got the motivation to do it by myself so any help and collaboration would be gratefully received.

As to where sp1 should go in harmonised codebase - it's probably libsrc/target/zx-common
Post Reply