[z88dk-dev] z80asm memory map composition using sections (problem)

Bridge to the z88dk-developers mailing list
Post Reply
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

[z88dk-dev] z80asm memory map composition using sections (problem)

Post by alvin »

I did a test assemble with this brief code:

;;; here's a memory map

section main
org 0
section PART_A
section PART_B
section PART_C

;;; here's some data placed in sections

section PART_B
INB: defs 2

section PART_A
INA: defs 1

section PART_C
INC: defs 3

;; here we introduce a new section and try to place it inside the defined memory map

section PART_B
section PART_OF_B
INB2: defs 4


z80asm -b -o=test test.asm


The output map file shows:

ASMHEAD_PART_A = 0000, G:
INA = 0000, L: test
ASMHEAD_PART_B = 0001, G:
INB = 0001, L: test
ASMHEAD_PART_C = 0003, G:
INC = 0003, L: test
ASMHEAD_PART_OF_B = 0006, G:
INB2 = 0006, L: test


Section "PART_OF_B" is appended to the output binary rather than following section "PART_B" as was intended.



I've been simplifying the memory map for compiles to include general containers like "code_clib" and "code_driver". The memory map only defines the sequencing of these broad containers and then in the library code, finer grained containers are defined such that they get placed into these broad containers by default. So, eg, strcpy() is defined like this:

SECTION code_clib
SECTION code_string

PUBLIC asm_strcpy

asm_strcpy:
....

The first "SECTION code_clib" is referencing a section defined in the memory map which is loaded by the crt. I'm expecting the new "SECTION code_string" to follow section code_clib unless it has been otherwise sequenced prior. The reason it's done this way is so that the general memory map can accommodate any sections the library or user might define and at the same time the user has the option of placing sections of code and data in memory with minimal effort.

Eg, the memory reserved for the heap might be defined like this:

SECTION bss_clib
SECTION bss_alloc_malloc_heap

PUBLIC __malloc_heap
__malloc_heap: defs CLIB_MALLOC_HEAP_SIZE


so by default it is placed in the crt's broadly defined BSS section. But the user could place the heap anywhere simply by referring to that section in the memeory map:


.... default memory map as usual

SECTION bss_alloc_malloc_heap
org 23295

Just by adding those lines to the file defining the memory map, the heap is removed from the default bss_clib and located at a specific address in memory (23295).


I actually thought this is how things worked before but I changed the memory map definition in the new clib tonight and discovered programs stopped working for the above reason. Is it a bug or is there anyway we can adopt this behaviour?



------------------------------------------------------------------------------
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

I'm reviving this topic since z80asm's method for forming memory maps is not scalable. There was some discussion on this before but unfortunately the emailed messages weren't recorded in the forum.

A review of the problem: it's not possible for users to create new sections and insert them into the output executable without editing the library's memory map. Also, if the library adds new sections, they must be explicitly added to the memory map.

Here's the memory map currently set up by the new c library:

Code: Select all

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; memory model ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

SECTION CODE

org __crt_org_code

section code_crt_init
section code_crt_main
section code_crt_exit
section code_crt_return
section code_crt_common

section code_driver
section code_font
section code_clib
include "../../clib_code.inc"
section code_lib
section code_compiler
section code_user

section rodata_driver
section rodata_font
section rodata_clib
include "../../clib_rodata.inc"
;;section rodata_error_strings
;;section rodata_error_string_end
;;defb 0
section rodata_lib
section rodata_compiler
section rodata_user

SECTION CODE_END

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

SECTION DATA

IF __crt_org_data

org __crt_org_data

ELSE

IF __crt_model

"DATA section address must be specified for rom models"

ENDIF

ENDIF

defb 0

section smc_driver
section smc_font
section smc_clib
include "../../clib_smc.inc"
section smc_lib
section smc_compiler
section smc_user

section data_driver
section data_font
section data_clib
include "../../clib_data.inc"
;;section data_fcntl_stdio_heap_head
;;section data_fcntl_stdio_heap_body
;;section data_fcntl_stdio_heap_tail
;;section data_fcntl_fdtable_body
section data_lib
section data_compiler
section data_user

SECTION DATA_END

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

SECTION BSS

IF __crt_org_bss

org __crt_org_bss

ELSE

IF __crt_model

org -1

ENDIF

ENDIF

defb 0

section BSS_UNINITIALIZED

section bss_driver
section bss_font
section bss_clib
include "../../clib_bss.inc"
section bss_lib
section bss_compiler
section bss_user

SECTION BSS_END

;; end memory model ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The indented includes are listings of every section created by the c library. I can show you one "code_clib.inc"


Code: Select all

;; clib code segments ;;;;;;;;;

section code_adt_b_array
section code_adt_b_vector
section code_adt_ba_priority_queue
section code_adt_ba_stack
section code_adt_bv_priority_queue
section code_adt_bv_stack
section code_adt_p_forward_list
section code_adt_p_forward_list_alt
section code_adt_p_list
section code_adt_p_queue
section code_adt_p_stack
section code_adt_w_array
section code_adt_w_vector
section code_adt_wa_priority_queue
section code_adt_wa_stack
section code_adt_wv_priority_queue
section code_adt_wv_stack
section code_alloc_balloc
section code_alloc_malloc
section code_alloc_obstack
section code_arch
section code_bifrost_h
section code_bifrost_l
section code_bifrost2
section code_compress_zx7
section code_ctype
section code_driver_general
section code_driver_character_input
section code_driver_character_output
section code_driver_memstream
section code_driver_terminal_input
section code_driver_terminal_output
section code_driver_tty
section code_error
section code_fcntl
section code_font_fzx
section code_fp_genmath
section code_fp_math48
section code_input
section code_inttypes
section code_l
section code_l_sccz80
section code_l_sdcc
section code_locale
section code_math
section code_network
section code_nirvanam
section code_nirvanap
section code_setjmp
section code_sound_bit
section code_stdio
section code_stdlib
section code_string
section code_temp_sp1
section code_threads
section code_threads_mutex
section code_z80

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
The fine grained allocation of library code to sections gives more flexibility in establishing difficult memory maps (code can be distributed among pockets in memory or in different memory banks fairly easily) and has the added bonus that a compile with --Cl-split-bin shows at a glance what is consuming the most memory in an output binary. The problem is every time a new section is created, it has to be manually entered into these files.

For users the problem is worse. One case is for 3rd party libraries. It's not possible to create new sections for user code and have the code made part of the output binary unless the user manually edits these files and adds the sections to them.



I have a proposal to fix this: if the linker encounters a new section it has not seen before, it should append the section to the previously active section in the current file.

An example:

Code: Select all

SECTION code_clib
SECTION code_string

PUBLIC asm_strlen

asm_strlen:

xor a
ld c,a
ld b,a

cpir

ld hl,$ffff
sbc hl,bc

ret
In this case, if section "code_string" has not yet been seen by the linker, it should append that section directly after the previously active one "code_clib". If there is no previously active section, the section should append to the unnamed section. Such a sequence of sections could be maintained in a linked list.

If at any point the linker encounters an ORG assignment for a section without one, that individual section should be extracted from the linked list individually. This section will then be output as a separate binary file. The special ORG value of -1 means the separate binary must be produced but the ORG should be set to the tail of the section it was previously appended to.

If things are set up this way it is easy to create new sections that are inserted into the memory map without editing them. It also leaves the door open for the user to optionally locate those sections at specific memory addresses or in specific memory banks by supplying an ORG address when desired.


Does this scheme sound workable?



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

In this case, if section "code_string" has not yet been seen by the linker, it should append that section directly after the previously active one "code_clib". If there is no previously active section, the section should append to the unnamed section. Such a sequence of sections could be maintained in a linked list.
The data structure would have to be a tree actually. When a particular section appends to another it has to be in its own branch of a parent so that if it gets an ORG later on, it can be disconnected from the parent without taking any other children with it.



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
pscust
Well known member
Posts: 194
Joined: Thu Jun 23, 2011 3:34 pm

Post by pscust »

I understand that the sections as defined today are rather simple-minded
and need some rework to allow the memory map to be defined in a finer
grained way. Also the lack of clear separation between assembly time and
link time and the ORG -1 are rather a hack than a feature.

The way I see it, the assembler should be responsible only for creating
relocatable object modules, and the linker/loader should be responsible for
grouping these modules and allocating memory addresses to them. The memory
map should be changeable just by relinking the code.

As I understand the problem, we need to:
- be able to define individual sections at the assembly phase
- be able to define section groups that are used at link phase to group all
the sections from the relocatable modules
- be able to allocate memory addresses to the start of each section group


Please check if this alternate proposal can suit the needs:
- define a GROUP statement that allows sections to be grouped into memory
chunks and define a start address
- allow section groups to be defined with wildcards, e.g. code_*
- each group will create one output binary file

The --split-bin option would make no sense in this setup; the information
of size of each section can be determined by the -v option. Also the ORG -1
would no longer be accepted.

For each memory map, one assembly file would have to be defined describing
the memory map to the linker, e.g.
memory_map.asm:
GROUP code = code_*, ...
GROUP data = bss_*, rodata_*, ...

The link command line would receive the start addresses of each group:
z80asm -b --origin=code,0x0000,data,0x4000
-> output one file code.bin with code at 0x0000 and data at 0x8000

or
GROUP blob = *
z80asm -b --origin=0x4000
-> output one file blob.bin with org at 0x4000


Please comment.

Regards,
Paulo


On Fri, Jul 22, 2016 at 5:39 PM, alvin (alvin_albrecht@...) <
lists@...> wrote:
In this case, if section "code_string" has not yet been seen by the
linker, it should append that section directly after the previously active
one "code_clib". If there is no previously active section, the section
should append to the unnamed section. Such a sequence of sections could be
maintained in a linked list.
The data structure would have to be a tree actually. When a particular
section appends to another it has to be in its own branch of a parent so
that if it gets an ORG later on, it can be disconnected from the parent
without taking any other children with it.




------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning
reports.http://sdm.link/zohodev2dev
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

Just adding this response to the forums so the discussion isn't lost again. It looks like responses will only be picked up from email if the primary recipient is the z88dk dev list.

===========================================

I understand that the sections as defined today are rather simple-minded and need some rework to allow the memory map to be defined in a finer grained way. Also the lack of clear separation between assembly time and link time and the ORG -1 are rather a hack than a feature.

The way I see it, the assembler should be responsible only for creating relocatable object modules, and the linker/loader should be responsible for grouping these modules and allocating memory addresses to them. The memory map should be changeable just by relinking the code.

As I understand the problem, we need to:
- be able to define individual sections at the assembly phase
- be able to define section groups that are used at link phase to group all the sections from the relocatable modules

- be able to allocate memory addresses to the start of each section group



Please check if this alternate proposal can suit the needs:
- define a GROUP statement that allows sections to be grouped into memory chunks and define a start address
- allow section groups to be defined with wildcards, e.g. code_*
- each group will create one output binary file


The --split-bin option would make no sense in this setup; the information of size of each section can be determined by the -v option. Also the ORG -1 would no longer be accepted.


For each memory map, one assembly file would have to be defined describing the memory map to the linker, e.g.
memory_map.asm:
GROUP code = code_*, ...
GROUP data = bss_*, rodata_*, ...


The link command line would receive the start addresses of each group:
z80asm -b --origin=code,0x0000,data,0x4000
-> output one file code.bin with code at 0x0000 and data at 0x8000


or
GROUP blob = *
z80asm -b --origin=0x4000

-> output one file blob.bin with org at 0x4000



Please comment.


Regards,
Paulo



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
alvin
Well known member
Posts: 1872
Joined: Mon Jul 16, 2007 7:39 pm

Post by alvin »

As I understand the problem, we need to:
- be able to define individual sections at the assembly phase
- be able to define section groups that are used at link phase to group all the sections from the relocatable modules
- be able to allocate memory addresses to the start of each section group

Please check if this alternate proposal can suit the needs:
- define a GROUP statement that allows sections to be grouped into memory chunks and define a start address
- allow section groups to be defined with wildcards, e.g. code_*
- each group will create one output binary file

The --split-bin option would make no sense in this setup; the information of size of each section can be determined by the -v option. Also the ORG -1 would no longer be accepted.
We also need to be able to group like data together and sequence groups relative to each other such that groups without org are appended to a previous group in address order.

In one sort of compile, one org address is needed to specify where a monolithic binary should be loaded in memory. A binary is formed out of CODE,DATA,BSS concatenated together. This is a ram model compile for programs run from ram.

In another sort of compile destined for rom, the portions destined for ram have to be orged at another address so there needs to be two binaries, one the CODE portion and the other the DATA,BSS sections together. However although the BSS section should append to the DATA section, it needs to be output as a separate binary so that the DATA section is in a separate file from the BSS section. This is what the -1 org is being used for. This allows the final binary to be formed out of the CODE section + either a copy of DATA or a compressed copy of DATA. The crt uses this stored copy to initialize ram at startup.

The above compile modes should be selectable without having to edit a memory map.

In binaries that are relocated by a target (and in this case I am taking SymbOS as the example), the output is a single binary that is composed of four(?) regions. A companion reloc file (what is produced by z80asm now is adequate) indicates what addresses in this monolithic binary need patching if the binary is relocated at load time by the target. Symbos examines each address indicated in the reloc file, determines which of the 4 regions the address at that location points to and adds the offset for that region to do the relocation. After patching all addresses, the monolithic binary's four parts are moved to their final destination addresses. This compile needs four groups to be generated into a single output binary but the groups need to be sequenced within that binary.

The above can all be done with the current section model. However there doesn't seem to be a way to sequence groups relative to each other to satisfy sequential placement in memory in the proposal.

There are more issues that are coming up that are not addressable by the current section model:

One is addressing the problem of holes in the memory map. We have targets that plop, for example, a display file right in the middle of memory. In order to maximize use of available ram, the program has to be flowed around the hole. Since there is no way to do this currently, the only solution as of now is a manual one where you create two sections - one before the hole and one after and you assign code and data to each section by trial an error such that code or data does not enter into the hole. Something that might solve this for groups is, in addition to an org attribute, to add optional size and spill attributes. If the size is exceeded, the spill attribute can indicate that code or data can overflow into another group.

An annoyance is there is no way to change org address without generating a separate binary. I don't want to violate the one org per section thing but there are applications where you may want to store a section someplace different than where the org is placed if the section is relocated at runtime. In old z80 assemblers you could do this with a DISP x directive which would cause the assembler to add x to all generated addresses. I ran into a use for this while creating compressed ram executables: http://www.z88dk.org/wiki/doku.php?id=l ... xecutables

The binary output by the compiler is compressed. A short program then decompresses this binary in ram and jumps to its start address. In order to do that it has to copy the stored compressed binary forward in memory (it expands into something larger during decompression) along with the decompression code before starting the decompression. So the decompression code needs to be ORGed at its execution address but stored at a different address. I solved this by creating two sections: one contains the startup code (+ compressed binary) that copies the compressed binary + decompressor forward in memory and then jumps to the decompressor. The second section contains the decompressor that is ORGed at its execution address. This code follows the compressed binary in address order but because the section has its own ORG, it is output in another file. So the user has to put the two pieces together manually to form the final compressed executable. That's the annoyance. One thi!
ng I was
thinking about was extending the ORG -1 idea by using a negative ORG < -1 whose absolute value would indicate an ORG for the section (the execution address of the decompressor, eg) while the linker would still append the actual bytes after the previous section as if there was no ORG indicated.


Also in the future I'd like to look at z180 and especially ez80 targets so if the assembler can prepare for that, that would be good. The z180 is not much different from the z80 and there's something in z80asm for it already. The ez80 is also not much different from the z80 except its register pairs are extended from 16-bit to 24-bit in ADL mode (the other mode is z80).



------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
Post Reply