FloppyBuilder

Work in Progress. Jump below to "old documentation" to get the original documentation for the system.

FloppyBuilder vs Sedoric

Sedoric (and TDOS, Randos, etc...) are designed to handle the fact that the content of a floppy can be modified by the user: Files get created, modified, erased, etc...

In order to handle this, the system needs to maintain complex storage structures to list the occupied sectors, the available space, and some mapping between names and files.

This also means that the DOS structure is complex, and contains a lot of code dedicated to manage these data structures

Most of this is totally unecessary for a game or a demo: 99% of the content is known at the time when the product is build, and in most cases the changes will be things like save games or high score tables, which can be preallocated and reserved at build time ("save slots")

This is the approach taken by the FloppyBuilder system, which results in something quite more complex to put in place, but also provides quite many advantages over using a general purpose disk operating system:

  • Storing files linearly ensures faster access time
  • Refering to files by index (integer) is much more efficient than using names (strings)
  • Significant increase in available disk space thanks to the removal of the DOS and metadata
  • Almost all of the 16KB of overlay memory are available to the user (the loader is about 0.5 kilobyte large)
  • No reserved locations in page zero and two (except during loading for the temporary buffer and a few zp variables)

Accessing files with FloppyBuilder

The FloppyBuilder "API" was designed to be minimalistic, and to behave similarly for both C and 6502 Assembler programs.

There is only one header file, currently called "loader_api.h", exposing only two functions: LoadFileAt and InitializeFileAt which both take the exact same parameters, the only difference is that the first one loads and returns to the program, while the second one is used to launch a second program.

Example:

//
// FLoppyBuilder sample code
// First part
// (c) 2015 Dbug / Defence Force
//

#include "lib.h"
#include "loader_api.h"

void main()
{
    // Load the first picture at the default address specified in the script
    LoadFileAt(LOADER_PICTURE_FIRSTPROGRAM,0xa000);

    // Quit and return to the loader, which will then load the second program
    InitializeFileAt(LOADER_PROGRAM_SECOND,0x400);
}

Note: We can probably come up with a better name for the second one (propositions are welcome)

The loader

The core of the system is the loader, which sits at the end of the addressable memory.

The loader is basically a never ending loop which loads data, jumps to the user code, and is reentrant so the user code can call the loader again.

The magic is done through the use of the variable area at the end of the memory:

_LoaderApiFileStartSector  .byt LOADER_GAME_PROGRAM_SECTOR       ; $FFEF
_LoaderApiFileStartTrack   .byt LOADER_GAME_PROGRAM_TRACK        ; $FFF0

_LoaderApiFileSize
_LoaderApiFileSizeLow      .byt LOADER_GAME_PROGRAM_SIZE        ; $FFF2

_LoaderApiJump             .byt OPCODE_JMP                       ; $FFF3
_LoaderApiAddress
_LoaderApiAddressLow       .byt LOADER_GAME_PROGRAM_ADDRESS     ; $FFF5
_LoaderXxxxxx_available    .byt 0                                ; $FFF6
_LoaderApiLoadFile         .byt OPCODE_JMP,LoadData   ; $FFF7-$FFF9

_VectorNMI    .word IrqDoNothing  ; $FFFA-$FFFB - NMI Vector (Usually points to $0247)
_VectorReset  .word IrqDoNothing  ; $FFFC-$FFFD - RESET Vector (Usually points to $F88F)
_VectorIRQ    .word IrqHandler    ; $FFFE-$FFFF - IRQ Vector (Normally points to $0244)

What the API functions presented above is simply to fill these values with the given parameters.

#define LoadFileAt(fileIndex,address)          
        LoaderApiEntryIndex=fileIndex;
        LoaderApiAddress=(void*)address;
        LoadApiLoadFileFromDirectory();

#define InitializeFileAt(fileIndex,address)    
        LoaderApiEntryIndex=fileIndex;
        LoaderApiAddress=(void*)address;
        LoadApiInitializeFileFromDirectory();

The two functions LoadApiLoadFileFromDirectory and LoadApiInitializeFileFromDirectory are implemented in loaderapi.s

_LoaderApiEntryIndex            .byt 0

_LoadApiInitializeFileFromDirectory
    ldx _LoaderApiEntryIndex

    lda FileStartSector,x
    sta _LoaderApiFileStartSector

    lda FileStartTrack,x
    sta _LoaderApiFileStartTrack

    lda FileSizeLow,x
    sta _LoaderApiFileSizeLow

    lda FileSizeHigh,x
    sta _LoaderApiFileSizeHigh
    rts

_LoadApiLoadFileFromDirectory
    jsr _LoadApiInitializeFileFromDirectory
    jmp _LoaderApiLoadFile

So basically what the macro does is to provide the file index, which is used to access the position and size on disk from four tables:

  • FileStartSector
  • FileStartTrack
  • FileSizeLow
  • FileSizeHigh
Which makes 4 bytes of metadata for each file stored on disk.

These values are then written into the variables in the loader memory.

Please note that if you already know all these informations, they can be directly written to the variables, which avoid having to store the dictionnary in memory at all.
Note: Probably add a LoadFile(sector,track,size) which poke directly in the variables


FloppyBuilder (old documentation)

Important: The rest of this documentation is going to try to explain how the whole process works, but the best example is to look at the FloppyBuilder sample in the OSDK\sample\floppybuilder sub-folder.

Description

The FloppyBuilder is a relatively advanced tool which can be used to generate optimized DSK files.

The Tap2Dsk tool is conceptually equivalent to the FloppyBuilder, the main difference being that Tap2Dsk generates Sedoric floppies while FloppyBuilder generates floppy files that do not rely on the disk operating system, allowing the user to use almost all the Oric memory and most of the floppy surface.

In order to use the FloppyBuilder, you need to create a description file using the syntax described in the following sections. From this description file will be generated both a DSK file and a header file.

The DSK file is a floppy disk image (in the usual Oric format at used in emulators), while the header file contains informations about the floppy layout which can be used by the programmer to load the files.

With this organisation comes an interesting chicken and egg problem: In order to generate the DSK file, the FloppyBuilder needs to have a list of files to write, but in order to access the files the user code needs to know where the files are located and how large they are, which in turn impacts the content of the DSK file which may mean that the location of the file changed. To solved this cyclical dependency the FloppyBuilder has been designed to be run in two different modes: The initialization mode and the building mode:

  • The FloppyBuilder will first be called in Initialization mode, which will generate a header file with enough information to make it possible to build the project without actually requiring that all files be present.
  • The user code is then assembled/compiled, it can reference and use the content of the header file, which will possibly not be entirelly correct but will allow the files to have the correct size.
  • The FloppyBuilder is then called in Building mode, which will require all files to be present, this pass will update the header file which should now contains correct data.
  • The user code is then assembled/compiled again, this time with all the correct information in the header file.
  • The FloppyBuilder is then called in Building mode a final time, we now have a valid DSK file.

This process is admitely not very elegant, but it has the benefit of being simple and automated, and it allows for basically optimal file referencement: You do not need to load a directory and search, you can just ask the loader to load a particular file immediately, identified by a single immediate value.

Invocation

The floppy builder does not take any optional switch or parameter at this point in time, the syntax is just one of these three situations:

    FloppyBuilder init description_file
    FloppyBuilder build description_file
    FloppyBuilder extract description_file
(The extract variant can be used to deconstruct an existing floppy content, assuming you can rebuild a description file for it.)
The Description File format

Description file is a simple ASCII text format containing commands followed by a number of parameters. Empty lines are ignored, and semi-colon characters (;) are considered as comments.

Here is a list of commands:

  • AddDefine define_name define_value
  • AddFile file address [optional_medatas]
  • DefineDisk sides tracks sectors
  • OutputFloppyFile dsk_file
  • OutputLayoutFile header_file
  • ReserveSectors number_of_sectors
  • SetPosition track sector
  • SetCompressionMode
  • WriteSector file

Here is an example of how these commands can be used:

;
; ODSK Floppy Builder template description file
;
FormatVersion 0.20                                    ; "New" format with breaking changes
DefineDisk 2 42 17                                    ; 2 sides, 42 tracks, 17 sectors
OutputLayoutFile floppy_description.h                 ; This file will be used by the loader
OutputFloppyFile ..\build\FloppyBuilderTest.dsk       ; The final DSK file containing the data

;
; This defines the bootsectors to use for the various operating systems
; - Jasmin loads the sector 1 of track zero in $400 and then runs it.
; - Microdisc loads the sector 2 of track zero, the address is different on Atmos and Telestrat
; - The system requires a third sector containing valid data
;
; Since we do not yet have a valid Jasmin reading code, all this bootsector will do is to 
; write a message saying that this floppy needs to be booted on a Microdisc compatible system.
;
SetPosition 0 1                                       ; Set the head to track 0, sector 1
WriteSector ..\build\files\sector_1-jasmin.o          ; Written to track 0, sector 1
WriteSector ..\build\files\sector_2-microdisc.o       ; Written to track 0, sector 2
WriteSector ..\build\files\sector_3.o                 ; Written to track 0, sector 3

;
; Now here is the loader code, that one is Microdisc only
;
SetPosition 0 4                                       ; Written to track 0, sector 4
WriteLoader ..\build\files\loader.o $fc00             ; and will be loaded in $fc00
AddDefine LOADER_SECTOR_BUFFER $200                   ; Loader: Load buffer address
AddDefine LOADER_BASE_ZERO_PAGE $00                   ; Loader: Zero Page variables address

;
; From now on we compress data (The loader should not be compressed)
;
SetCompressionMode FilePack                           ; So far only two modes: 'None' and 'FilePack'

;
; Then the files used in this demo
;
AddFile ..\build\files\testdemo.o                     ; can be loaded by using this define
AddDefine LOADER_TEST_DEMO {FileIndex}                ; The main application

AddFile ..\build\files\test_picture.hir               ; load with LOADER_FIRST_INTRO_PICTURE define
AddDefine LOADER_FIRST_INTRO_PICTURE {FileIndex}      ; A hires picture

This may look a bit complicated, but the beauty of the system is that the entire process is data driven because the generated data can be used almost transparently from either your assembler or C code.

The generated Header file

So, what does the generated header file looks like? If you would put the sample description file through the FloppyBuilder is here what you would get as a result:

//
// Floppy layout generated by FloppyBuilder 0.15
// (The generated floppy is missing some files, a new build pass is required)
//

#ifdef ASSEMBLER
//
// Information for the Assembler
//
#ifdef LOADER
FileStartSector .byt 4,8,9
FileStartTrack .byt 0,0,0
FileStoredSizeLow .byt <1024,<44,<44
FileStoredSizeHigh .byt >1024,>44,>44
FileSizeLow .byt <1024,<44,<44
FileSizeHigh .byt >1024,>44,>44
FileLoadAdressLow .byt <64512,<1024,<40960
FileLoadAdressHigh .byt >64512,>1024,>40960
#endif // LOADER
#else
//
// Information for the Compiler
//
#endif

//
// Summary for this floppy building session:
//
#define FLOPPY_SIDE_NUMBER 2    // Number of sides
#define FLOPPY_TRACK_NUMBER 42    // Number of tracks
#define FLOPPY_SECTOR_PER_TRACK 17   // Number of sectors per track

//
// List of files written to the floppy
//
// Entry #0 '..\build\files\loader.o'
// - Loads at address 64512 starts on track 0 sector 4 and is 4 sectors long (1024 bytes).
// Entry #1 '..\build\files\testdemo.o'
// - Loads at address 1024 starts on track 0 sector 8 and is 1 sectors long (44 bytes).
// Entry #2 '..\build\files\test_picture.hir'
// - Loads at address 40960 starts on track 0 sector 9 and is 1 sectors long (44 bytes).
//
// 9 sectors used, out of 1428. (0% of the total disk size used)
//
#define LOADER_TEST_DEMO 1
#define LOADER_FIRST_INTRO_PICTURE 2
#define LOADER_LAST_PICTURE 3

//
// Metadata
//
#ifdef METADATA_STORAGE

#endif // METADATA_STORAGE

As you can see, the file contains some preprocessor information, tests, defines and actual entries:

  • Some #ifdef to handle both C and 6502 code. If ASSEMBLER is not defined at build time, the code will be assumed to be loaded from a C module
  • Some loader specific information, in the form of arrays of data with information for each file (location on track, size, loading address, compressed size, ...)
  • Some #define describing the format of the floppy (used by the loader to detect when it needs to change track or side)
  • Some #define that can be used to identify the files by ID (created by the AddDefine commands)
  • Some final #ifdef for the METADATA section (optional, more on that later)
The loader

We are now reaching the 'can do better' part. The loader is pretty much still a work in progress, if you take the source code of Pushing The Envelope as a base you could just keep it as is, the only thing you may need to change is the ldx #LOADER_SLIDESHOW in loader.asm if you changed the define. The rest can be kept as is.

The loading API

If you kept the loader as is, you have access to two functions in the loading API:

  • LoadData (in $FFF7)
  • SetLoadAddress (in $FFF4)

The function calls have been wrapped in 'loader_api.s', but basically all you need to do is to load a file is to pass the file entry number in the register X and then jmp to $fff7. This will use the default address specified in the description file.

If you want to load the data at another location, you can call the SetLoadAddress function in a similar way: Set X with the file entry number, A for the lower part of the address and Y for the high part, then call $fff4. Subsequent calls to $fff7 for this file will use the new location (the call patched the loader table)



Known issues


Reported by Dbug the Sun 31st May 2015


Issue #8: Use symbols in description file
Details: It is currently possible to have FloppyBuilder generate #define symbols to use from inside the program, but it would be nice to be able to use program symbols inside the description files, for example to use a label instead of a hardcoded loading address




Resolved issues


Reported by Chema the Fri 24th November 2017
Fixed in OSDK 1.11


Issue #21: Generated floppies have invalid CRCs
Details: Hi ThomH! I'm glad to read you have tested my game. This CRC bug is something related to FloppyBuilder, so it is Dbug who should fix it.


Reported by Dbug the Tue 14th July 2015
Fixed in OSDK 1.7


Issue #10: Add command to reserve space in description file
Details: It would be nice to be able to reserve a certain number of sectors without having to add a file. That would be particularly useful to reserve space for saving data in a game for example.


comments powered by Disqus
Coverity Scan Build Status
History
Version 1.2
  • Hopefully the CRC of floppies should now be correct (thanks ISS and ThomH)
  • Gap2 value changed from 0x22 to 0x4E)
Version 1.1
  • Added support for sector interleave: The 'DefineDisk' now has a additional parameter which indicates how far the next sector is (defaults to 1 for a linear search)
Version 1.0
  • Added a 'FormatVersion' command to help handle the lack of backward compatibility
  • Added a 'WriteLoader' command to simplify the handling of loader specific parameters (the loader cannot be compressed, should not be in the directories, etc...)
  • Three new defines are automatically created: FLOPPY_LOADER_TRACK, FLOPPY_LOADER_SECTOR and FLOPPY_LOADER_ADDRESS. They are designed to be used by the boot sectors to help load the loader.
  • Added a new set of macro variables: {FileTrack}, {FileSector}, {FileSize} and {FileSizeCompressed}
  • It is now possible to use the -D switch on the command line parameters to add a number of defines to the list of defines exported to the header file.
Version 0.19
  • Improved some error message to make them more useful when a problem happens.
  • Made it possible to use the system without having to delete the build folder if for some reason the size of a sector file got too large.
Version 0.18
  • Added a 'ReserveSectors' command that can be used to leave room on the disk for save games or stuff like that
Version 0.17
  • A macro expansion now accept the new value {FileSize} which gets expanded to the size of the previous file inserted in the script
Version 0.16
  • Added a mode where data can be extracted from an existing DSK file
  • The parser now accepts quoted strings
Version 0.15
  • The output file now clearly states how much free room is available in bytes on the disk
Version 0.14
  • The MetaData tables will now not contain any information after the last file that declared metadata, this allows to not waste room in the loader for dummy data
Version 0.13
  • Added a new parameter to make it possible to bootstrap the floppy building process: With 'init' a description fill be generated even if data is missing, this makes it possible to do a multi-pass build process which will not fail because it depends on things not yet generated :)
Version 0.12
  • The 'DefineDisk' command accepts a variable set of track definition values
Version 0.11
  • Added support for metadata that can be used later on by the programmer
Version 0.10
  • The compression code now generates correct data (it was using the Atari ST mode encoding, making the unpacking code not happy)
  • Added to the report file the occupation ratio of the floppy (by maintaining an internal list of used sectors also used to check if there's no overlap)
Version 0.9
  • Added the 'SetCompressionMode' command. Possible parameters are 'None' (default value) and 'FilePack'
Version 0.8
  • Cleaned up a bit the output description generation
Version 0.7
  • The code now automatically compute the gaps values based on the floppy structure parameters
  • The 'DefineDisk' command now works (at least for 2 sided, 42 tracks and 17 sectors floppies)
Version 0.6
  • Added the 'LoadDiskTemplate' and 'DefineDisk' commands (and removed these parameters from the command line)
  • Added the 'AddTapFile' command, similar to 'AddFile' but automatically removes the header and extract the start address of the file
Version 0.5
  • Fixed parsing of comments
  • added a 'OutputFloppyFile' command
  • validated that the number of sectors and tracks is correct in the 'SetPosition' command.
  • removed some unused variables
  • cleaned the offset/track/sector management code
  • the 'SetBootSector' command is now 'WriteSector' and automatically move to the next sector after writing data
Version 0.3
  • Work started in 2013 by Mickaël Pointier for the Oric 30th birthday
Version 0.2
  • Makedisk (c) 2002 Jérome Debrune, used on all Defence Force demos until 2013