Friday, November 3, 2023

When Emulation goes wrong

My name is Dave and I'm a software developer.

I both create and fix defects in code that I've written.   Hopefully I find the bugs quickly, but sometimes they linger for weeks, months, years, before being discovered or reproduced.

My story is not unique.  It is difficult or impossible to write software without introducing defects.  Computers are complex systems, but very obedient.   They do exactly as they are told if you know how to speak their language, precisely, without misunderstanding or any inaccuracies.  Problems can crop up if the developer didn't understand the system requirements in the first place (design/requirements issues), didn't thoroughly test all inputs and code paths, didn't understand concurrency issues, made assumptions, or makes the simplest of mistakes.   As humans, we learn to overlook some inconsistencies, mistakes of others, so as to not disrupt the social norm.  But with most of the languages we use with computers (Assembly, C, C++, C#, Java, etc.) many of these don't allow for the slightest of change. Some scripting languages (HTML, JavaScript, CSS, etc.) allow for some variance, but still are expecting a valid input.

Am I making an excuse for my mistakes?  No, but explaining that they will exist.  I wrote a limited Commodore/6502 emulator a while back and have ported it to various platforms with demonstrable results.  I can run Commodore 64 and related systems at the BASIC prompt and it appears to mostly function normally.

Then I ported to Vic-20 including adding full-screen editing support and conversion from Commodore 64 key scans to Vic-20 key scans.   And it did not work.   But I swore up and down that I did everything correctly (ha!).  Then dove in deeper to see what was different.   Turned out the ROL ABS and ASL ABS instructions were doing the value shift but not storing the result in memory.  A normal 6502 knows how to do this just fine, but the emulator needed to accurately replicate this behavior to be correct.  The fix was to store the result at the absolute 16-bit address.

My first mistake was that I never performed a formal test of the instruction set of the emulated 6502.  I knew they existed, but when bringing up the C64 I only spot checked a few instructions, and even compared traces between Vice and my emulator to note differences, and correct the emulation in a few places.  (There's a reason my trace looks exactly like that from Vice -- so the comparison would be easy!)  But once the system appeared to function correctly, I stopped testing.

Fast forward over 3 years later, and I know that some of my systems are not 100% correct.  The effort I had brushed away came back to nag me.  I really should test better.  I have a saying that if it's not tested, it's not working.  And I had ignored that saying.  But I buckled down and did the effort of running a 6502 test suite against my emulator.  And sure enough there were problems.

And since I have multiple emulator instances, the problems were slightly different, as I hadn't applied fixes consistently between all of them (still have missed a few at date of this article!  STM32, mbed, TypeScript still have known bugs). [update 2023-11-05: TypeScript emulator updated with fixes]

Here are the instructions and features of the 6502 that required fixes in my emulator:

ROL (absolute address)

LSR (absolute address)

PHP (B flag)

PLP (B flag)

IRQ (push processor state without B flag) 

(zp,X) indirect zero page indexed by X addressing not truncated to byte address when overlap page, affecting both load value and store value

SBC resulting in a negative result when in decimal mode

Many changes are now checked in for C++, C#, Arduino, M5, Teensy and related projects.

There are remaining bugs in emulation, probably closer to individual system systems outside the 6502 emulation.  A few issues can be reproduced on demand, so I can tackle those next.

But the 6502 emulation should be good, at least as much as it has been tested.

Thanks to Klaus Dormann for these 6502 tests.

Reference: 6502_65C02_functional_tests

Friday, June 30, 2023

Extremely small emulated C64 and C128

 


This "portable" Commodore 64 and 128 emulator (m5 source code branch) is my work in progress, one in a series of minimalist emulators ported to different hardware targets. Only text (on LCD) with background, foreground, border colors, keyboard entry via USB serial tethered web browser, and general 6502/6510 and C64 memory management emulation is present (no, won't play games, make sound, or do bitmapped graphics) with some D64 emulation [added 7/2/2023].

(Update 7/28/2023) Now with GO 128 command.

GO 128 command


Even my son asked, "Why do you need to do that?"  Well, he has a point.  I wanted a C64 that fit in my pocket or even on my wrist.  And targeting new hardware platforms with my emulator is part of my hobby.

How does it work?  Check out my highly technical drawing.

Now I already here you asking why I didn't connect Bluetooth to the M5Core, because certainly it has Bluetooth as well, and why didn't I use a USB keyboard connected to CoreS3, because it includes USB Host.  But I've had trouble tracking down examples of HID Host examples for M5; client examples are prevalent, but host?

Pictured here is a phone is running Chrome with a custom copy of the html/javascript keyboard adapter including web-serial-polyfill because mobile Chrome doesn't directly include Serial API support.  A Palm Pilot foldable keyboard has a Bluetooth adapter, paired with the phone.   HID keystrokes are captured by the web page, converted to C64 key scan codes, and a list of the active key scan codes (or 64 when keys released) is sent over USB Serial to the M5Core device which is running the C64 ROMs which are tricked into thinking a real keyboard is attached; keystrokes are processed by the C64 KERNAL IRQ as normal.

The M5Core is being powered by the phone.   Why M5Core?  Because it's a polished packaged solution.

Yeah, we could just run a Commodore 64 emulator on the phone, but this way, I could have complete control over the keyboard emulation, what keys are present, how CTRL and Commodore keys work, etc.  And it's just because I can, not because I should.

Why the Palm Keyboard?  Because it folds in my pocket!  And because I had one from back in the day.  Any keyboard you can attach to a phone or computer should work.  And this Bluetooth adapter just makes it so cool, and easier than a tethered keyboard.

The next step is to merge this solution with my Commodore 128 keyboard adapter to completely reject the portability feature.   That would look really cool hooked up to my phone!  Update (7/31/2023): check out YouTube for connection from ItsyBitsy/keyboard to Core Port.A.


C128 Keyboard Adapter Breadboarded Prototype


7/23/2023: PCB prototype keyboard adapter w/ ItsyBitsy

I am excited about my nonsense crazy adventures. Even if only I enjoy them.

=====

Update (7/2/2023): D64 support is currently working with Core2 only (Basic Core doesn't usually have the additional SPI RAM, but not yet sure why CoreS3 is failing to attach SD).

Update (7/3/2023): Got CoreS3 working with SD switching header to M5Unified.h for that target (was M5Cores3.h) and adding special definition, override logic for SD_CS to use GPIO_NUM_4 instead of default.  See updates to M5Core.h.

Update (7/28/2023): Commodore 128D extended keyboard working with UART connection to Port.A of Core, and Commodore 128 emulation is ported as well.

=====


Tuesday, May 30, 2023

Bible with 8-bit feel interface

Here is my third iteration of an eBible interface -- browsing the Bible (KJV) by electronic means.  First was a tiny 8x2 text screen on mbed NXP LPC1768 (ARM M3), second was a Nokia cell phone screen on mbed NXP LPC1114 (ARM M0 through hole package), and this third one looks very similar to a familiar 8-bit system, running in the browser.  

The third revision required no soldering, and works with desktop, tablet, mobile phone devices, anything with a modern web browser.  This current iteration builds on cbmish-script, providing the Commodore-like look and feel.  

History

Back 35+ years ago I brainstormed on how to put the Bible on a Commodore computer.  For example, with the Bible containing 66 books, I had planned to combine I John, II John, III John as one book, saving two so the book number would be 0..63 fitting in 6 bits.  Nah, that's probably silly, one byte is proper to fit the book number. But I never got that far on the Commodore.

With my first mbed based eBible iteration, I focused on being able to navigate book to book, chapter to chapter, and verse to verse.  So I built an index that was stored on the SD card, FAT filesystem.

The second eBible iteration had a different screen and used simpler block flash memory device (no filesystem whatsoever), with text, indexes, and bookmarks stored at different offsets.

Just prior to the third iteration, I restored the indexing code, porting from its original state working with mbed and linux, now to Microsoft Visual C++ on Windows.  With indexing working, I was playing around and output the Bible text to a flat JSON format now shared on github.

Then I created a few JavaScript routines to navigate the text and perform basic operations required for navigation.  I hosted the solution on StackBlitz to begin with as it is a good solution for shared web development with no local tools needed.  I just wrote some JavaScript that output very simple HTML, and it was doing something useful very quickly.  With some additional polish, it switched from outputting fixed results to populating drop down controls for the books, chapter numbers, and verse numbers, resulting in rudimentary navigation.  This solution was shared back to a GitHub repository as well, and has since been revised to work more standalone from a local file system folder in addition to being hosted on the web.

All along the way I had planned to revive my Bible on Commodore project, and the idea was to leverage my existing cbmish-script solution, which provides a Commodore look while programming in modern TypeScript.

So I set out on that journey, and saw results right away.  Almost all the bitmapped text on the screen is a link.   First the navigation was limited to book/chapter/verse, allowing selection of book, book by book selection of chapter numbers, chapter by chapter selection of verse numbers, and navigation of the text verse by verse, with back links to book, chapter, and verse selections.

Added on was an about page, stats page, link to cbmish samples, and search for words.

Links are almost everywhere

Virtually every word and number on the screen is a link.  Clicking on text will perform a search of that word showing other references to it.  One of the latest features added is being able to replace the word (or phrase) searched for, to truly make this solution interactive.

How information is accessed, efficiency

How is the latest incarnation indexed?  It is not.  Each entry has book name, chapter number, verse number, and verse text.  It is one long array of verses with this context in order.

To retrieve a specific verse, code scans the entire array for that data, returning a match.

How much data?  About 4 megabytes.  Or 6 megabytes with references per entry.  Almost 32,000 records/verses in this flat file format.

The operations involved include retrieving the book names, counting the chapters in a book, counting the verses in a specific book/chapter, retrieving the text of a verse, retrieving all the verse records for a chapter, and finding the verse records containing a specific word or word portion.

Is it slow?  No.  It just works.  Scanning the 6 megabytes of data is fine on a modern desktop computer or mobile device.  Even retrieving a list of all verses containing an "e" or "the" seems instantaneous.

I was not used to this kind of performance or memory or storage capability when originally coding Commodore or embedded development platforms.  But fast processors with lots of resources handle these tasks with ease.  It's almost like they were designed to handle big 3D virtual games and a few megabytes of text is no big deal.

Is it a waste?  Is it inefficient?  Sure, but today megabytes are measly, and cycles are cheap too.  When should we optimize for speed and energy?  Of course the answer is that it depends.  As this is an interactive application used by a single user occasionally, and performance is absolutely fine, no optimizations are necessary.  We are done!

If you are developing a multi-user system in a VM process in a server farm where you are paying for resources, you may need to prioritize your optimizations for cost, energy use (be a good steward!), and performance.

It would be easy enough to adapt the flat file into a hierarchical structure, and add indexing for words and text.  Handling this complexity is a useful skill even if the performance/processing optimizations are not at all necessary.  For now it's just useful enough to consider what effort would be necessary rather than actually performing the extra work.

Comparison with Commodore

It looks like a Commodore 64 because the resolution, fonts, colors, and behaviors are the same or similar.  

Though a port to a Commodore would require some special work including the indexing.

Storage - original Commodores didn't have megabytes of storage.  Using an SDIEC or similar may be possible, or another hard disk solution.  Another choice would be to offload to an add-on board such as ESP32 and optionally a web service.

Most Commodores didn't have a mouse back in the day, and mouse handling would require extra work too.  Additional software and hardware is required, or use of emulation.

All in all, developing the cbmish-script environment alleviates much of the resource limitations of the original Commodore microcomputer and while it may not be a technically accurate representation, it provides much of the look and feel of the original system.

Cbimsh-script technical details

cbmish-script uses multiple canvases to provide the border, background, foreground, and sprite layers which are drawn independently, and combined by the browser software.  Commodore fonts are drawn pixel by pixel with both character sets available to be displayed simultaneously.  Transparent pixels allow a lower level behind to show through (like background).

Mouse support is provided with click, mousemove, mouseleave window event handlers for the sprite (highest visible) layer.  Clickable regions and buttons are drawn highlighted (or unhighlighted) when hovered to demonstrate they may be interacted with the mouse.  Similarly they animate when clicked on a touchscreen device.  The cbmish-script system keeps track of a list of active buttons and is able to compute which ones are being interacted with.

This Bible software leverages these regions to also act as a text input field.  When the region is clicked on, it activates a bounding box for the region to limit screen updates and cursor movement to that region, exited when Enter or Escape keys are pressed, or the mouse clicks outside the region.  An onexit handler regains control and is able to grab the modified text from the screen.

Before these changes, cbmish-script had limited input capabilities.  It had operated in a display only and echo only capacity, displaying what is typed, not intelligently responding to input.

What could be added next? 

Next I should add more input and graphical capabilities to cbmish-script.  And/or a responsive mode that changes resolution and orientation with the mobile device or browser window.  Maybe add a Bible reading plan/tracker, bookmarks, multiple search terms, and a master word list.  Add another Commodore emulator to my repos?  Just ideas for now.

Links

Thursday, January 5, 2023

What is Cbmish?

 


Is this Commodore?

If it looks like a Commodore, acts like a Commodore, and feels like a Commodore, is it a Commodore?  Not always.  This cbmish is not a Commodore, and not made by Commodore Business Machines (CBM).  But it is like a Commodore, thus the "-ish" (means like).  

Cbmish is like a Commodore.  This web software solution provides the feel of a Commodore in a web browser.  There's a retro look and feel.  But with a more modern programming language.

READY.

But it says "READY."  Is that BASIC?  No, it is lying to you.  It is not ready for commands.  It won't PRINT or GOTO.  It is only ready for typing at this point.   You can navigate the screen with cursor keys and typing letters, numbers, punctuation, and other symbols.  You can change colors, type PETSCII, change to upper/lowercase, switch to reverse characters. It does feel like a Commodore.

Mouse

It also responds to mouse motion and clicks.  That's not a Commodore I ever owned.  Those rounded corners and text are buttons, and there are links too.  Clicking on a rounded button will present the user with a relevant sample screen, or navigate to another web page.  Colors may change, graphics may be drawn, text will display, and sprites may move.  

Why?

So why do this?  The about screen goes into some detail.  I'm ready to move on from BASIC (gasp!) and 6502 Assembly (gasp!) and I don't really want to program on my Commodore (gasp!), but I do want to continue the look and feel of Commodore.  My current efforts are web based and I have been using TypeScript from some years now.  My skill is mostly is C languages (C, C++, C#) and while I had shunned web development for sometime, my professional and hobby efforts have collided with doing full-stack development including C# and also TypeScript.

While interpreted languages aren't usually my thing (though I love me some AWK) I have seen features of JavaScirpt and functional programming influencing C#, and of course there are underpinnings of C in TypeScript too, and similar functional programming in JavaScript/TypeScript too.  While they are all quite distinct languages, there is enough familiarity between them, also also gotchas that keep them interesting when they are different.  It's fun to know multiple languages!

This is stuff you could do with your real Commodore, but it's done here using TypeScript written code.  TypeScript that is transpiled to JavaScript so it runs in the user's web browser.  A combination of HTML, CSS, and TypeScript results in code to provide the cbmish interfaces to present things like one of the favorite 80s retro systems.

You can code interactively using the browser developer tools, or you can launch Visual Code (or similar) with typescript compiler dependency to have code transpiled as soon as you save.  Instructions for both are on github in the README (see link below). 

10 PRINT "Hello ";: GOTO 10

So do you really want the equivalent of PRINT/GOTO? Well this will have to do, see immediate mode JavaScript entered into the console of the development tools of Google Chrome web browser.

PRINT/GOTO equivalent

Conclusion

So long story short, it's a fine time to do Commodore like programming on the web, in a modern language.  Take the best of modern languages and pair that with the look and feel of a great retro computer.   Are you keeping up with the Commodore?  Because cbmish is keeping up with the web.

Links

    open source (MIT License) at github.com
    live demo (try it now!)

Samples

mouse over button

Colors

petscii

chr$()s

Maze

Keyboard

lores

sine wave

dissolve

knight vs. dragon
sprite animation

about


Monday, September 19, 2022

HIRES for Vic-20 (BASIC 2.0 extension) is READY.

 


The BASIC commands are implemented, small font created, and demonstrations implemented.   Please enjoy this Space War and Omega Race inspired flying ship.

Source, disk image, and more details are on github for Commodore Vic-20 with memory expansion.

BASIC extension command syntax is:

COLOR [foreground[+8][,[background][,[border][,auxillary[,inverse]]]]]

COLOR [foreground[+8]] @ x1,y1 [TO x2,y2]

TEXT

HIRES xresolution, yresolution [,fillbyte]

DELAY jiffies

PLOT COLOR ON|OFF

PLOT [NOT|CLR] (@ x1,y1)|(TO x2,y2)...    **

PLOT 0|1|2|3 (@ x1,y1)|(TO x2,y2)...

PLOT "ABC" @ x,y [,addr [,width,height [,bytes]]]

RECT [NOT|CLR] [@] x1,y1 TO x2,y2

RECT 0|1|2|3 @ x1,y1 TO x2,y2

SHAPE GET|PUT|OR|XOR|AND|NOT|CLR addr @ x1, y1 TO x2, y2


** only first @ optional, when not multi-color

Monday, August 29, 2022

Vic-20 Hi-Res BASIC extension progress

 


Shape operations are functional, and work has begun on leveraging BASIC keyword extensions (borrowing from my own HIRES for C64).

4UPART program listing


While the SYS syntax is implemented, the BASIC keyword work has just begun.

Current vs. Proposed Syntaxes

Source and disk image are on github



Monday, August 8, 2022

Knight vs. Dragon prototype on Commodore 128



My son Ben (age 9) designed all the graphics, background and colors.  He edited the background screen strictly with inverse spaces.  Sprites were edited on the Commodore 128 with SPRDEF.  The programming manual has instructions on saving the sprite memory to disk.  And I was able to save both the text screen and colors to disk also using the dual screen capability of the Commodore 128 to enter the commands via the 80 column screen (ESC+X).  My memory is that I used MONITOR to save using hex addresses, whereas I'm showing BSAVE commands here for more symmetry with the program listing.

BSAVE "SPRITES",B0,P3584 TO P4096

BSAVE "BACKGROUND",B0,P1024 TO P2048

BSAVE "COLORS",B15,P55296 TO P56320

I wrote the BASIC demo to randomly move the characters, and flash the border red when the sprites collide.

My son and I made a great team on this project.   Thanks to his art and I am very pleased to have introduced him to the Commodore 128 and have him involved in computer animation.

Links: 

dragon.d64 disk image

Commodore 128 Programmer's Reference Guide