Oldskooler Ramblings

One of the side effects of designing a tracker completely from scratch, using the best software design I can muster, is that everything is virtualized. A song is an object; the player engine is an object; the output device is an object. I’m probably going to make the editor an object itself, if just to keep the source consistent.  The nice thing about objects is that they can be abstracted and used as “parents” for descendant objects. So I have done just that for the internal speaker output. So that leads to some good news and bad news:

The good news is that the tracker will probably come with basic support for the Tandy/PCjr sound chip, and Adlib, and heck probably the CMS as well, out of the box, all in addition to the internal speaker.  This is because once the parent object is done, it’s easy to create a descendant for each output device.  And it will make it easier to add more complex internal speaker support in the future, such as mixing 3 or 4 voices realtime for “true” multichannel output from the speaker.  Just don’t expect the Adlib to sound better than the rest — I have a common denominator to target, and it ain’t 9-channel FM instruments.

The bad news about all this is that the tracker is still mostly in my head and not on paper. Which means I may only have a tracker to enter into the wild compo at Block Party instead of a demo. Or neither, considering that the presentation I’m giving takes precedence over compos :)

If I were never releasing the code, I could’ve hacked something together by this time already.  But I am, and I want people to be impressed by the code as much as music (democoder background, remember?  Both form and function are equally important :-)

I taught myself Pascal in high school. I was not admitted to the AP course my high school was offering due to my overall GPA, so I taught myself to 1. spite the arrogant jerk of a teacher who denied me, and 2. follow along with the course so I could spend time with my friends. I had been taught other languages (LOGO when I was 12, and BASIC when I was 13) so I wasn’t a stranger to programming. What I was a stranger to, however, was discipline.

My early code, which includes my early demo coding, is sloppy. Oh, the code is nice to look at and doesn’t contain spelling errors, but it is incredibly sloppy. Instead of using REP MOVSW to copy memory around, or even Turbo Pascal’s move() procedure, I would copy one array to another byte by byte. Instead of sorting an array by inserting an array set into a sorted binary tree and then copying it back to the array (heapsort), I would loop through the array and swap crap as necessary (bubblesort). Instead of learning how to use pointers, I would try to trick the compiler into giving me multiple data segments (which had the side effect of bloating the executable). Essentially, I lacked experience, insight, and wisdom — and my programs were buggy and slow as a result.

Today, nearly 20 years later, that is not the case. I still program in Turbo Pascal, but I use pointers/heap, objects, inheritance, and especially in-line assembler in places that need it. I understand proper data structures a lot better now (still struggling with tries, but I’ll get it eventually).  I write my code as if I am delivering it to someone else to maintain.  I can’t say that I am ashamed of anything I’ve written past 1997.

I mention all this because the PC Speaker tracker, something that most people bang out in a day, has taken over two weeks and I haven’t written more than 50 lines of code.   What I have done, however, is:

• Spent 3 hours writing up various design and structure ideas in a notes file
• Spent 2 hours last night laying in bed trying to visualize object-oriented tracker design while preventing playback performance from turning to shit (answer: a Song object that controls entering and retrieving note/effect data into the song, and a Player object that retrieves song data one row at a time… still thinking about this one, since a single object makes more sense, but would be harder to virtualize playback methods for other output devices)
• Spent an hour researching the frequency and amplitude characteristics of Vibrato (not just the human voice, but woodwinds and stringed instruments) so I could be sure to make the best use of the effect bits I have available (answer: most pleasing vibrato has a frequency within 4 to 7Hz and amplitude of about 10 cents)
• Spent an hour revisiting various tracker formats, looking for ideas on how to pack notes+volume+effects into two bytes (answer: 7 bits for note, 3 bits for effect, 3 for effect 1st parm, 3 for effect 2nd parm)

…but no code.  Yet.

I believe it was the demogroup Silents who proclaimed, “If you can’t do it better, why do it at all?”  That is simultaneously democoding’s greatest motto, motivator, and curse.

All (and I really do mean all) of the hardware info for low-level PC programming available on the internet today is for ATs and later. Which didn’t help me debug my keyboard handler, as I’m on an XT and the behavior is different.

So I got fed up with looking and decided to just locate the BIOS’ own keyboard handler and see what it’s doing:

c:\>debug
-u f000:0d78
F000:0D78 50            PUSH    AX
F000:0D79 53            PUSH    BX
F000:0D7A 51            PUSH    CX
F000:0D7B 52            PUSH    DX
F000:0D7C 56            PUSH    SI
F000:0D7D 57            PUSH    DI
F000:0D7E 1E            PUSH    DS
F000:0D7F 06            PUSH    ES
F000:0D80 FC            CLD
F000:0D81 E88EEC        CALL    FA12
F000:0D84 E460          IN      AL,60
F000:0D86 93            XCHG    BX,AX
F000:0D87 E461          IN      AL,61
F000:0D89 8AE0          MOV     AH,AL
F000:0D8B 0C80          OR      AL,80 <--Hey, that's
F000:0D8D E661          OUT     61,AL <--new!!
F000:0D8F 86E0          XCHG    AH,AL
F000:0D91 E661          OUT     61,AL
F000:0D93 FB            STI
F000:0D94 93            XCHG    BX,AX
F000:0D95 B44F          MOV     AH,4F
F000:0D97 F9            STC

So, after reading the scan code from port 60h like on ATs and later, the PC/XT is also finding it necessary to acknowledge the scancode by setting bit 7 on port 61. Thanks for nothing, intarweb!

This is an example of why I tell my co-workers at my Unix sysadmin job that my old DOS knowledge from 1983 helps me in all of my computer work: Because basic troubleshooting skills are basic troubleshooting skills. Computers are pretty much the same; if you train yourself to think a certain way, it will lend itself to other computer problems.

If this post seems incoherent and not an example of my best writing, that’s because I just got finished playing my first game of Team Fortress 2 and my vision is still blurred. The last online FPS I played, Counter-Strike, was in 2004… I’m way out of practice and still quite disoriented.

I couldn’t save my old XT keyboard (enter key died, and that’s pretty much the #1 most important key on the damn thing) so it has been tossed into the spare parts bin. My remaining keyboard, however, survived my pokings and I have a functional XT keyboard again. My last, but beggars can’t be choosers.

Thanks to a functional keyboard, I started working on MONOTONE again, starting with the interface (the design docs are done — yes, design docs — hey, stop laughing!) except that I was immediately sidelined by my custom keyboard handler.

A keyboard handler, for the uninitiated, is a little routine that “replaces” the “hold-key-down-and-watch-it-repeat-after-a-short-delay” behavior with “hold-key-down-and-it-stays-down-without-beeping-incessantly” behavior. This is good behavior, if you want to make your keyboard act like a piano. But my trusted old handler code locked up the machine after the first keypress.

Here is the code in question, which I had been using since 1994:

Procedure New9handler; Interrupt;
Var
  b: Byte;
Begin
  port [$20] := $20; {send end-of-interrupt to PIC}
  b := port [$60];   {read scancode from keyboard}
  If b < 128
    Then kbd [b] := True
    Else kbd [b And 127] := False;
End;

On keyboard interrupt, grab a friggin’ character and stuff it in a bitmask array. Easy as pie. Yet the XT locked up, so I am clearly doing something that the XT is allergic to (or, more likely, forgetting to do something). So now I get to research early XT keyboards/signals and try to figure out what I’m doing wrong. Luckily, I have a lot of programming books to consult; here are the ones I’m going to take to bed:

• Compute’s Mapping the IBM PC and PCjr by Russ Davies
• The Undocumented PC by Frank Van Gilluwe (founder of V Communications — thanks for Sourcer, Frank!)
• Sam’s IBM PCjr Assembler Language by David C. Willen (why a PCjr-only book? Because eventually monotone will support PCjr — ssh, keep it a secret!)
• Compute’s Beginner’s Guide to Machine Language on the IBM PC and PCjr by Christopher D. Metcalf and Marc. B. Sugiyama
• Assembly Language Primer for the IBM PC & XT by Robert Lafore

Overkill, but I want to check them all so that I can get all the info on handling a keyboard interrupt and then pick out what I need.

I know all this seems stupid and unnecessary, and makes me seem like a freak, but honestly it is the reason over the years I have gravitated towards older and slower platforms to code for fun on. It’s the same reason people still code demos on the Commodore 64 and other legacy platforms: They are fixed in nature, which means you can truly discover the absolute fastest way to accomplish a particular task on them. It’s impossible to do this on a modern winbox, because winboxen are moving targets. It also explains perfectly why modern demos have evolved in the last decade the way they have, but that’s a topic for another day.

PS: The last book by Robert Lafore is the best book you can read to learn assembler on an IBM PC. It teaches you the basics by making you assemble, by hand, in DEBUG. It sounds incredibly scary and hardcore, but it’s actually very fun!

I haven’t watched many modern demos in the last 10 years; I usually watch maybe 1 or 2 a year.  My demoscene friends are always shocked when I tell them that, given my past involvement with all things sceneish.  In fact, they’re always shocked, no matter how many times it comes up.  I’m going to try to explain why, briefly, and then hopefully either I will change, or it will never come up again.

In a nutshell, I feel shame and self-loathing when I watch modern Windows demos.   Before that completely confuses you, I’ll hurriedly explain that I always regretted not having the time, skill, intelligence (my math has always been poor), and inclination to create Windows demos.  So when I watch one, I am reminded of how I have failed at that particular aspiration.  I have succeeded at others, but watching a 64K from Fairlight or Farbrausch just gets me depressed.  Yes, I’m screwed up.  The better and more impressive the demo/intro/4k is, the worse I feel.  I know it’s not rational.

So now you know.  This also conveniently explains why I have actually gone backwards in my democoding (I started out on 386+SB+VGA and my latest efforts are geared toward an 8088+PC speaker+CGA) instead of most people, who go forwards.  For example, I’m writing a small utility as a favor to a friend, but it needs to support long filenames in Windows XP and memory over 640K.  Regardless, I’m writing it in DOS, on the XT, because I feel safe and comfortable doing so.  Once it runs within the restrictions of DOS, I’m going to port the code over to WinXP and compile it with a WinXP compiler.

Related topic change: We’ve started talking about MindCandy volume 3.

Several times in my life, I’ve asked a question that nobody could answer. Sometimes I wait patiently and the answer comes one decade later (such as a walkthrough to Tass Times in Tonetown) or even two decades later (the solution to Hack). Other times, my need for information is too great and I research it myself (8088 Corruption) or architect a solution that can provide me the answer (MobyGames). The latter is a lot of work — I greatly prefer the former, as does everyone. But it doesn’t happen often, and sometimes I write off the entire cause and forget about it.

Today, however, I had one of those moments. I’m a compression geek, with a focus on high-performance decompression. I’ve always wondered how well real-time disk compression products like Stacker, Drivespace, Doublespace, SuperStor, EZDrive, etc. performed. How fast were they at compression? Decompression? How much compression did each achieve on a set of test data? Well, today I stumbled across this:

Real-time Data Compression Algorithms’ Benchmarks

After everyone is in bed tonight, I am setting aside an entire hour with a comfy chair to sit in and a pop to just… soak it all in.

Just a quick note that I’ve rewritten the 8088 Corruption page substantially, posted a new video player, and posted all of the sample videos in my talk along with a few extra ones.

I appreciate all of the nice comments I’ve gotten regarding my lack of motivation (ie. possible depression); again, thanks.  I’m battling it now with a mixture of work, playing Worms with the kids, trying to speed up 8088flex even more, and a mystery fourth ingredient.  What could it be?  Why, it’s mysterious!  When I finish ingesting the fourth ingredient, I’ll post about it here.  It’s… immersing.

I had hoped to completely update the 8088 Corruption webpages before posting this, but it’s going to be at least another week and people have been asking me for it, so: An edited video of my NOTACON/Block Party 8088 Corruption Explained talk is available at archive.org. All of the embarrassing and missing parts have been fixed, added, edited, massaged, spindled, and mutilated, and it should be completely watchable. I replaced most of the bad video-camera-aimed-at-the-monitor footage with the actual conversion footage, filled in the hey-where’d-my-electricity-go? missing section with a voiceover, replaced all filmed slides with the actual slides, and took out two embarrassing swears (embarrassing not because they were swear words, but because I was nervous and stumbled over them).

While it is tempting to watch the flash version in a browser, I went through a great deal of trouble to make the MPEG-2 version perfect, including true 60Hz video in places. If you can spare the time, grab the MPEG-2 version and watch it on a real set-top dvd player for full effect. (Or a software player that isn’t broken; for example, use my favorite MPEG-2 player, VLC, with Deinterlace set to Linear.)

Work and home have been particularly busy this week and will be next week, so I apologize in advance for not having the extra movies, updated 8088 player, full source code, etc. available on the website yet. When I do, I’ll make a note of it in this blog.

I’m rewriting the 8088 Corruption player for the talk I’m giving at Block Party. No doubt this is a form of procrastination (I haven’t even done any PowerPoint slides yet) but I also have a fear of the player crashing when I’m showing it off, which would be a serious blow to my fragile ego. For one particular demonstration, I was going to wow the audience with a double-rate version of Corruption. Yes, 60 frames per second on an XT. The memory speed is there, but there’s so little CPU time left over that the poor disk can’t keep memory filled, and it rebuffers constantly — every four seconds, which is hardly fun to watch.

(This is going somewhere; trust me.)

Although it would be “cheating”, I have a hardware LIM EMS 4.0 board in my 8088 with 2MB of RAM in it. Last night I started to write a version of the player that buffered to EMS, then to system RAM where it could be transferred to CGA RAM. I figured, hey, maybe I can delay the rebuffering for a few more seconds for the presentation so I don’t look like an idiot.

I found that doing so was slower than simply reading the data from disk right into regular memory. You read that correctly: RAM buffering was slower than hard disk buffering. WTF?

I thought about why this might be, and when I figured it out, I started laughing. It’s a perfect example of just how wrong it is what I’m doing. Here’s the explanation:

• Old IBM PCs have a DMA controller in them to move memory around instead of forcing the CPU to do it. On a PC/XT, it’s primarily used when reading from disk, because the slower the disk, the slower the transfer and the longer the CPU would be hung up. The DMA controller can move a byte of memory in one cycle, giving it a maximum memory move speed of about 960KB/s.
• My hard disk, however, is not what the DMA chip was designed to help with — it’s a 340MB IDE drive connected to an 8-bit IDE controller. It’s about 5-10x faster than what was available in 1983. It maxes out around 300KB/s.
• Moving memory around maxes out at a top speed of 240KB/s.

Do you see where this is going? I’ve put a hard disk in my machine whose transfer speed, thanks to copying help from the DMA controller, exceeds the CPU’s ability to move memory around. Let’s word that another way in case it’s not obvious: The hard drive is faster than the RAM.

That’s just wrong.

It is probably the only configuration in the entire IBM PC/compatible family where this is the case (where the hard drive is faster than RAM). Any 286 or higher, with its 16-bit memory accesses, would be faster at moving memory around. Crazy upside-down world of 8088 hardware!!

So I’ll use this newfound realization to write a DMA version of the player to keep memory filled while playing, right? Nope. The only way to do that is to switch to reading absolute disk sectors, and that means the player would work on my machine only. Not an option.

In my last post, I stated that interrupts were supposed to be disabled on an 8088 for a MOV to the stack segment register (SS). Turns out that interrupts are supposed to be disabled during ALL segment register moves, not just SS! That explains why the procedure works to test if you have a buggy CPU or not:

Start DEBUG and run the following commands:

-A 100
MOV ES, AX
INC AX
NOP
-T

All registers are displayed at this point, where you want to look at the value in the AX register. If it is “0000″ your 8088 CPU has the bug. AX = “0001″ means your CPU passes the test & does not have the bug.

Why does this work? The T(race) command tries to single-step the MOV ES,AX instruction, but on a working cpu, the trace will include the INC AX as well, before the cpu stops so that you can check the result (value in AX). You can just as easily check the IP value, i.e. the position of the next opcode to be executed: If this is INC AX then you have a buggy cpu.

Many thanks to the venerable Terje Mathisen and DJ Delorie for cluing me in.