fluxengine/doc/technical.md

How the FluxEngine works
========================

## The hardware design

It's essentially very simple, although the details are complicated. In
essence, all the firmware does is to measure the time between flux transition
pulses and send them to the PC.

There's an 8-bit counter attached to an 12MHz clock. This is used to measure
the interval between pulses. A Cypress datapath state machine thingy using
Verilog turns the timer, pulse and index information into a bytecode stream
which encodes intervals, pulses, and whether the index hole has been seen.

This is then streamed back to the PC, where offline software decodes it: it
searches for known bit patterns (which depend on the format), uses those to
determine the clock rate, and then turns the pulses into a nice, lined up bit
array and from there decodes the file system format.

Writing back to disk works the same way: bytes are streamed to the
FluxEngine, where a different datapath state machine thingy (the PSoC5LP has
24, all independently programmable) to interpret the bytecodes, generate a
stream of pulses to the disk.

The bytecode format represents an interval between pulses as a byte, a pulse
as a byte, and the index hole as a byte. Timer overflows are handled by
sending multiple intervals in a row. However, the USB transport applies a
simple compression system to this in order to get the USB bandwidth down to
something manageable.

An HD floppy has a nominal pulse frequency of 500kHz, and we use a sample
clock of 12MHz (every 83ns). This means that our 500kHz pulses will have an
average interval of 24. This gives us more than enough resolution. At this
speed, in the 200ms that a 3.5" disk takes to rotate, we will see about
100,000 pulses. Each one is encoded as two bytes, one for the interval and
one to generate the pulse; so that revolution generates 200kB of data.
(Extremely approximately. The actual figure is less.)

(The clock needs to be absolutely rock solid or we get jitter which makes the
data difficult to analyse, so 12 was chosen to be derivable from the
ultra-accurate USB clock.)

Some useful and/or interesting numbers:

  - nominal rotation speed is 300 rpm, or 5Hz. The period is 200ms.
  - a pulse, as generated by the floppy drive electronics, is 150ns to 800ns
    long.
  - a 12MHz tick is 83ns.
  - MFM HD encoding uses a clock of 500kHz. This makes each recording cell 2us,
    or 24 ticks. For DD it's 4us and 48 ticks.
  - a short transition is one cell (2us == 24 ticks). A medium is a cell and
    a half (3us == 36 ticks). A long is two cells (4us == 48 ticks). Double
    that for DD.
  - pulses are detected with +/- 350ns error for HD and 700ns for DD. That's
    4 ticks and 8 ticks. That seems to be about what we're seeing.
  - in real life, pulses start going astray on a 3.5" drive after about 128
    ticks == 10us. I haven't tried with a 5.25" drive yet as I don't have a
    5.25" scratch disk.

## Why don't I use an Arduino / STM32 / ESP32 / Raspberry Pi / etc?

I've got a _lot_ of questions on this, and multiple Github issues of people
debating it. It's complicated, but it's essentially a tradeoff between speed
and complexity.

FluxEngine's read process involves generating a lot of data using a fairly
brute force sampling approach --- about 150kB per disk revolution, and
sometimes it needs to record multiple revolutions. Most microcontrollers
don't have enough RAM to buffer this, so instead I have to stream it over USB
back to the host PC in real time. The disk won't wait, so I need to stream data faster
than the disk is producing it: the total is about 800kB/s.

Handling USB is pretty CPU-hungry, so my candidate microntroller has to be
able to cope with the ruinously strict real-time requirements of the
sampler's 12MHz clock as well as keeping up with 13,000 USB interrupts a
second (one for each 64-byte frame) in order to transfer the data.

The Atmels and STM32s I found were perfectly capable of doing the real-time
sampling, using hand-tool assembly, but I very much doubt whether they could
do the USB streaming as well (although I want to move away from the Cypress
onto something less proprietary and easier to source, so I'd like to be
proven wrong here).

The Raspberry Pi easily has enough processing power and memory, but it's also
got terrible GPIO pin read performance --- [about
1kHz](https://raspberrypi.stackexchange.com/questions/9646/how-fast-is-gpiodma-multi-i2s-input/10197#10197).
That's a long way from the 12MHz I need.

The PSoC5LP part I'm using has enough CPU to handle the USB side of things,
and it _also_ has a whole set of FPGA-like soft programmable features,
including 24 mini-ALU systems that are ideally suited to exactly this kind of
sampling. I can read the disk and generate the byte stream describing the
flux pattern entirely in 'hardware', without involving the main CPU at all.
This is then DMAed directly into a set of ring buffers read for the USB
system to pick up and relay back to the PC. It's incredibly simple and works
well. (The same applies to writing flux back onto the disk.)

The development board I'm using, the
[CY8CKIT-059](https://www.cypress.com/documentation/development-kitsboards/cy8ckit-059-psoc-5lp-prototyping-kit-onboard-programmer-and),
also has another big advantage: it's the right shape. It's got 17 holes in a
row connected to GPIO pins, and it's a native 5V part, which means I can just
connect a floppy drive connector directly to the board without needing to
build any hardware. No adapter board, no level shifting, no special cable,
nothing. This makes the FluxEngine hardware incredibly easy to assemble,
which therefore means cheap.

Speaking of which, the CY8CKIT-059 is $10. (Before shipping, which is
admittedly expensive.)

### Some useful links

  - [The floppy disk user's
    guide](http://www.hermannseib.com/documents/floppy.pdf): an incredibly
    useful compendium of somewhat old floppy disk information --- which is
    fine, because floppy disks are somewhat old.

  - [The TEAC FD-05HF-8830 data
    sheet](https://hxc2001.com/download/datasheet/floppy/thirdparty/Teac/TEAC%20FD-05HF-8830.pdf):
    the technical data sheet for a representative 3.5" drive. Lots of useful
    timing numbers here.

  - [The Mitsubishi M4851 data
    sheet](http://www.bitsavers.org/pdf/mitsubishi/floppy/M4851/TJ2-G30211A_M4851_DSHH_48TPI_OEM_Manual_Nov83.pdf):
    the equivalent data sheet for a representative 5.25" drive.

  - [The DRG Business Machines YD-174 manual](https://electrickery.hosting.philpem.me.uk/comp/divcomp/doc/YE_Data_YD-174_8inchFloppyDriveTechnicalManual.pdf):
	the equivalent manual (data sheets hadn't been invented then) for a
	representative 8" drive.

  - [KryoFlux stream file
    documentation](https://www.kryoflux.com/download/kryoflux_stream_protocol_rev1.1.pdf):
    the format of KryoFlux stream files (partially supported by FluxEngine)