Using a FluxEngine ================== So you've [built the hardware](building.md), programmed and tested it! What now? ## The programs I'm sorry to say that the client program is very badly documented --- it's moving too quickly for the documentation to keep up. It does respond to `--help` or `help` depending on context. There are some common properties, described below. If possible, try using the GUI, which should provide simplified access for most common operations.
screenshot of the GUI in action
### Core concepts FluxEngine's job is to read magnetic data (called _flux_) off a disk, decode it, and emit a filesystem image (called, er, an _image_); or, the other way round, where an image is read, converted to flux, and written to a disk. A file system image typically has the extension `.img`. It contains a sector-by-sector record of the _decoded_ data on the disk. For example, on a disk with 512 byte sectors, one sector will occupy 512 bytes. These are typically what you want in everyday life. FluxEngine supports a variety of file system image formats, including [LDBS](http://www.seasip.info/Unix/LibDsk/ldbs.html), [imd](http://dunfield.classiccmp.org/img/index.htm), Macintosh's [DiskCopy 4.2](https://en.wikipedia.org/wiki/Disk_Copy) and some others, including Amiga's `.adf`, Atari ST's `.st`, and so on. Flux, however, is different. It represents the actual magnetic data on the disk. This has to be decoded before anything useful can be done with it (like turning it into a file system image). It's possible to read the flux data off the disk and write it to a file, known as a _flux file_, which allows you to fiddle with the decode parameters without having to touch the disk again, which might be fragile. FluxEngine supports several different kinds of flux file, including its own, SuperCard Pro's `.scp` format, and the Kryoflux stream format. A flux file will typically contain from 80 to 150 kilobytes of data per track. In general, FluxEngine can use either a real disk or a flux file interchangeably: you can specify either at (very nearly) any time. A very common workflow is to read a disk to a flux file, and then reread from the flux file while changing the decoder options, to save disk wear. It's also much faster. ### Connecting it up To use, simply plug your FluxEngine (or [Greaseweazle](greaseweazle.md) or [Applesauce](applesauce.md)) into your computer and run the client. If a single device is plugged in, it will be automatically detected and used. If _more_ than one device is plugged in, you need to specify which one to use with the `--usb.serial` parameter, which takes the device serial number as a parameter. You can find out the serial numbers by running the command without the `--usb.serial` parameter, and if more than one device is attached they will be listed. The serial number is also shown whenever a connection is made. You can list all the detectable devices with: ``` $ fluxengine test devices ``` This will show you their serial numbers. You _can_ work with more than one FluxEngine at the same time, using different invocations of the client; but be careful of USB bandwidth. If the devices are connected via the same hub, the bandwidth will be shared. ### Basic use The FluxEngine client is a command line program. As parameters it takes one or more words telling it what to do, and then a bunch of configuration options. Configurations can be specified either on the command line or in text files. Here are some sample invocations: ``` # Read an PC 1440kB disk, producing a disk image with the default name # (ibm.img) $ fluxengine read -c ibm --1440 # Write a PC 1440kB disk to drive 1 $ fluxengine write -c ibm --1440 -i image.img -d drive:1 # Read a Eco1 CP/M disk, making a copy of the flux into a file $ fluxengine read -c eco1 --copy-flux-to copy.flux -o eco1.ldbs # Rerun the decode from the flux file, tweaking the parameters $ fluxengine read -c eco1 -s copy.flux -o eco1.ldbs --cylinders=1 ``` ### Configuration Configuration options are represented as a hierarchical structure. You can either put them in a text file and load them from the command line: ``` $ cat config.textpb encoder { ibm { trackdata { emit_iam: false } } } $ fluxengine write -c ibm --1440 -c config.textpb -i image.img ``` ...or you can specify them on the command line: ``` $ fluxengine write -c ibm --1440 -i image.img --encoder.ibm.trackdata.emit_iam=false ``` Both the above invocations are equivalent. The text files use [Google's protobuf syntax](https://developers.google.com/protocol-buffers), which is hierarchical, type-safe, and easy to read. The `ibm` string above is actually a reference to an internal configuration file containing all the settings for writing PC disks, and the `--1140` refers to a specific definition inside it. You may specify as many profile names or textpb files as you wish; they are all merged left to right. You can see all these settings by doing: ``` $ fluxengine write -c ibm --1440 --show-config ``` The `--config` option will cause the current configuration to be dumped to the console, and then the program will halt. Going into the details of the configuration is complicated and mostly futile as it's likely to change as things get modified. Brief but up-to-date information about each configuration setting is available with the `--doc` option. Note that not all combinations of settings make sense. ### The tools The FluxEngine program has got multiple sub-tools, each of which performs a different task. Run each one with `--help` to get a full list of (non-configuration-setting) options; this describes only basic usage of the more common tools. - `fluxengine read -c -s -o ` Reads flux (possibly from a disk) and decodes it into a file system image. `` is a reference to an internal input configuration file describing the format. `` may be any combination of options defined by the profile. - `fluxengine write -c -i -d ` Reads a filesystem image and encodes it into flux (possibly writing to a disk). `` is a reference to an internal output configuration file describing the format. - `fluxengine rawwrite -s -d ` Reads flux from a file and writes it (possibly to a disk) without doing any encoding. You can specify a profile if you want to write a subset of the disk. - `fluxengine inspect -s -t -h -B` Reads flux (possibly from a disk) and does various analyses of it to try and detect the clock rate, display raw flux information, examine the underlying data from the FluxEngine board, etc. There are lots of options but the command above is the most useful. - `fluxengine rpm` Measures the rotation speed of a drive. For hard-sectored disks, you probably want to add the name of a read profile to configure the number of sectors. - `fluxengine seek -c ` Seeks a drive to a particular cylinder. There are other tools; try `fluxengine --help`. **Important note on `rawwrite`:** You can't use theis tool to copy disks, in most circumstances. See [the FAQ](faq.md) for more information. ### Flux sources and destinations FluxEngine supports a number of ways to get or put flux. When using the `-s` or `-d` options (for source and destination), you can use any of these strings: - `drive:` Read from or write to a specific drive. - `` Read from or write to a native FluxEngine flux file. - `` Read from or write to a Supercard Pro `.scp` flux file. - `` Read from a Catweasel flux file. **Read only.** - `dmk:` Read from a Catweasel CMK directory. **Read only.** - `` Write to a AppleSauce flux file. **Write only.** - `kryoflux:` Read from a Kryoflux stream, where `` is the directory containing the stream files. **Read only.** - `flx:` Read from a FLUXCOPY stream, where `` is the directory containing the stream files. **Read only.** - `erase:` Read nothing --- writing this to a disk will magnetically erase a track. **Read only.** - `testpattern:` Read a test pattern, which can be written to a disk to help diagnosis. **Read only.** - `au:` Write to a series of `.au` files, one file per track, which can be loaded into an audio editor (such as Audacity) as a simple logic analyser. **Write only.** - `vcd:` Write to a series of `.vcd` files, one file per track, which can be loaded into a logic analyser (such as Pulseview) for analysis. **Write only.** ### Image sources and destinations FluxEngine also supports a number of file system image formats. When using the `-i` or `-o` options (for input and output), you can use any of these strings: - ``, ``, ``, ``, `` Read from or write to a simple headerless image file (all these formats are the same). This will probably want configuration via the `input/output.image.img.*` configuration settings to specify all the parameters. - `` Read from or write to a [DiskCopy 4.2](https://en.wikipedia.org/wiki/Disk_Copy) image file, commonly used by Apple Macintosh emulators. - `` Read a [Sydex Teledisk TD0 file](https://web.archive.org/web/20210420224336/http://dunfield.classiccmp.org/img47321/teledisk.htm) image file. Note that only uncompressed images are supported (so far). - `` Read from a JV3 image file, commonly used by TRS-80 emulators. **Read only.** - `` Read from a [DIM image file](https://www.pc98.org/project/doc/dim.html), commonly used by X68000 emulators. Supports automatically configuring the encoder. **Read only.** - `` Read from a [FDI image file](https://www.pc98.org/project/doc/hdi.html), commonly used by PC-98 emulators. Supports automatically configuring the encoder. **Read only.** - `` Read from a [D88 image file](https://www.pc98.org/project/doc/d88.html), commonly used by various Japanese PC emulators, including the NEC PC-88. FluxEngine is currently limited to reading only the first floppy image in a D88 file. When writing, a single unnamed floppy will be created within the image file. The D88 reader should be used with the `ibm` profile and will override most encoding parameters on a track-by-track basis. The D88 writer should likewise be used with the `ibm` profile in most circumstances as it can represent arbitrary sector layouts as read from the floppy. - `` Read from a [imd image file](http://dunfield.classiccmp.org/img/index.htm), imd images are entire diskette images read into a file (type .IMD), it's purpose is to recreate a copy of the diskette from that image. A detailed analysis is performed on the diskette, and information about the formatting is recorded in the image file. This allows ImageDisk to work with virtually any soft- sectored diskette format that is compatible with the PC's type 765 floppy diskette controller and drives. FluxEngine is able to read from and write to an imd image file. The imd reader should mostly be used with the `ibm` profile or ibm deratives and will override most encoding parameters on a track-by-track basis. The imd writer should likewise mostly be used with the `ibm` profile in most circumstances as it can represent arbitrary sector layouts as read from the floppy. With options it is possible to add a comment for the resulting image when archiving floppies. By default imd images assume MFM encoding, but this can by changed by suppling the option RECMODE_FM. - `` Read from a [NFD r0 image file](https://www.pc98.org/project/doc/nfdr0.html), commonly used by various Japanese PC emulators, including the NEC PC-98. **Read only.** Only r0 version files are currently supported. The NFD reader should be used with the `ibm` profile and will override most encoding parameters on a track-by-track basis. - `` Write to a [LDBS generic image file](https://www.seasip.info/Unix/LibDsk/ldbs.html). **Write only.** - `` Write to a [D64 image file](http://unusedino.de/ec64/technical/formats/d64.html), commonly used by Commodore 64 emulators. **Write only.** - `` Write undecoded data to a raw binary file. **Write only.** This gives you the underlying MFM, FM or GCR stream, without actually doing the decode into user-visible bytes. However, the decode is still done in order to check for correctness. Individual records are separated by three `\\0` bytes and tracks are separated by four `\\0` bytes; tracks are emitted in CHS order. ### Manipulating flux files `fluxengine fluxfile` contains a set of tools for examining or manipulating individual flux files. (Note that this only works on flux files themselves, not sources.) - `fluxfile ls -f ` Shows all the components inside a flux file. - `fluxfile rm -f -t ` Removes flux from a flux file. All reads are removed from the specified track; use the normal `c0h0` syntax, including using ranges, such as `c0-9h0-1`. - `fluxfile cp -i -o -t ` Copies flux from one file to another. All reads are copied from the source file and _appended_ to the relevant track in the destination file. You can use ranges etc. ### High density disks High density disks use a different magnetic medium to low and double density disks, and have different magnetic properties. 3.5" drives can usually autodetect what kind of medium is inserted into the drive based on the hole in the disk casing, but 5.25" drives can't. As a result, you need to explicitly tell FluxEngine on the command line whether you're using a high density disk or not with the `--drive.high_density` configuration setting. **If you don't do this, your disks may not read correctly and will _certainly_ fail to write correctly.** You can distinguish high density 5.25" floppies from the presence of a traction ring around the hole in the middle of the disk; if the ring is not present, the disk is probably high density. However, this isn't always the case, and reading the disk label is much more reliable. [Lots more information on high density vs double density disks can be found here.](http://www.retrotechnology.com/herbs_stuff/guzis.html) ### Other important flags These flags apply to many operations and are useful for modifying the overall behaviour. - `--drive.revolutions=X` When reading, spin the disk X times. X can be a floating point number. The default is usually 1.2. Some formats default to 1. Increasing the number will sample more data, and can be useful on dubious disks to try and get a better read. - `--drive.sync_with_index=true|false` Wait for an index pulse before starting to read the disk. (Ignored for write operations.) By default FluxEngine doesn't, as it makes reads faster, but when diagnosing disk problems it's helpful to have all your data start at the same place each time. - `--drive.index_mode=X` Set the source of index pulses when reading or writing respectively. This is for use with drives which don't produce index pulse data. `X` can be `INDEXMODE_DRIVE` to get index pulses from the drive, `INDEXMODE_300` to fake 300RPM pulses, or `INDEXMODE_360` to fake 360RPM pulses. Note this has no effect on the _drive_, so it doesn't help with flippy disks, but is useful for using very old drives with FluxEngine itself. If you use this option, then any index marks in the sampled flux are, of course, garbage. - `--tracks='c0h0 c1h0 c3-5h1' Constrains which tracks to access. These are always specified as _physical_ locations (i.e. which cylinders your drive will actually seek to). Inside the track specifier, you can use comma-separated values which can either be a raw number, a range (`3-5`), or a range with a step (`3-10x2`). So, this is possible: `c0,2,3,9,15-20x2h0-1` ## Visualisation When using `fluxengined read` (either from a real disk or from a flux file) you can use `--decoder.write_csv_to=output.csv` to write out a CSV file containing information about the location of every sector on the disk. You can then use `fluxengine analyse layout` to produce a graphical visualisation of this. Here's a IBM PC 1232kB disk: ![A disk visualisation](./visualiser.jpg) Blue represents data, light blue a header, and red is a bad sector. Side zero is on the left and side one is on the right. The visualiser is extremely primitive and you have to explicitly tell it how big your disk is, in milliseconds. The default is 200ms (for a normal 3.5" disk). For a 5.25" disk, use `--visualiser-period=166`. ## Extra programs Supplied with FluxEngine, but not part of FluxEngine, are some little tools I wrote to do useful things. These are built alongside FluxEngine. - `brother120tool`, `brother240tool` Does things to Brother word processor disks. These are [documented on the Brother disk format page](disk-brother.md). ## The recommended workflow So you've just received, say, a huge pile of old Brother word processor disks containing valuable historical data, and you want to read them. Typically I do this: ``` $ fluxengine read -c brother240 -s drive:0 -o brother.img --copy-flux-to=brother.flux --decoder.write_csv_to=brother.csv ``` This will read the disk in drive 0 and write out an information CSV file. It'll also copy the flux to `brother.flux` (replacing any old one) and write out a CSV file for used when making a visualisation. If I then need to tweak the settings, I can rerun the decode without having to physically touch the disk like this: ``` $ fluxengine read -c brother -s brother.flux -o brother.img --decoder.write_csv_to=brother.csv ``` Apart from being drastically faster, this avoids touching the (potentially physically fragile) disk. If the disk is particularly dodgy, you can force FluxEngine not to retry failed reads with `--decoder.retries=0`. This reduces head movement. **This is not recommended.** Floppy disks are inherently unreliable, and the occasional bit error is perfectly normal; FluxEngine will retry and the sector will read fine next time. If you prevent retries, then not only do you get bad sectors in the resulting image, but the flux file itself contains the bad read, so attempting a decode of it will just reproduce the same bad data. See also the [troubleshooting page](problems.md) for more information about reading dubious disks.