Stuck making Subsurface on Mac

[Dirk Hohndel]

On Sat, May 24, 2014 at 08:05:25AM -0700, Steve Ratts wrote:
> Yes, you definitely have two more testers! My buddy (Darell) and I are both
> very technically oriented guys who enjoy working on problems like this. I
> believe Darell only or primarily uses Windows machines; but he's very
> technically savvy (he builds and operates electrical testing equipment for
> a living). I'm an engineer with coding experience under linux, but none
> working with gui based apps, and obviously I use both Windows and OSX.

Excellent. It's easy for me to create Windows (and Mac) test binaries.
I actually fixed another Uemis bug since 4.1 was released, so I can make
new binaries available for the two of you.

> Are you serious that Uemis intends to offer a rebranded Subsurface log
> software?

Yes. It's a complicated story.

Uemis went bankrupt almost three years ago (literally four weeks after I
bought my SDA). It hung in limbo for quite a while and then after a few
failed attempts two brothers in Switzerland bought the assets out of
bankruptcy. So the new company Swissdivetec is now the owner of Uemis.

And Swissdivetec is working with me on creating UEMISDiveLog which is a
very lightly tuned Subsurface.

> how much help they've been in your effort to get the SDA working with
> Subsurface.

The old Uemis was actively hostile to our attempts as they had this
idiotic vision of the captive portal that would be a revenue source for
them, so they did not want you to be able to download your dives any other
way.
Swissdivetec is very supportive - but the engineers who wrote the horrible
firmware / download crap are no longer working there. So while the new
engineers there have spent some time helping me reverse engineer things
and verify some of my assumptions, it's not as straight forward as it may
seem.

> I would imagine that having them working on this with you would
> be a help - surely they have insights into their own equipment. I've always

See above :-/

> had the impression that they want to somehow own the social networking
> aspect related to dive logs and such. Hence their MyUemis site, etc. The
> trouble is their web interface is so incredibly kludgy for editing dive
> info. Have they seen the light and realized Subsurface can do that so much
> better?

Yes, that's what's happening with Swissdivetec.

> If you wouldn't mind, I'd very much like to know more about the problem
> with the SDA file system getting full while downloading dives. In my
> experience, when the SDA file system choked it wasn't possible to see if
> any dives had downloaded. It seemed as if every time we cycled the power
> and began again we were starting from the same place, and so were doomed.

That was the case in 4.0 on Windows. NOTHING ever got downloaded,
actually. But in 4.1 that should be fixed. I actually created a Windows
box so I could test this :-/

Here is the quick version of how the downloading works. Make sure you are
sitting and that you have a puke bucket ready before reading this.

The SDA creates a fake FAT filesystem that it exposes via USB.
In the root of that filesystem is a file req.txt. You truncate this file,
write things to it, then overwrite parts of it and sync. Once you do this,
the SDA parses the content and executes the commands that you wrote into
it (brilliant. not).
Then it writes its responses into clear text files under the the ANS
folder.
In the directory structure, 4000 such files are precreated. But the files
themselves only get populated once you access them. The SDA keeps track of
the highest number that you have accessed and then writes its responses to
your last command into the next files. No more than 512 bytes are written
and the first three are a tag that tells you if this is part of a multiple
block set that needs to be assembled.
The data in those files is an odd structured text representation of data,
mixed in with base64 encoded binary blobs.
I have completely reverse engineered what's in there, but we don't read
all of the extra databases - we check weight, dive location and a few
other things, but there's more data there that we aren't parsing as it
doesn't really relate to things we track in Subsurface.
Now the challenge is what happens if you have requested enough data to
fill all 4000 of these ANS files. Remember, whenever you check if the SDA
has responded to your request, you potentially 'waste' one of those files;
once a file has been read by the host there is no way the filesystem can
invalidate that file and recreate different content - FAT has no concept
of that as it assumes only the host modifies the filesystem. That's part
of the reason why the download is so painfully slow.

So regardless which software downloads the data - once enough data
(samples are the biggest chunk) has been read and the 4000 files are full,
all you can do is ask the user to disconnect the SDA, wait a moment, and
reconnect it. Doing that resets the filesystem to a clean state (all
caches on the host are invalidated if you disconnect a USB drive) and you
start filling those 4000 ANS files again.

Yes, whoever came up with this has a serious drug problem.

> Was that just a bug in 4.0?

The way we created the file names to access the ANS files had a typo that
only affected Windows. So the files were never opened, but the downloading
algorithm of course thought it did open them and incremented its counter.

> If so, why didn't rolling back to 3.0.2 work for us?

It should have worked if you unplugged the SDA, plugged it back in and
then accessed it from 3.0.2. But if you had tried 4.0 and then 3.0.2
without unplugging it, the SDA would be in an inconsistent state and 3.0.2
wouldn't be able to download, either.

> If it's possible, I'd like to see Subsurface gain a more robust
> ability to inform the user what it's downloaded at each step so that the
> user can know for sure that they're making progress.

4.1 shows you progress text of what you are downloading, doesn't it? Can
you tell me what else you would like it to show?


/D