Missing pO2 samples from CCR download [was: 4.7.8: a couple of questions]

[Anton Lundin]

On 21 June, 2018 - Dirk Hohndel wrote:

> Hi Davide,
> 
> We just merged the patches from Anton, a new test build should show up in
> https://github.com/Subsurface-divelog/subsurface/releases/tag/continuous <https://github.com/Subsurface-divelog/subsurface/releases/tag/continuous>
> soon.
> This link always gets you the latest test installer - and sometimes it may be
> missing because it's just being built (or because there was an error in the
> build). If that's the case (a) wait 10 minutes and (b) if it's still missing, send
> an email to the list so we know to kick it :-)
> 
> > On Jun 18, 2018, at 9:15 PM, Davide DB <dbdavide at gmail.com> wrote:
> > 
> > I'm sure that this problem will solved.
> 
> The frustrating part is that it took so long. And that is definitely my fault
> because as I said, I didn't pay enough attention to the issue.
> 
> > As you see my report is dated back on december 2017. I cannot blame
> > nobody except my self for being lenient checking my profiles after
> > discovering that bug. On the other side I know well how open-source
> > projects works. People write code or generally help (like me) for what
> > is important for themselves and not so quickly for other things that
> > are not so important for themselves. No one gets paid for the work, so
> > one cannot force anybody.
> 
> So true

Long story on short format. I had other things on my mind, like a new
born.
 
> > Hence this bug was thought provoking for me. I've been always a
> > minority but with a divecan shearwater controller I thought I was
> > mainstream but I was wrong. This bug hit me hard. I have over 60 ccr
> > maniacally logged dives with completely wrong or missing pO2 samples.
> 
> I am reasonably certain that we can rectify this.
> 
> Please make a copy of your dive data (simply export them to an XML
> file), open that XML file, and then re-download from the Shearwater.
> In the Download dialog check the two boxes (sorry, English text, I'm
> sure you can figure out the Italian version):
> "Force download of all dives"
> "Always prefer downloaded dives"
> 
> With these options checked it should keep all of your other data like
> notes, buddies, etc, but replace the downloaded data with the new
> version which we assume will be corrected with the fix in place.
> 
> So this should be reasonably easy to recover from.
> 
> > A CCR dive without pO2 samples is like a cake with no sugar. If I
> > still have them on my Petrel (I never investigated how many profiles
> > it stores) I should import everything from scratch into Subsurface and
> > compile everything again. Not a nice perspective for the future. Right
> > now I'm diving two/three times per week so the count will increase
> > rapidly. I've tons of notes, wrecks positions, all dives have GPS
> > locations...
> 
> See above - you should be able to do this without having to manually
> re-edit all of this work!

I would rather describe it as without the pretty icing on the cake.
Every thing else about the profile is correct. Time, depth, temp and so
on. The pO2 is just affecting your deco and the monitoring of your
rebreather performance. Thats my view on it.

> > I think I will use Shearwater desktop for the time being
> > and leave old dives in Subsurface. When a complete solution will be
> > found I will evaluate how much effort is needed to realign everything
> > because I do not see a real solution coming soon.
> 
> This is where my ignorance shows. I thought Anton's fix solves the problem
> for you. Can you explain in non-CCR diver terms what is still missing?
> 
> > Anton (and now Aaron) patches are blessed workarounds but  don't solve
> > the inner problem:  for some mysterious reason, on some devices we
> > cannot use/decode Shearwater data.
> 
> If someone can extract this as one or two sample dives that show the
> problem and add words that make enough sense to me, I'll reach out
> to Shearwater again to see if they have suggestions. I know I sent them
> one of the earliest exchanges on this topic but neither they nor I really
> understood the problem and it got dropped

The calibration constants stored in the dive header are the default
value of 2100, and not the actual calibration constant.

First, that caused us to incorrectly convert the mV values from the
sensors into po2 values, and the later code did just skip exposing any
po2 values at all.

Now we fall back to the averaged/voted po2 value stored in a different
position in the samples.

> > Moreover other questions remained unanswered; In one year my unique
> > Petrel was identified as 3 different devices with random number of
> > sensors. My logbook is a mess and nobody knows the reason.
> 
> Also something I can ask about if you give me more information.

I rarely thing its random.

The old code exposed 1 value, the averaged/voted one.

Later, we exposed the individual sensor values.

Now we will expose the individual sensor values if we can find sane
calibration constants, otherwise the averaged/voted one. We will also
add a row in the "Extra info" pane saying which one it is.

> > Frankly speaking I think nobody with a shearwater petrel ccr
> > controller knowing in advance this bug  would use Subsurface as their
> > logbook. My profiles shows 2,1 bar of pO2 at 80 meters. Nothing you
> > would show to anybody.
> 
> That would be, err, unsafe. Even I know that.

Still, this is just that the icing on the cake is the wrong color.
Sure, the data is invalid, and just bogus, but it doesn't affect
anything else than the posibility to validate the deco calculations done
by the computer and the monitoring of how your rebreather preformed.

Nothing "important" thought you had a po2 of 2.1. It was just Subsurface
who thought that.

> > Months ago, given your good relationship with Shearwater, I warmly
> > recommended to contact them privately asking info on this nasty
> > behaviour of some devices.
>
> I will look again if there are more emails that I've missed.

The devices behave just fine. Its just our reverse engineered code who
doesn't work any longer.

There has bin some email exchanges, (don't remember who talked to them
or when), about how to use the calibration constants and the comment
from Shearwater back then was that this is a internal detail they didn't
want to say anything about it.
The answer might change when someone else asks them, but thats the last
thing I heard.

I don't have any Shearwater devices nor any contacts at that company. I
just stared at the data long enough and figured out a (what later turned
out to be a faulty) relationship between them. Later Jef figured out
what the real relation ship was which matched the data way better.
(To my defence, my formula matched all the test data I had =)


//Anton


-- 
Anton Lundin	+46702-161604