RFC: Statistics in Subsurface

Sat May 16 09:22:25 PDT 2020

Hi Willem,

That's a great detailed writeup. I'll comment in line and try to keep all of the pictures in place!

> On May 16, 2020, at 7:13 AM, Willem Ferguson <willemferguson at zoology.up.ac.za> wrote:
> 
> This is just an attempt to enumerate how many types of graphs one is likely to need, given the discussion until now. As a basis I use Dirk's Proposal for selecting a appropriate graph.
> 
> 
> In the above diagram, the different types of variables have different colours.
> 
> 1) The yellow ones are just totals (Total # dives, Total no. minutes/hours) that are unlikely to have any associated minimum or maximum.
> 
> 2) The blue ones are variables defined in terms of categories. Date : day, week, month, etc; Trip : trip locality; suit: type of dive suit; tags : tag text. There is no dive suite value inbetween wetsuit and semidry suit because they are two distinct categories. 
> 
> 3) The white ones are continuous numeric variables. Duration can potentially have any arbitrary number of minutes or hours. The same goes for Max_depth, Min_temp, SAC, and all the other white ones. Inbetween any two arbitrary depths there are innumerable intermediate depths and depth only becomes a value along a continuous scale.
> 

Technically... we store depth in mm (and round when displaying)... yeah, you know where I'm going here. I'm being a childish stickler for details that these are of course discrete values, but for the purpose of this discussion you are correct. They have "nearly continuous" values and in that are distinctly different from the blue categories.

I have two questions: 
- why is 'dive type' not blue. That's likely just an oversight, right? It's not like depth, duration or temp that are "nearly continuous".
- and what about date. that one is kinda weird - but I'd say for someone with 8000 dives since 1983, date is "more continuous" than temperature (we dive in water typically between about -2C and +40C, so even at 0.01 degree intervals that's only ~4200 values - and 0.01deg C is way beyond the measurement accuracy here)... I think 'by date' is either white or it is 'special'

> The type of graph that best depicts a relationship between two types of variables depends on the colour that each of the variables above has. I need to emphasize that the graphs below are totally open to discussion. The purpose here is to assess how many types of graphical elements one would need for a basic statistics tab in Subsurface.
> 
> Plotting a yellow variable against a blue variable is probably best represented by a simplistic bar graph like:
> 
> 
> There are no min and max values to indicate. The different suit categories are indicated along the horizontal axis. There is no need to specify a degree of "granularity" or increment along the horizontal axis and no min or max values are involved.
> 

Yes, I think that's reasonable. People like me will run into trouble here because of the way I name my suits. "dry, Whites Fusion with Weezle", "wet, 3mm, with 2/1 hooded vest"

> If one plots a yellow variable against a white (continuous) variable, then a granularity/increment needs to be specified. In the image below, an increment of 20m was used.
> 
> 
> Basically the same type of graph as the one used above. No need for min/max values.
> 

The challenge here lies in the ability to come up with "clever" groupings. To a human this is likely somewhat obvious, but you need to invest some thought / algorithm / heuristics into this to get this right. Especially if we include 'date' as a white variable (as I think we should).
Clearly something that is "solvable", just not necessarily straight forward.

> Of course, as was well argued previously, the bar graph can be horizontal in the case of long names on the horizontal axis, e.g. dive site names:
> 
> 
> While I personally have no qualms with horizontal diagrams where needed, I would argue it is a regression to default to horizontal orientations for all bar graphs.
> 

I think for blue variables horizontal might be easier to get right. But I'm willing to have us consider this a "phase 2" think and start with just vertical and see into how much trouble we run there...

> The above graphs deal with yellow variables in Dirk's proposal. Now about the other categories. Plotting a White variable against a Blue variable has several options, including box and whisker plots that are not popular in this discussion. My proposal two days ago was something like this and there was some discussion around it:
> 
> 
> 
> 
> Here SAC is a white (continuous) variable and Suit is a blue (catagorical) variable. A graphical element that is likely to differ sharply from the bar graphs used above. Here again, because the horizontal axis comprises categories, there is no need to specify a granularity/increment. For lack of a better name (there is actually a esoteric statistical name for this graph) I call this a dot graph.
> 

The more I look at this, the more I like it.

> What about plotting a Blue (categorical) variable against a White (continuous) variable? For our case the order in which the blue and white variables are selected probably does not matter and the dot graph shown above (or some derivative of it) should suffice. 
> 

I had never thought of blue categories being plotted against the white categories. What's the meaning of a plot of my tags over the depth? Or trips over temperature? I'm not saying that this is wrong or not useful, I'm just trying to understand how that would look and what information the user would get from it?

> What if a white (continuous) variable is plotted against another white variable (e.g. dive duration against dive depth). The most appropriate type of graph is probably as scatter diagram:
> 
> 
> The raw data are indicated on the graph. There is no need for specifying a granularity value because there in no grouping of values along the horizontal or vertical axes. If a clear relationship between the two variables exists, it is clearly visible on the graph as in this case.
> 

Yes, this seems like the obvious choice here, much better than creating artificial columns by intervals of depth.

> We have now dealt with 
> 
> Yellow/white
> 
> Yellow/Blue
> 
> White/Blue and Blue/White
> 
> White/White
> 
> What about Blue/Blue?
> 

Same question as above. What does a graph of, say, suit over tags look like? What does it tell me?
> There is another type of graph that is potentially extremely useful : introduce a *third* variable to the graph. For instance, in the case of the second blue bargraph towards the start of this message (No.dives vs depth) one could ask what the distribution of a third category is. For instance, how long did I use various dive suits at different depths? Or how many dives did I use different dive suits at different depths? This is the above barchart, divided into the values for different dive suits. This is also useful to analyse variables used as tags, e.g. the use of air/nitrox/trimix during dives, the number of boat/shore dives, the number of training dives compared to fun dives, the number of dives using different dive modes as a function of depth, dive duration, temperature, or whatever white variable has been selected.
> 
> 
> Since the horizontal axis corresponds to a white (continuous) variable, one would need to specify a granularity/increment. The UI cost for this would be an additional dropdown list/comboox to select the appropriate categorical variable to appropriately subdivide each bar of the graph (Dirk's Granularity??). This diagram handles cases of graphs with a blue(categorical) variable plotted against another blue (categorical) variable, although a third variable needs to be specified to form the unit of measurement (e.g. dive duration in the above graph). This can probably be selected using Dirk's Granularity Combobox in his proposal.
> 

We are of course going to get ourselves into trouble here. Yes, the addition of a third variable and the use of color for that is really cool. But that does add significant complexity. And BTW, I'm sure the next person is going to say "I want to add that to a scatter plot"... e.g., in the duration over depth chart, you could use dot color to indicate the suit, or you could pick some tags to be shown (think nitrox, air, trimix). Which are all wonderfully cool graphs, but the complexity is quickly going up.

How about a fourth axis? Think about this scatter plot where you track suit over depth/duration - would you want to add different dot shapes to indicate your dive type?

I am half joking, but we all know that someone will ask for this as soon as we release the new statistics :-)

> This handles basically all the possibilities of the different combinations of Yellow, White and Blue variables in Dirk's proposal. There are fundamentally FOUR types of graphs that would be required, forming the basis of visual presentation of the Statistics tab.
> 

I'd argue that's six (but that's because of my potential additions):
two values:
- Bar chart
- Dot graph
- Scatter plot
three values:
- Stacked bar chart
- Dot graph with colors
- Scatter plot with colors.

> I hope this appears somewhat useful in the present discussion.
> 

Yes, I think this was super useful.

There are a few questions in my response that I think would benefit from more thought / responses. 

And then maybe some more exploration of how things would work to
- pick which categories to show (for some the number is likely small enough, but for tags, full text, and possibly trip you'd want to be able to select those that you want to plot
- pick the grouping of date
- enter the granularity for "near continuous" variables (depth, duration, temperature)

Once we have sketches / proposals for those, I think we have the UI side handled and can design the backend data model that would be able to provide those data.

And then start implementing and iterating :-)

Thanks

/D
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gfnhmollagimhhko.png
Type: image/png
Size: 141121 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hnljlgceengjopkb.png
Type: image/png
Size: 15773 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mengpnbdjbogjbcm.png
Type: image/png
Size: 14416 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ffkgopmmdoajfbbm.png
Type: image/png
Size: 13128 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jijkgdeoohbbfibe.png
Type: image/png
Size: 13009 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dmdckpjgcadkbmmd.png
Type: image/png
Size: 18330 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: omjglffkhhbefejk.png
Type: image/png
Size: 21213 bytes
Desc: not available
URL: <http://lists.subsurface-divelog.org/pipermail/subsurface/attachments/20200516/8723b1aa/attachment-0013.png>