services should be back up - power outage

[Jeroen Massar]

On 20201001, at 18:23, Dirk Hohndel <dirk at hohndel.org> wrote:

> [..]
>> Just rsync/git-pull the stuff over, so that if the primary dies, that you can switch over manually.
> 
> I actually use some simple git hooks to completely automate that. The system is actively keeping its standby in sync, and the standby does the same with the live system if it were to become the live system. So yes, one could switch between them.
> BUT - you can't switch in the middle of a transaction, and there's of course a short delay, so a simple web round-robin style handoff creates a mess.
> Additionally, I have not thought through the solution to the authentication problem. That mysql database also runs on a VM in that same room - and making that be redundant and consistent... that's usually when I decide that this isn't a life-and-death critical system and go back to doing something else :-)

Git-hooks are indeed great for that purpose.

Store the auth info in git, really, just have a passwd file or a file per user (htpasswd is a great format).
Of course, if you are facebook that does not scale, as the Ops/s on git are not very high.

Master-slave MySQL can be done, but can give you big headaches which typically means re-doing the initial sync. A network disruption and you typically see the slave log stop following the master; I have only found it finicky.
Master-Master MySQL.... just don't bother.

>> Split-brain is the biggest issue in these kind of setups.
> 
> With the way git works, as long as you don't switch in the middle of a transaction it's actually fairly resilient. But still, I'm sure I'm missing subtle issues that could go wrong.

You could do it with a multi-stage git commit:
 - commit happens
 - in git-hook:
   - flock(LOCK_EX)
   - rsync or git-pull/push the tree to a different directory (git-hook still has a lock on the first thus it won't change)
   - rsync or git-pull/push the secondary directory to remote 'syncdir'
   - trigger remote to swap directories from 'syncdir' to 'active'
   - flock(LOCK_UN)

That way, you only are syncing 'clean' repos, never half ones and 'syncdir' could be dirty but 'active' is not.

Thus might be a bit too convoluted, I am sure others have another idea which might be better.

>> That said though, I use private colo'd servers (because of home connectivity not always being superb and could not get more than slow cable speeds and upload at home still is slow), if you need a secondary/third VM somewhere on a static IP (colo'd), don't hesitate to yell, more than happy to donate one for Subsurface (and I am sure others here can do the same; it is always funny the names you see on this list ;) ).
> 
> Symmetrical Gb/s plus backup 200Mb/s on a different provider. Great UPS setup (cough, cough), soon a solar setup plus Tesla PowerWall to take the UPS setup to the ridiculous level :-)

Well, many 1U boxes have dual-power feeds, thus just use two UPSs and two power feeds.

>> On Oct 1, 2020, at 5:57 AM, Attilla de Groot <attilla at attilla.nl> wrote:
>> 
>>> On 1 Oct 2020, at 07:00, Dirk Hohndel via subsurface <subsurface at subsurface-divelog.org> wrote:
>>> 
>>> The beauty of the current situation is that outside the (grumble, grumble) hours that I spend on this
>> 
>> How / where can we do an anti-grumble, pro-infrastructure donation?
> 
> 
> Thank you. I got a lot of generous offers of donations and support. I really appreciate them. But I have a simple rule. I never take money for the work on Subsurface that I do. That makes things so much simpler. And my day job does cover my basic and not so basic needs.
> 
> 
> All of you: thank you. What a nice thread to wake up to. This community is what really makes it worth investing in Subsurface. Be it time or money.

I think many of us here on the list have provided 'community service' in one way or another, hence, knowing that a 'thank you' is something that goes a long way.

That said, if other types of resources are needed, I bet there is a pizza place that can take orders ;)
And otherwise we'll have a drink of one or another one day!

Greets,
 Jeroen