More good news about the #mastodonau network tidy-up - everything seems to be working properly now. (Well, almost).

A few devices need upgrades and reboots - just to make sure everything is on the same version - so there'll be temporary glitches and dropouts. After everything is upgraded, rebooted, and checked, I'll FINALLY be able to unplug that bloody switch that has been causing most of the problems.

Phew.

OK, Today is going to be ATTEMPT TWO at decommissioning that switch that has been causing me grief here at #mastodonau - The plan is to do it in two stages this time.

Step 1 (in the next half hour or so) is to move the single remaining 10gbit connection from old switch to new switch. This worked fine last time.

Step 2 (in which everything exploded for no reason I could fathom last time) is to unplug everything ELSE from that old switch. That will happen tonight, after I am 100% certain that whatever caused it all to explode LAST time won't happen again.

YOU HAVE BEEN WARNED.

The reason this is not as simple as most people would think is that it's slightly complicated by 'the single remaining 10gbit connection' actually being a 3km long dark fibre link that runs between two geographically separate datacenters.

Yes, I seem to enjoy overcomplicating things for no sensible reason.

@xrobau #mastodonAU excellence right there :akko_fistup: :akko_fingerguns: :awesome:

@billyjoebowers 😂 I should probably tag this as #ode to #mastodonAU and Rob's efforts :bill: :awesome:

So here's a better post-mortem of what happened last night with the #mastodonau outage.

All the network changes have (hopefully) finished, and the new fibre link to NextDC B2 (Brisbane) and M1 (Melbourne) is going live (hopefully) on the 31st!

So I wasn't expecting any more problems. Until last night, when I was JUST starting a new #BaldursGate3 campaign with my daughter, and m.au stopped responding. And then I started getting alerts.

What SEEMED to be happening was similar to the network issues of last week - no connectivity to the file server. But, the file server was sort of responding. But it wasn't responding well enough that I could even log into it.

Due to those SPECIFIC network changes that are happening next, I had temporarily disconnected the 'remote console'. So I couldn't even connect to it and see what was on the screen. Which meant a trip to the datacenter, to physically plug a screen and keyboard into it to see what was happening.

It had hung. I turned it off, turned it back on, and this was all I saw on the screen as it was booting. It couldn't read the operating system disks.

(Thread continued...)

@xrobau tadaah we're back up! what did we decommission at this morning's #mastodonAu outage?

This morning's outage was brought to you by having two switches, both unlabeled and right next to each other. ONE of them was to be decommissioned. The other one was still in production, and should not have been touched. Guess which one was decommissioned? #mastodonau #sorry

Hallo Sommer...auch wieder zurück vom Urlaub?! :doge:

#sommer #sonne #sonnenschein #wetter #Deutschlandwetter #endlichsommer #endlichsonne #socialmedia #mastodonau

Ooh I think REALLY REALLY we're on the downhill stretch for all the network changes. Finally! #mastodonau

@xrobau hah is the curtain about this little dude not showing up anymore? thanks in any way for keeping the #mastodonAu boat afloat also awesome donation suggestion 👏 :akko_fistup:

I was asked about ways to donate to #mastodonau today, and respectfully declined. However, if people ARE looking for a place to donate to, can I suggest the Gladstone Womens Health Center?

https://www.gladstonewomenshealth.org.au/about/donate

They're good people, and your donation is tax deductible. Win win!

I've had a REALLY PLEASANT few decades where I stopped needing to care about dedicated links and waves and stuff. I must be nuts to be pulling new fibres around the country. On the upside, the new 10gbit dedicated fibre to Brisbane and Melbourne for #mastodonau (and other things that actually make money!) will be FINALLY active in about 8 weeks.

It's annoying when impossible things happen. Apparently a UPS, a Generator and an automatic power transfer switch is NOT GOOD ENOUGH. Sigh. Sorry for the outage #mastodonau

@ecoscore I hope that's true for both Australian instances as #mastodonAu also had an influx from great folks around the globe interested in the antipodean and marsupilean :awesome: ☺️

does #MastodonAu have a patreon or somesuch for the community to contribute to the running costs?

Temporary Limit on techhub.social is lifted now.

If you encounter spam, please report, then block. Reporting allows us to block the account for the entire instance quickly, and it gives us a record that could be useful when considering further action.

There are a couple of things you can do on your end for spam protection:

a. Block DMs from accounts you do not follow

b. Use filters to block keywords or links that show up specifically in these DMs

(if your app does not give you these options, log in via your web browser to find access to full features)

#mastodonAU

Temporary Limit on mastodon.world is lifted.

Today it is a temporary Limit on techhub.social.

Mastodon devs and instance admins are looking into options such as captcha and prevention of big influxes in signups.

#mastodonAU

also someone flicked the switch :awesome: 🥳 👏 #mastodonAu

@xrobau this is very cute 😊 thanks #mastodonAU

@xrobau this is very cute 😊 thanks #mastodonAU