Rob Thomas · @xrobau
303 followers · 456 posts · Server mastodon.au

More good news about the network tidy-up - everything seems to be working properly now. (Well, almost).

A few devices need upgrades and reboots - just to make sure everything is on the same version - so there'll be temporary glitches and dropouts. After everything is upgraded, rebooted, and checked, I'll FINALLY be able to unplug that bloody switch that has been causing most of the problems.

Phew.

#mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
303 followers · 453 posts · Server mastodon.au

OK, Today is going to be ATTEMPT TWO at decommissioning that switch that has been causing me grief here at - The plan is to do it in two stages this time.

Step 1 (in the next half hour or so) is to move the single remaining 10gbit connection from old switch to new switch. This worked fine last time.

Step 2 (in which everything exploded for no reason I could fathom last time) is to unplug everything ELSE from that old switch. That will happen tonight, after I am 100% certain that whatever caused it all to explode LAST time won't happen again.

YOU HAVE BEEN WARNED.

The reason this is not as simple as most people would think is that it's slightly complicated by 'the single remaining 10gbit connection' actually being a 3km long dark fibre link that runs between two geographically separate datacenters.

Yes, I seem to enjoy overcomplicating things for no sensible reason.

#mastodonau

Last updated 1 year ago

Heliograph · @Heliograph
879 followers · 7904 posts · Server mastodon.au

@xrobau excellence right there :akko_fistup: :akko_fingerguns: :awesome:

#mastodonau

Last updated 1 year ago

Heliograph · @Heliograph
837 followers · 7780 posts · Server mastodon.au

@billyjoebowers πŸ˜‚ I should probably tag this as to and Rob's efforts :bill: :awesome:

#ode #mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
302 followers · 444 posts · Server mastodon.au

So here's a better post-mortem of what happened last night with the outage.

All the network changes have (hopefully) finished, and the new fibre link to NextDC B2 (Brisbane) and M1 (Melbourne) is going live (hopefully) on the 31st!

So I wasn't expecting any more problems. Until last night, when I was JUST starting a new campaign with my daughter, and m.au stopped responding. And then I started getting alerts.

What SEEMED to be happening was similar to the network issues of last week - no connectivity to the file server. But, the file server was sort of responding. But it wasn't responding well enough that I could even log into it.

Due to those SPECIFIC network changes that are happening next, I had temporarily disconnected the 'remote console'. So I couldn't even connect to it and see what was on the screen. Which meant a trip to the datacenter, to physically plug a screen and keyboard into it to see what was happening.

It had hung. I turned it off, turned it back on, and this was all I saw on the screen as it was booting. It couldn't read the operating system disks.

(Thread continued...)

#mastodonau #baldursgate3

Last updated 1 year ago

Heliograph · @Heliograph
822 followers · 7593 posts · Server mastodon.au

@xrobau tadaah we're back up! what did we decommission at this morning's outage?

#mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
290 followers · 412 posts · Server mastodon.au

This morning's outage was brought to you by having two switches, both unlabeled and right next to each other. ONE of them was to be decommissioned. The other one was still in production, and should not have been touched. Guess which one was decommissioned?

#mastodonau #sorry

Last updated 1 year ago

Noot Noot · @noot_noot
7 followers · 74 posts · Server social.tchncs.de
Rob Thomas · @xrobau
279 followers · 404 posts · Server mastodon.au

Ooh I think REALLY REALLY we're on the downhill stretch for all the network changes. Finally!

#mastodonau

Last updated 1 year ago

Heliograph · @Heliograph
760 followers · 7096 posts · Server mastodon.au

@xrobau hah is the curtain about this little dude not showing up anymore? thanks in any way for keeping the boat afloat also awesome donation suggestion πŸ‘ :akko_fistup:

#mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
269 followers · 390 posts · Server mastodon.au

I was asked about ways to donate to today, and respectfully declined. However, if people ARE looking for a place to donate to, can I suggest the Gladstone Womens Health Center?

gladstonewomenshealth.org.au/a

They're good people, and your donation is tax deductible. Win win!

#mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
266 followers · 389 posts · Server mastodon.au

I've had a REALLY PLEASANT few decades where I stopped needing to care about dedicated links and waves and stuff. I must be nuts to be pulling new fibres around the country. On the upside, the new 10gbit dedicated fibre to Brisbane and Melbourne for (and other things that actually make money!) will be FINALLY active in about 8 weeks.

#mastodonau

Last updated 1 year ago

Rob Thomas · @xrobau
261 followers · 371 posts · Server mastodon.au

It's annoying when impossible things happen. Apparently a UPS, a Generator and an automatic power transfer switch is NOT GOOD ENOUGH. Sigh. Sorry for the outage

#mastodonau

Last updated 1 year ago

Heliograph · @Heliograph
693 followers · 6555 posts · Server mastodon.au

@ecoscore I hope that's true for both Australian instances as also had an influx from great folks around the globe interested in the antipodean and marsupilean :awesome: ☺️

#mastodonau

Last updated 1 year ago

does have a patreon or somesuch for the community to contribute to the running costs?

#mastodonau

Last updated 1 year ago

Help · @help
24857 followers · 96 posts · Server mastodon.au

Temporary Limit on techhub.social is lifted now.

If you encounter spam, please report, then block. Reporting allows us to block the account for the entire instance quickly, and it gives us a record that could be useful when considering further action.

There are a couple of things you can do on your end for spam protection:

a. Block DMs from accounts you do not follow

b. Use filters to block keywords or links that show up specifically in these DMs

(if your app does not give you these options, log in via your web browser to find access to full features)

#mastodonau

Last updated 1 year ago

Help · @help
24857 followers · 95 posts · Server mastodon.au

Temporary Limit on mastodon.world is lifted.

Today it is a temporary Limit on techhub.social.

Mastodon devs and instance admins are looking into options such as captcha and prevention of big influxes in signups.

#mastodonau

Last updated 1 year ago

Heliograph · @Heliograph
499 followers · 4637 posts · Server mastodon.au

also someone flicked the switch :awesome: πŸ₯³ πŸ‘

#mastodonau

Last updated 2 years ago

Heliograph · @Heliograph
291 followers · 2789 posts · Server mastodon.au

@xrobau this is very cute 😊 thanks

#mastodonau

Last updated 2 years ago

Heliograph · @Heliograph
291 followers · 2788 posts · Server mastodon.au

@xrobau this is very cute 😊 thanks

#mastodonau

Last updated 2 years ago