I need this!
#devoops

#sre #devoops #aws #cloudops #cloud

OH from slack: "I asked the waitress if the AWS IPA was local or if it was from Virginia and likely to not be available and she was confused"

🎼 Zero Instances?
🎶 What is happening?
🎵 It's confusing,
🎶 I should have some workloads.

🎵 Then I notice,
🎶 in the upper right,
🎶 I'm in the wrong region,
🎵 I've been fooled again!

🎵 Oh console
🎶 Take me home
🎵 To the region
🎶 I belong

🎵 NOT Virginia!
🎶 I'm in Frankfurt!
🎵 eu-central-1!
🎶 Take me home!

#Devops #Devoops #AWS

Can a RasPi3 sitting on my desk with 1GB RAM with Tailscale + Syncthing + Salt manage a small fleet of personal machines - baremetal and VMs, over a 4G connection?

Seems to work with 4 "minions" with minimal configuration so far.

#Automation #TailScale #SyncThing #SaltStack #DevOops

TIL that AWS Kinesis Data Firehose will happily batch 500+ megs of logs via a CloudWatch Subscription Filter, serialize it into a single line, and compress that down to 5 megs. Fluentd doesn't like that. It also makes troubleshooting hard. Since opening the file is hard.
#DevOops

I've just discovered a bug in Windows Server Failover Clustering (WSFC)... FUCKING WSFC..

That product was first introduced in Windows NT Server 4.0 Enterprise Edition back in 1997... and I'm still finding bugs.

If there is a bug in something, I will find it. #DevOops

OH: #GCP just released the beta of their new data lake in Paris #sre #devoops #devops

When you’re unsure if resources or entries are safe for deleting remember the following: “the proof of the pudding is in the yeeting” #devoops

One of the Junior devops engineers made a mistake...

Let thou who has not terminated production using badly written infracode cast the first stone

#DevOops

sunday funday means #devoops memelording

"This week, we discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository." https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ #DevOops

Oops! 😅 Github accidentally committed their on SSH private key to a public github repo. They rightly have changed the keys, so you'll get bit scary “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” messages.

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/
#ssh #github #devoops

@mjeaton #devoops

#devoops.....
when you get nine minutes into an #terraform apply and realize with the new workflow you needed to do:

TF_VAR_key_filepath=~/.keys/weird_key.pem terraform apply

@sre @devops #sre #DevOPs #devOops

OH:
User: This queue is backing up
Me: Oh, that sucks
User: Why?
Me: Cause you're not processing messages in it fast enough
User: Oh
10xs their worker
User: My database is falling over!
:picardfacepalm:

Networking. If it's not DNS, it's probably routing.

#DevOops

Found out my #selfhosted Mastodon instance wasn't cleaning it's assets up properly and the S3 bucket had accrued 12Gb of media in it #fail #devoops

🤦 I forgot to turn off fathom while working on my blog and now have these silly spikes in page views (all just me on localhost) #fail #devoops

Plugged in a machine with no OS installed to find it booting netboot.xyz via network.

Seems I setup a TFTP server serving netboot.xyz previously and then entirely forgot I did it.

#DevOops #Homelab

#devoops #devops #sre
OH: Give a man a fish, and he eats for a day. Teach a man to fish, and you're stuck on perpetual tech support.