The great thing about the ix-denver.mm.fcix.net #MicroMirror is that it's our only 10Gbps server running off an SSD, so it's the ideal host to cache fill new servers off of once they're deployed.

Well... nerts. We upgraded the Ohio IX #MicroMirror from our 1Gbps model appliance to a 10Gbps appliance, and it promptly rolled over and died after a few hours.

The NIC seems to have blown its brains out. But then after a reboot it's fine. So what the heck.

I hate intermittent failures

Woke up in that mood again where I just want someone to hand me a briefcase with $100k in cash so I can go salt the earth with #MicroMirror #Linux servers.

So as part of the #MicroMirror constellation setup, we use push mirroring to tell all of the nodes the moment that our main mirror.fcix.net server is done pulling in updates from upstream.

This has the upside that the stampeding herd is all in sync so it vastly saves on disk IOPS, but on the other hand, the CPU usage starts to get a little hilarious on our main box...

New point on the global #MicroMirror map: Germany. Thanks Level66!

The interesting thing about our new Denver #MicroMirror POP https://ix-denver.mm.fcix.net/ is that it is our first deployment of a variant of our appliance where we're using an HP T620 plus instead of the T620 non-plus. Slightly thicker, but still 10" x 10"

This thin client is interesting because it has a PCIe slot, so we're able to install a Mellanox X3 10G NIC in it and have a 10Gbps server that fits in a USPS flat rate box.

Initial benchmarking is showing that single TLS flows top out at 1.5Gbps, TLS maxes out the CPU at 5Gbps, but it's still able to saturate the NIC for HTTP traffic, so these boxes with SSDs are still looking good for serving RPM updates.

The Creeperhost #MicroMirror in GB hosting the @fedora repo kind of cracks me up because the Fedora tier1 mirrors in the US are a dumpster fire, so we had to resort to pulling from the Hochschule Esslingen tier1 mirror in Germany.

So when Fedora releases new packages, they go:
Fedora tier0 in the USA → Hochschule Esslingen tier 1 in Germany → mirror.fcix.net in CA, USA → creeperhost Micro Mirror in the UK.

Those bits really go the distance before users can finally download them.

I guess a new stable version of #LibreOffice shipped today.

They didn't like the idea of a single organization spinning up 6 new 10Gbps mirrors for them, sooooo... 🤷 #MicroMirror

This is definitely more spicy than we like for a 1Gbps #MicroMirror node at our UK node generously hosted by CreeperHost.

I consider the ideal daily traffic level for a 1Gbps node to be 1TB. Not... almost 4TB.

We just aren't in the position to do it, but while we continue to stamp out #Linux mirrors in North America, clearly the same thing can be done to positive effect everywhere.

@HMHackMaster maybe, but all you are doing is putting a cdn behind a cdn, which seems like a potentially expensive thing to do.

As for extremely fast download experience there are so many knobs, buttons, etc there that it's hard to generalize *BUT* but what we are doing with MM is pushing small mirrors out onto closer networks to try and keep more traffic "in network" (as lets be honest most internet bottlenecks these days are at the peering points themselves not the networks beyond them) and that results in some ridiculously fast speeds depending on what you are after.

#MicroMirror

I enjoy how many projects we've got seriously looking at standing up their own MirrorBits/MirrorBrain instances because of the #MicroMirror project.

Got interviewed by a PhD student today about the #MicroMirror project.

Pretty much every question he asked about why things are so broken in Linux mirroring got an answer that boiled down to "politics from a previous decade"

We got a bug report against the #MicroMirror project, apparently every link on our mirrors to filenames with a colon in it were broken in the index pages.

XSLT continues to prove to be seemingly much more trouble than it was worth.

Another dot on the #MicroMirror map! Thanks to CreeperHost in England for offering to host one of our Linux mirror appliances to help make the Internet and free software better!

https://www.creeperhost.net/
http://creeperhost.mm.fcix.net/

So I added some code to the metrics server last night to let me know, every 10K events, how far behind it is...

*watches as the metrics server slowly falls behind*

*OK*

*FINE*

I'll learn Python multiprocessing!

#MicroMirror

So while the #MicroMirror project is primarily focused on the US, only because we both live here and it's so much easier to ship computers here vs to anywhere else, EdgeUno was kind enough to offer dealing with all the customs hassle to get one of our Micro Mirror appliances to Colombia.

This one server has been very interesting. The Micro Mirror project has been designed to focus on the short tail of requests. We only host the most popular projects on these, and even then often trim down the less popular arches per project so we're only carrying the x86-64 builds.

We rely on other mirrors for the long tail, and all the Micro Mirrors are doing is peeling off the hottest load such that larger mirrors can spend more of their resources focusing on what we aren't able to host.

This kind of falls apart once you drop one of our Micro Mirror appliances in South America. There just isn't enough other mirrors to lean on, so we see a ton more IO cache thrashing on this node because there's clients asking for pretty much every byte we host on it.

So this box is sitting at a load average of around 8-10, which is BAD. Kind of makes me wish we had sprung for an all-SSD node for that POP. Guess we just need to keep throwing more appliances at the problem...

Just opened a ticket with Arch Linux asking for them to add FOUR additional mirrors to their project load balancer.

Happy Thursday folks. #Linux #Archlinux #MicroMirror

The #MicroMirror fleet as a whole is answering 15,000 to 25,000 queries per second.

@kwf @warthog9 This thread served as my introduction to the #MicroMirror project. I like everything about it. The re-use of modest hardware and the re-purposing of underutilized egress bandwidth sounds awesome. As a Fedora, CentOS and RHEL user, you have my thanks. One quick/naive question about this specific graph. Surely everyone would be better "served" if AWS EPEL download demand was provided from within AWS. Any idea why so much download traffic found its way out to your modest server?

Thanks to @warthog9 getting the #MicroMirror telemetry sort of working again after I set up a DDoS against that poor little box, I can finally show you how utterly ridiculous the load on our NNENIX server is.

There's every other project served per pop on the bottom, and then there's the 20TB/day that just our one server in Maine is serving of just the EPEL project, mostly to AWS.