And another smallish article, this time about a Consul error I recently encountered, with some explanation what a Consul Connect Service Mesh is, and how to debug certificate expiration issues in it:

https://blog.mei-home.net/posts/consul-mesh-problem/

#blog #HomeLab #HashiCorpConsul #HashiCorpNomad

The Consul update went through quickly. It seems that doing a quick Consul update while the Nomad jobs are still running works perfectly fine. All the Envoy proxy sidecars stayed up, and all even got a fresh TLS cert. Now let's see what those cert's expiration dates say come Sunday.

#HomeLab #HashiCorpConsul

The Consul team is really fast. They've just released a patch for the issue of the expiring Consul Connect certs issue I reported on Monday.

So now on to updating Nomad. I think I will take the time to test whether I can update Consul on a running cluster. I've got three server nodes, so that's not a problem, but I don't know whether the individual Consul clients react well to being restarted while there are services still running.

#HomeLab #HashiCorpConsul

Downgraded Consul to 1.14.5 again until the bugs are fixed in 1.15.x

The one that finally did me in was that the leaf certs were not getting to the clients and causing the service mesh to fail.

#HashiCorpConsul

It is a weird feeling sitting here waiting for my cluster to crash again in the hopes that the debug logs show something more.

#HomeLab #HashiCorpNomad #HashiCorpConsul

Just to reinforce: It happened, on the minute, exactly three days after the services were started again after the last occurrence. Not three days after the last occurrence - three days after the services were started again. Something must break in Consul Connect after three days.

Hmmmm, don't Consul connect mTLS certs have a 72 hour TTL, now that I think about it?

#HomeLab #HashiCorpNomad #HashiCorpConsul

And it happened again. My entire Nomad cluster broke. Again same picture, all Jobs are up, most health checks green.

This time, I used nsenter on one of the services and tried connecting to their upstream services in the Mesh via curl. Got connection reset by peer.

The most significant thing: It happened precisely three days after the services came back up again after the last occurrence. Still not enough info to write a useful bug, though.

#HomeLab #HashiCorpNomad #HashiCorpConsul

Got the new Consul Cluster (3 nodes) up and configured with TLS and auto encrypt enabled. Then got the Nomad cluster with 3 servers and 4 clients.

Then migrated workloads over to it from the old cluster, and updated the configuration for anti-social and the haproxy server.

Then I deployed a registry container for my custom images.

Going to work on keycloak tomorrow, assuming I don't have to take one of the kids to Urgent Care.

#HashiCorp #HashiCorpConsul #HashiCorpNomad

Spent the better part of last night and this morning troubleshoot an issue with Consul UI, to only just 30 minutes ago deciding to check if it is a known issue with version 1.15.0.

Github Issues confirms it is a known bug, and will be fixed in 1.15.1.

I wish I had checked that before starting to stand up a new cluster for Consul, Nomad, and hey while I am at it lets toss Vault in there too.

I may have wanted to do that anyway, before I put a lot of "production" stuff in the Nomad Cluster anyway.

At least I know I didn't screw up the update.

#HashiCorpConsul
#HashiCorpNomad
#HashiCorpVault

Updating my Nomad and Consul Versions. Hold on to your butts...

#HashiCorpConsul #HashiCorpNomad

LibreTranslate is running in containers on my Nomad Cluster, and mapped to the local port via the service mesh.

The translate request is passed to 1 of 4 translate containers I created.

It has sped up the translate process significantly.

#MastoAdmin #HashiCorpConsul #HashiCorpNomad

Mastodon Update for anti-social.online will happen later today. I am also going to look at moving the translate feature to containers running on nomad, and have it be connected over the service mesh.

#Today #HashiCorpConsul #HashiCorpNomad

This medium article Nik Long is the start of a series on getting the HashiStack working together.

Finding a comprehensive example of the Hashicorp Stack working together is not so easy to find. This the first of a series of articles called Getting Started with Hashicorp Stack on AWS. link.medium.com/twZgZYwj6wb

#Hashicorp #HashicorpVault #HashiCorpConsul #HashiCorpNomad #howto #guide