#FluxFramework Operator Python SDK? A prototype for a multi-tenant MiniCluster? Say what! Yes this is underwaaaay! https://twitter.com/vsoch/status/1629995329482539008

Did you know that #FluxFramework makes it easy to interact with a flux instance on... another #HPC cluster!? 🫢 We just added the first in a new series of "Command Tutorials" for how to do this! - check it out 👉 https://flux-framework.readthedocs.io/en/latest/tutorials/commands/ssh-across-clusters.html

More command tutorials on the way! 🦸

Some fun, and exciting work I am finishing up today for the #FluxFramework operator! It was the first time I created a local volume for a #Snakemake workflow. Seeing everything cleanup after the jobs finished and the complete output directory of results and logs? Joy! 🥰

The schedule is out for #Kubecon in April! 🎉

I'll be presenting work we are doing for the #FluxFramework Operator along with Michał Woźniak from Google, check out our talk! 👉 https://sched.co/1HyaG

Have you ever wondered how #FluxFramework compares to other resource managers? 🤔 As promised, we just added a new comparison table (in two colors)! 🎉 https://flux-framework.readthedocs.io/en/latest/tables/comparison-table.html. We care about accuracy! Please open an issue to add a correction or update. https://github.com/flux-framework/flux-docs/issues ❤️

We are working on improving the #FluxFramework documentation - and it's so gorgeous I have to share! https://flux-framework.readthedocs.io/en/latest/quickstart.html 🙌

Next will be a table to compare Flux to similar tools, and a concepts page and more work on content. What else would you like to see? 🤔 I'm all 👂s!

In preparing to run experiments using the #FluxFramework Operator on #Kuberentes, I wanted a way to automate and make them reproducible. Did I absolutely need to make a custom tool for myself? Probably not. But it's much more fun if I do! Here is the start of Flux Cloud - a tool that explicitly helps you run workflows using the Flux Operator on a cloud.

https://twitter.com/vsoch/status/1609025858580381698

It's very early in development and not ready for others to use, but I'm really excited and wanted to share.

@accessrel another idea is a more scoped set of builds (for OS and compilers) and this is generally a better strategy, we do this here https://github.com/rse-ops/docker-images but again it’s arduous to debug new failures and see previously working builds break. What would help is if package maintainers would take on the burden of testing their stuff, and we do that with #FluxFramework. https://github.com/flux-framework/spack.

This isn't sophisticated but it was fun to figure out! For the #FluxFramework #Kubernetes Operator we want to be able to maintain a set of CRDs as examples that are always tested. This means in a GitHub action being able to:

🥑 Start a #minikube cluster
🥑 Install the operator
🥑 Pull a job container
🥑 Run the job
🥑 Check output
🥑 Check containers exit with 0

I got a matrix setup working! 🥳 https://github.com/flux-framework/flux-operator/actions/runs/3755306087/jobs/6380266335. It's a start. I think we could eventually do this with #kuttl. I ❤️ this stuff!

This isn't sophisticated but it was fun to figure out! For the #FluxFramework #Kubernetes Operator we want to be able to maintain a set of CRDs as examples that are always tested. This means in a GitHub action being able to:

🥑 Start a #minikube cluster
🥑 Install the operator
🥑 Pull a job container
🥑 Run the job
🥑 Check output
🥑 Check containers exit with 0

I got a matrix setup working! 🥳 https://github.com/flux-framework/flux-operator/actions/runs/3755306087/jobs/6380266335. It's a start. I think we could eventually do this with #kuttl. I ❤️ this stuff!

New "living documentation" for #FluxFramework is underway! https://twitter.com/vsoch/status/1604989861022179328.

🥑 repository renders from data that is updated nightly.
🥑 release posts are derived automatically
🥑 the site serves a static API of said data it uses for graphics
🥑 most content page are also generated from it.

Check out the longer thread above for more details!

New "living documentation" for #FluxFramework is underway! https://twitter.com/vsoch/status/1604989861022179328.

🥑 repository renders from data that is updated nightly.
🥑 release posts are derived automatically
🥑 the site serves a static API of said data it uses for graphics
🥑 most content page are also generated from it.

Check out the longer thread above for more details!

I’m going to be adding tests to our #FluxFramework #Kubernetes operator this week! I’ve seen scorecard for the operator-sdk but I’m looking for examples in the wild using best practices, and possibly not sticking to those suggested by that SDK. The e2e and unit tests in the #kubeflow MPI operator with ginkgo and gomega look nice! 👍 https://github.com/kubeflow/mpi-operator/blob/master/v2/test/e2e/mpi_job_test.go Anyone have other examples they like or experiences to share? 🤔

Dino #koobernottie

I’m going to be adding tests to our #FluxFramework #Kubernetes operator this week! I’ve seen scorecard for the operator-sdk but I’m looking for examples in the wild using best practices, and possibly not sticking to those suggested by that SDK. The e2e and unit tests in the #kubeflow MPI operator with ginkgo and gomega look nice! 👍 https://github.com/kubeflow/mpi-operator/blob/master/v2/test/e2e/mpi_job_test.go Anyone have other examples they like or experiences to share? 🤔

Dino #koobernottie

We just finished a demo using #nextflowio with #FluxFramework for a #ML pipeline! 👉 https://github.com/rse-ops/flux-hpc/tree/main/nextflow/ml-hyperopt 🎉

Thanks to the maintainers (Paolo!) for speedy responses, top notch memes 😁 and fun collaboration despite an 8 hour time difference! 🙌

And check out other Nextflow Community pipelines! https://nf-co.re/. Maybe it's time the #HPC community tries out some of these #bioinformatics pipeline tools for our workloads too? 🤔

If you have a basic thing you know you can run on #HPC with #FluxFramework
, how do you start to think about running in Kubernetes? If you are empowered to write simple containers that harness the same workflow manager, that's a start! And that's why this basic example is useful.

Some cool work today for #FluxFramework
! If you are working to containerize #HPC workflows, you often know the basics of containers and have an end goal of #Kubernetes. "But wait!" you say. "I'm not a Koobernottie yet!" 😭

Have no fear... there is a nice way to transition! 🦋

The #FluxFramework executor is freshly merged into #nextflow! Until there is a release, you can use it installing from the master branch.

https://github.com/nextflow-io/nextflow/commit/cc9fc3f0b9d7c1f3e59df2e00870415019a9c403

Time to get on some workflows for the #FluxOperator! 🥳

Cross posted from Twitter!

Hey #HPC community! I'm doing some fun work to containerize and test traditional workloads with some cool #FluxFramework
orchestration - I've already done #amg #lammps and #qmcpack dummy examples and #Snakemake 🐍 is up next! Can you point me to others you like? Thank you! 🙏