Took a fun couple hours creating a function that does his slowest part as a service (not doing his work for him--we realized this needs to exist outside of his work) and then also making a parallelized helper for it.

Yep, 10-15x speedup using 16 cores. I think the operational machines have 32 or more cores, so this is great. Gets our final runtimes down to <1s (easy case) and <10s (hard case).

#python #multiprocessing #threads #software #space #orbitalmechanics

Last updated 1 year ago

@the_curiostech I don't have a particular go-to. My frequent use-case is embarassingly parallel so I use mp.Pool.map().

No rpc, no queues or locks or anything. Just "please blast this code across 10000 items and give me the results".

I find if I write "base code" any more complicated than that the bugs I encounter are too hard for my tiny brain.

#multiprocessing #python

Last updated 1 year ago

Peng Qian · @qtalen
2 followers · 3 posts · Server me.dm

🚀 Supercharge your projects with Aiomultiprocess! Easily integrate & with this powerful library. Learn how through a real-world web scraping example. 🔥

📖 Read more:
qtalen.medium.com/aiomultiproc

#python #multiprocessing #asyncio #aiomultiprocess #webdevelopment #webdev #programming #coding

Last updated 1 year ago

Thomas Arildsen · @arildsen
115 followers · 443 posts · Server fosstodon.org

@jannem in my course, we use both and . It's amazing how easily you can get by with accelerating something way more by just throwing in a JIT compile decorator than by rewriting your code to run multiprocessing. It easily makes multiprocessing seem not worth the work.

#numba #multiprocessing

Last updated 2 years ago

MiaPlan.de · @miaplan
39 followers · 339 posts · Server social.tchncs.de

Wer multiprocessing.Pool nutzt, sollte auch
set_start_method('spawn')
nutzen. Warum und wo das Problem liegt steht hier:

pythonspeed.com/articles/pytho

#multiprocessing #python

Last updated 2 years ago

aegilops :github::microsoft: · @aegilops
109 followers · 373 posts · Server fosstodon.org

@joxean Can you instead collect the status/results from each worker as it finishes the tasks? That's the usual way to do it.

If you want realtime communication from a worker then you can use a Queue or a Pipe, depending on whether you need one way or two way messaging.

You set the Queue or Pipe object as part of the initialisation of the worker, with one end in the controller script and the other in the worker, and it abstracts away locking for you.

#python #multiprocessing

Last updated 2 years ago

aegilops :github::microsoft: · @aegilops
108 followers · 347 posts · Server fosstodon.org

@joxean what state do you need to share across workers with a Pool?

Can you just do the thing needing synchronization at the controller process, instead of in the workers? e.g. do expensive work in the workers, and print results as they come back, instead of in each worker.

Sometimes when I think I need to sync my workers I realise I can dedupe the workload upfront.

Can you share the code or problem to understand why you need locking?

#multiprocessing #python #concurrency

Last updated 2 years ago

++Out of Cheese++ · @out_of_cheese
8 followers · 44 posts · Server mastodon.social

Another very long article on a topic I am not using very often. However I learned something new and hope to remember it if I ever need it.
The author gives some nice to follow examples how to screw things up using multiprocessing and states strategies on fixing the shown race conditions

superfastpython.com/multiproce

#python #multiprocessing

Last updated 2 years ago

Johannes Hentschel · @johentsch
22 followers · 7 posts · Server hostux.social

This year's programming revelation for me was the library . Now that I've discovered how easily you can adapt existing code to make it scale up to any number of nodes, I can never go back to Python's built-in multiprocessing, especially considering all the limitations that have suddenly evaporated in a ray :mind_blown: Also, the interface is so dead simple, I can't believe it.

#multiprocessing #ray #python #programming #DataScience #BigData #scalability #parallelization #interfaces #ux

Last updated 2 years ago

CodeRefinery · @coderefinery
54 followers · 32 posts · Server fosstodon.org

schedule update:

- Now (xx:10), Parallel in . is useful for many people, is more as a demo. A good 50-min intro if you want to know what's available.

#PythonForSciComp #python #multiprocessing #mpi #teaching #livestream

Last updated 2 years ago

CodeRefinery · @coderefinery
54 followers · 32 posts · Server fosstodon.org

is going well. Coming up:

- Now: library ecosystem
- xx:15 or so: getting data from web APIs
- In about an hour: ( in )

twitch.tv/coderefinery

#PythonForSciComp #scipy #parallel #multiprocessing #mpi #python

Last updated 2 years ago

CodeRefinery · @coderefinery
54 followers · 32 posts · Server fosstodon.org

resumes tomorrow morning 9:50 EET / 8:50 CET, with these four lessons:

- scripts (moving from notebooks to reusable interfaces with )
- T+~1h: library ecosystem ( ecosystem)
- T+~1h: Getting data from web APIs with
- T+~2h: Parallel code in Python (, , and a bit more)

All are relative basic level, designed to introduce new programmers to the topics.

#PythonForSciComp #livestream #jupyter #commandline #scipy #requests #multiprocessing #mpi

Last updated 2 years ago

IT News · @itnewsbot
1661 followers · 238595 posts · Server schleuss.online

Fork and Run: The Definitive Guide to Getting Started With Multiprocessing - Since the early 2000s, the CPU industry has shifted from raw clock speed to core c... - hackaday.com/2022/09/15/fork-a

#fork #thread #openmp #featured #multicore #multiprocessing #softwaredevelopment

Last updated 2 years ago

Tech News Worldwide · @TechNews
11267 followers · 97982 posts · Server aspiechattr.me
CSDUMMI✝️🕊️🏛️ · @csddumi
121 followers · 1009 posts · Server norden.social

: I know thath this already exists, but I am bothered by the lack of a useful library.

So, I developed a tool that walks through a website and checks reachability of all sites linked and records their latency.

This was the first time that I properly deployed parallel processing in Python with multiprocessing.

codeberg.org/developers/mainta

#rfc #foss #cli #codeberg #multiprocessing #python

Last updated 3 years ago

Fred · @fredds
199 followers · 2490 posts · Server mamot.fr

passer de 45 minutes de traitement a 4 minutes, merci le
de 😍

#multiprocessing #python

Last updated 5 years ago