I’ve regarded round rather a lot to attempt to discover the reply to this however have to this point been unsuccessful. I’m programming in Python, and I’ve code that takes a very long time to run (hours to months relying on the applying) and I’m attempting to optimize it. On bigger thread methods (as soon as I get above ≈8 threads, however particularly >18 threads), I run right into a scenario the place the system is utilizing up a considerable amount of my CPU as an alternative of the particular code I wish to run. On a 12-thread machine as I sort this, the system is utilizing ≈25–30% of my whole CPU. If I attempt to run it on a 36-thread machine, the system takes >70% of the CPU which is solely unworkable (although operating two cases of the code, limiting every to 18 threads, cuts this overhead again, oddly sufficient).
The Exercise Monitor signifies that taskgated is utilizing not less than 10%, whereas notifyd, logd, and launchd are additionally utilizing a number of %, every (together with mds at 1% so I ought to flip off Highlight, and typically lsd spikes to >40%, however that is extra uncommon … word that that course of is one other launch system daemon).
My older speculation was that it was a disk I/O challenge, because the code was writing and studying many small information to attempt to preserve monitor of sure issues and recuperate if I wanted to cease it.
My now-working speculation, based mostly on what I may discover on-line in regards to the interactions between taskgated and launchd, is that this explicit chunk of code is spawning a lot of processes and macOS’s launching and safety daemons are taking giant quantities of CPU to verify these processes are protected. These are issues like calling “mv” and “rm” from the command line (os.system(…) in my Python code), and spawning different code that take a second or two to run (corresponding to in a unique conda atmosphere when two have conflicting installs of mandatory parts). I rely not less than 40 potential spots the place this chunk of code is presumably spawning little one processes, and I thread it in order that it is doing that concurrently nevertheless many threads there are (so, on a 12-thread machine, 12×40 over the course of ~10 seconds). It would not appear to me like this ought to be making my system take up a lot CPU, however that is my finest guess in the meanwhile.
Probably associated, the taskgated is continually spitting out to log information, “MacOS error: -67062”, which once more I’ve looked for and located no luck with diagnosing the problem. And, diskarbitrationd is spawning numerous “
My apologies for rambling a bit right here, however I am attempting to offer the knowledge I’ve, and hopefully somebody right here has an concept. If I can do away with this 25% or bigger challenge, that may save months of time.
For what it is price, I am operating macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop computer. Identical challenge for all. Working in Linux on an almost equivalent 36-thread system construct doesn’t have the overhead challenge (however I actually do not wish to change to Linux) which is another excuse why I do not suppose it is a disk I/O drawback.