Continued corespotlightd process CPU overload issues

Follow-up to my earlier observations:

Yesterday, after using my M1 MBP for several hours with no issues to do the same kind of document editing use-case that I had going before, I decided to turn off Apple Intelligence on my M3 iMac, and so far it although I do see corespotlightd occasionally spiking in CPU, it comes back down in a reasonable amount of time, and is not causing random freeze conditions, nor has it caused a CPU panic at this time.

I have a follow-up call with Apple Support tomorrow morning. (I started squawking about this with them last week when the problem resurfaced after a re-install of the OS).

Correlation? It seems concerningly probable that something to do with Apple Intelligence integration is related to this issue.

Slightly OT, but I've often had the same questions about what Apple means with "%CPU." Obviously Apple does not make single-core systems anymore; my two Intel Macs have 8-core CPUs (one is a Core i7 and the iMac Pro is a Xeon) with hyperthreading turned on, meaning (in some sense) they have 16 cores. My Mac Studio has a 20-core M1 Ultra and my MBP has a 12-core M2 Max.

So if corespotlightd is using, say, 750% of CPU time on an 8-core CPU with hyperthreading, what does that actually mean? In my experience very few processes impose any kind of load on the 8 virtual cores. You'd think with Apple Silicon it would be more straightforward, with no virtual cores (although performance vs. efficiency probably complicates matters somewhat). But what's puzzling to me is that I'll watch my Mac Studio spin-lock as corespotlightd spikes up to say 225% CPU, but looking at CPU history in Activity Monitor, I'll see that eight of 20 cores have essentially zero load on them.

I know that some processes have to run on an individual core and cannot be split across multiple cores. But it sure seems like even if corespotlightd were pinning four cores, other processes, like say typing into an email, should still be able to use other processors that are seemingly idling.

Is this an issue with the Mach scheduler? I mean ultimately it's hard not to blame a process that can soak up nearly 800% of "%CPU" but still….

ericmurphysf wrote:

So far I've deleted the contents of the referenced folders on the two Intel systems I own—a 2020 27-Inch iMac and an iMac Pro—and saw immediate performance gains, especially with anything having to do do with search: Spotlight searches, smart folders in Mail, etc. Corespotlightd also seems to have calmed down significantly, generally using less than 15% of available CPU time (yesterday I saw it go as high as 1,400% on an 8-core system). Prior to deleting this metadata, it could take upwards of five minutes just to log out of my account; now it's just however long it takes to quit all running apps).

Progress report: since deleting all this metadata, not only has the system seemingly stopped writing more of it (or at least slowed way, way down in adding to it), but corespotlightd has been at barely above 0% in terms of CPU utilization. I'm not sure this is a magic bullet in resolving this issue, but so far it seems to have resolved a plethora of issues I've had with my Intel Macs.

Unfortunately for me none of the notable "fixes" listed in this thread have worked much for me. Multiple system processes in general are out of wack with this update, which I feel are related, and the only actual thing that works in keeping my SSD at any level of "normal" is not having Pages open at all. My document isn't nearly as large as what has been listed here, mine is only 1.3MB; a couple hundred pages with zero images and some hyperlinks, and yet opening it, even if I don't do anything with it, will spike disk reads up to ridiculous levels and cause the CPU to rise high enough to make the fans kick. If I were to make any edits, even just a few letters, disk write will multiply several times over compared to the idle. Feel pretty defeated on this especially since without this issue I'd otherwise be loving my new Mac that I just spent thousands on, and especially since this feels like one of those things Apple will never acknowledge.

Mitch Stone said:
Curiously, deleting the Spotlight plist file does tame this behavior for better part of a day.
Turning off the "help Apple" selection in the Spotlight settings as suggested by another user 
did as well. But this is not a sticky fix. The problem always returns, at least for me.

Mitch Stone, you've been advocating for deleting the Spotlight plist file. The reason I opted for deleting the contents of the CoreSpotlight and SpotlightKnowledgeEvents folders was because you said on Dec. 30 that deleting the Spotlight plist file was a fix that would only last about a day. Am I misunderstanding something?

I should probably clarify that whether or not I have multiple Numbers files open on various computers seems not to affect either how much CPU time corespotlightd uses or how quickly Spotlight metadata piles up.

Having even one Pages file open, by contrast, has a huge effect on both.

sugarskyline wrote:

If the folder is the core problem, what would happen if someone locked it (Get Info > Locked) ? Not well versed on how the OS works so I don't want to attempt in case it breaks something, but theoretically, would be there any value in locking the folder/relevant innermost folder to prevent anything from being written?

Interesting question. I don't know the answer.

I do know that file and folder locking in the current MacOS is a very old legacy of the Classic MacOS, and that the read/write permissions of OS X basically override it.

Of course, you cold also change that folder to read only. I don't know what would happen, but I'm wary of trying it because a bunch of spotlight processes expect to be able to write to that folder. I worry that some stuff could go haywire if you randomly blocked their access.

The good thing about my current workaround is that it minimizes the potential problems, but doesn't break anything.

Mitch Stone wrote:

Having tried this myself, I must report a non-confirmation. I opened a large Pages file and watched the Corespotlight folder file size. It started out at 60.35 GB and remained exactly this size after a half hour, even though the process showed as being very active (100+ percent) for part of this time. I don't doubt that deleting it has a temporary effect but it's also clear that this folder growing in size cannot be triggered predictably by opening a Pages file.

This is helpful. Some key questions:

1. When you say "the process showed as being very active" do you mean corespotlightd? For the record, when I repeatedly watch the corespotlight folder grow (with a Pages file open), it is NOT associated with the corespotlightd process. To the contrary, it's mdworker and mdstores that are writing all the data to that folder.
2. Do you have optimize iCloud storage turned ON, on your machine?

On my Mac with optimize OFF, the corespotlight folder always grows with a Pages file open. But on my Mac without optimize storage ON, I do not see the growth as consistently.

I have another question: do people with Apple Silicon systems ever see the size of their Spotlight metadata folders decline? I've got two Intel systems and two AS systems, and while I never have seen these folders' sizes shrink (other than when I manually delete them) on Intel systems, I often see them get smaller on their own on AS systems, especially on my M2 Max MBP.

I'm not sure this is actually happening (I suspect it's just the Finder having difficulty giving accurate sizes to folders with a lot of subfolders, which is more the case with AS systems). But I have noted that AS systems do not seem to see as rapid growth of Spotlight metadata as my Intel systems do. I've repeatedly deleted metadata on Intel systems only to see it grow back to an absurd extent. But I've never deleted the metadata out of either AS system, and so far neither one has ever gotten any bigger than 47 MB, and currently neither system has more than 25 GB worth of Spotlight metadata.

Regarding the two large folders (NSFileProtectionCompleteUntilFirstUserAuthentication & Priority) under the ~Library/Metadata/CoreSpotlight folder; I don't have "Advanced Data Protection" turned on, so how is it that the NSFile... folders exist - is this part of Apple Intelligence? The NSFile... folders do not exist on my Intel iMac, only on the M2 laptop.

Has anybody removed (or renamed) these large directories, turned AI off and restarted - I just wondered if they get recreated.

Also, has Apple Support shed any light as yet?

Nothing in my understanding of this problem, which echoes the conclusion that Eric Murphy mentioned early today (namely, the root of the issue is how spotlight indexes pages documents) – nothing there gives me good reason to believe that this could be fixed server side.

Nonetheless, in the interests of scientific transparency, I have to report that so far today the core spotlight folders (in ~/Library/Metadata) are not growing in size at all, despite my having a Pages document sitting open. This is the case on my M2 Pro through 3 hours of work this morning, and repeated on an Intel iMac through another 3 hours this afternoon. In both cases the core spotlight folder is not budging, while a Pages document remains open.

I can't explain it. Could just be random. But it seems conspicuous to me that the problem seems to have disappeared not long after sugarskyline's long support session with a higher-up apple tech. ??