The “encryption” bug post mortem.
If you updated to macOS 12.3 recently you may have noticed some strange things like preferences, bookmarks, and serial numbers going missing.
We now have a fix for this and will release it shortly. While I wait for feedback from beta testers, I thought I’d write this post mortem.
Stacks v4.2.4 should be out soon. Watch your updates later today.
What happened
It looks like new security features of the app sandbox or a bug in the macOS user-preferences daemon (a daemon is a background app that runs all the time taking care of some system function). The macOS change causes that daemon to occasionally lock up in certain odd circumstances. It doesn’t crash, but it does reject every request from that point on.
The result is that all your apps – even outside RapidWeaver – seem like they’re brand new again. When your apps ask macOS for the preferences and macOS says, “No. Go away.”
I don’t think Stacks or RapidWeaver have changed their preferences substantially in a long long time. And I think it’s safe to say that this daemon should never lock up or behave in this strange way – even if you attack it with too much garbage.
The app sandbox and hardened runtime in modern macOS are designed expressly to prevent this sort of thing from happening: from one app changing the behavior of another, either accidentally or maliciously.
Why Stacks
Stacks and especially the Stack Updater are somewhat unique. Although they behave just like other updaters in many apps, the stack updater sends out many requests for updates all at once. If you have hundreds or even thousands of stacks installed, that can mean contacting A LOT of developers and checking A LOT of version numbers.
About the Updater
Stacks uses a set of efficient threaded tasks to perform many checks concurrently. How many depends on your computer hardware and whether it’s doing the check after you click the Check Now button or you let Stacks automatically look for updates in the background.
The updater uses Sparkle – the same updater that many apps use. Because Stacks is a plug-in (I suppose I should say “was a plugin”), and RapidWeaver contains Sparkle too, the copy of Sparkle inside of Stacks is heavily modified and customized so that we can update many things all at once and don’t interfere with RapidWeaver’s copy. But some things are still plain-old-sparkle. And the update prefs-file is one part that was largely untouched.
Sparkle stores a preference file for each app (or stack) that contains info about the last update and any preferences for skipping, background-installing, and ignoring specific updates. In fact users can modify EVERY PROPERTY of any update simply by modifying the preferences file.
But Stacks largely ignores these files and manages updates as a single group. There are too many stacks to tinker with one at a time. So all these intricate preference files are totally superfluous. But I have never modified that code (until today) because it worked, didn’t seem to hurt anything – and you know, basically if it ain’t broke don’t fix it.
What broke
The preferences daemon in macOS 12.3 seems to get upset about Sparkle writing the same preference keys to many preference different preference files. Perhaps it’s a new heuristic targeted to prevent malware? I don’t really know. Whether the preference daemon allows the writes or blocks them, it doesn’t affect stacks. Like I mentioned above, these reads and writes are entirely superfluous.
Strangely the preferences daemon seems to get a lot more upset with some users than with others. We are not sure why. It could be that there is yet an unknown ingredient that makes things much worse.
What’s the fix
Of course I can’t fix the underlying bug in macOS, only Apple can do that. But if I know Apple, once they’ve shipped something, these changes can remain for a very long time. So it’s up to me to work around the problem.
So, I simply stopped reading/writing those superfluous preference files.
Fortunately sparkle is quite modular and though it’s complex in some places, this part is pretty simple.
Speed bonus
Although the macOS preference system is very very fast – it can synchronize reading and writing thousands of preferences in a second, there is a hitch. It does this by reading that file into memory and then synchronizing the info to the slow file later on. The key here is that the FILE reads and writes that are very very slow. Even a fast SSD is 10 - 100 times slower then reading from memory.
But sparkle keeps a separate file for every single update. That means macOS has to do a file read for each one, and synchronize each with a file-write later on.
By just avoiding ALL of these little reads and writes I think it runs a bit faster. It’s probably not huge change for most users, but to folks with hundreds or even thousands of stacks installed, I think the difference should be visible immediately.
What was that “encryption” stuff about
It was just a red herring – a poor assumption based on some confusing info and unfortunately magnified by a misleading post by someone with a large platform.
But let me clear the air and explain in more detail were the assumptions likely went wrong.
When stacks starts up it reads in a little bit of info about each stack that it needs to display in the Stacks Library and then the things in the layout on the page you’re looking at. Some stacks use a lightweight encryption feature I provided to stack developers to protect discourage copying by other developers or unethical plug-ins.
While stacks loads it decrypts the necessary bits. Stacks uses its own copy of the an open source library called OpenSSL that’s baked right into the Stacks plug-in (statically linked).
But Stacks didn’t always “bake in” OpenSSL. Years ago macOS came with the OpenSSL library so there was no need, every Mac program in those days could just use the shared macOS library. But changed things up and now has their own SSL encryption library and stopped included the open source OpenSSL library. When Apple changed, we started baking in OpenSSL so Stacks and the stack developers didn’t have to change anything. You probably didn’t even notice. ;-)
When this very confusing bug hit the first users, it stopped preference info from being read – and all apps behave very strangely, including RapidWeaver. It reset RapidWeaver to load plug-ins from the default location inside located deep in the RW’s app sandbox group-container – a very well hidden spot some power users haven’t in a very long long time. For some this default location still has a very out-of-date set of add-ons from long ago, right before they switched to the custom location. For at least a couple folks, this included a very old copy of Stacks.
When RapidWeaver tried to load this old set of things the resulting errors and crashes were pretty confusing. It caused old dynamic libraries to load too, like the ancient OpenSSL – still included, but hidden and deprecated on Intel macs and completely missing on M1 Apple Silicon Macs. Needless to say the crash-logs were super confusing. I admit I even doubted myself for a few minutes and had to double check my build settings to make sure I was doing it right.
Unfortunately not everyone double checked and it sent several developers and users off in the wrong direction.
This is a good lesson for us all: try hard not to jump to conclusions too fast or ascribe blame before the problem is understood completely. It only gets in the way of actually solving problems and helping users quickly.