@Ruyton Oh, yes, always! In fact I make date sensitive project files named something like: myproject-2022-08-22.rw8
… and I always make sure to use RW’s ability to send a zipped backup to the host once a day.
… otherwise hell could, no probably would, break loose!
“So I’m guessing the solution is to go through all projects and delete the shared folder if it’s got random content.”
I’d suggest doing what I recommended in post #2 — deleting the images but keeping the folder and the plist, but also deleting the ‘Subsandwich’ in the plist where the images are also stored. RW/Stacks is clearly going to look for a YHStacksPlugin folder and to want to read the plist in it. What we want is for there to be nothing else there in either of them.
Yep. My resources take up one cabin in SS RapidWeaver — all the rest is other people’s random stuff.
“Wait, Dan, what was that juddering on the starboard side as we passed the great big white thing?”
I’ve been very busy prepping a new upgrade to Stacks, so I’m afraid I’ve missed a lot of the chatter here. So first off, let me apologize for that.
But after reading a few of the posts here I can see that some folks are definitely confused about what’s going on and why.
So I’ve made a little video about Externals that I hope answers at least a few of the questions here. I tried to cover three things:
Even though I’m moving pretty fast, it’s a complex topic and talking about it in detail takes some time – so it is a 20 minute video. But do take the time to watch, I hope it has enough detail that everyone will learn a little bit they didn’t know.
But let me also say I’m sorry this part of the Stacks experience is a bit confusing. When I designed some of these things RapidWeaver worked a bit differently and in a way that made it easier for Stacks to keep track of where something like an External or a Partial was was needed and where it wasn’t.
In today’s RapidWeaver only parts of a project are active at any one time. This is great as it allows projects to open in a fraction of the time – but comes with a hitch – sometimes the project can change even when Stacks is not yet active.
I demonstrate this is the video. Hopefully it’s clear why Stacks sometimes behaves conservatively to preserve data even in cases where you didn’t expect it to.
And finally I demonstrate how to eliminate unwanted externals from your project once and for all. But please be aware this can be quite a bit more difficult on a large project where there are many pages that have Externals data stored in them.
If you follow along and still can’t seem to get rid of some pesky externals data – send me a copy of your project and I’ll see what I can do.
I should also say that Partails, Externals, and Templates all behave slightly different in these cases. Templates, because they’re locked down, we can make some assumptions there. Partials live in only a single document, so we never have to worry if they’re needed in another project file.
But externals, with their super-power of being able to be shared from project to project – saved and sent to colleagues – this super power comes with a super weakness too: we must always worry that the data is needed in any open file and migrate it whenever there’s any chance of that.
For that reason you need to be a little bit careful when you use externals yourself or share them with others – because they may jump from project to project or move along with pages where you didn’t intend for them to move.
Lastly, I think it’s worth mentioning that I’ve been working on a Stacks app for later this year. One of the advantages of Stacks owning the whole app is that cases like Externals can be managed a bit more smoothly. Stacks always knows every page that needs a externals and can easily decides whether the data should be migrated from one project to another. Because the project files were designed from the ground up to work with complex data we can manage all this data in a way that makes sense and doesn’t require so much duplication.
Deleting the shared plug-in data is… scary. Even for me, I’m not sure I would recommend that. However, if you’ve not experienced any negative side effects, and it seems to be getting the results that you like i have a couple things to say:
What you’re doing, from a high level perspective, is deleting one of the places Stacks stores partial/external/template data.
Let me see if I can explain how it works, without getting too technical.
The first thing you should know is that there is NO WAY for Stacks to communicate from one page to another page in any predictable way. Pages can only communicate with each other when they are both active and loaded into memory. In a large project it’s very rare for that to happen to all your pages. In most cases, people work on a subset of their project, upload those changes, then Quit. In that normal use case, only the pages that you visit are loaded into memory – the others are all “asleep” – when they might wake up is impossible to tell.
Once you know that little detail the use of the “shared” data area might become a bit easier to guess. It’s used as a communication channel from page to page.
OK, let’s imagine a project – and see how that project communicates from page to page. I’m going to describe a pretty strange project. It’s unrealistic – but just to keep things simple enough to describe in words.
Let’s imagine a medium sized project that has 26 pages named A → Z. Let’s say that on January 1st you edit page A, and then on Jan 2nd you edit page B, and so on. Each day editing one page until Jan 26 when you arrive on page Z and publish your finished site.
Now let’s imagine that on every page there is a shared partial at the bottom called “Footer”. Every page on your site utilizes this Footer partial.
On each day of the month you add a new image to the footer partial. On Jan 26th there are 26 photos in the footer.
The homepage of the site (like my example videos) is NOT a stacks page. Let’s say it’s a blog. And on each day, after adding the image, you switch to the blog and write a nice blog entry for the day, save your project and Quit RapidWeaver.
On the first day you add the first image to the Footer partial. This makes the Footer partial version 1.0.0 – then you add your blog entry, save, and quit.
In this case Footer v1.0.0 is stored inside the Page A data-file. Since you have not visited any other pages, they haven’t loaded into memory and won’t get to know about Footer v1.0.0 yet.
On the second day you open your project and it, of course, opens where you left off: the Blog page. When this happens the Stacks plug-in is barely “running”. Stacks kicks into gear when you move to page B and go to add the second image to the Footer partial.
But wait – Page B wasn’t loaded on your on Jan 1st when you added the first image and saved Footer v1.0.0 into the Page A data-file.
Now on Jan 2nd, page A is dormant and has no way to communicate to Page B. How will page B get Footer v1.0.0???
This is where the “shared data” comes in.
When you saved Footer v1.0.0 to Page A, Stacks also saves a copy of this partial to the “shared” area of your project file. When you open page B, Stacks will see that the page B copy of Footer is still back at v0.0.0 and the shared version of Footer is at v1.0.0. So Footer updates to v1.0.0 before displaying the page to you.
And so it will go on each day that the shared data will act as a conduit to transfer the Footer partial to each next page – and each page will display the correct newer version – even though each page has no way to directly talk to any other page since none are loaded on the same day.
And on the final day Jan 26th you make the final change to Footer – it’s now v26.0.0. Then you choose “Publish” and since every page has been edited, they will each need to export their data to send to your web-host. When each opens up, they will each find that the Footer v26 is the latest and will each incorporate that latest version into their page using the shared data rather than the copy of they have in their own data-file.
So when the site publishes it will have only v26 on every page. Perfect!
And it all happened without any page being able to directly talk with any other page.
This, of course, is a very contrived example. In most cases there will be a few pages loaded during any session – and those pages can communicate their partial/external changes to each other and don’t need to use the shared-data as a communication conduit. So in many cases the shared-data area is completely redundant. But not always…
In many cases it is necessary and definitely will be used.
Nuking it, will nuke that communication conduit. In most cases this will not cause any side-effects – but it’s not 100% safe – and it might be very difficult to immediately see what side effects there are.
I could write for days more about this topic. There are A LOT of details I’ve left out here:
there are many corner cases.
but please know that in all cases stacks behaves in a conservative way to ensure data integrity.
Things will be quite a bit different in StacksPro (the app). For obvious reasons I’d like to keep the details of the project-file as secret as possible for as long as I can. But I’ll say what is probably obvious: site-wide project info, like partials and externals, is stored in its own special place within the project. This one simple change makes a world of difference. It makes the entire system simpler, easier to understand, and vastly more efficient.
But for many folks the first version of the app may not have 100% of the features they need to build their site – and for that we to have the Stacks Plug-In running the same Stacks5 core rendering engine inside.
This new Stacks Plug-In upgrade – that should come in the next few weeks – provides some nice new UI features, but also absolutely more important: it synchronizes the core Stacks5 rendering engine with the coming app.
That way you’ll be able to easily use both, copy and paste and drag and drop between them.
I have a ton to tell everyone about the new Stacks Plug-In upgrade that’s coming in a few weeks, and about the StacksPro app that’s coming in a few months, and about the Stacks5-Rendering Engine inside them both – but at the bottom of a thread about data-file bloat probably isn’t right. LOL 😆
Assuming I have a few big brain-dumps about the new stuff coming, what would you guys rather see:
If you have answer to this, please, instead of posting them here let me know on @isaiah on Twitter: https://twitter.com/isaiah – that way i’ll definitely see it and it won’t pollute this thread with other info
Isaiah, I had caught something about externals before, but I didn’t pay any attention to it because I’ve never used one. I do use partials occasionally, but I find the two-step editing a bit annoying and try to avoid them. Otherwise I’m very happy with Stacks.
My problem, though, is not with externals or partials, but with my RW files filling up with images that are not mine, have never been used in any of my projects and which, in the vast majority of cases, I have no idea where they come from. And they take up a lot of disk space — about 46MB per file. Since I do back up regularly, this means I have a lot of 50MB RW files that should be no more than 4 or 5 MB. (This becomes even more of a nightmare when RW backs these files up to my server, taking up vast amounts of space that I’m renting).
I should add, too, that I haven’t deleted the places where Stacks is putting these images. I’ve left the folder and deleted the images. Likewise I’ve left the plist but deleted the images in it. And that seems to work. Well, I’d expect it too — I can’t imagine that any of these pictures is essential for Stacks to run. But I just don’t want them there. And if this wasn’t so extreme — if it was only adding a few 100kb to my files, I’d be happy to let it go. But this 46MB seems to be a constant, half from images in the folder, half from those same images in the plist. And none of these are my project images: those don’t end up there. These images are just some terrible detritus that my files have picked up from somewhere. Like the digger image. It’s not as if I even open other people’s files — apart from a couple of projects from Gary and Stuart, and a collection of demos from stackmakers, I don’t have anyone else’s files.
As this random artifacts drove me crazy, I followed some time ago the advice at the Stacks Slack and started deleting the whole Shared Plugin Data folder from all of my projects. This finally removed the vagabounding images and, more important, the random partials I had in every project.
Sometimes I had to click through the pages to get my own partials back (not visibile initially), but that’s okay for me.
Yes, that’s the issue here that is causing large files to be created nd then to potentialy cross contaminate other files and make them even larger. I had a quick look and found a RW project file that I created was 220Mb. From the images inside I traced it back to another file that I was sent while trying to offer help to a forum user. Perhaps I had it before that.
From James and Mitch’s detective work, I now know what to do, to avoid this.
Others may not be comfortable editing files so there is potential for users not to offer help by asking for a version of their project file.
@jamessouttar BTW I also have that same digger image, but there is no real point in starting a witch hunt. I expect we will all have the digger image and more, soon.
Haha, I don’t have anything against the owner of the digger image — it’s just this surreal situation, in that borderland between the tearing-ones-hair-out frustrating and the laugh out loud hilarious, where I have hundreds of files populated by this same bunch of bizarre, random images and I’m paying hosting fees for even more duplicates to be housed on the server.
Isaiah, this post of yours has been very important, to me at least. But I do not use Twitter (or any other similar platform), so this way of communicating with you, or getting information from you, is impossible for me. Could you, please, continue posting on this forum all the new info about your progress? You could use one of already existing threads, or create a new one.
@fapkogi - yes. i always post in as many places as possible. i don’t post Stacks info to twitter too often – it’s primarily a channel for personal communications and the goings on in my life as a developer and dad.
i always try to “make the rounds” and post to the forums when i can. but forums have info continually – but there are now (discord (private) slack (public and private), Dan’s forum, this forum, Adam’s forum, and Joes forum, and a few twitter accounts, and the indomitable Facebook (which I try very hard to boycott for their privacy issues) – that’s at least 9 and i suspect i’m forgetting a few.
i rely on a few of the nice people who run these forums and channels to pass along info when i get busy with coding (like right now) – i hope you understand. :-)
isaiah
i don’t know if you noticed, but in the last update of Stacks I made the Image Backup (which is sort of the same redundant store – but for images) a more public and easily accessible thing.
My guess is that your libraries include some Template stacks that contain image data that is then finding its way into your projects.
As with any complex issue – I would suggest:
when you add in something that causes unwanted images to be included, then we have our culprit. if you manage to track it down, let me know. perhaps there is a way for me to mitigate this particular problem without degrading the integrity of other things.
StacksPro will solve those issues. Until then:
Backup, backup, backup…