the Sim Settlements forums!

Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Long Save Research Tool

Had a random thought this evening while researching this - if this is an issue with stack dumps (ie. too many scripts running), then Skyrim will have the problem as well... sure enough: https://github.com/Wildlander-mod/S...-quicksaves-not-recommended-for-use-in-ultsky

That simultaneously gives me peace of mind, as I didn't know what I would do if the issue were the esm changes in 2.0.0, but also means I've got some serious coding optimization sessions ahead.

@Mystical Panda It seems like you're getting closer to have a piece of software/methodology that can analyze this stuff in realtime and figure out which scripts are causing all this traffic. Having some starting points would definitely help. In addition, if you can get it streamlined enough to distribute to others we could widen the net to find out which scripts are causing all the backup.

Does any of you have any "long-saves" with a bunch of ActiveScripts? I've noticed that since we fixed the spotlight bug outside GNN, the saves folks are sending me for the long save issue tend to have pretty minimal numbers of ActiveScripts - which is what I would normally look at to trace back where the backups were occurring.

All that said, one of the things that still has me questioning the "too many scripts" theory is that I occasionally get the long saves while doing my Let's Play during the Q&A sessions. During that part of my stream, I set the gametime to 0.1 so that the weather doesn't change, which means the timers effectively never trigger to run all of the systems in SS2. When checking those saves in Fallrim tonight, they all had 0 ActiveScripts.

You could say that.
 
I wonder if there is significance in the fact the VM was frozen and it took ~1 min from the save call to the thaw VM call? :scratchhead
The save apparently is partially complete when it 'goes dark' or hangs then completes some time later (the 'long save' delay). Which is the interval between the save and thaw.
 
I wonder if the the long saves where there aren't stack dumps just have a significant amount of references tangled up in the scripts that are running.

For example, does it take longer to freeze 100 running scripts or 1 script involving 100 different references? We don't really know, but that seems reasonable that both scenarios could be problematic - and there are plenty of times where SS2 is iterating over tons of items.
How can we test this? The papyrus log doesn't seem to give enough information and when we get to the save files they are virtually identical.

I'm up for some testing, but at this point, I'm way beyond my knowledge level (CK and papyrus I mean) and I'm basically firing at all directions to see if I hit something xD

Used a few hours of my free time today to look at code instead of playing the game. Basically, I attempted to do what I proposed to you. I've downloaded v1.0.17b (could not find 1.1.0) and 2.0.0. Created a local rep in my machine with the 1.x code as the first commit, replaced the files in the folder with the 2.x and passed a few hours looking at the code. It was amusing to see how it works behind the scenes, but as I suspected, I have no idea what might be causing the problem. I was a little scared when I started seeing locks flying around, but then I realized that they were just control bools (flags) and not real locks (as in multi-threading locks). I've learned that function calls are heavier than casts in papyrus, which blows my mind, as I always was thought that casting is the heaviest thing you can do in any language, but that's Todd for you xD. Also, I consulted the papyrus documentation on tips for saving and, while it is impossible for me to say if all of the code is following the good practices, what I can say is that as soon as you start a new game none of it matters. Problems only start to appear when you update mods in the same save. Once again, Thank you Todd!

I've paid more attention to the data structures and the SimPlot files. By comparison with what was there before, nothing appears syntactic and semantic different. I don't have enough knowledge in papyrus to say if they are optimized or not, but the new code seems consistent with the old code, and the old code worked. But these are just two files of a 300+ files commit. Just by looking at a glance, it seems like 1/3 of the files were modified while 2/3 are new files.

Another thing I noticed is that the old code had a lot of work already prepared for chap 2. I'm assuming these functions and scripts were never called and were waiting for the rest of the implementation, as by the end of 1.x life cycle makes sense that the team was working both on bug fixes and implementing the new content.


TLDR: I have no Idea what I'm doing, I'll just keep playing and testing other people's ideas xD
 
I wonder if the the long saves where there aren't stack dumps just have a significant amount of references tangled up in the scripts that are running.

For example, does it take longer to freeze 100 running scripts or 1 script involving 100 different references? We don't really know, but that seems reasonable that both scenarios could be problematic - and there are plenty of times where SS2 is iterating over tons of items.
I was wondering that also- if a shared resource was being used by one script while another was waiting for it. There was a problem awhile back with game engine list mitigation- and thus random crashes. The one approach I'm trying to work code around... The active stacks show what's currently executing and the suspended show what's waiting. If we can combine the two is a readable 'stack trace' we should be able to see what is calling what, and in what order. The profiler (TGPG 1) does this, if I'm correct, by logging sequentially what the VM is doing, but I don't recall seeing time stamps. Time stamps would show the 'long save' gap, and what was the last script trying to execute.

What I'm seeing by looking at the dumps is many stacks each have 1 frame and the frame has an 'onLoad' call. If these are rendered 3d items events this would make sense if your in a settlement- the item is loaded to be rendered by the game engine while in the cell, but should those show up out of a settlement? I'm pretty sure I was getting dumps outside of settlements also.
 
How can we test this? The papyrus log doesn't seem to give enough information and when we get to the save files they are virtually identical.
As PITA as this sounds- The only way I can see to do this, maybe from a logistics point of view, is to have each script (or more painfully) each function log in the papyrus an 'I've started...', 'I've completed...' message, thus adding more information. Even selectively- with so many scripts at a time doing this, and if they don't seem to have a problem disable or remove the logging code. Then check those time stamps before and during save (or just after if the game engine halts logging messages during the save process) and see which scripts haven't completed yet, or just completed some time later.
 
Most of the onload calls I saw in your screenshots were from the information icons, and that system was the same in Chapter 1, so I've been treating those as a non-issue since this problem started with 2.0.0, my assumption is it is either new code in 2.0.0, or new code in WSFW that's at fault.

In the HUD options you can actually turn off those floating icons, and they should stop generating all that traffic for you - might help to eliminate that from your logs.
 
As PITA as this sounds- The only way I can see to do this, maybe from a logistics point of view, is to have each script (or more painfully) each function log in the papyrus an 'I've started...', 'I've completed...' message, thus adding more information. Even selectively- with so many scripts at a time doing this, and if they don't seem to have a problem disable or remove the logging code. Then check those time stamps before and during save (or just after if the game engine halts logging messages during the save process) and see which scripts haven't completed yet, or just completed some time later.
That actually shouldn't be too hard to do right? Just regex edit all the script files searching for Function *() and EndFunction to add in the debug stuff and then bulk recompile them with the CK.
 
As PITA as this sounds- The only way I can see to do this, maybe from a logistics point of view, is to have each script (or more painfully) each function log in the papyrus an 'I've started...', 'I've completed...' message, thus adding more information. Even selectively- with so many scripts at a time doing this, and if they don't seem to have a problem disable or remove the logging code. Then check those time stamps before and during save (or just after if the game engine halts logging messages during the save process) and see which scripts haven't completed yet, or just completed some time later.
Good old C style debugging... well, if there is no other way, I might build a simple script that adds a debug log after each function declaration and another just before every return and pass the program to the devs. It shouldn't be that hard, but is it worth the trouble?

If you think it is worth it I'll code it tomorrow.
 
Most of the onload calls I saw in your screenshots were from the information icons, and that system was the same in Chapter 1, so I've been treating those as a non-issue since this problem started with 2.0.0, my assumption is it is either new code in 2.0.0, or new code in WSFW that's at fault.

In the HUD options you can actually turn off those floating icons, and they should stop generating all that traffic for you - might help to eliminate that from your logs.
Dunno if you knew (you probably did), but you can scrap the icons by mistake xD refreshing the plot brings it back to normal.
 
That actually shouldn't be too hard to do right? Just regex edit all the script files searching for Function *() and EndFunction to add in the debug stuff and then bulk recompile them with the CK.
Unfortunately, Ihave very little scripting experience or with using the CK. If you can lay me out some steps or point me in the right direction to get started it's definitely something I'm interested in learning as there are some 'base' changes I'd like to add to the game. Such as locking things behind a faction wall... such as... your spouse is alive and in stasis at the institute, and the one that was killed at the beginning of the game is a synth... just like you as your there too. Along with many many more...
 
Last edited:
Good old C style debugging... well, if there is no other way, I might build a simple script that adds a debug log after each function declaration and another just before every return and pass the program to the devs. It shouldn't be that hard, but is it worth the trouble?

If you think it is worth it I'll code it tomorrow.
That's just it. I'm trying to determine out how to figure out which call is hanging up (if that's what's happening) and if the papyrus log text is formatted a certain way, we can use the time stamps to determine how long it took for the function to complete. It'll 'spam load' the papyrus logs that's for sure- being a combination of, in a way, a profiler log and papyrus log. I don't really have scripting experience or the available functions with the CK, if it's possible to wrap it around an "if functionTrace=1", it wouldn't spam unless turned on, which could be done from an MCM option. I've done some MCM stuff years ago with Fallout 3, but nothing recent.

The biggest problem is it will increase the size of the compiled scripts, but if it's only as a test branch, hopefully it'll be just for those testing and trying to figure out the cause for this.

Also, SS1-SS2 always does a ton of things, script wise, and there were not 'stack dumps' continuously before. Which would make the number of active scripts, or SS2 processing being done, not necessarily the problem. However, it wouldn't mean there were no 'suspended stacks' prior to this at all, just not enough to kick it over the limit and dump into the papyrus log. And definitely not enough to draw attention to papyrus logs like now as were starting to spotlight the 'long save' issue.

That's the problem I'm trying to work towards- how to determine what might be temporarily stopping the VM 'pipe' enough to allow things to start stacking, and kicking suspended stacks (or frames) over the suspended limit.
 
if it's possible to wrap it around an "if functionTrace=1", it wouldn't spam unless turned on, which could be done from an MCM option. I've done some MCM stuff years ago with Fallout 3, but nothing recent.
My assumption was we'd do it as a fork, not in the main code base.
 
Don't know how much I can help but wanted to add my two cents. Started a new playthrough and wanted to "walk my saves" until I started running into long saves. I have no scrap mods. In the attached file:
  • Save 6 is Starlight Drive-In completely scrapped as allowed by vanilla. I saved it in Starlight before building a beacon. No long save present.
  • Save 7, built the beacon, Jake builds the first Residential Plot and goes to Concord to check on his workshop. I saved it in Starlight again. No long save present.
  • Save 8, met Jake at his Concord Workshop and he gave me the Holotape. Saved in his Workshop at Concord. No long save present.
  • Save 9, built and completed the Agricultural Plot. Saved in Starlight. No long save present.
  • Save 10, Industrial Plot completed. Met Old Paul and Jake leaves. Start the quest "If I Had A Hammer". Long Save
In my previous playthrough, my long saves happened in or around the "Main Settlement" of Sim Settlements 2. First playthrough before I scrapped it, anytime I saved in or around Sanctuary, long save. This one, Starlight Drive-In, same thing. The "Main Settlement" is where my problems are. Hope that helped.
 

Attachments

  • BlackKnightOfGileadLongSaves.7z
    277.7 KB · Views: 0
Don't know how much I can help but wanted to add my two cents. Started a new playthrough and wanted to "walk my saves" until I started running into long saves. I have no scrap mods. In the attached file:
  • Save 6 is Starlight Drive-In completely scrapped as allowed by vanilla. I saved it in Starlight before building a beacon. No long save present.
  • Save 7, built the beacon, Jake builds the first Residential Plot and goes to Concord to check on his workshop. I saved it in Starlight again. No long save present.
  • Save 8, met Jake at his Concord Workshop and he gave me the Holotape. Saved in his Workshop at Concord. No long save present.
  • Save 9, built and completed the Agricultural Plot. Saved in Starlight. No long save present.
  • Save 10, Industrial Plot completed. Met Old Paul and Jake leaves. Start the quest "If I Had A Hammer". Long Save
In my previous playthrough, my long saves happened in or around the "Main Settlement" of Sim Settlements 2. First playthrough before I scrapped it, anytime I saved in or around Sanctuary, long save. This one, Starlight Drive-In, same thing. The "Main Settlement" is where my problems are. Hope that helped.
That's the pattern I observed in all my playthroughs. As your settlements grow, you'll start having problems outside the main settlement as well. Eventually, around 10-15 settlements, if you have the option for them to grow without you being present, you'll start getting long saves as you receive notifications of buildings/upgrades in settlements.
 
it seems to be related to scripts instead of ESMs (whatever that means... once again, I know nothing of this engine)
Based on what I have seen/read, I see Fallout 4.exe as the main controller while the Papyrus VM is a separate thread. The .esm, .esp and .esl files are the initial data that create an "initial" database with the included records being the last one loaded. (aka the load order) The save file is a "runtime" database of record changes made during gameplay + the "initial" database. (save baked data) When a Papyrus script function call is made, a stack is created in the VM. The VM asks the game engine for the required data and pauses (suspends) the script function call (stack) while it waits for a return. This happens during the time slice allowed per cycle.

This is just my observation based on things I have learned while creating mods. I'm not a coder by trade or even a script kiddie. (or anything related to IT...) so I have no clue as to how accurate this is. ;)
changes inside the settlement can trigger long saves
The main scripts dealing with plots would be SimSettlementsV2:ObjectReferences:SimPlot SimSettlementsV2:Quests:PlotManager and SimSettlementsV2:Quests:ResourceManager Maybe SimSettlementsV2:Quests:EmpireManager is involved as well.
So about to test Notepad++ compare plugin...
:shok1 Link please??? That sounds really useful. :giver:
Just regex edit all the script files searching for Function *() and EndFunction to add in the debug stuff and then bulk recompile them with the CK.
How do you do this? This sounds incredibly useful.
The profiler (TGPG 1) does this, if I'm correct, by logging sequentially what the VM is doing, but I don't recall seeing time stamps.
Yeah, it would be nice if there were time stamps. IIRC, when calling console commands from a text file (bat console command) the called commands do not wait for a return... :( So something like:
Code:
; SaveGameAndProfile.bat
TogglePapyrusGlobalProfiler ; TPGP
Save "ProfiledGameSave"
TogglePapyrusGlobalProfiler
and using the console command bat SaveGameAndProfile would turn off profiling during the save. I know you can call console command functions from a script, but I do not know how. (F4SE?) I'll do a little looking and see if I can come up with something if you think it would be useful to only have global script profiling active while a save is in progress.
 
That's just it. I'm trying to determine out how to figure out which call is hanging up (if that's what's happening) and if the papyrus log text is formatted a certain way, we can use the time stamps to determine how long it took for the function to complete. It'll 'spam load' the papyrus logs that's for sure- being a combination of, in a way, a profiler log and papyrus log. I don't really have scripting experience or the available functions with the CK, if it's possible to wrap it around an "if functionTrace=1", it wouldn't spam unless turned on, which could be done from an MCM option. I've done some MCM stuff years ago with Fallout 3, but nothing recent.

The biggest problem is it will increase the size of the compiled scripts, but if it's only as a test branch, hopefully it'll be just for those testing and trying to figure out the cause for this.

Also, SS1-SS2 always does a ton of things, script wise, and there were not 'stack dumps' continuously before. Which would make the number of active scripts, or SS2 processing being done, not necessarily the problem. However, it wouldn't mean there were no 'suspended stacks' prior to this at all, just not enough to kick it over the limit and dump into the papyrus log. And definitely not enough to draw attention to papyrus logs like now as were starting to spotlight the 'long save' issue.

That's the problem I'm trying to work towards- how to determine what might be temporarily stopping the VM 'pipe' enough to allow things to start stacking, and kicking suspended stacks (or frames) over the suspended limit.
I'll try to make a simple C# program today that edit all the scripts recursively. It will add an "I'm starting" with the name of the function as the first line of the function and in all returns and EndFunction will print "I'm stoping".

Shouldn't be that hard, I'll report on progress.

This is just my observation based on things I have learned while creating mods. I'm not a coder by trade or even a script kiddie. (or anything related to IT...) so I have no clue as to how accurate this is. ;)
I really appreciate the time you're taking to explain stuff to me. It has been much helpful :agree:
 
Last edited:
Don't know how much I can help but wanted to add my two cents. Started a new playthrough and wanted to "walk my saves" until I started running into long saves. I have no scrap mods. In the attached file:
  • Save 6 is Starlight Drive-In completely scrapped as allowed by vanilla. I saved it in Starlight before building a beacon. No long save present.
  • Save 7, built the beacon, Jake builds the first Residential Plot and goes to Concord to check on his workshop. I saved it in Starlight again. No long save present.
  • Save 8, met Jake at his Concord Workshop and he gave me the Holotape. Saved in his Workshop at Concord. No long save present.
  • Save 9, built and completed the Agricultural Plot. Saved in Starlight. No long save present.
  • Save 10, Industrial Plot completed. Met Old Paul and Jake leaves. Start the quest "If I Had A Hammer". Long Save
In my previous playthrough, my long saves happened in or around the "Main Settlement" of Sim Settlements 2. First playthrough before I scrapped it, anytime I saved in or around Sanctuary, long save. This one, Starlight Drive-In, same thing. The "Main Settlement" is where my problems are. Hope that helped.

Pretty much covers how i received how the LS start to manifest... i wonder if somehow the Virtual Resorce Storage is involved as it seems to just run fine when only Residential or Agricultural Plots are present.
Was there any change regarding Industrial Plots and Virtual Resources between 1.x and 2.0?
 
Not a raw data solution but thinking out loud, sharing experience to help to "paint the picture" as maybe why things happen.

The previous version "Chapter 1" didn't trigger long saves, however if you're levelled up in the game it could be become slow in and around Settlements, influencing overall gameplay. I'm talking about level 30 / 45 here and 20 to 30 unlocked settlements. Since I had no interest in the financial simulation part of SS2, I would turn off all costs options and calculation options in SS2 at that point. Instant game boost. At that point act 2 of the game would start / continue and I've completed Chapter 1 with Jake. With those options off all the great little SS2 Quests ran smoother became more enjoyable could be completed without hiccups plus all the other quests that I'm running meanwhile re-building the Commonwealth with SS2 plots / addons and vanilla building. Absolute blast.

Where I'm going with this description is that if the features introduced with Chapter 2 heavy rely upon or work scriptwise the same as the calculation ones in Chapter 1, I can imagine it will freeze the saving process. If the calculating costs is a seperate scripting thread, the disease a seperate scripthing thread and so on, it might be worth to compile a version that would delay for example the disease. It's already starting very early in a new game. If this is also possible for another feature, it would put less stress on the engine and we would see an imidiate improvement with the long saves. Well in theory :)
 
Top