the Sim Settlements forums!

Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!

Long Save Research Tool

Granted, this is only for this session and it might 'produce' under other circumstances.
I don't think you will get long saves anymore, that's fixed :grin

This mean that if something like SS2 comes in behind that, it'll get dumped also since the limit was already breached, and it's scripts are running too and coming in after the fact.
This is true, but if the load order is minimal and the only thing script heavy you have is SS2, then it is safe to assume that SS2 filled the stack memory and kept calling more functions, so it got dumped.

Suspended stacks should be a different animal
I'm still not 100% comfortable with the concept of a suspended stack. When a script is suspended? Is it only when the cell is unloaded or there are other conditions, like the wait function? And if they are suspended, why would they be fighting for resources? They should be sleeping, like a well-behaved sleep thread xD

Anyway, keep up the good work. Today and tomorrow I'll be busier than usual with work stuff, but once I can I'll look at the code again. Gonna check my stack dumps and see what the functions are doing to try to understand what's going on. I'll leave the heavy-duty of building real-time analysis tools to you :grin
 
I'm still not 100% comfortable with the concept of a suspended stack. When a script is suspended? Is it only when the cell is unloaded or there are other conditions, like the wait function? And if they are suspended, why would they be fighting for resources? They should be sleeping, like a well-behaved sleep thread xD
I would venture a guess it's because of... let's say there are 10 scripts running calling other scripts and various intrinsic functions (any combination of things). With each subsequent call a 'stack frame' is created to provide information 'transversal' between different threads and that 'frame' has to wait until what it called either completes or returns the requested information all the way back up the chain... but, before that happens, a called function then in turn calls something else (script, function, etc.,.) that is needing information on an let's say npc or world object while another script's function call is trying to do the same. It's a collision. So one will have to wait on the other if it's 'instanced' locked for dedicated access- which I believe objects are, such as the player's data structures. Especially if one script is removing an item from a 'list' while another is waiting to read said 'list', disabling or removing objects from the game world is another example.

Now, if you have 1000 world objects and you attach scripts to each one of these, even if it's very small and event driven (I use that term loosely) for let's say 'on attach' or 'on load', then when those objects load or are attached and if they're in the same cell or surrounding cells all of the object's scripts are going to try fire almost simultaneously (if the objects can be loaded that fast), resulting in a VM slam. The bigger the mod or more objects you have the worse it'll get. You can see this as script lag since the VM is trying to muscle through 100 stack frames all from the same script (attached to each object) while other scripts have to wait their turn. Now that I think of it, I believe I read that there is a way to assign script priority, but don't quote me on that one.

I'm sure this happens all the time and part of the reason for Beth's simplistic approach to many things. Instead of building the tools around the story, they build the story around the limitations of the engine and tools. So we never really notice it until the game starts to get heavily modded.
 
These might help explain the suspended stack: https://www.creationkit.com/fallout4/index.php?title=Threading_Notes_(Papyrus) https://www.creationkit.com/fallout4/index.php?title=Performance_(Papyrus)

They don't mention it explicitly, but they explain how the game engine handles threading.
OMG. what a nightmare! Jesus... why??? Who thought that was a good idea? Do you see how easy it is to mess up and get things out of sync?? (ofc you do... I remember the hud numbers from v1 xD)

Now while ObjectC's thread is processing SomeThing1(), ObjectB's thread is done with DoSomethingElse() on ObjectD, but before it runs Something2() it now must wait in line until ObjectC's done with Something1() and itself moves out to ObjectD's DoSomethingElse(). As soon as ObjectC's thread heads out to DoSomethingElse() in ObjectD, ObjectA waves in ObjectB's thread who is returning from ObjectD's DoSomethingElse() and will now be allowed to continue on with Something2()... and so on...
This is bad enough, but I can get my head around it. It's like a script wide lock... inefficient as hell, but it works.

SetStage is Latent, therefore myPropery may change, even if you just use “SetStage(10)” and your script extends Quest and is attached to the quest.

In the case of the fragment – it is entirely likely that myProperty will no longer equal 5 by the time the fragment runs. Remember: you called setStage, NOT a function. Fragments execute “some time in the future” (which happens to be very soon in most cases). This is one case where a function call is much more useful than a fragment.
If myProperty is NOT in your local script, then “myProperty += 1” may not increment. This is because it resolves to “myProperty.set(myProperty.get() + 1)”, and myProperty’s value may change between the get and set calls since it isn’t on your own script.

Now, this?? Really? For someone who used OOP almost 100% of my academic/professional life, this breaks my brain! And you guys did near 1M lines of code with this? Jesus...
1643929038657.png

Ok... compose yourself...

Tkx @kinggath, I would never get there by myself. Gonna take that into account when I analyze the code :grin

PS: Really? They made it as a function call is better than just simple increment a Fing variable??..... I'm out of words...

EDIT: What the hell is a fragment?? ... don't answer, I'll google it :P
 
Ok, so far so good... until now. For some reason I can't get anywhere near Sanctuary as you can see from all the crashes- the code index for the crash is the same. I can view Sanctuary from a distance, but getting within 'Workshop range' results in a instant crash 100%. I used the Buffout 4 log scanner and it's showing a potential power grid problem there? (see the screenshot). If that's the case, is there a way to completely clear out and reset that settlement remotely? Without being in Sanctuary? This is the first time I've seen this happen, so I'm not sure exactly what the cause might be. One thing to consider, and this is just a toss, if a complex structure can't be gc-ed, would that necessarily imply that some sub-structures or references might still be in use? and forcing them clear would null ref some stuff every now and then?

There's not a lot of RoTC settlements going- maybe 6, and the papyrus log doesn't show any 'stack dumps' that could be associated with the crash.
 

Attachments

  • screenshot-07.png
    screenshot-07.png
    2 MB · Views: 8
  • crash-2022-02-04-03-54-12-AUTOSCAN.txt
    6.7 KB · Views: 2
Ok, so far so good... until now. For some reason I can't get anywhere near Sanctuary as you can see from all the crashes- the code index for the crash is the same. I can view Sanctuary from a distance, but getting within 'Workshop range' results in a instant crash 100%. I used the Buffout 4 log scanner and it's showing a potential power grid problem there? (see the screenshot). If that's the case, is there a way to completely clear out and reset that settlement remotely? Without being in Sanctuary? This is the first time I've seen this happen, so I'm not sure exactly what the cause might be. One thing to consider, and this is just a toss, if a complex structure can't be gc-ed, would that necessarily imply that some sub-structures or references might still be in use? and forcing them clear would null ref some stuff every now and then?

There's not a lot of RoTC settlements going- maybe 6, and the papyrus log doesn't show any 'stack dumps' that could be associated with the crash.
I have some ideas for some remote tools to help with resetting settlements remotely. I'll see if I can make time for them this weekend, I'm going to be out of town for family stuff, but I can still sneak in some coding time in the evenings.
 
From what I understood from the papyrus logs, this means this stack has 5 function calls
utility.Wait()
simplot.SpawnStageModel()
simplot.UpdateIndicators()
simplot.PostInitialization()
simplot.OnTimer()
When a plot is placed, when the script initializes, it checks some conditions. If those conditions fail, it calls a timer which calls PostInitialization to finish init. Then it spawns the plot icons and the building model. (usually the construction materials) I took a look at SpawnStageModel...
Code:
Function SpawnStageModel(Form aRequestedForm = None, Bool abFade = false, Bool abUseOffsets = true, Float[] aPositionOffsets = None)
    if(bGrabbed || bForceLocked)
        return
    endif
    
    ;    snip - code to get building model
    
    ModTraceCustom(GetLogName(), "[SimPlot] " + Self + " Placing StageModel: " + aRequestedForm)
    
    ObjectReference thisStageModel = PlaceAtMe(aRequestedForm, abInitiallyDisabled = abFade, abDeleteWhenAble = false)
    
    ; snip - code to move model into position
    
    if(thisStageModel)
        if(abFade)
            thisStageModel.Enable(abFade)
            Utility.Wait(2.0)    ; wtf is this wait for? could this be what you found? 2 sec is a long time!
        endif
        
        thisStageModel.AddKeyword(PlotSpawnedKeyword)
        thisStageModel.SetLinkedRef(kLinkedRefHolder, StageModelLinkKeyword)
        thisStageModel.SetLinkedRef(Self, WorkshopStackedItemParentKEYWORD)
        thisStageModel.SetLinkedRef(GetWorkshop(), WorkshopItemKeyword)
        thisStageModel.AddKeyword(PreventExportKeyword)
        
        kCurrentStageModel = thisStageModel
    endif
EndFunction
I would assume that is the wait you are referencing. I'll throw a question to Kinggath on the internal bug tracker as to why there is a 2 sec wait call. (Issue#816) That seems like a long time to me especially after we are waiting for the latent function Enable with abFadein=true (the fade in takes about 0.5 sec) to return.
 
As for the HUD updates, what concerns me a little is the fact that, unlike in other places, I can't find any control variable to slow down chain calls.
For plots, this is handled by SimPlot.
Code:
Function UpdateInformationIcons()
    if(bGrabbed)
        return
    endif
    
    if(bUpdateInformationIconsBlock)
        return
    endif
    
    bUpdateInformationIconsBlock = true
    
    int i = 0
    while(i < AccessoryRefs.Length)
        SimSettlementsV2:ObjectReferences:InformationIconSet_Plot asPlotIconSet = AccessoryRefs[i] as SimSettlementsV2:ObjectReferences:InformationIconSet_Plot
        
        if(asPlotIconSet)
            asPlotIconSet.UpdateDisplay()
        endif
        
        i += 1
    endWhile
    
    bUpdateInformationIconsBlock = false
EndFunction
For the settlement needs and virtual resource elements, StartTimer calls are used to control spam. (remember each time a new timer is created, the old one, if applicable, is reset) There are also control bools in place as well.
 
It's like a script wide lock... inefficient as hell, but it works.
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.
What the hell is a fragment?? ... don't answer, I'll google it :P
I don't know that a google search will give you a good answer. ;) Script fragments can run on a quest stage set, dialog, perk, terminal menus and most likely more things I'm forgetting. Simply put, you can think of it as a function that's called when an event happens. i.e. stage 10 on a quest is set. If there is a fragment defined for that quest stage, the code will run.
 
Ok, so far so good... until now. For some reason I can't get anywhere near Sanctuary as you can see from all the crashes- the code index for the crash is the same. I can view Sanctuary from a distance, but getting within 'Workshop range' results in a instant crash 100%. I used the Buffout 4 log scanner and it's showing a potential power grid problem there? (see the screenshot). If that's the case, is there a way to completely clear out and reset that settlement remotely? Without being in Sanctuary? This is the first time I've seen this happen, so I'm not sure exactly what the cause might be. One thing to consider, and this is just a toss, if a complex structure can't be gc-ed, would that necessarily imply that some sub-structures or references might still be in use? and forcing them clear would null ref some stuff every now and then?

There's not a lot of RoTC settlements going- maybe 6, and the papyrus log doesn't show any 'stack dumps' that could be associated with the crash.
It might just be a corrupted spawn point. I'd try sleeping in an interior cell somewhere for five days.
 
Ok, so far so good... until now. For some reason I can't get anywhere near Sanctuary as you can see from all the crashes- the code index for the crash is the same.
If it is a power grid corruption, we do have two options in Workshop Framework’s settings to attempt to repair it. Remote tools would be great, though! There used to be a remote reset option in SS1
 
I have some ideas for some remote tools to help with resetting settlements remotely. I'll see if I can make time for them this weekend, I'm going to be out of town for family stuff, but I can still sneak in some coding time in the evenings.
I wonder if it's because Jake is stuck in Red Rocket instead of Sanctuary. Here's the scenario I have... I'm in the creek between Red Rocket and Sanctuary. When I get within 'Workshop' range of Sanctuary it crashes, so I head back to Red Rocket and get inside the settlement, no crash. As I walk past Jake he says... "As I was saying...", I keep walking and it crashes again. Same crash offset as before.

Could it be that the engine is trying to 'spawn' Jake into Sanctuary but can't if he's location elsewhere? and thus be a 'corrupted' spawn point problem? Like we occasionally get with companions when fast traveling or going through doors into interior cells. Which sometimes is related to either a missing or broken navmesh, or the engine can't load enough data... something.

Let me see if I can get you a bit more information on this and try different scenarios... like just being in Red Rocket but not getting near Jake to trigger the conversation, and triggering it then speaking with him. Then if it doesn't crash see if I can get back into Sanctuary.
 
It might just be a corrupted spawn point. I'd try sleeping in an interior cell somewhere for five days.
Let me try testing something else out, and I'll give that a try. Would you happen to know what the 'corruption' is? Is this actual data corruption or a condition the engine isn't handling?
 
If it is a power grid corruption, we do have two options in Workshop Framework’s settings to attempt to repair it. Remote tools would be great, though! There used to be a remote reset option in SS1
Using Workshop framework what options would I use? I think I remember a reset all settlements? Is that it? When I looked at the Buffout 4 log, and according to the auto-scanner (which I think looks for matching crash signatures- certain text or text strings in the log), it brought up the power grid. When I go the log, it appears (at least on the surface) to be a 'chain' of nil? Which I'm can only assume is a broken linked list. Thus a broken power grid and why it was reported:

(std::_Func_impl<std::_Callable_obj<<lambda_8ecb0e8b234fd35dc5ae505953507ccb>,0>,std::allocator<std::_Func_class<BSContainer::ForEachResult,PowerUtils::GridAdjacencyMapNode &,std::_Nil,std::_Nil,std::_Nil,std::_Nil,std::_Nil,std::_Nil> >,BSContainer::ForEachResult,PowerUtils::GridAdjacencyMapNode &,std::_Nil,std::_Nil,std::_Nil,std::_Nil,std::_Nil,std::_Nil>*)

... let me fire Fallout 4 back up and take a look at the Workshop options and tinker around with this a bit more.
 
I wonder if it's because Jake is stuck in Red Rocket instead of Sanctuary. Here's the scenario I have... I'm in the creek between Red Rocket and Sanctuary. When I get within 'Workshop' range of Sanctuary it crashes, so I head back to Red Rocket and get inside the settlement, no crash. As I walk past Jake he says... "As I was saying...", I keep walking and it crashes again. Same crash offset as before.

Could it be that the engine is trying to 'spawn' Jake into Sanctuary but can't if he's location elsewhere? and thus be a 'corrupted' spawn point problem? Like we occasionally get with companions when fast traveling or going through doors into interior cells. Which sometimes is related to either a missing or broken navmesh, or the engine can't load enough data... something.

Let me see if I can get you a bit more information on this and try different scenarios... like just being in Red Rocket but not getting near Jake to trigger the conversation, and triggering it then speaking with him. Then if it doesn't crash see if I can get back into Sanctuary.
Here's what I've found so far... I still can't go into Sanctuary, but it doesn't crash at Red Rocket, at least after I've talked with Jake, but it did when I went into the console and clicked on him to get his reference. Though it might not be that, but something that's 'out of sequence' somewhere.

When starting a new game I received a 'SS2 failed to start', but then appeared to recover itself. I went to Concord got Garvey and gang, after using the MCM Manager option to set all MCM variables to a preset which included SS2 options. I didn't use to holo-tape. After dropping a planner's desk and firing off the RoTC city plan, the guys at Sanctuary starting building. Jake appeared. I ignored him completely and went to Red Rocket and slept. When I woke up Jake is there and seems to call Red Rocket home as moving him on the road outside of Red Rocket causes him to head back there instead of Sanctuary. I travelled around building settlements coming back to Sanctuary early on, but mostly staying out in the Commonwealth.

The crash in Red Rocket didn't happen when I talked to Jake but when I went in to the console to get his reference. When I reloaded the game just after I talked to him I waited a few seconds and I received 3 'building plans just unlocked' notifications (one includes Mama's Murphy's house- which is some of the first to show). After that I could get his reference. That's how I was able to move him into the streets to see if he would path back to Sanctuary.

Trying to sleep for 5 days straight, and just before the 1st day was complete, again resulting in a crash- this was at Red Rocket. Though just being in Red Rocket doesn't cause the crash. It seems to be data related or data 'expected' somehow.

The one thing I haven't tried yet is to do the 'Laying the Foundations' quest at Red Rocket and see if that makes a difference, along with a few other things I might be able to try.
 
Let me try testing something else out, and I'll give that a try. Would you happen to know what the 'corruption' is? Is this actual data corruption or a condition the engine isn't handling?
Its not actual data corruption. More of something like a NPC from a mod that didnt spawn correctly. The problem can heppen more on modded installs. Has been an issue with Bethesda games since Morrowind days. I'd try sleeping in an interior cell somewhere like Dugout Inn in Diamond City.
 
If it is a power grid corruption, we do have two options in Workshop Framework’s settings to attempt to repair it. Remote tools would be great, though! There used to be a remote reset option in SS1
I’d really like A remote settlement scrap option it was one of my favorite features in ss1 conquers.
 
isn't there a function in workshop framework that tries to fix if possible and destroy if unable to fix a broken grid?
if i remember correct this made is specifically into wsfw to prevent such crashes due to broken grids.
 
isn't there a function in workshop framework that tries to fix if possible and destroy if unable to fix a broken grid?
if i remember correct this made is specifically into wsfw to prevent such crashes due to broken grids.
I saw an option there also- power grid tools has a console fix option which I have installed. The problem is, you have to be in a settlement to use both, and I can't get near that settlement without crashing. I even tried changing the speedmult and running in, but to no avail.
 
Last edited:
Top