Discussion Understanding CTDs - Triangle of Death

[RayBo]

Precombines/Previs
The other half of the performance system described by Snarkywriter

The technical details (only the first page of the thread is important, don't let the length scare you)

Short user friendly video on precombines

To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Accept third party cookies

Demanding, that is one amazing Link "The technical details"! A huge archive of Information that I now will be visiting, Often. I am really happy you shared that. Wow, just wow.

RedRocketTV is also doing some excellent youtube vids that I think folks young and old should check out. I've been bookmarking them as he's been putting them out.

I keep thinking to myself. There are two games here. I need to better understand the technical one in order to enjoy the potential in the other.

I don't think you need to "argue" any of the above points. If you talk, you have my attention. Then, I go away and find myself agreeing with you and enjoying the game just that much more. Even appreciating what Bethesda has provided and feeling that much more grateful to all the folks who have contributed to make it what I see and play most evenings.

Thank you for sharing!

 

[Catallaxis]

I realise this would be annoying, but it seems like removing a small convenience from the player may actually help with workshop script problems. That is, disable the workshop mode shortcut button that allows you to instantly activate workshop mode anywhere in a settlement. In other words, force the player to activate workshop mode from the workbench itself, usually located near the middle of a settlement. This will give the scripts more time to update when the player arrives in a settlement, encourage players to load in the whole settlement whenever they visit, and also make them more purposeful about when they enter and exit workshop mode.

Does such a mod exist, or would it be possible to create? I know it wouldn't fix the problem, but it might help mitigate it.

Besides, on Xbox, I'm always accidentally activating workshop mode whenever I want to just tweak the camera, because they're mapped to the same button. More than once I've wished I could disable it somehow.

 

[Catallaxis]

The technical details (only the first page of the thread is important, don't let the length scare you)

Sclerocephalus says:

"Scripted items (they all have a WorkShopObjectScript attached) include everything that produces resources (including radios etc.) or can be assigned to an NPC (including beds), while non-scripted ones include all walls, fences, shack parts, most decorations, etc."

I'm guessing that idle markers, or furniture attached to idle markers, are also scripted?

 

[snarkywriter]

Sclerocephalus says:

"Scripted items (they all have a WorkShopObjectScript attached) include everything that produces resources (including radios etc.) or can be assigned to an NPC (including beds), while non-scripted ones include all walls, fences, shack parts, most decorations, etc."

I'm guessing that idle markers, or furniture attached to idle markers, are also scripted?

Yes indeed. As are doors, TVs, each individual crop, water pumps/purifiers, generators, turrets, guard posts, traps, spotlights, animals, cages, beds, all of the power routing objects, scrap stations, recruitment beacons, and basically anything else that "does" something beyond looking pretty. Pretty much anything that has more than a mesh, texture, and snap points is usually scripted in some way. You see the problem.

 

[snarkywriter]

But let's be honest here - no one is going to read anything we put on the wiki that says "do triangle of death settlements one at a time". They'll just go and kick them all off and expect everything to be fine, and when the engine shits itself, they'll complain anyway.

With the complaint... Your mod broke my game! Fix it!
LOL

I'm absolutely certain that there a quite a few people out there that will do exactly this. However, I don't think most of them are doing it out of spite. I think they (a) don't understand that correlation is not causation, (b) don't understand how the systems work so they can only go off correlation to solve their problem, and (c) would rather spend the time it would take to learn how the systems work and solve their problem actually working the systems.

Despite that, I'm personally still willing to help them. I won't speak for anyone else, but I feel like I have a duty to (a) do my due diligence to make the information that they need to make informed choices easily accessible and available, (b) talk them down after they ignore that information completely, shoot themselves in the foot, and fly off the handle about it, and (c) help them understand how to avoid shooting themselves in the foot in the future. My personal goal is to help make Fallout 4 and modding as accessible as possible from a player's perspective.

 

[ApoAlaia]

I'm absolutely certain that there a quite a few people out there that will do exactly this. However, I don't think most of them are doing it out of spite. I think they (a) don't understand that correlation is not causation, (b) don't understand how the systems work so they can only go off correlation to solve their problem, and (c) would rather spend the time it would take to learn how the systems work and solve their problem actually working the systems.

Despite that, I'm personally still willing to help them. I won't speak for anyone else, but I feel like I have a duty to (a) do my due diligence to make the information that they need to make informed choices easily accessible and available, (b) talk them down after they ignore that information completely, shoot themselves in the foot, and fly off the handle about it, and (c) help them understand how to avoid shooting themselves in the foot in the future. My personal goal is to help make Fallout 4 and modding as accessible as possible from a player's perspective.

You definitely live up to the 'writer' part of your handle but you are far too calm, collected and level headed so I am going to call it out: the 'snarky' part is a fail.

 

[Elgered]

Are there any options or .ini tweaks that could help players on Windows?
My System isn't the newest, but far below real heavy load, but it seems FO4 doesn't take full advantage of what is available (i7 4790k, 32GB RAM, GTX1080). After playing with some multithreading settings I get the CPU up to ~75%, the GPU to 100%, but neither RAM nor GPU Memory are completely used. Instead the game (my fault, installed it to a HDD Raid5 instead of the SSD, because my Skyrim directory already hit the 300gb there) always unloads and reloads (and seems to stop scripts or have them run to timeouts or whatever - I don't really want to learn to understand the logs)...
Would maybe something like
uGridsToLoad=6
uExterior Cell Buffer=49
help in this case? Or does the game hit some sort of softcap in memory usage (I haven't seen it using more than ~8GB of RAM)?

 

[snarkywriter]

Are there any options or .ini tweaks that could help players on Windows?
My System isn't the newest, but far below real heavy load, but it seems FO4 doesn't take full advantage of what is available (i7 4790k, 32GB RAM, GTX1080). After playing with some multithreading settings I get the CPU up to ~75%, the GPU to 100%, but neither RAM nor GPU Memory are completely used. Instead the game (my fault, installed it to a HDD Raid5 instead of the SSD, because my Skyrim directory already hit the 300gb there) always unloads and reloads (and seems to stop scripts or have them run to timeouts or whatever - I don't really want to learn to understand the logs)...
Would maybe something like
uGridsToLoad=6
uExterior Cell Buffer=49
help in this case? Or does the game hit some sort of softcap in memory usage (I haven't seen it using more than ~8GB of RAM)?

Unfortunately, no. The way the engine mechanics work means there're indeed a few "soft caps" and this isn't something we can brute force past with stronger hardware or setting changes. I'm actually writing a bit on that now, and will update as soon as our code experts have had a chance to correct anything I've messed up.

 

[snarkywriter]

Would maybe something like
uGridsToLoad=6
uExterior Cell Buffer=49
help in this case?

Quite the opposite, in fact! Increasing these two values to uGridsToLoad=7 (it can only be an odd number, using an even number will crash the game on load) and uExterior Cell Buffer=64 (this is supposed to be =(1+ uGridsToLoad)^2 ) just makes the yellow squares I drew around your character to represent rendered cells one cell larger on every side, and now the game pre-loads and holds twice as many cells unrendered in memory, meaning the problems will happen even more often. As you increase those values, you are adding exponentially more for the engine to try to render and process in a given frame per second, which is the key to the whole thing. You are also setting up a situation where you trigger the "premature" partial resource calculations even further away from the settlement.

 

[ApoAlaia]

Did I misread your post or did you just say that we cannot 'brute force' it but we can?

If papyrus is multithreaded and the limitation is the available time to process the scripts then the higher the instructions per clock, clock frequency and available cores/threads the more papyrus can process in the offered window?

Therefore we do need your hypothetical helium cooled supercomputer to break the triangle of death by sheer computational power alone?

So renting computing time from the Amazon S3 servers to run the papyrus VM is the way forward? :P

 

[snarkywriter]

If papyrus is multithreaded and the limitation is the available time to process the scripts then the higher the instructions per clock, clock frequency and available cores/threads the more papyrus can process in the offered window?

As far as I am aware (coding guys, correct me if I'm wrong here) sheer computational power won't do it because the memory allocation sub-system was designed to run on both consoles and PCs. The memory allocation sub-system has to be able to work for Xbox and Playstation, as well as a PC. I don't know enough yet about how it all works to say how or why, but I'm pretty sure each of the different consoles and PC systems have different methods for determining how system resources are distributed. The Creation Engine's memory allocation sub-system appears to have been designed with a standard in mind that can accomodate the majority of the systems it is meant to run on.

To use an analogy:

Imagine Fallout 4 and your computer as a car. The motor is your computer. Everything else, the body, interior, frame, steering system, suspension, tires, power transfer system, cooling system, braking system, etc, is Fallout 4.

Fallout 4 is a mid-sized, 4-door Sedan. Every one of its systems was designed to run best with a 4-cylinder, factory engine.

Your computer is the motor off a Bugatti Veyron.

You take your mid-sized sedan, and shoe-horn in the motor from a Bugatti Veyron. It'll drive, handle, brake, steer, accelerate, and idle just like a Veyron, right?

 

[ApoAlaia]

As far as I am aware (coding guys, correct me if I'm wrong here) sheer computational power won't do it because the memory allocation sub-system was designed to run on both consoles and PCs. The memory allocation sub-system has to be able to work for Xbox and Playstation, as well as a PC. I don't know enough yet about how it all works to say how or why, but I'm pretty sure each of the different consoles and PC systems have different methods for determining how system resources are distributed. The Creation Engine's memory allocation sub-system appears to have been designed with a standard in mind that can accomodate the majority of the systems it is meant to run on.

To use an analogy:

Imagine Fallout 4 and your computer as a car. The motor is your computer. Everything else, the body, interior, frame, steering system, suspension, tires, power transfer system, cooling system, braking system, etc, is Fallout 4.

Fallout 4 is a mid-sized, 4-door Sedan. Your computer is the motor off a Bugatti Veyron. You take your mid-sized sedan, and shoe-horn in the motor from a Bugatti Veyron. It'll drive, handle, brake, steer, and idle just like a Veyron, right?

I am not a coding guy; regardless I was just joking.

The way I understand it you would need as many parallel threads dedicated exclusively to the VM as scripts/fragments you are running at any given time for this to make any kind of difference.

If you don't then you would need a hierarchical overview of that entire 'spider's web' of scripts and dependencies so the VM can run them in parallel but also in order so the scripts on whose output other scripts depend to run are run first and so on and so forth.

Without this, with the scripts running on a 'first come/first served' fashion you may have to store any number of outputs in memory and what scripts those outputs are relevant to while the next 'batch' is run, which it turn may produce outputs that are still not relevant to the next 'batch' and have to be stored too.

This might be way off though because as I said I am not one of the coding guys though so take with a pinch of salt.

 

[snarkywriter]

I am not a coding guy; regardless I was just joking...

No problem at all! I feel like the blind man who is using his hands to not only figure what an elephant is, but how all of its internal organs are connected to each other.

 

[Yagisan]

As far as I am aware (coding guys, correct me if I'm wrong here) sheer computational power won't do it because the memory allocation sub-system was designed to run on both consoles and PCs. The memory allocation sub-system has to be able to work for Xbox and Playstation, as well as a PC. I don't know enough yet about how it all works to say how or why, but I'm pretty sure each of the different consoles and PC systems have different methods for determining how system resources are distributed. The Creation Engine's memory allocation sub-system appears to have been designed with a standard in mind that can accomodate the majority of the systems it is meant to run on.

OK, this is something I can help with. As you said, we have up to 2.4ms to do as much Papyrus VM work as we can. We can't push the frame rate above 60fps, due to it controlling papyrus timings. However - we can partially brute force it. A faster CPU (single core performance here) can push through more of the Papyrus scripts in that 2.4ms it has to process. Multiple cores aren't going to help this particular performance aspect (but they do help others). Can you brute force it? well, the answer is a "yes but". Yes, but if you are doing this, you'll get diminishing returns.

Now, what happens if you overload the Papyrus VM - it starts dumping to memory. You probably won't notice until it hits swap - then it will crash/grind to a halt. But by this time, you are beyond what your CPU can handle in that 2.4ms. The occasional overload is OK - it will catch up in a less busy time - but constant overloads - that's when you need to reduce your mod list, and cut down on your 60 settler magnum opus, because the reality of having a very powerful computer is that you have a little bit of "wriggle room" to push the system before you'll also crash.

 

[woodfuzzy]

But Snarkywriter!! Can't I just uncap my FPS and let the game run at 100+ FPS? That would allow more processing time!
Woodfuzzy pops in to smart off..
Why sure you can.
But, you'll CTD at some point because letting the game render unfettered, effect the time scale the scripts use, which also will cause scripts to run when they shouldn't, NPC's to move faster, animations to run faster.. etc.
The game engine has built in errors, who'd a thunk, right?!, that normally are dealt with and you never notice. But, unlocking the FPS, causes the errors to pile up so fast the engine can't deal with them.. boom CTD.

You can also achieve the same effect by set timescale to 120. Within just a few minutes it will crash.
Even setting it to 30 will cause it crap out, although it will take a lot longer.
The entire papryus system is actually very fragile. It can be pushed a little.. but one good shove and it will take its toys and go home.

@snarkywriter You are correct on the more power doesn't = more performance. People with 6, 8 or 27 bazillion core processors have more problems than systems with 4 cores. Dunno why!! But it's been reported many times. My son even seen a noticeable improvement by restricting F04 to only 4 cores.

 

[Zonary]

Also thanks for the compliment and I hope you didn't think I was scolding you for your previous post! I really wasn't, I just thought it was a common misconception and an explanation would be good.

Don't worry I didn't think you were scolding me! Your explanation was insightful.

Frame rate is the key. Remember, Creation Engine was designed with a target and maximum frame rate of 60 frames-per-second for computers, and 30 frames-per-second for consoles.

Here is the thing. I'm playing on a 144hz monitor, so when I started playing FO4, the fps went above 100 and I got some really weird bugs and glitches, like the npcs lip sync faster then the voice audio, Cooke and Paul fist fight resulting in both pulling out guns and shoot each other infinitely, and etc. So after some research (a.k.a. asking on reddit, not always the smartest choice lol), people said that I should NEVER play above 60fps and use a fps capper at 60fps. I indeed used a fps capper (nvidia profile inspector) and capped at 100fps instead. My framerates vary between 45 (in places lke downtown boston, lvl3 rotc cities, etc) and 100 (inside cells are always 100), and I never got any bug or glitch, after 200 hours of gameplay. So if I understand correctly, even that I have fps above 60, the engine is capping the script execution as if I had 60fps, is that right?

 

[Zonary]

But Snarkywriter!! Can't I just uncap my FPS and let the game run at 100+ FPS? That would allow more processing time!
Woodfuzzy pops in to smart off..
Why sure you can.
But, you'll CTD at some point because letting the game render unfettered, effect the time scale the scripts use, which also will cause scripts to run when they shouldn't, NPC's to move faster, animations to run faster.. etc.
The game engine has built in errors, who'd a thunk, right?!, that normally are dealt with and you never notice. But, unlocking the FPS, causes the errors to pile up so fast the engine can't deal with them.. boom CTD.

You can also achieve the same effect by set timescale to 120. Within just a few minutes it will crash.
Even setting it to 30 will cause it crap out, although it will take a lot longer.
The entire papryus system is actually very fragile. It can be pushed a little.. but one good shove and it will take its toys and go home.

@snarkywriter You are correct on the more power doesn't = more performance. People with 6, 8 or 27 bazillion core processors have more problems than systems with 4 cores. Dunno why!! But it's been reported many times. My son even seen a noticeable improvement by restricting F04 to only 4 cores.

HAHAHA @woodfuzzy the timing was perfect! Look at my post above. Weird is that I never got any CTD. Toughts...?

 

[ApoAlaia]

But Snarkywriter!! Can't I just uncap my FPS and let the game run at 100+ FPS? That would allow more processing time!
Woodfuzzy pops in to smart off..
Why sure you can.
But, you'll CTD at some point because letting the game render unfettered, effect the time scale the scripts use, which also will cause scripts to run when they shouldn't, NPC's to move faster, animations to run faster.. etc.
The game engine has built in errors, who'd a thunk, right?!, that normally are dealt with and you never notice. But, unlocking the FPS, causes the errors to pile up so fast the engine can't deal with them.. boom CTD.

You can also achieve the same effect by set timescale to 120. Within just a few minutes it will crash.
Even setting it to 30 will cause it crap out, although it will take a lot longer.
The entire papryus system is actually very fragile. It can be pushed a little.. but one good shove and it will take its toys and go home.

@snarkywriter You are correct on the more power doesn't = more performance. People with 6, 8 or 27 bazillion core processors have more problems than systems with 4 cores. Dunno why!! But it's been reported many times. My son even seen a noticeable improvement by restricting F04 to only 4 cores.

The guy that coded the ENB 'engine' wrote once: mo' cores mo' problems.
Given his 'pedigree' I take his word at face value.

 

[jsalemi]

My son even seen a noticeable improvement by restricting F04 to only 4 cores.

How do you do this? May solve some of my issues....

 

[ApoAlaia]

How do you do this? May solve some of my issues....

I use:

https://www.nexusmods.com/fallout4/mods/102/

You can change this directly in the .ini files but I like a nice GUI me.