Call of Duty: Black Ops II [BLUS31011] Game freezes randomly

[Darkhost1999]

#13602 (comment)

Old issue

When booting BLUS31011 updated to 1.19 the game crashes at the press start screen if using 100% default configuration. But by changing zcull to approximate or relaxed we can actually play the game. This same fatal error and device lost affects so many of my games that I am glad to have obtained this game and am able to make a replicable debug output log. ``` ·F 0:00:58.200436 {RSX [0x80558bc]} SIG: Thread terminated due to fatal error: Assertion Failed! Vulkan API call failed with unrecoverable error: Device lost (Driver crashed with unspecified error or stopped responding and recovered) (VK_ERROR_DEVICE_LOST) (in file C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\rpcs3\Emu\RSX\VK\VKQueryPool.cpp:45[:4], in function poke_query) (in file C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\rpcs3\Emu\RSX\VK\vkutils\shared.cpp:107[:24], in function die_with_error) ```

Attach a log file

Debug.log
Crash.log
Working-Approximate Zcull Accuracy.log

System configuration

Ryzen 5 3600
GTX 1660 Super
Windows 11

[kd-11]

If it's consistent on your system you can install SDK and capture a log with debug output enabled.
Also missing steps to reproduce, it's a large game, I don't know if it crashes in the menus or a specific level, etc.

[Darkhost1999]

If it's consistent on your system you can install SDK and capture a log with debug output enabled.

Debug.log

Also missing steps to reproduce, it's a large game, I don't know if it crashes in the menus or a specific level, etc.

When booting BLUS31011 updated to 1.19 the game crashes at the press start screen if using 100% default configuration.

[kd-11]

I'm unable to reproduce on RTX 3050. The debug output also has no information about any violation that could cause a crash.
There is some log spam about use of barycentrics, but that shouldn't crash and has nothing to do with zcull accuracy. You can easily prove that this is not the culprit by setting shader quality to low.

[kd-11]

I have an idea - try this build: https://cirrus-ci.com/task/5489746785337344
Let's see if it solves the nvidia silent crash problem.

[Darkhost1999]

There was a change. The fatal error is now before the intro cutscene rather than after.
Debug.log
RPCS3.log

[kd-11]

We just have to wait for nvidia to implement VK_EXT_device_fault then. AMD already ships that extension, basically whenever the driver crashes it should give some more information to us about why it did so. Until then I think it's not very useful to keep fighting the NVIDIA driver.

[kd-11]

Try with #13875

[Darkhost1999]

This is using bc96333

RPCS3.log
-Edit add debug output log-
RPCS3.log

[Shoegzer]

Getting something similar here, though not quite the same. The game doesn't crash at the start screen on default settings, but rather freezes ingame after a while (between 3-15 minutes), usually in the "zombies" mode. Using default settings except for Approximate ZCull accuracy and Ordered&Atomic RSX FIFO accuracy. Note though, the "device lost" message isn't thrown to the log file. The log below is quite large as it took about 15 minutes of play to trigger the freeze.

OS: Linux Mint 21.1 / kernel 5.15
CPU: Intel i7-10870H
GPU: Nvidia GTX 3070 / driver 530.41
Commit tested: 1371bf8
Log: RPCS3.log.zip

[Shoegzer]

Update: Gave this a closer look. As I mentioned I don't see the VK_ERROR_DEVICE_LOST exception in the log, though I do see CELL_NET_CTL_ERROR_NOT_CONNECTED as @Darkhost1999 has reported:

19386, spu:0]; [soft:665 hard:727]; Switches:[vol:187178 unvol:14150]; Wait:[9.577s, spur:131010]
·E 0:21:02.668006 {PPU[0x1000000] main_thread} SYS: Stat: 'cellNetCtlGetInfo' failed with 0x80130108 : CELL_NET_CTL_ERROR_NOT_CONNECTED [x2012]

This is soon followed by other entries that seem to reflect a problem, for example:

·W 0:21:02.668038 {PPU[0x100000a] Thread (hang detect) [0x0033d844]} PPU: 'sys_timer_usleep' aborted (9.539612s)

Failed attempts to resolve include:

• SPU Decoder: ASMJIT
• Framelimit: 30
• Sleep timers accuracy: [all settings]
• Driver wake-up delay: 200us
• RSX FIFO accuracy: Atomic&Ordered
• Accurate PPU 128 Reservations: Always Enabled
• PPU Thread Count: 1

Note that I cannot test OpenGL as rpcs3 hangs during the initial shader build process (though this happens with all titles, not just this one).

[Darkhost1999]

Issue not occurring on RPCS3 v0.0.29-15463-edf4f7ea Alpha
RPCS3.log

[Shoegzer]

Still freezing here unfortunately, after several single-player zombie mode games. Tested against commit edf4f7e with new cache and default configuration as well. Please re-open.

·E 0:25:41.404741 {Emulation Join Thread} SIG: Thread [rsx::thread] is too sleepy. Waiting for it 4226396.399µs already!

Log: RPCS3.log.zip

[Darkhost1999]

Looks like when I tried today. Reinstall your game update files and try again.
https://wiki.rpcs3.net/index.php?title=Help:Installing_Game_Updates#Removing_game_updates/game_data

[Shoegzer]

Thanks for reopening. I tried what you suggested but it still froze. Update path was from 1.00 > 1.09 > 1.19, was that yours?

Log: RPCS3.log.zip

[Asinin3]

Test with #14657 (if its merged when you're reading this, just test latest RPCS3 build instead)

[Shoegzer]

Thanks, still freezes after 5-15 minutes after going ingame in testing a build with #14657. Again I do not see VK_ERROR_DEVICE_LOST in the logs but rather the others as I noted above.

New log in case it helps: RPCS3.log.zip

[Darkhost1999]

Yea freezes for me as well.
I don't obtain freezes when zcull accuracy is changed from precise.

[Shoegzer]

Well I can certainly create a new issue for this, as this one's title may be misleading assuming the fatal device error isn't a cause of freezing any more.

With regards to zcull accuracy, I've tested using precise and approximate and freezing occurs with either (though perhaps not as frequent with approximate). Play in zombie survival mode, specifically starting in the bus station, and you should see the same thing?

[kd-11]

Does the driver still crash @Darkhost1999 ?

[Darkhost1999]

RPCS3.log.gz
On RPCS3 v0.0.33-16999-938306a7
I can not finish the first mission when using precise zcull.
My old comment I assume holds true

I don't obtain freezes when Zcull Accuracy is changed from precise.

[kd-11]

Well, the log doesn't show any RSX crashing though. The only difference between precise and approximate is that precise makes the RSX code run slowly through the same steps.
I've updated the title to reflect that this issue is not causing a device lost error anymore.

[Shoegzer]

Freezing still occurs with ZCULL accuracy set to approximate, unfortunately. Other settings for this test: RSX FIFO accuracy was set to Atomic and Driver Wake-Up Delay at 280. Tested against the latest commit 6bb0106. I don't believe there's any way to prevent this title from freezing, I've tried many different settings.

Here's a log from it freezing after about 30 minutes of play in zombie mode, single player: RPCS3.log.zip

[kd-11]

Thanks for confirming. In that case, this likely has nothing to do with RSX anymore.

[Shoegzer]

Sure thing. Looking at the logs you'll see the same CELL_NET_CTL_ERROR_NOT_CONNECTED followed by Thread (hang detect) [0x0033d844]} PPU: 'sys_timer_usleep' aborted exceptions being thrown as noted above from older builds. I'm not sure if that's a clue as to the root cause, it may simply be a benign message about not connecting to network services, but it looks possible to me at least.

[Shoegzer]

Update: Actually scratch that, CELL_NET_CTL_ERROR_NOT_CONNECTED is probably a red herring as it doesn't seem to be an unintended error but rather signifies an inability to get information about the connecting system since network services aren't turned on. Would make sense anyway.

I've adjusted just about every setting exposed to at least reduce freezing occurrences (same for several others like #16393 above and #16087), but all these titles eventually freeze regardless.

[Darkhost1999]

I have settings I've used for the Assassin's Creed series to almost remove all freezing.
The settings were

• PPU Interpreter (cpu tab)
• Disable PPU non-java mode fixup (advanced tab)
• Enable PPU non-java mode (debug tab)
• Ordered and Atomic RSX accuracy (advanced tab)
• Possibly Accurate RSX Reservation Access (advanced tab)

There were changes to Accurate RSX Reservation Access since my last full test. And these Call of Duty Black ops games display different reasons for freezing than the games these settings have been tested with. Still worth trying any combination of these settings if desired.

[Shoegzer]

Thanks @Darkhost1999, that's helpful and I'll try those combinations with some of the other titles. Which AC titles exhibited freezing in your testing? I ask because I tested AC1 to completion and it never froze.

It's also important to distinguish between freezing and crashing. The titles above all freeze, while others (such as in #16452) cause RPCS3 to crash entirely. I don't know if your settings would help with crashing issues but I can try them anyway.

[Darkhost1999]

Which AC titles exhibited freezing in your testing?

Brotherhood and beyond.
So from the middle of the Ezio trilogy to every single game after that released on PS3.
1 and 2 unharmed.
They exhibit behavior as if pausing while music keeps running in the background

[Shoegzer]

That's exactly what happens with #16087 (music playing, gameplay frozen), I haven't checked logs for comparative indicators but I'll try the settings there anyway.

Also, realizing we're going OT but if you have a chance to document your settings in relevant entries like here I'm sure that would help others in the future.

[Darkhost1999]

Also, realizing we're going OT but if you have a chance to document your settings in relevant entries like here I'm sure that would help others in the future.

#12807 (comment)

[Shoegzer]

Ah thanks, I was mostly thinking of the two non-java mode settings but I suppose the important bits are already in there.

[Shoegzer]

One more question, I noticed you mentioned this setting but I could not find it:

Enable PPU non-java mode (debug tab)

The closest I could find was in the game-specific config file, there's this:

PPU Accurate Non-Java Mode: false

...which seems to be the same as "Accurate PPU Non-Java Mode" though it's greyed-out on this test system for some reason. Is that what you meant?

[Darkhost1999]

"Accurate PPU Non-Java Mode" seems to be greyed out on this system for some reason. Is that the setting you mean?

Yes, that's it and it's not implemented for PPU LLVM so it needs to be used with PPU Interpreter.