Lua on each mapgen thread #13092

sfan5 · 2022-12-28T23:02:43Z

Fixes #3182
this adds a separated lua environment on each emerge thread, for obvious reasons mods will have to be adjusted to take advantage of this

Merging

squash everything, don't keep the commit messages

To do

This PR is Ready For Review.

~~access to node meta ← feasible?~~ postponed
allow mods to register stuff to be imported at env init time
a way for "communication" with the game env (gennotify)
expose relevant mapgen apis
expose relevant env apis
transfer core.registered_* into mapgen env
fix metatable stuff for registered_items table

How to test

The easiest (but not intended) way is to edit builtin/emerge/init.lua and add code there.

core.register_on_generated(function(vm, pos1, pos2, blockseed)
	print("generating chunk " .. core.pos_to_string(pos1) .. " -> " ..
		core.pos_to_string(pos2))

	local mid = pos1:add(pos2:subtract(pos1):divide(2)):apply(math.floor)
	core.set_node(mid, {name="default:mese"})
end)

This will place a mese node in the middle of each generated map chunk all without blocking the main thread (wow!).

You can find a complete example that showcases the intended way here →→ https://gist.github.com/sfan5/1a124eb496a39734f53fb4daffc47bee

Another fun thing you can do is this:

core.register_on_generated(function(vm, blockseed)
	local data = vm:get_data()
	local N1, N2 = #data / 2, #data + 1

	local tmp
	for i = 1, N1 do
		tmp = data[N2-i]
		data[N2-i] = data[i]
		data[i] = tmp
	end
	vm:set_data(data)
	vm:get_param2_data(data)
	for i = 1, N1 do
		tmp = data[N2-i]
		data[N2-i] = data[i]
		data[i] = tmp
	end
	vm:set_param2_data(data)
end)

For some reason it confuses our mapgens to a great deal and they never stop generating surface.

appgurueu · 2022-12-29T09:25:14Z

Oh wow this is awesome. This could make Spiraling Down playable again (currently mapgen takes avg. 200 ms per chunk; it is hard to optimize it further). It would be great if the mapgen didn't block the main thread.

debagos · 2023-01-03T15:15:25Z

Thanks for investing your time and knowledge in such a complicated task @sfan5!
In my opinion this is an even more beneficial goal than refactoring formspecs for example /2cents
Sorry for the OT, but I have had to say that.

Lazerbeak12345 · 2023-02-17T03:57:21Z

This might also make fractured faster - and perhaps fractured-style-layering more viable across other games.

MisterE123 · 2023-03-13T18:34:55Z

This PR needs to be rebased to work with the latest mtirrlicht

MisterE123 · 2023-03-13T18:44:25Z

It would be nice to have a way to send data/code to the lua environment. Perhaps that is possible with writing a lua script to the world directory and then calling it from the mapgen environment?

My usecase is to adapt luamap to run on the separate thread, but the whole point of luamap is to let other mods register mapgens using luamap's simplified API (which means less control, but also less confusion for noob modders)

So, I want to build my own API around the mapgen API. Currently that seems to be thwarted since globals are not available. Perhaps register_mapgen_dofile should be able to pass variables to the environment, such as a table containing functions to call

sfan5 · 2023-03-13T19:23:24Z

In your case I would require the other mod to call register_mapgen_dofile (after yours) with its own implementation of luamap.logic.

MisterE123 · 2023-03-13T20:14:40Z

Ah, so, I set up the luamap functions in the mapgen environment under a lumap global, and then other mods will be able to see the luamap global.

MisterE123 · 2023-03-13T21:01:29Z

I just found out that minetest.get_perlin_map is not available in the environment, only PerlinNoiseMap
Its nice how minetest.get_perlin_map adds the worldseed to the perlin seed automatically

MisterE123 · 2023-03-13T21:22:40Z

apparently minetest.get_mapgen_setting_noiseparams is not available either, which would be good to include for consistency

MisterE123 · 2023-03-13T21:56:11Z

Apparently minetest.registered_biomes is missing

MisterE123 · 2023-03-13T22:11:06Z

mapgenmods.zip
here's a couple things to play with: Luamap and mandeland (a mandelbrot mapgen) modified to work with this PR
Until I get minetest.registered_biomes, I can't make biomegen work with this PR, so its just rock for now.

MisterE123 · 2023-03-13T22:55:13Z

minetest.generate_decorations(vm) is missing, I'm not sure about generate_ores

I can't convert theTermos's islands mapgen until that's fixed

Concise list of things that I'm aware of that are missing in order of importance:

minetest.generate_decorations
minetest.generate_ores ???
minetest.registered_biomes
minetest.get_mapgen_setting_noiseparams
minetest.get_perlin_map

appgurueu

I've now tested this with Spiraling Down. It required quite a few changes, but nothing that wasn't doable in an hour.

The effect is amazing. Finally the main thread is free to do actual game logic again.

One caveat: Lua mapgens like to use singlenode. This means that mapblocks not generated yet will temporarily be air until the Lua register_on_generated has finished, which is problematic because the player probably has already dropped through a few blocks by then.

Other caveats: OOMing, lighting bugs, "couldn't grab block we just generated".

sfan5 · 2023-03-17T18:34:22Z

This means that mapblocks not generated yet will temporarily be air until the Lua register_on_generated has finished, which is problematic because the player probably has already dropped through a few blocks by then.

This is supposed to be impossible, the block doesn't leave the server until the callbacks are done.

"couldn't grab block we just generated".

reproducer please

appgurueu · 2023-03-17T18:49:59Z

"couldn't grab block we just generated".

reproducer please

Here's a hackypatch.txt (Git patch) for https://github.com/spiraling-down/game latest master. Fly around quickly until the error appears on your screen.

Lazerbeak12345 · 2023-03-17T21:59:50Z

I've seen similar behavior before on single-threaded lua mapgen before.

MisterE123 · 2023-05-20T15:54:50Z

I see there have been updates; its not entirely clear what effect they have. What's changed, and is it time to experiment with this again (have some of the above issues mentioned been fixed, so its time to test its limits again?)

sfan5 · 2023-05-20T18:10:49Z

I added a bunch of APIs you were missing but I'll ping you if you can re-test.

MisterE123 · 2023-05-24T02:55:52Z

Well, first thing I notice is that minetest.get_perlin_map is still missing. Is that intentional? All get_perlin_map does is to wrap PerlinNoiseMap by adding the world seed to the seed of the noise. While you can do that in lua it makes sense to continue to offer that in the environment for consistency, imo

local worldseed = minetest.get_mapgen_setting("seed")

minetest.get_perlin_map = minetest.get_perlin_map or function(np, size)
    np.seed = (np.seed or 0) + worldseed
    return PerlinNoiseMap(np,size)
end

minetest.get_perlin = minetest.get_perlin or function(np)
    np.seed = (np.seed or 0) + worldseed
    return PerlinNoise(np)
end

register_on_shutdown() is missing... which I discovered while trying to convert mapgen_rivers

gaelysam · 2024-01-04T23:48:15Z

If you are interested, I made my own version of Mapgen Rivers (independently from MisterE) that allows to switch between standard and multithreaded modes: https://gitlab.com/gaelysam/mapgen_rivers/-/tree/mapgen_thread (set mapgen_rivers_use_mapgen_thread = true in minetest.conf)

Concerning minetest.register_on_generated(vm, minp, maxp, seed), I noticed that vm comes first and offsets other params compared to standard version. Why not adding it last instead? This would make it easier to write flexible code like what I did (mapgen code that should be able to run either in main or mapgen thread), even if it's not complicated to workaround.

As well, the docs should explain that vm:write_to_map() is done implicitly.

Anyway I would really like to see this merged, as this feature has been long-awaited and this seems to work well. Congratulations.

sfan5 · 2024-01-10T20:36:14Z

Why not adding it last instead? This would make it easier to write flexible code like what I did (mapgen code that should be able to run either in main or mapgen thread), even if it's not complicated to workaround.

I was thinking it discourages people from just copying code from the normal environment and expecting it to work.
For example set_node is quite inefficient and you should be interfacing with the Vmanip. Also metadata is not available.

As well, the docs should explain that vm:write_to_map() is done implicitly.

I guess they could but I don't think it's any different to the old callback, is it?

MisterE123 · 2024-01-11T00:29:28Z

For the most part, I had a lot of success just copying (mapgen) code and getting it to work, although I did have to change it for the differences in the api. Its more valuable to have a consistent api than the "stop and think" effect obtained with a slightly different arguement order...

grorp · 2024-01-11T15:56:47Z

As well, the docs should explain that vm:write_to_map() is done implicitly.

I guess they could but I don't think it's any different to the old callback, is it?

IIRC, you have to call write_to_map manually in regular/synchronous on_generated callbacks.

https://github.com/minetest/minetest/pull/13092/files#r1278370082

It would of course be nice to get rid of this inconsistency between on_generated on the main thread and on_generated on mapgen threads.

sfan5 · 2024-01-15T18:52:02Z

IIRC, you have to call write_to_map manually in regular/synchronous on_generated callbacks.

You're right. Updated the docs to be clearer about this.

doc/lua_api.md

src/emerge.cpp

src/emerge_internal.h

src/script/lua_api/l_mapgen.cpp

doc/lua_api.md

src/script/lua_api/l_vmanip.cpp

Co-authored-by: SmallJoker <[email protected]>

…inux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX. Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called Linux, and many of its users are not aware that it is basically the GNU system, developed by the GNU Project. There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called Linux distributions are really distributions of GNU/Linux!

MisterE123 · 2024-02-04T20:03:24Z

... is this ready?

SmallJoker

☑️ Gen notify positions are saved and retrieved correctly
☑️ Both, VManip and core.set_node do work in the main thread (add onto the terrain generated by the mapgen script)
☑️ The functions provided in the example work. That's already a large part what's needed for Lua mapgen development.

What I don't like:

Code duplication. To be addressed later. This could be worked around by using helper functions that are used by the VManip and regular API using (less efficient) lambdas/callbacks.
The Lua callback execution order is yet not documented. What if other mods register mapgen scripts? Will all mapgen scripts be executed in order of registration, then the main thread in order of registration? --> see comment

sfan5 · 2024-02-08T20:34:30Z

I did some more editing on the docs to make them hopefully clearer.

SmallJoker

Thank you for the improvements. There's also an attempt to make the documentation easier to comprehend.

doc/lua_api.md

SmallJoker

Thank you for the changes. Looks good. 👍

grorp

Just a little nitpicking on the documentation... I don't think these comments are particularly important, I just want the documentation to be as good as (easily) possible.

grorp · 2024-02-13T20:27:42Z

doc/lua_api.md

+    * Saves data for retrieval using the gennotify mechanism (see [Mapgen objects]).
+    * Data is bound to the chunk that is currently being processed, so this function
+      only makes sense inside the `on_generated` callback.
+    * `id`: user-defined ID (a string)


newline not rendered
"user-defined ID (a string) By convention these ..."

I can't remember how to fix that, so I'll leave it 😅

two trailing spaces result in a <br>

doc/lua_api.md

Desour

Sorry, I'm too late.

(Just skimmed the PR.)

Desour · 2024-02-13T21:08:11Z

builtin/emerge/env.lua

+
+function core.get_node_or_nil(pos)
+	local node = core.vmanip:get_node_at(pos)
+	return node.name ~= "ignore" and node


Suggested change

return node.name ~= "ignore" and node

return node.name ~= "ignore" and node or nil

Desour · 2024-02-13T21:52:32Z

src/script/lua_api/l_server.cpp

+	CHECK_SECURE_PATH(L, path.c_str(), false);
+
+	// Find currently running mod name (only at init time)
+	lua_rawgeti(L, LUA_REGISTRYINDEX, CUSTOM_RIDX_CURRENT_MOD_NAME);


This mod name is used for check for insecure environment, if I understand correctly.

But this means that if a trusted mod A calls a function from another mod B, B can in this function register their own script under the name of A, and can hence request insec env.

sfan5 added @ Script API @ Mapgen WIP Feature ✨ PRs that add or enhance a feature labels Dec 28, 2022

rubenwardy mentioned this pull request Jan 14, 2023

[Content] Engine updates for December luanti-org/blog#54

Closed

sfan5 force-pushed the mglua branch 2 times, most recently from b1ebf14 to 067fd4b Compare January 23, 2023 22:19

sfan5 force-pushed the mglua branch from 067fd4b to 6acda91 Compare March 13, 2023 20:13

appgurueu reviewed Mar 17, 2023

View reviewed changes

sfan5 force-pushed the mglua branch 2 times, most recently from 59a43fe to fd17164 Compare May 17, 2023 14:16

sfan5 changed the title ~~Lua on each mapgen thread (proof of concept)~~ Lua on each mapgen thread May 17, 2023

doc

c53cf99

Merge remote-tracking branch 'upstream/master' into mglua

cf25f41

SmallJoker reviewed Jan 22, 2024

View reviewed changes

sfan5 and others added 3 commits January 23, 2024 15:20

Apply suggestions from code review

76b1a16

Co-authored-by: SmallJoker <[email protected]>

review

f14b815

SmallJoker self-requested a review February 5, 2024 19:52

SmallJoker reviewed Feb 6, 2024

View reviewed changes

sfan5 added 2 commits February 8, 2024 21:10

Merge remote-tracking branch 'origin/master' into mglua

f1f621f

docs

f4608c3

SmallJoker reviewed Feb 10, 2024

View reviewed changes

docs2

4d68241

SmallJoker approved these changes Feb 13, 2024

View reviewed changes

SmallJoker added the One approval ✅ ◻️ label Feb 13, 2024

grorp reviewed Feb 13, 2024

View reviewed changes

docs3

7344972

sfan5 merged commit 3cac17d into luanti-org:master Feb 13, 2024
15 checks passed

sfan5 deleted the mglua branch February 13, 2024 21:47

Desour reviewed Feb 13, 2024

View reviewed changes

This was referenced Feb 14, 2024

[no squash] Remove insecure environment from async and emerge environment #14370

Merged

Constantly having huge desync issues on macOS M series. #14375

Closed

This was referenced Mar 15, 2024

[Mod Request]: realms minetest-whynot/whynot-game#92

Closed

Improve biomes minetest-whynot/whynot-game#32

Open

	return node.name ~= "ignore" and node
	return node.name ~= "ignore" and node or nil

Lua on each mapgen thread #13092

Lua on each mapgen thread #13092

Conversation

sfan5 commented Dec 28, 2022 • edited Loading

Merging

To do

How to test

appgurueu commented Dec 29, 2022

debagos commented Jan 3, 2023

Lazerbeak12345 commented Feb 17, 2023

MisterE123 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023 • edited Loading

sfan5 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023 • edited Loading

MisterE123 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023

MisterE123 commented Mar 13, 2023 • edited Loading

appgurueu left a comment • edited Loading

Choose a reason for hiding this comment

sfan5 commented Mar 17, 2023

appgurueu commented Mar 17, 2023 • edited Loading

Lazerbeak12345 commented Mar 17, 2023

MisterE123 commented May 20, 2023

sfan5 commented May 20, 2023

MisterE123 commented May 24, 2023 • edited Loading

gaelysam commented Jan 4, 2024

sfan5 commented Jan 10, 2024

MisterE123 commented Jan 11, 2024

grorp commented Jan 11, 2024 • edited Loading

sfan5 commented Jan 15, 2024

MisterE123 commented Feb 4, 2024

SmallJoker left a comment

Choose a reason for hiding this comment

sfan5 commented Feb 8, 2024 • edited Loading

SmallJoker left a comment

Choose a reason for hiding this comment

SmallJoker left a comment

Choose a reason for hiding this comment

grorp left a comment

Choose a reason for hiding this comment

grorp Feb 13, 2024

Choose a reason for hiding this comment

sfan5 Feb 13, 2024

Choose a reason for hiding this comment

grorp Feb 14, 2024

Choose a reason for hiding this comment

Desour left a comment

Choose a reason for hiding this comment

Desour Feb 13, 2024

Choose a reason for hiding this comment

Desour Feb 13, 2024

Choose a reason for hiding this comment

sfan5 commented Dec 28, 2022 •

edited

Loading

MisterE123 commented Mar 13, 2023 •

edited

Loading

MisterE123 commented Mar 13, 2023 •

edited

Loading

MisterE123 commented Mar 13, 2023 •

edited

Loading

appgurueu left a comment •

edited

Loading

appgurueu commented Mar 17, 2023 •

edited

Loading

MisterE123 commented May 24, 2023 •

edited

Loading

grorp commented Jan 11, 2024 •

edited

Loading

sfan5 commented Feb 8, 2024 •

edited

Loading