MultiEventProjection CustomGrouper documentation and tests are misleading

[migajek]

Given the fact that multistream projections are async by default (for a very good reason, as explained in the docs) I'd expect the documentation example of the multistream projection with a custom grouper to behave properly when registered as async.

This is however not the case, and the only reason the provided sample code actually works, is because it's tested in the inline mode

The test does the operations in steps; each step committed separately (and thus, projection is updated each time, for the given batch only).

However in the async mode it might (and most likely will) fail - because the custom grouper looks up the UserFeatureToggles documents - if the UserLicenseAssigned event end up in the same batch as the LicenseFeatureToggled (for given user/license), the query returns no results and therefore the grouper can't figure out which UserFeatureToggles documents should be affected.

BTW, isn't the grouper doing exactly what the docs warn us against?

If your grouping logic requires you to access the aggregate view itself, ViewProjection will not function correctly

conclusion
If I'm not mistaken, the scenario provided in the tests is still not solvable with the custom grouper (unless we set BatchSize to 1)?

Inspired by a recent question on the discord, I've crafted a solution that - instead of looking up the documents (which is the source of the issue), looks up the events - combines the past events with current batch and does a simple aggregation.
While it technically works, I don't think this is a good solution, it might get very tricky for more complex aggregations, and I'm not even sure whether this feels right.

var licenseIdsDistinct = licenseIds.Distinct().ToArray();
var commitedAssignments = await session.Events
    .QueryRawEventDataOnly<UserLicenseAssigned>()
    .Where(x => x.LicenseId.IsOneOf(licenseIds))
    .ToListAsync();
var uncommitedAssignments = events
    .OfType<IEvent<UserLicenseAssigned>>()
    .Select(x => x.Data)
    .Where(x => x.LicenseId.IsOneOf(licenseIdsDistinct))
    .ToArray();
var allEvents = commitedAssignments.Concat(uncommitedAssignments);
var streamIds = allEvents.GroupBy(x => x.LicenseId, x => x.UserId)
    .ToDictionary(x => x.Key, x => x.ToArray());

[jeremydmiller]

@migajek Sorry, I don't scan the discussions very closely -- hence the pinned discussion urging you to use Discord instead.

That's a really, really old example that hasn't been touched in a long time. By and large, I think you'll have bad performance from having to query back at the old events to do grouping like that. You might try for some kind of multi-stage thing where you do a lightweight Inline project that just writes the relationships to a document or even flat table, then use that in a custom slicer so you can batch the lookups. And maybe even cache the relationship lookup

[migajek]

@jeremydmiller thanks, I am aware of the Discord, however I thought it's good for prompt subjects. This is by no means a high priority thing, so decided to raise it here instead.

I agree on the performance penalty of querying events for aggregation, and - personally, after approaching the problem in a different ways, decided to go with the "helper" projection that is inline.

What I believe is a missing piece of puzzle is some kind of marking the dependency between projections. Rebuilding the main projection without ensuring the helper projection is rebuilt first (in case of any changes to the logic, etc) will result in integrity issues.
I handle it manually now, that is - the documentation, the comments in the code etc, but I wonder whether you considered any mechanisms in Marten itself (or the command line tooling) to mark the dependencies between projections