Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit disconnect_document with errorType to FRS on FF client disconnects #23958

Draft
wants to merge 72 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
c8fb4b5
Cherry-pick loader and container-definitions from #23268
RishhiB Jan 14, 2025
4e867dc
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 14, 2025
36988b3
api extractor, format and dev tools
RishhiB Jan 14, 2025
b3664f9
document delta connector cherry-pick
RishhiB Jan 14, 2025
f1bd575
cherry-pick- server fluidContainer
RishhiB Jan 15, 2025
b5bf87d
add jsdoc and format
RishhiB Jan 15, 2025
089e75f
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 15, 2025
9214ea2
extract string from error in disconnectCore
RishhiB Jan 15, 2025
590d091
remove extra jsdoc
RishhiB Jan 15, 2025
c19ab83
move DisconnectReason to core-interfaces
RishhiB Jan 15, 2025
1a482d5
remove type
RishhiB Jan 15, 2025
3a78424
Update packages/common/core-interfaces/src/disconnectReason.ts
RishhiB Jan 15, 2025
49b3125
feedback
RishhiB Jan 15, 2025
863e7d5
enforce error object when Corruption reason
RishhiB Jan 15, 2025
8421721
wip
RishhiB Jan 27, 2025
0f7d687
format
RishhiB Jan 27, 2025
861e707
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 27, 2025
deff9ff
fix back compat
RishhiB Jan 27, 2025
4ab55a7
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 27, 2025
eb1a10f
undo random delete
RishhiB Jan 27, 2025
95f63ed
format
RishhiB Jan 27, 2025
19ebadb
prettier for server
RishhiB Jan 27, 2025
1d5e99c
undo loader changes
RishhiB Jan 27, 2025
8730cb1
fix typo
RishhiB Jan 27, 2025
65960f4
api-doc
RishhiB Jan 27, 2025
a5d3b35
remove IContainer changes
RishhiB Jan 27, 2025
5c14139
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 27, 2025
fe68cc7
change to internal
RishhiB Jan 27, 2025
03db41d
wip
RishhiB Jan 29, 2025
bf9844a
isCorruption
RishhiB Jan 29, 2025
767ffe2
undo loader change
RishhiB Jan 29, 2025
7006e3a
wip
RishhiB Jan 29, 2025
403e769
wip
RishhiB Jan 29, 2025
bf681b1
to string
RishhiB Jan 29, 2025
c906a32
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Jan 29, 2025
a38ae82
attempt at tests
RishhiB Feb 7, 2025
dfcbea1
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 7, 2025
d2d82c8
wip
RishhiB Feb 7, 2025
59df842
use correct error objects
RishhiB Feb 7, 2025
21cdedd
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 7, 2025
ebd26f0
change to promise
RishhiB Feb 10, 2025
175d8ca
format
RishhiB Feb 10, 2025
007829e
move comment
RishhiB Feb 10, 2025
c10ef59
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 10, 2025
51007f5
directly disconnect with Corrution error instead of flowing it from d…
RishhiB Feb 10, 2025
8a33aa0
remove error on dispose
RishhiB Feb 10, 2025
3e8df2e
use helpers to generate tests
RishhiB Feb 13, 2025
e756050
format
RishhiB Feb 13, 2025
726eca4
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 13, 2025
c49ad13
Revert "use helpers to generate tests"
RishhiB Feb 18, 2025
0a1701a
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 18, 2025
69e6b18
add error back in
RishhiB Feb 20, 2025
386d400
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 20, 2025
87e872e
remove data processing error
RishhiB Feb 24, 2025
ff6b536
design change
RishhiB Feb 27, 2025
61b36ed
fix tests
RishhiB Feb 27, 2025
3b83f4a
dont log if errorType is undefined
RishhiB Feb 27, 2025
a61344f
update to use dispose instead of disconnect
RishhiB Feb 27, 2025
95940af
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 27, 2025
7701d53
fix tests
RishhiB Feb 27, 2025
f6550cc
Revert "update to use dispose instead of disconnect"
RishhiB Feb 27, 2025
3e3b052
generate tests
RishhiB Feb 27, 2025
159ad19
zach's feedback
RishhiB Feb 28, 2025
cf3ffa4
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 28, 2025
054c761
jason's feedback
RishhiB Feb 28, 2025
d4e64bd
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 28, 2025
a99e8e6
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Feb 28, 2025
9e30cc1
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Mar 3, 2025
b4add46
format
RishhiB Mar 9, 2025
b858eb7
Merge branch 'main' into cherry-pick-disconnectReason-loader
RishhiB Mar 9, 2025
6c1df19
cherry pick tests and server changes
RishhiB Mar 9, 2025
d82e712
remove arrow functions
RishhiB Mar 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions packages/drivers/driver-base/src/documentDeltaConnection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@ export class DocumentDeltaConnection
);
}

protected disconnect(err: IAnyDriverError) {
protected disconnect = (err: IAnyDriverError) => {
// Can't check this.disposed here, as we get here on socket closure,
// so _disposed & socket.connected might be not in sync while processing
// "dispose" event.
Expand Down Expand Up @@ -449,7 +449,7 @@ export class DocumentDeltaConnection

// Let user of connection object know about disconnect.
this.emit("disconnect", err);
}
};

/**
* Disconnect from the websocket.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,13 @@ export class R11sDocumentDeltaConnection extends DocumentDeltaConnection {
url: getUrlForTelemetry(this.url, socketIoPath),
};
}

/**
* Disconnect from the websocket
*/
protected disconnectCore(err: IAnyDriverError): void {
// tell the server we are disconnecting this client from the document
this.socket.emit("disconnect_document", this.clientId, this.documentId, err?.errorType);
super.disconnectCore(err);
}
}
289 changes: 178 additions & 111 deletions packages/drivers/routerlicious-driver/src/test/r11sSocketTests.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,17 @@

import { strict as assert } from "assert";

import { FluidErrorTypes } from "@fluidframework/core-interfaces/internal";
import { IClient } from "@fluidframework/driver-definitions";
import {
DriverErrorTypes,
IResolvedUrl,
type IAnyDriverError,
} from "@fluidframework/driver-definitions/internal";
import {
DataProcessingError,
DataCorruptionError,
} from "@fluidframework/telemetry-utils/internal";
import { stub } from "sinon";
import { Socket } from "socket.io-client";

Expand Down Expand Up @@ -69,129 +74,191 @@ describe("R11s Socket Tests", () => {
)) as DocumentService;
});

it("connect_document_error with Token Revoked error", async () => {
const errorToThrow = {
code: 403,
message: "TokenRevokedError",
retryAfterMs: 10,
internalErrorCode: "TokenRevoked",
errorType: DriverErrorTypes.authorizationError,
canRetry: false,
};
const errorEventName = "connect_document_error";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName, errorToThrow },
function runConnectDocumentErrorTest(
description: string,
errorToThrow: any,
expectedErrorType: string,
expectedInternalErrorCode: string,
) {
it(description, async () => {
const errorEventName = "connect_document_error";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName, errorToThrow },
});
await assert.rejects(
mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
),
{
errorType: expectedErrorType,
scenarioName: errorEventName,
internalErrorCode: expectedInternalErrorCode,
},
"Error should have occurred",
);
});
}

await assert.rejects(
mockSocket(socket as unknown as Socket, async () =>
function runSocketErrorTest(
description: string,
errorToThrow: any,
expectedErrorType: string,
expectedInternalErrorCode: string,
) {
it(description, async () => {
const errorEventName = "connect_document_success";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName },
});
const connection = await mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
),
{
errorType: DriverErrorTypes.authorizationError,
scenarioName: "connect_document_error",
internalErrorCode: "TokenRevoked",
},
"Error should have occurred",
);
});

it("Socket error with Token Revoked error", async () => {
const errorToThrow = {
code: 403,
message: "TokenRevokedError",
retryAfterMs: 10,
internalErrorCode: "TokenRevoked",
errorType: DriverErrorTypes.authorizationError,
canRetry: false,
};
const errorEventName = "connect_document_success";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName },
);
let error: IAnyDriverError | undefined;
connection.on("disconnect", (reason: IAnyDriverError) => {
error = reason;
});
socket.sendErrorEvent(errorToThrow);
assert(
error?.errorType === expectedErrorType,
`Error type should be ${expectedErrorType}`,
);
assert(error.scenarioName === "error", "Error scenario name should be error");
assert(
(error as any).internalErrorCode === expectedInternalErrorCode,
`Error internal code should be ${expectedInternalErrorCode}`,
);
});
}

const connection = await mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
);
let error: IAnyDriverError | undefined;
connection.on("disconnect", (reason: IAnyDriverError) => {
error = reason;
function runClientErrorTest(
description: string,
clientError: any,
expectedErrorType: string,
) {
it(description, async () => {
const socketEventName = "connect_document_success";
socket = new ClientSocketMock({
connect_document: { eventToEmit: socketEventName },
});
const connection = await mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
);
const disconnectEventP = new Promise<{ clientId: string; errorType: string }>(
(resolve) => {
assert(socket !== undefined, "Socket should be defined");
socket.on(
"disconnect_document",
(clientId: string, _documentId: string, errorType: string) => {
resolve({ clientId, errorType });
},
);
},
);
(connection as any).disconnect(clientError);
const disconnectResult = await disconnectEventP;
assert.strictEqual(
disconnectResult.clientId,
connection.clientId,
"Client ID should match",
);
assert.strictEqual(
disconnectResult.errorType,
expectedErrorType,
`Error type should be ${expectedErrorType}`,
);
});
}

// Send Token Revoked error
socket.sendErrorEvent(errorToThrow);

assert(
error?.errorType === DriverErrorTypes.authorizationError,
"Error type should be authorizationError",
);
assert(error.scenarioName === "error", "Error scenario name should be error");
assert(
(error as any).internalErrorCode === "TokenRevoked",
"Error internal code should be TokenRevoked",
);
});
// connect_document_error tests
const connectErrorTests = [
{
description: "connect_document_error with Token Revoked error",
errorToThrow: {
code: 403,
message: "TokenRevokedError",
retryAfterMs: 10,
internalErrorCode: "TokenRevoked",
errorType: DriverErrorTypes.authorizationError,
canRetry: false,
},
expectedErrorType: DriverErrorTypes.authorizationError,
expectedInternalErrorCode: "TokenRevoked",
},
{
description: "connect_document_error with Cluster Draining error",
errorToThrow: {
code: 503,
message: "ClusterDrainingError",
retryAfterMs: 1000,
internalErrorCode: R11sServiceClusterDrainingErrorCode,
errorType: RouterliciousErrorTypes.clusterDrainingError,
canRetry: true,
},
expectedErrorType: RouterliciousErrorTypes.clusterDrainingError,
expectedInternalErrorCode: R11sServiceClusterDrainingErrorCode,
},
];

it("connect_document_error with Cluster Draining error", async () => {
const errorToThrow = {
code: 503,
message: "ClusterDrainingError",
retryAfterMs: 1000,
internalErrorCode: R11sServiceClusterDrainingErrorCode,
errorType: RouterliciousErrorTypes.clusterDrainingError,
canRetry: true,
};
const errorEventName = "connect_document_error";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName, errorToThrow },
});
connectErrorTests.forEach((test) =>
runConnectDocumentErrorTest(
test.description,
test.errorToThrow,
test.expectedErrorType,
test.expectedInternalErrorCode,
),
);

await assert.rejects(
mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
),
{
errorType: RouterliciousErrorTypes.clusterDrainingError,
scenarioName: "connect_document_error",
// Socket error tests
const socketErrorTests = [
{
description: "Socket error with Token Revoked error",
errorToThrow: {
code: 403,
message: "TokenRevokedError",
retryAfterMs: 10,
internalErrorCode: "TokenRevoked",
errorType: DriverErrorTypes.authorizationError,
canRetry: false,
},
expectedErrorType: DriverErrorTypes.authorizationError,
expectedInternalErrorCode: "TokenRevoked",
},
{
description: "Socket error with Cluster Draining error",
errorToThrow: {
code: 503,
message: "ClusterDrainingError",
retryAfterMs: 1000,
internalErrorCode: R11sServiceClusterDrainingErrorCode,
errorType: RouterliciousErrorTypes.clusterDrainingError,
canRetry: true,
},
"Error should have occurred",
);
});

it("Socket error with Cluster Draining error", async () => {
const errorToThrow = {
code: 503,
message: "ClusterDrainingError",
retryAfterMs: 1000,
internalErrorCode: R11sServiceClusterDrainingErrorCode,
errorType: RouterliciousErrorTypes.clusterDrainingError,
canRetry: true,
};
const errorEventName = "connect_document_success";
socket = new ClientSocketMock({
connect_document: { eventToEmit: errorEventName },
});
expectedErrorType: RouterliciousErrorTypes.clusterDrainingError,
expectedInternalErrorCode: R11sServiceClusterDrainingErrorCode,
},
];

const connection = await mockSocket(socket as unknown as Socket, async () =>
documentService.connectToDeltaStream(client),
);
let error: IAnyDriverError | undefined;
connection.on("disconnect", (reason: IAnyDriverError) => {
error = reason;
});
socketErrorTests.forEach((test) =>
runSocketErrorTest(
test.description,
test.errorToThrow,
test.expectedErrorType,
test.expectedInternalErrorCode,
),
);

// Send Token Revoked error
socket.sendErrorEvent(errorToThrow);
// Client error tests
runClientErrorTest(
"Client Data Corruption error",
new DataCorruptionError("Data corruption error", { driverVersion: "1.0" }),
FluidErrorTypes.dataCorruptionError,
);

assert(
error?.errorType === RouterliciousErrorTypes.clusterDrainingError,
"Error type should be clusterDrainingError",
);
assert(error.scenarioName === "error", "Error scenario name should be error");
assert(
(error as any).internalErrorCode === R11sServiceClusterDrainingErrorCode,
"Error internal code should be R11sServiceClusterDrainingErrorCode",
);
});
runClientErrorTest(
"Client Data Processing error",
DataProcessingError.create("Data processing error", "test", undefined, {
driverVersion: "1.0",
}),
FluidErrorTypes.dataProcessingError,
);
});
14 changes: 14 additions & 0 deletions server/routerlicious/packages/lambdas/src/nexus/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -642,5 +642,19 @@ export function configureWebSocketServices(
}
disposers.splice(0, disposers.length);
});

socket.on(
"disconnect_document",
(clientId: string, documentId: string, errorType?: string) => {
if (errorType === undefined) {
return;
}
Lumberjack.error(
"Client disconnected due to error",
{ clientId, documentId, errorType },
new Error(errorType),
);
},
);
});
}
Loading