Add outgoing metadata to TestContext

[bradwilson]

Now that v3 has a TestContext object, we should provide a way for test authors (and extensibility authors who are writing test extensions) a way to record additional information that can be reported (which I'll call "outgoing metadata").

One example of such a request is UI-based testing systems which would like to attach screenshots to failed test records.

Let's assume a model that is like traits (aka "incoming metadata"): introduce name/value pairs of string => string, where the data is then added to the "results" object model, which then in turn gets placed into XML on the way out for reporting (this XML can be written directly to disk, and is also the basis of our transformation system to produce other report files, like competing/compatible XML report files and an HTML-based report).

I believe the use of strings here rather than arbitrary types (i.e., string => object) places a reasonable limit on the outgoing data to be able to ensure that it is serializable; using arbitrary objects then places an additional serialization burden on the end user, and at the end of the day, still needs to get translated into a string to be placed into XML. The usefulness of a Base64 encoding of a .NET binary serialized object is effectively zero; forcing the user to think about how to make a string value useful, therefore, ensures that this design consideration is not overlooked. It makes more sense, then, to just use string as the value type in the key/value pair, rather than object + yet another arbitrary serialization system.

What remains, then, is the decision about how much we should expect from the end user about the string values, and whether we should attempt to suggest or enforce some restrictions. For example, should they be limited in size? Should we suggest or mandate that they be human readable? If we don't mandate human readability and/or limit the size, does that mean we shouldn't render their values into the HTML report, which is intended to be relatively compact and human readable?

In the aforementioned example of screenshots, I feel a bit torn by this requirement and how best to support it (indirectly).

There are two obvious (and perhaps other non-obvious) implementations here.

1. Write the screenshot to disk and use the outgoing string value to point to the screenshot file.
2. Use a compacted version of the screenshot, and then Base64 encode that into the outgoing string value.

Option 1 is more readable in report form, but has an inherent external dependency (the file on disk) that may not have been preserved in some circumstances, like a continuous integration environment. Option 2 is not reasonable readable in a generic report, and would require some additional transformation during display that would not necessarily be inherent in the built-in HTML reporting.

As such, I'm looking for feedback on the design of this outgoing metadata system. I like the self-contained nature of the XML, and am concerned that introducing the extra concept of files necessarily complicates that model. If we directly support files, what's the best way to represent this? Perhaps the outgoing metadata is a trio of information ("key", "value", and "value-data-type") which can be used to help reports determine what to do with that information? Do we also need to provide an arbitrary external storage mechanism for values, or should we assume that specific value types are actually just Base64-encoded binary values?

[HoraceBury]

@bradwilson Any news on this issue? So desperately want to report out screenshots! Thanks for the hard work.

[bradwilson]

@HoraceBury It's on the v3 roadmap. It will be part of v3, but has not been implemented yet. No other information is available at this time.

[naeemakram]

Hey! What's up with this one? I would love to add some screenshots to my test reports.

[JamesTerwilliger]

OK, I'm digging into this item in my brain just a little bit just out of curiosity. Please excuse my ignorance. Some questions:

1. It seems as if options 1 and 2 above are kind of the same option if one leaves it open to the user or the extensibility point to decide how to use it. I would imagine that whoever is calling the reporting methods in the tests is at least talking to the person writing the UI for the test output, so whether the caller writes a URI or a base64 encoded string is up to the caller, and the UI writer will just need to know what to do with it, and from the xUnit perspective, a string is a string is a string. Or am I missing something?
2. Poking around at the TestContext object, there already appear to be some very handy SendDiagnosticMessage methods. These obviously don't do quite the right thing, but maybe have the right interface? As in, "I have a context object, I have some of my own context, let's write it down!" So perhaps what's needed are (1) methods that look like ReportState(string key, string message) with possibly some of the same overloads as SendDiagnosticMessage for templating, but also (2) save that new state alongside the key metadata at that point of the execution so that it can be collated later, and finally (3) a sink to which to write all of that delicious, delicious new state when it's all done. Is that right?
3. And if that's right, does that mostly mean that we need an interface to specify that sink and possibly what serialization format to use (XML versus JSON) alongside 1 and 2 above?

[bradwilson]

whether the caller writes a URI or a base64 encoded string is up to the caller

I don't think that's sufficient, though, because if someone just writes a file to a disk, how do you translate that into a durable URI?

Poking around at the TestContext object, there already appear to be some very handy SendDiagnosticMessage methods. These obviously don't do quite the right thing, but maybe have the right interface?

I don't think so, because this is more like traits than diagnostic messages. If the user wants to report something which is a formatted string, they can just use string.Format or interpolation on their own.

does that mostly mean that we need an interface to specify that sink and possibly what serialization format to use (XML versus JSON) alongside 1 and 2 above?

The discussion here for XML is the reporting format that already exists. What would show up to the runners would likely just be a Dictionary<string, string> like traits today.

[bradwilson]

This has been implemented as "attachments", which can come in two forms: plain text strings or binary data (byte[]) along with a media type (like image/png).

This is available in v3 0.2.0-pre.46 https://xunit.net/docs/using-ci-builds

Adding attachments

Adding attachments is done via two overloaded TestContext.Current.AddAttachment methods.

Add a plain-text string value:

xunit/src/xunit.v3.core/TestContext.cs

Lines 221 to 228 in cbee584

	/// <summary>
	/// Adds an attachment that is a string value.
	/// </summary>
	/// <param name="name">The name of the attachment</param>
	/// <param name="value">The value of the attachment</param>
	public void AddAttachment(
	string name,
	string value)

Add a binary value w/ media type:

xunit/src/xunit.v3.core/TestContext.cs

Lines 245 to 260 in cbee584

	/// <summary>
	/// Adds an attachment that is a binary value (represented by a byte array and media type).
	/// </summary>
	/// <param name="name">The name of the attachment</param>
	/// <param name="value">The value of the attachment</param>
	/// <param name="mediaType">The media type of the attachment; defaults to "application/octet-stream"</param>
	/// <remarks>
	/// The <paramref name="mediaType"/> value must be in the MIME "type/subtype" form, and does not support
	/// parameter values. The subtype is allowed to have a single "+" to denote specialization of the
	/// subtype (i.e., "application/xhtml+xml"). For more information on media types, see
	/// <see href="https://datatracker.ietf.org/doc/html/rfc2045#section-5.1"/>.
	/// </remarks>
	public void AddAttachment(
	string name,
	byte[] value,
	string mediaType = "application/octet-stream")

Attachments can only be added in the context of an executing test, which means effectively during:

• Test class creation
• BeforeAfterAttribute.Before
• Test method invocation
• BeforeAfterAttribute.After (*)
• Test class disposal (*)

During the last two steps marked with (*), you will have access to the proposed result state of the test, via TestContext.Current.TestState?.Result. The TestState property is nullable, because it will not contain any data until after test method invocation is complete. Note that this is considered a proposed result state, because any errors caused during BeforeAfterAttribute.After or test class disposal can cause a previously passing test to ultimately fail.

Attachments recorded during the test will be available to runners via ITestFinished:

xunit/src/xunit.v3.common/Abstractions/Messages/ITestFinished.cs

Lines 10 to 16 in cbee584

	/// <summary>
	/// Gets any attachments that were added to the test result via <see cref="M:Xunit.TestContext.AddAttachment"/>.
	/// </summary>
	// Due to the potential serialization size of this information, it was decided to put this only
	// in ITestFinished and not in ITestResultMessage, because otherwise the information would be
	// duplicated on the wire.
	public IReadOnlyDictionary<string, TestAttachment> Attachments { get; }

Updated native XML reports

The native XML output format has been expanded to record the attachments:

xunit/src/xunit.v3.runner.common/Sinks/ExecutionSink.cs

Lines 444 to 472 in cbee584

	void HandleTestFinished(MessageHandlerArgs<ITestFinished> args)
	{
	var finished = args.Message;
	if (finished.Attachments.Count != 0 && testResultElements.TryRemove(finished.TestUniqueID, out var testResultElement))
	{
	var attachmentsElement = new XElement("attachments");
	foreach (var attachment in finished.Attachments)
	{
	var attachmentElement = new XElement("attachment", new XAttribute("name", attachment.Key));
	if (attachment.Value.AttachmentType == TestAttachmentType.String)
	attachmentElement.Add(new XCData(attachment.Value.AsString()));
	else
	{
	var (byteArray, mediaType) = attachment.Value.AsByteArray();
	attachmentElement.Add(new XAttribute("media-type", mediaType));
	attachmentElement.SetValue(Convert.ToBase64String(byteArray));
	}
	attachmentsElement.Add(attachmentElement);
	}
	testResultElement.Add(attachmentsElement);
	}
	if (options.AssemblyElement is not null)
	metadataCache.TryRemove(finished);
	}

This manifests in slightly different form for string values:

<attachments>
  <attachment name="NAME">STRING_VALUE</attachment>
</attachments>

vs. binary values:

<attachments>
  <attachment name="NAME" media-type="MEDIA_TYPE">BASE64_ENCODED_BYTES</attachment>
</attachments>

Updated JUnit XML reports

The JUnit XML output format does not have official support for attachments, but the common pattern is to add them to the <properties> node of the <testcase> with names that indicate they are attachments. There is an informal format to represent Base 64 encoded binary values.

Updated XSL-T template:

xunit/src/xunit.v3.runner.common/Transforms/templates/JUnitXml.xslt

Lines 92 to 103 in cbee584

	<xsl:template match="attachment">
	<property>
	<xsl:attribute name="name">attachment:<xsl:value-of select="@name"/></xsl:attribute>
	<xsl:choose>
	<xsl:when test="@media-type">data:<xsl:value-of select="@media-type"/>;base64,<xsl:value-of select="."/></xsl:when>
	<xsl:otherwise>
	<xsl:value-of select="."/>
	</xsl:otherwise>
	</xsl:choose>
	<xsl:if test="@media-type"></xsl:if>
	</property>
	</xsl:template>

String values:

<properties>
  <property name="attachment:NAME" value="STRING_VALUE" />
</properties>

vs. binary values:

<properties>
  <property name="attachment:NAME" value="data:MEDIA_TYPE;base64,BASE64_ENCODED_BYTES" />
</properties>

Updated CTRF JSON reports

There is no official CTRF JSON support for attachments, so we have added them to the extra section of the test result:

xunit/src/xunit.v3.runner.common/Transforms/TransformFactory.cs

Lines 385 to 399 in cbee584

	var attachmentsXml = testXml.Element("attachments")?.Elements("attachment");
	if (attachmentsXml is not null)
	using (var attachmentsJson = extraJson.SerializeObject("attachments"))
	foreach (var attachmentXml in attachmentsXml)
	if (attachmentXml.Attribute("name") is XAttribute nameXml)
	{
	if (attachmentXml.Attribute("media-type") is not XAttribute mediaTypeXml)
	attachmentsJson.Serialize(nameXml.Value, attachmentXml.Value);
	else
	using (var attachmentJson = attachmentsJson.SerializeObject(nameXml.Value))
	{
	attachmentJson.Serialize("media-type", mediaTypeXml.Value);
	attachmentJson.Serialize("value", attachmentXml.Value);
	}
	}

String values:

{
  "attachments": {
    "NAME": "STRING_VALUE"
  }
}

vs. binary values:

{
  "attachments": {
    "NAME": {
       "media-type": "MEDIA_TYPE",
       "value": "BASE64_ENCODED_BYTES"
    }
  }
}

[bradwilson]

Sample images posted to our Mastodon account:

Source:

Native XML:

JUnit XML:

CTRF JSON:

[Piotrreek]

What am I doing wrong in here ?

I would expect to see attachment in VS Test Explorer with the zip file, but there is nothing to see.
I am migrating from NUnit and previously it was done like that and attachments were visible in Test Explorer locally (and also in Azure DevOps ci/cd pipeline tests summary) for failed tests.

And another question. How can I retrieve an XML test results file that was sent in previous comment ?

[bradwilson]

xunit/visualstudio.xunit#425