From 0d19d92ee46e5ae2a3568354106e8fc0b6101fc9 Mon Sep 17 00:00:00 2001 From: Dave Pagurek Date: Tue, 12 Dec 2023 17:33:38 -0500 Subject: [PATCH] Update WebGL contributor docs --- contributor_docs/images/caps.svg | 16 + contributor_docs/images/flags.svg | 45 ++ contributor_docs/images/joins.svg | 21 + contributor_docs/images/line-diagram.svg | 3 + contributor_docs/webgl_contribution_guide.md | 161 +++++++ contributor_docs/webgl_mode_architecture.md | 416 +++++++++++-------- 6 files changed, 496 insertions(+), 166 deletions(-) create mode 100644 contributor_docs/images/caps.svg create mode 100644 contributor_docs/images/flags.svg create mode 100644 contributor_docs/images/joins.svg create mode 100644 contributor_docs/images/line-diagram.svg create mode 100644 contributor_docs/webgl_contribution_guide.md diff --git a/contributor_docs/images/caps.svg b/contributor_docs/images/caps.svg new file mode 100644 index 0000000000..29da5f48de --- /dev/null +++ b/contributor_docs/images/caps.svg @@ -0,0 +1,16 @@ + + +Bounding Quad +EXTEND +ROUND +SQUARE + + + + + + + + + + diff --git a/contributor_docs/images/flags.svg b/contributor_docs/images/flags.svg new file mode 100644 index 0000000000..1954abb75e --- /dev/null +++ b/contributor_docs/images/flags.svg @@ -0,0 +1,45 @@ + + + + + + + + +-1 +1 +-3 +3 +-2 +2 + + + + + + + + + +-1 +1 +-2 +2 + + + + + +-1 +-2 +1 +2 +Segment +Tangent In +Tangent In +Tangent Out +Center Point +Side +Cap +Join + diff --git a/contributor_docs/images/joins.svg b/contributor_docs/images/joins.svg new file mode 100644 index 0000000000..e46d48ad5b --- /dev/null +++ b/contributor_docs/images/joins.svg @@ -0,0 +1,21 @@ + + + + + + + + + + + + + + + + +Bounding Quad +MITER +ROUND +BEVEL + diff --git a/contributor_docs/images/line-diagram.svg b/contributor_docs/images/line-diagram.svg new file mode 100644 index 0000000000..31d25794a7 --- /dev/null +++ b/contributor_docs/images/line-diagram.svg @@ -0,0 +1,3 @@ + + +CapSegmentJoinCenter Line \ No newline at end of file diff --git a/contributor_docs/webgl_contribution_guide.md b/contributor_docs/webgl_contribution_guide.md new file mode 100644 index 0000000000..cbdc2b0011 --- /dev/null +++ b/contributor_docs/webgl_contribution_guide.md @@ -0,0 +1,161 @@ +# WebGL Contribution Guide + +If you're reading this page, you're probably interested in helping work on WebGL mode. Thank you, we're grateful for your help! This page exists to help explain how we structure WebGL contributions and to offer some tips for making changes. + + +## Resources + +- Read our [p5.js WebGL architecture overview](webgl_mode_architecture.md) to understand how WebGL mode differs from 2D mode. This will be a valuable reference for some implementation specifics for shaders, strokes, and more. +- Read our [contributor guidelines](https://p5js.org/contributor-docs/#/./contributor_guidelines) for information on how to create issues, set up the codebase, and test changes. +- It can be helpful to know a bit about the browser's WebGL API, which is what p5.js's WebGL mode is built on top of: + - [WebGL fundamentals](https://webglfundamentals.org/) goes over many core rendering concepts + - [The Book of Shaders](https://thebookofshaders.com/) explains many techniques used in WebGL shaders + + +## Planning + +We organize open issues [in a GitHub Project](https://github.com/orgs/processing/projects/5), where we divide them up into a few types: + +- **System-level changes** are longer-term goals with far-reaching implications in the code. These require the most discussion and planning before jumping into implementation. +- **Bugs with no solution yet** are bug reports that need some debugging to narrow down the cause. These are not yet ready to be fixed: once the cause is found, then we can discuss the best way to fix it. +- **Bugs with solutions but no PR** are bugs where we have decided how to fix it and are free for someone to write code for. +- **Minor enhancements** are issues for new features that have an obvious spot within the current architecture without needing to discuss how to fit them in. Once agreed that these are worth doing, they are free to write code for. +- **2D features** are ones that already exist in p5.js but not within WebGL mode. The expected behavior of the feature, once implemented, is to match 2D mode. We may need to discuss the best implementation, but the user requirements for these are clear. +- **Features that don't work in all contexts** are ones that exist in WebGL mode but do not work in all the ways one can use WebGL mode. For example, some p5.js methods work with both 2D coordinates and 3D coordinates, but others break if you use 3D coordinates. These are generally free to begin working on. +- **Feature requests** are all other code change requests. These need a bit of discussion to make sure they are things that fit into WebGL mode's roadmap. +- **Documentation** issues are ones that don't need a code change but instead need better documentation of p5.js's behavior. + + +## Where to Put Code + +Everything related to WebGL is in the `src/webgl` subdirectory. Within that directory, top-level p5.js functions are split into files based on subject area: commands to set light go in `lighting.js`; commands to set materials go in `materials.js`. + +When implementing user-facing classes, we generally try to have one file per class. These files may occasionally have a few other internal utility classes. For example, `p5.Framebuffer.js` includes the class `p5.Framebuffer`, and also additionally consists of a few framebuffer-specific subclasses of other main classes. Further framebuffer-specific subclasses can go in this file, too. + +`p5.RendererGL` is a large class that handles a lot of behavior. For this reason, rather than having one large class file, its functionality is split into many files based on what subject area it is. Here is a description of the files we split `p5.RendererGL` across, and what to put in each one: + + +#### `p5.RendererGL.js` + +Initialization and core functionality. + + +#### `p5.RendererGL.Immediate.js` + +Functionality related to **immediate mode** drawing (shapes that will not get stored and reused, such as `beginShape()` and `endShape()`) + + +#### `p5.RendererGL.Retained.js` + +Functionality related to **retained mode** drawing (shapes that have been stored for reuse, such as `sphere()`) + + +#### `material.js` + +Management of blend modes + + +#### `3d_primitives.js` + +User-facing functions that draw shapes, such as `triangle()`. These define the geometry of the shapes. The rendering of those shapes then happens in `p5.RendererGL.Retained.js` or `p5.RendererGL.Immediate.js`, treating the geometry input as a generic shape. + + +#### `Text.js` + +Functionality and utility classes for text rendering. + + +## Testing WebGL Changes + +### Testing Consistency + +There are a lot of ways one can use the functions in p5.js. It's hard to manually verify all of it, so we add unit tests where we can. That way, when we make new changes, we can be more confident that we didn't break anything if all the unit tests still pass. + +When adding a new test, if the feature is something that also works in 2D mode, one of the best ways to check for consistency is to check that the resulting pixels are the same in both modes. Here's one example of that in a unit test: + +```js +test('coplanar strokes match 2D', function() { +  const getColors = function(mode) { +    myp5.createCanvas(20, 20, mode); +    myp5.pixelDensity(1); +    myp5.background(255); +    myp5.strokeCap(myp5.SQUARE); +    myp5.strokeJoin(myp5.MITER); +    if (mode === myp5.WEBGL) { +      myp5.translate(-myp5.width/2, -myp5.height/2); +    } +    myp5.stroke('black'); +    myp5.strokeWeight(4); +    myp5.fill('red'); +    myp5.rect(10, 10, 15, 15); +    myp5.fill('blue'); +    myp5.rect(0, 0, 15, 15); +    myp5.loadPixels(); +    return [...myp5.pixels]; +  }; +  assert.deepEqual(getColors(myp5.P2D), getColors(myp5.WEBGL)); +}); +``` + +This doesn't always work because you can't turn off antialiasing in 2D mode, and antialiasing in WebGL mode will often be slightly different. It can work for straight lines in the x and y axes, though! + +If a feature is WebGL-only, rather than comparing pixels to 2D mode, we often check a few pixels to ensure their color is what we expect. One day, we might turn this into a more robust system that compares against full image snapshots of our expected results rather than a few pixels, but for now, here is an example of a pixel color check: + +```js +test('color interpolation', function() { +  const renderer = myp5.createCanvas(256, 256, myp5.WEBGL); +  // upper color: (200, 0, 0, 255); +  // lower color: (0, 0, 200, 255); +  // expected center color: (100, 0, 100, 255); +  myp5.beginShape(); +  myp5.fill(200, 0, 0); +  myp5.vertex(-128, -128); +  myp5.fill(200, 0, 0); +  myp5.vertex(128, -128); +  myp5.fill(0, 0, 200); +  myp5.vertex(128, 128); +  myp5.fill(0, 0, 200); +  myp5.vertex(-128, 128); +  myp5.endShape(myp5.CLOSE); +  assert.equal(renderer._useVertexColor, true); +  assert.deepEqual(myp5.get(128, 128), [100, 0, 100, 255]); +}); +``` + + +### Performance Testing + +While not the #1 concern of p5.js, we try to make sure changes don't cause a large hit to performance. Typically, this is done by creating two test sketches: one with your change and one without the change. We then compare the frame rates of both. + +Some advice on how to measure performance: + +- Disable friendly errors with `p5.disableFriendlyErrors = true` at the top of your sketch (or just test `p5.min.js`, which does not include the friendly error system) +- Display the average frame rate to get a clear sense of the steady state frame rate: + +```js +let frameRateP; +let avgFrameRates = []; +let frameRateSum = 0; +const numSamples = 30; +function setup() { +  // ... +  frameRateP = createP(); +  frameRateP.position(0, 0); +} +function draw() { +  // ... +  const rate = frameRate() / numSamples; +  avgFrameRates.push(rate); +  frameRateSum += rate; +  if (avgFrameRates.length > numSamples) { +    frameRateSum -= avgFrameRates.shift(); +  } + +  frameRateP.html(round(frameRateSum) + ' avg fps'); +} +``` + +Here are cases we try to test since they stress different parts of the rendering pipeline: + +- A few very complicated shapes (e.g., a large 3D model or a long curve) +- Many simple shapes (e.g., `line()` called many times in a for loop) diff --git a/contributor_docs/webgl_mode_architecture.md b/contributor_docs/webgl_mode_architecture.md index 3a50eca920..4cff1152dc 100644 --- a/contributor_docs/webgl_mode_architecture.md +++ b/contributor_docs/webgl_mode_architecture.md @@ -1,179 +1,263 @@ -# p5.js WEBGL mode architecture +# p5.js WebGL Mode Architecture -This document describes the structure of the p5.js WEBGL mode for p5.js contributors and maintainers—and any other interested parties. If you're interested in using 3D graphics in your sketches, [view this tutorial](https://github.com/processing/p5.js/wiki/Getting-started-with-WebGL-in-p5) instead. +This document is intended for contributors and library makers who want to extend the WebGL codebase. If you are looking for help using WebGL mode in your sketches, consider reading the WebGL tutorials [on the p5.js Learn page](https://p5js.org/learn/) instead. -## Overview of Structure and Objects -The core objects in the WEBGL architecture are the p5.Renderer.GL, p5.Shader, p5.Texture, and p5.Geometry. -A single instance of p5.Renderer.GL manages its own p5.Shaders, p5.Textures, and p5.Geometry—a goal for this is to allow offscreen rendering using WebGL, but that hasn’t been tested yet. -Shaders and textures are associated with a specific GL context via the renderer. +## What is WebGL mode? -### p5.RendererGL -The p5.RendererGL object is the renderer for WEBGL / 3D Mode in p5.js. -Derived from p5.Renderer, it provides additional functionality not available in 2D mode, such as: `box()`, `cone()`, use of shaders, accelerated texture rendering, and lighting. +There are two renderers that p5.js can run in, 2D and WebGL mode. WebGL mode in p5.js allows the user to make use of the [WebGL API](https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API) built into the web browser for rendering high-performance 2D and 3D graphics. The key difference between 2D mode and WebGL mode is that the latter provides more direct access to the computer's GPU, allowing it to performantly render shapes in 3D or perform other graphics and image processing tasks.  -* Manages—creates and caches—shaders (p5.Shader objects) and textures (p5.Texture objects) -* Prepares coordinates of shapes to be used as attribute arrays by shaders. -* Selects the correct shader for use when stroke, fill, texture, and various lighting methods are called, then supplies the appropriate uniforms to those shaders. -* Maintains data about lighting: a count of how many various types of lights are used and their parameters. -* Caches geometry for 3D primitives (in retained mode), with the exception of shapes created with begin/endShape, which are dynamically generated and pushed to GL every time one is drawn (in immediate mode). +We keep track of the progress of WebGL issues in [a GitHub Project.](https://github.com/orgs/processing/projects/20) -The renderer manages choosing an appropriate p5.Shader for the current drawing conditions. -### p5.Shader -The p5.Shader class provides access to the uniforms and attributes of a GL program. +## Goals of WebGL mode -* Compiles and links the vertex and fragment shader components -* Provides an API for accessing and setting shader attributes and uniforms -* Binds textures the shader needs before a shape is rendered -* Provides the bindShader() method for use in the render before a shape is drawn, then unbindShader() after the shape is drawn. +When evaluating a new feature, we consider whether it aligns with the goals of p5.js and WebGL mode: -There are four default shaders, as documented in the Shader section. +1. **Features should be beginner friendly:** It should provide a **beginner-friendly introduction to WebGL** and the features it offers. This means that we should offer simple APIs for 3D shapes, cameras, lighting, and shaders. We can still support advanced features, but only if they do not interfere with the simplicity of core features. +2. **Improving feature parity with 2D mode:** It should be a **frictionless transition from 2D mode,** making 3D and WebGL "click" more easily for users. This means that we try to create features that work in 2D mode and also in WebGL mode. Since WebGL also has 3D and shader features, this means WebGL mode aims to have a superset of 2D mode's features. +3. **Simplicity and Extensibility are paramount:** It should **have a small core and be extensible for libraries.** Keeping WebGL mode small makes it easier to optimize core features and reduce bug surface area. Extension provides an avenue to include more advanced features via libraries. +4. **Improve p5.js performance:** It should **run as fast as possible without interfering with the previous goals.** Good performance keeps sketches accessible to a wide variety of viewers and devices. When designing new APIs, we try to ensure the design has a performant implementation. However, we give preference to simplicity and parity with 2D mode. -### p5.Texture -The p5.Texture object manages GL state for a texture based on a `p5.Image`, `p5.MediaElement`, `p5.Element`, or `ImageData`. -* Internally handles processing image data based on type, so that the p5.Renderer implementation doesn’t have to make special exceptions in its own methods when handling textures -* Updates conditionally every frame by making a best guess at whether or not image data has changed. Tries not to upload the texture if no change has been made, to help performance. +## Design Differences with 2D Mode -### p5.Geometry -Shapes rendered in retain mode are stored as p5.Geometry objects in a cache in the p5.Renderer object. -The renderer maps strings to p5.Geometry objects based on the shape drawn and its parameters (for example, geometry for a box created with `box(70, 80, 90, 10, 20)` is mapped from `'box|70|80|90|10|20’`). Calls to functions that have retained geometry create a p5.Geometry object on the first pass and store it in a geometry hash using the aforementioned string ID. Later calls look for that ID in the hash, and if it is found use that to reference the existing object rather than creating a new one. +The browser's 2D and WebGL canvas context APIs offer very different levels of abstraction, with WebGL being generally lower-level and 2D being higher-level. This motivates some fundamental design differences between p5.js's WebGL and 2D modes. + +- **WebGL mode creates more deeply nested data structures.** 2D mode generally passes commands to the browser, leading to relatively shallow call stacks. In contrast, WebGL mode is responsible for breaking down shapes into triangles, rendering them, and often caching them for future reuse. The more complicated rendering logic necessitates splitting code into a number of classes like `p5.Texture`, `p5.RenderBuffer`, and `p5.DataArray` to keep implementations readable and maintainable. +- **WebGL mode offers more customization.** For example, while 2D mode does not control how curves get rendered, WebGL is responsible for converting them into triangles. While it picks a sensible default, the `curveDetail()` API lets users control how many line segments to use, as we cannot predict the best balance of quality and performance for every use case. +- **WebGL mode must balance high- and low-level APIs.** Since finer-grained control is available with the browser WebGL API, p5.js's WebGL mode is able to offer users some of that control where 2D mode cannot. We then are faced with the task of picking the right level of abstraction for users. Too high, and they are unable to take advantage of some of what the browser has to offer; too low, and we pass too much of the work of managing complexity and performance onto the user. + + +## Drawing Shapes + +### Creating Shapes: Fills, Strokes, and 3D Geometry + +Everything drawn by p5.js, both in 2D and WebGL, consists of fills and strokes. Sometimes, we only draw one or the other, but every shape must be ready to draw either component. + +All shapes in webGL are composed of triangles. When a user calls a function like `circle()`, `beginShape(),` or `vertex()`, the renderer must [break the shape down into a series of points](https://github.com/processing/p5.js/blob/main/src/webgl/3d_primitives.js). The points are connected into lines, and the lines into triangles. For example, `circle()` uses trigonometry to figure out where to place points along a circle. `curveVertex()` and `bezierVertex()` create look-up tables to turn any Bezier curve into points. + + +#### Fills + +To create fills, the outline of a shape needs to be filled in with triangles. Some drawing functions like `beginShape(TRIANGLE_STRIP)` already provide triangles for fills. If the shape being drawn is not already made using triangles, we use the library [libtess](https://github.com/brendankenny/libtess.js/) to break it down: in [p5.RendererGL.Immediate.js](https://github.com/processing/p5.js/blob/main/src/webgl/p5.RendererGL.Immediate.js), we run polygon outlines through `_processVertices()`. After libtess turns them into triangles, they are in a format where a shader can draw them to the screen. + + +#### Strokes + +Despite their name, strokes also need to be filled in to support strokes of varying widths and styles. The lines along the outline of a shape need to expand out from their centers to form shapes with area. Expansion of strokes creates three types of shapes: joins, caps, and segments, illustrated below. + +Illustration of the segment, join, and cap shapes created by a stroke. + + +Where two line segments connect, we add a join.  + +- A miter join extends the edges of the rectangles until they intersect at a point.  +- A bevel join connects the corners of the rectangles with a straight line.  +- A round join connects the corners with a circular arc.  + +We support switching join styles without having to recompute line triangles. To do this, we make each line join a quad, composed of two triangles. The quad extent bounds all possible join styles. We use a shader to only display pixels within the quad that are present in the selected join style. We illustrate how each style fits into the quad below. + +The bounding quad of a stroke join and how each join style fits into it. The bounding quad is a square. Miter joins fill the whole square. Round joins round off one corner of the square. Bevel joins cut the square in half in a straight line. + +We use a similar strategy for stroke caps, present at the disconnected ends of lines. Each cap is a quad that bounds round, square, and extended cap styles. The line shader determines which pixels it needs to draw within those bounds. Below, we illustrate how each style fits into the quad. + +The bounding quad of a stroke cap and how each style fits into it. The bounding quad is a rectangle. The extend cap fills the whole rectangle. The round cap places a semicircle inside the rectangle. The square cap leaves the rectangle empty. + +3D shapes can also have strokes, but stroke shapes are calculated in 2D. This means they can change based on the camera's perspective. We want to avoid as much recalculation as possible, so we store all the information about the line that is not camera-dependent: + +- We include the **center points of the line** in model space, shown below in red. +- We include the **direction of the line**, its tangent, at the start and end of each shape, shown in blue and pink, respectively. This helps us compute the shape of joins where two lines connect. +- We include **a flag that uniquely identifies each corner of the shape.** Combined with the tangent and the normal (a 90-degree rotation of the tangent), it helps determine in what direction to expand the line to give it thickness. + +To draw the line, we combine that information with camera intrinsics in a shader to produce the final line positions in screen space. + +The information stored about lines, and the final shapes that they turn into. + + +### Rendering Shapes: Immediate and Retained Modes + +There are two routes that p5.js uses to draw shapes onto the screen: **immediate mode** and **retained mode.** + +**Immediate mode** is optimized for shapes that change every frame. If you were drawing a curve that changes each frame, its shape data would be different every time you drew it. Because of this, immediate mode fits it best. It indicates to p5.js that it does not need to spend time storing the shape for reuse, and it saves graphics memory from being filled up with all the shape variations over time. The following functions use this mode: + +- `vertex()`, `curveVertex()`, `bezierVertex()`, and `quadraticVertex()`, called between `beginShape()` and `endShape()` +- `rect()` when using rounded corners +- `bezier()` +- `curve()` +- `line()` +- `image()` + +Retained mode is optimized for shapes that you will need to keep redrawing and don’t change shape. Once a shape is made of triangles and has been sent to the GPU to draw, retained mode keeps it there. It can then be drawn again without having to spend time re-triangulating it or sending it to the GPU again. The saved shape data is kept in a p5.Geometry object. p5.Geometry stores triangle data and keeps track of its uploaded buffers on the GPU. Calling `freeGeometry()` clears the GPU data to make space. Drawing it again after that will re-upload the data. Many 3D shape drawing functions in p5.js, such as `sphere()` or `cone()`, use this internally. + +You can use buildGeometry() to make a p5.Geometry out of immediate mode commands. You call it with a function that runs a series of any p5.js shape drawing functions. It runs the function, collects the shapes into a new p5.Geometry, and returns it. The p5.Geometry can then be drawn and redrawn efficiently in the future. + + +## Materials, Lights, and Shaders + +Every shape we draw uses a single shader for its fills, and a single shader for its strokes. There are a few default shaders that one can pick from in p5.js. You can also write and use your own shader instead of the default ones. + +The default shaders work with p5.js's lighting and materials system. The user can specify what lights are in the scene with a shape and how each object reacts to light, including color and shininess. This information is given to the shader for each object being drawn. Custom shaders can also access the same lighting and material information, allowing users and library makers to extend the default rendering behavior. + + +### Shaders + +P5 has a few shaders built in: + +- **Color Shader:** for drawing flat colors, activated by using `fill()` or `stroke()`. +- **Lighting Shader:** for drawing 2D and 3D shapes with complex lighting and textures. Activated by calling `lights()`, `ambientLight()`, `directionalLight()`, `pointLight()`, and `spotLight()`. Each adds a light to the lighting list. All added lights contribute to the shading of the shape. If you do not use lights, the shape will be drawn using the color shader, which only uses the fill color. +- **Normal/Debug Shader:** for drawing 2D and 3D shapes using the surface normal as a color. It is activated by calling `normalMaterial()`. + + +### Lights + +Users can set `ambientLight()`, `directionalLight()`, `pointLight()`, and `spotLight()`. Each adds a light to the lighting list. All added lights contribute to the shading of the shape. They all have a color, and some have a few other parameters, such as position or direction. + +For all but `ambientLight()`, each light added contributes to the view-independent lighting of shapes and the view-dependent reflections. The reflection of each light, by default, matches the color of that light. If desired, one can change the color of all reflections by setting `specularColor()`. + +If you do not use lights, the shape will be drawn using the **color shader,** which only uses the fill color. + + +### Materials + +Each 3D object has a few material properties that can be set by the user: + +- **Fill color:** This is the color you see when there are no lights. Set it with `fill()`. If done before drawing the shape, it applies to the whole shape. If called before a `vertex()` between `beginShape()`/`endShape()`, it will apply only to that vertex. A texture, if present from a call to `texture()`, will override the fill. When there are lights, this color will be mixed with the diffuse component of the light. The diffuse component describes the bright and dark areas of the surface due to light being directly cast on it. +- **Specular material:** This is the color that gets mixed with the specular component of the light. The specular component describes the reflected highlights seen on the shape's surface. Set this parameter with `specularMaterial()`. If unspecified, the shape will have no reflections. +- **Shininess:** Set with `shininess()`, this is how sharp the specular reflections are. +- **Ambient material:** This is the color that gets mixed with the ambient light. Ambient light describes constant omnidirectional light on the shape. Set this parameter with `ambientMaterial()`. If unspecified, it will be the same as the fill color. +- **Emissive material**: Set with `emissiveMaterial()`, this adds a constant color to the lighting of the shape, as if it were producing its own light of that color. + + +### Shader Implementation + +The lighting and material parameters get turned into shader attributes and uniforms. If you reference them in a custom shader, p5.js will supply them automatically. + +While advanced shader writers can take advantage of these properties, it can be unclear for new users. In the Future Goals section, we describe some plans for improving the API. We may want to improve it before publicly documenting and supporting it. + + +#### Global + +For all objects in all contexts, the following global uniforms are available: + +- `uniform mat4 uModelViewMatrix`: A matrix to convert object-space positions into camera-space +- `uniform mat4 uProjectionMatrix`: A matrix to convert camera-space positions into screen space +- `uniform mat3 uNormalMatrix`: A matrix to convert object-space normals into camera-space + +Additionally, these per-vertex properties are available as attributes: + +- `attribute vec3 aPosition`: The position of the vertex in object space +- `attribute vec3 aNormal`: For fills, a direction pointing outward from the surface +- `attribute vec2 aTexCoord`: For fills, a coordinate between 0 and 1 in x and y referring to a location on a texture image +- `attribute vec3 aVertexColor`: For fills, an optional per-vertex color + + +#### Lights + +- `uniform bool uUseLighting`: Whether or not lights have been provided + +If `uUseLighting` has been set, further lighting information will be passed in: + + +##### Ambient lights + +- `uniform int uAmbientLightCount`: How many ambient lights are present, up to a maximum of 5 +- `uniform vec3 uAmbientColor[5]`: Ambient light colors + + +##### Directional lights + +- `uniform int uDirectionalLightCount`: How many directional lights are present, up to a maximum of 5 +- `uniform vec3 uLightingDirection[5]`: Light directions +- `uniform vec3 uDirectionalDiffuseColors[5]`: Light diffuse colors +- `uniform vec3 uDirectionalSpecularColors[5]`: Light specular colors + + +##### Point lights + +- `uniform int uPointLightCount`: How many point lights are present, up to a maximum of 5 +- `uniform vec3 uPointLightLocation[5]`: Locations of point lights +- `uniform vec3 uPointLightDiffuseColors[5]`: Diffuse colors of point lights +- `uniform vec3 uPointLightSpecularColors[5]`: Specular colors of point lights + + +##### Spot lights + +- `uniform int uSpotLightCount`: How many spot lights are present, up to a maximum of 5 +- `uniform float uSpotLightAngle[5]`: Spot light cone radii +- `uniform float uSpotLightConc[5]`: Concentration (focus) of each spot light +- `uniform vec3 uSpotLightDiffuseColors[5]`: Light diffuse colors +- `uniform vec3 uSpotLightSpecularColors[5]`: Light specular colors +- `uniform vec3 uSpotLightLocation[5]`: Spot light locations +- `uniform vec3 uSpotLightDirection[5]`: Spot light directions + + +#### Materials + +##### Fill color + +- `uniform vec4 uMaterialColor`: shape fill color +- `uniform bool uUseVertexColor`: Whether there are per-vertex colors that override the shape fill color +- `attribute vec4 aVertexColor`: Per-vertex fill color +- `uniform bool isTexture`: Whether a texture is specified +- `uniform sampler2D uSampler`: A texture +- `uniform vec4 uTint`: A tint multiplier for the texture + + +##### Specular reflections + +- `uniform bool uSpecular`: Whether to show reflections +- `uniform float uShininess`: The shininess of the material +- `uniform vec4 uSpecularMatColor`: The material color to blend with specular light + + +##### Ambient color + +- `uniform bool uHasSetAmbient`: Whether an ambient color has been set or if it should default to the fill color +- `uniform vec4 uAmbientMatColor`: The material color to blend with ambient light + + +#### Other shader inputs + +##### Lines + +The default line shader is automatically supplied with these global properties in uniforms: + +- `uniform vec4 uViewport`: A vector containing `[minX, minY, maxX, maxY]` of the screen rectangle +- `uniform int uPerspective`: A boolean specifying whether to apply perspective projection to line thickness +- `uniform int uStrokeJoin`: An enum representing the join style. 0, 1, and 2 represent `ROUND`, `MITER`, `BEVEL`. +- `uniform int uStrokeCap`: An enum representing the cap style. 0, 1, and 2 represent `ROUND`, `PROJECT`, `SQUARE`. + +It also has the following per-vertex attributes: + +- `attribute vec3 aTangentIn`: The 3D direction at the start of the line segment +- `attribute vec3 aTangentOut`: The 3D direction at the end of the line segment +- `attribute float aSide`: An enum representing what part of the stroke shape the point is on. See the Strokes section for details. + + +##### Points + +- `uniform float uPointSize`: The radius of the point being drawn + + +## Classes + +The entrypoint to most WebGL code is through **p5.RendererGL**. Top-level p5.js functions are passed to the current renderer. Both 2D and WebGL modes have renderer classes that conform to this common interface. Immediate mode and retained mode functions are split up into **p5.RendererGL.Immediate.js** and **p5.RendererGL.Retained.js**. + +Within the renderer, references to models are stored in the **retainedMode.geometry** map. Each value is an object storing the buffers of a **p5.Geometry**. When calling model(yourGeometry) for the first time, the renderer adds an entry in the map. It then stores references to the geometry's GPU resources there. If you draw a p5.Geometry to the main canvas and also to a WebGL p5.Graphics, it will have entries in two renderers. + +Each material is represented by a **p5.Shader.** You set the current shader in the renderer via the shader(yourShader) function. This class handles compiling shader source code and setting shader uniforms. + +When setting a shader uniform, if the uniform type is an image, then the renderer creates a p5.Texture for it. Each p5.Image, p5.Graphics, p5.MediaElement, or p5.Framebuffer asset will get one. It is what keeps track of the image data's representation on the GPU. Before using the asset in a shader, p5.js will send new data to the GPU if necessary. For images, this happens when a user has manually updated the pixels of an image. This happens every frame for assets with data that may have changed each frame, such as a video or a p5.Graphics. + +Textures corresponding to **p5.Framebuffer** objects are unique. Framebuffers are like graphics: they represent surfaces that can be drawn to. Unlike p5.Graphics, framebuffers live entirely on the GPU. If one uses a p5.Graphics as a texture in a shader, the data needs to be transferred to and from the CPU. This can often be a performance bottleneck. In contrast, when drawing to a framebuffer, you draw directly to its GPU texture. Because of this, no extra data transfer is necessary. WebGL mode tries to use p5.Framebuffers over p5.Graphics where possible for this reason. + + +## Future Goals + +Currently, WebGL mode is functional for a variety of tasks, but many users and library makers want to extend it in new directions. We aim to create a set of building blocks for users and library makers from which they can craft extensions. A block can be considered "done" when it has an extensible API we can confidently commit to supporting. A major milestone for WebGL mode will be when we have a sufficient set of such blocks for an ecosystem of libraries. The main areas currently lacking in extension support are geometry and materials. + +- **Extend p5.Geometry to support richer content.** One can create geometry, but many tasks a user might want to accomplish are not yet supported with a stable API. One might want to efficiently update geometry, which is necessary to support animated gltf models. One might want to group multiple materials in one object, if they are present in an imported model. One might want to add custom vertex attributes for a shader to work with. These tasks are currently unsupported. +- **Enable less brittle custom shaders.** To create a shader that integrates p5.js's lighting and materials system, a user is currently forced to create shaders from scratch. These shaders often copy and paste parts of default shaders. This may break between versions if internal naming or structure changes. To be less brittle, libraries should be able to import and reuse default pieces. This lets libraries reuse positioning logic or augment positioning logic but reuse shading logic. There is currently [an issue open for this task.](https://github.com/processing/p5.js/issues/6144) +- **Improve performance.** WebGL mode tries to strike a balance between features and performance. One method is to introduce APIs to tune output quality, like how `curveDetail()` allows faster but lower-quality curves. Line rendering is one of the common performance bottlenecks in its present state, and it could benefit from having lower fidelity but higher performance options. Another method is to introduce new types of objects and rendering methods that are optimized for different usage patterns, like how `endShape(shouldClose, count)` now supports WebGL 2 instanced rendering for more efficient drawing of many shapes. -* Stores vertices, normals, faces, line vertices, line normals, and texture coordinates for the geometry primitives -* Provides methods for computing the faces, normals, line vertices, and line normals for a set of vertices - -## Immediate Mode -All attributes for drawing with Immediate Mode are stored in an object in the renderer, used to draw to the GL drawing context, and then discarded. - -## Geometry: Retain and Immediate Mode -Retained geometry is used for 3D primitives, while immediate mode is used for shapes created with begin/endShape. - -|Functions with retained geometry| Functions with immediate mode geometry | -|--------------------------------|----------------------------------------| -|plane() | bezier() | -|box() | curve() | -|sphere() | line() | -|cylinder() | beginShape() | -|cone() | vertex() | -|ellipsoid() | endShape() | -|torus() | point() | -|triangle() | curveVertex() | -|arc() | bezierVertex() | -|point() | quadraticVertex() | -|ellipse() | -|rect() | -|quad() | -|text() | - - - - - - -## Texture Management -A p5.Renderer.GL instances manages an array of p5.Textures objects on an as-needed basis. -Textures are created for images and videos used with the `texture()` method or as uniforms provided to custom shaders. - -When the renderer needs a texture, it first checks if one has already been created for a given image/video, then provides that to a shader for rendering. A new texture is only created if an existing one cannot be found for the image/video. - -## Shaders - -### Types of Shaders - -#### Color Shader -Provides flat shading of objects, based on the current fill color. - -#### Light Shader (for lighting AND textures) -Accounts for: -* Lighting parameters set by `ambientLight()`, `directionalLight()`, `pointLight()`, `spotLight()` and `specularColor()` -* Material parameters set by `ambientMaterial()`, `emissiveMaterial()` and `specularMaterial()` -* Texture parameters, set by `texture()` - -#### Normal Shader -The normal shader is set when `normalMaterial()` is in use. It uses the surface’s normal vector to determine a fragment color. - -### Shader Parameters - -#### Standard Model View & Camera Uniforms -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|---------------------------------|-----------|---------------|------------|-------------|------------| -|`uniform mat4 uModelViewMatrix;` |x |x |x |x |x | -|`uniform mat4 uProjectionMatrix;`|x |x |x |x |x | -|`uniform vec4 uViewPort;` |x | | | | | -|`uniform vec4 uPerspective;` |x | | | | | - - -#### Geometry Attributes and Uniforms -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|---------------------------------|-----------|---------------|------------|-------------|------------| -|`attribute vec3 aPosition;` |x |x |x |x |x | -|`attribute vec3 aNormal;` | |x | |x | | -|`attribute vec2 aTexCoord;` | |x | |x | | -|`uniform mat3 uNormalMatrix;` | |x | |x | | -|`attribute vec4 aDirection;` |x | | | | | -|`uniform float uStrokeWeight;` |x | | | | | - -#### Material Colors -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|---------------------------------|-----------|---------------|------------|-------------|------------| -|`uniform vec4 uMaterialColor;` |x |x | | |x | -|`attribute vec4 aVertexColor;` | | |x | | | - -#### Light Parameters - -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|-----------------------------------------------|-----------|---------------|------------|-------------|------------| -|`uniform int uAmbientLightCount;` | |x | | | | -|`uniform int uDirectionalLightCount;` | |x | | | | -|`uniform int uPointLightCount;` | |x | | | | -|`uniform int uSpotLightCount;` | |x | | | | -|`uniform vec3 uAmbientColor[8];` | |x | | | | -|`uniform vec3 uLightingDirection[8];` | |x | | | | -|`uniform vec3 uDirectionalDiffuseColors[8];` | |x | | | | -|`uniform vec3 uDirectionalSpecularColors[8];` | |x | | | | -|`uniform vec3 uPointLightLocation[8];` | |x | | | | -|`uniform vec3 uPointLightDiffuseColors[8];` | |x | | | | -|`uniform vec3 uPointLightSpecularColors[8];` | |x | | | | -|`uniform float uSpotLightAngle[8];` | |x | | | | -|`uniform float uSpotLightConc[8];` | |x | | | | -|`uniform vec3 uSpotLightDiffuseColors[8];` | |x | | | | -|`uniform vec3 uSpotLightSpecularColors[8];` | |x | | | | -|`uniform vec3 uSpotLightLocation[8];` | |x | | | | -|`uniform vec3 uSpotLightDirection[8];` | |x | | | | -|`uniform bool uSpecular;` | |x | | | | -|`uniform bool uEmissive;` | |x | | | | -|`uniform int uShininess;` | |x | | | | -|`uniform bool uUseLighting;` | |x | | | | -|`uniform float uConstantAttenuation;` | |x | | | | -|`uniform float uLinearAttenuation;` | |x | | | | -|`uniform float uQuadraticAttenuation;` | |x | | | | - -#### Texture Parameters - -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|--------------------------------------|-----------|---------------|------------|-------------|------------| -|`uniform sampler2D uSampler;` | |x | | | | -|`uniform bool isTexture;` | |x | | | | - -#### General Parameters - -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|--------------------------------------|-----------|---------------|------------|-------------|------------| -|`uniform float uResolution;` | | |x | | | -|`uniform float uPointSize;` | | |x | |x | - -#### Varying Parameters - -|Parameter |Line Shader|TexLight Shader|Color Shader|Normal Shader|Point Shader| -|--------------------------------------|-----------|---------------|------------|-------------|------------| -|`varying vec3 vVertexNormal;` | |x | | | | -|`varying vec2 vVertTexCoord;` | |x | | | | -|`varying vec3 vLightWeighting;` | |x | | | | -|`varying highp vec2 vVertTexCoord;` | | | |x | | -|`varying vec4 vColor;` | | |x | | | -|`varying float vStrokeWeight` | | | | |x | - -## Next Steps - -Coming soon!