Merge branch 'feature/implied-leading' of github.com:sjgallagher2/pyg…

…erber into feature/implied-leading
Argmaster · Nov 19, 2024 · 872fd5d · 872fd5d
2 parents f7d4ac7 + c526dd5
commit 872fd5d
Show file tree

Hide file tree

Showing 42 changed files with 483 additions and 100 deletions.
diff --git a/docs/LICENSE.md → docs/10_license.md b/docs/LICENSE.md → docs/10_license.md
diff --git a/docs/Changelog.md → docs/30_changelog.md b/docs/Changelog.md → docs/30_changelog.md
diff --git a/docs/40_gerber/00_introduction.md b/docs/40_gerber/00_introduction.md
@@ -0,0 +1,124 @@
+# Introduction
+
+## What you will find here
+
+Welcome to the documentation of Gerber tooling within PyGerber library. This part of
+documentation will cover most of the features of PyGerber which have to do with
+consuming and changing Gerber files.
+
+In particular, you can learn here how to:
+
+- [quickly render PNGs, JPEGs or SVGs from Gerber files](./20_quick_start/01_single_file.md),
+- [stack many of them on top each other so they align well](./20_quick_start/02_multi_file.md),
+- [consume Gerber Job files](./20_quick_start/03_project_gerber_job.md),
+- [perform analysis of Gerber files](./30_analysis/00_introduction.md),
+- [modify them](./40_modify_optimize/00_introduction.md),
+- [and format them](./60_formatter/00_introduction.md),
+- [collect diagnostics](./70_diagnostics/00_introduction.md),
+- [use language server](./80_language_server/00_introduction.md),
+- [or even extend PyGerber](./90_extend/00_introduction.md).
+
+Generation of Gerber files is covered separately by
+[Gerber code generation](../80_code_generation/00_introduction.md) guide.
+
+Adding rendering backends is covered by documentation of
+[Virtual Machines](../60_rendering_backends/00_introduction.md).
+
+## Design overview
+
+In this section we will cover the general design of Gerber tooling within PyGerber.
+Let's start with basic stages of Gerber processing which have to be performed to create
+PNG image out of Gerber file.
+
+### Parser
+
+First of all, Gerber files have to be loaded into memory and parsed into
+[Abstract Syntax Tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) (AST). This
+is done by a parser, currently PyGerber has only one Gerber parser, based on
+[pyparsing](https://pypi.org/project/pyparsing/) library, available as
+[`pygerber.gerber.parser.pyparsing.Parser`](../reference/pygerber/gerber/parser/pyparsing/parser.md#pygerber.gerber.parser.pyparsing.parser.Parser)
+class. Currently work is being done to provide plugin interface which would allow to use
+parsers implemented as separate packages. Alongside that initiative, a
+[C++ based Gerber parser](https://github.com/PyGerber/pygerber_gerber_parser_cpp) is
+being implemented.
+
+### Compiler
+
+After obtaining AST it is converted to a special assembly like format called Rendering
+Virtual Machine Commands (RVMC). This conversion is handled by a
+[Compiler](../reference/pygerber/gerber/compiler/compiler.md#pygerber.gerber.compiler.compiler.Compiler)
+class. RVMC representation is much simpler, it does not differentiate between
+rectangles, obrounds, circles and other Gerber constructs, but rather represents
+everything as shapes built out of lines and arcs. This simplification allows it to be
+very generic and quite easy to implement.
+
+### Virtual Machine
+
+Once RVMC is obtained, it is passed to one of the classes inheriting from
+[VirtualMachine](../reference/pygerber/vm/vm.md#pygerber.vm.vm.VirtualMachine) class,
+which are responsible for "executing" the RVMC (nomenclature based on analogy to
+executing assembly code). Depending on the VM chosen, there are different configuration
+options available and different type of image will be obtained. Since at the begging we
+assumed we want to walk through the process of rendering PNG image, we would have to
+choose
+[PillowVirtualMachine](../reference/pygerber/vm/pillow/vm.md#pygerber.vm.pillow.vm.PillowVirtualMachine),
+which is capable of rendering raster images with
+[Pillow](https://pypi.org/project/pillow/) library and exporting them in different
+raster formats.
+
+```mermaid
+flowchart TD
+  src@{ shape: st-rect, label: "Gerber source" }
+  parser@{ shape: proc, label: "Parser" }
+  ast@{ shape: st-rect, label: "Abstract Syntax Tree" }
+  compiler@{ shape: proc, label: "Compiler" }
+  rvmc@{ shape: st-rect, label: "RVMC" }
+  vm@{ shape: proc, label: "Virtual Machine" }
+  image@{ shape: st-rect, label: "Image" }
+
+src --> parser
+
+subgraph Parsing
+    parser --> ast
+end
+
+subgraph Translation
+    ast --> compiler
+    compiler --> rvmc
+end
+
+subgraph Rendering
+    rvmc --> vm
+    vm --> image
+end
+
+```
+
+## Additional processing
+
+Although rendering is probably the most useful feature of PyGerber, it is not the only
+one. PyGerber is capable of performing analysis, formatting and modification of existing
+Gerber files. This is done by different classes, which are not part of the rendering
+pipeline. All those tools require AST as input, so you have to first parse Gerber file
+before you can use them. For example here is processing flow for formatting:
+
+```mermaid
+flowchart TD
+  src@{ shape: st-rect, label: "Gerber source" }
+  parser@{ shape: proc, label: "Parser" }
+  ast@{ shape: st-rect, label: "Abstract Syntax Tree" }
+  formatter@{ shape: proc, label: "Formatter" }
+  formatted@{ shape: st-rect, label: "Formatted Gerber source" }
+
+src --> parser
+
+subgraph Parsing
+    parser --> ast
+end
+
+subgraph Formatting
+    ast --> formatter
+end
+
+formatter --> formatted
+```
diff --git a/..._gerber/90_migrate_from_2_4_x_to_3_x_x.md → ..._gerber/10_migrate_from_2_4_x_to_3_x_x.md b/..._gerber/90_migrate_from_2_4_x_to_3_x_x.md → ..._gerber/10_migrate_from_2_4_x_to_3_x_x.md
diff --git a/..._gerber/20_quick_start/00_introduction.md → ..._gerber/20_quick_start/00_introduction.md b/..._gerber/20_quick_start/00_introduction.md → ..._gerber/20_quick_start/00_introduction.md
diff --git a/...0_gerber/20_quick_start/01_single_file.md → ...0_gerber/20_quick_start/01_single_file.md b/...0_gerber/20_quick_start/01_single_file.md → ...0_gerber/20_quick_start/01_single_file.md
@@ -55,18 +55,18 @@ Check out [Custom color maps](./10_custom_color_maps.md) for more details.
 [`set_parser_options()`](../../reference/pygerber/gerber/api/__init__.md#pygerber.gerber.api.GerberFile.set_parser_options)
 allows you to modify advanced parser settings. It is available to allow tweaking
 predefined parser behavior options. If you need more control than provided here, please
-check out [Advanced Guide](../30_advanced_guide/00_introduction.md). `**options` are
+check out [Extending Guide](../90_extend/00_introduction.md). `**options` are
 intentionally not precisely defined here, as they are different for different parser
 implementations, only way to use this method is to already understand what you are
 doing.
 
 [`set_compiler_options()`](../../reference/pygerber/gerber/api/__init__.md#pygerber.gerber.api.GerberFile.set_compiler_options)
 allows you to modify advanced compiler settings. It is available to allow tweaking
 predefined compiler behavior options. If you need more control than provided here,
-please check out [Advanced Guide](../30_advanced_guide/00_introduction.md). `**options`
-are intentionally not precisely defined here, as they are different for different
-compiler implementations, only way to use this method is to already understand what you
-are doing.
+please check out [Extending Guide](../90_extend/00_introduction.md). `**options` are
+intentionally not precisely defined here, as they are different for different compiler
+implementations, only way to use this method is to already understand what you are
+doing.
 
 ## Rendering Gerber file
 

diff --git a/...70_gerber/20_quick_start/02_multi_file.md → ...40_gerber/20_quick_start/02_multi_file.md b/...70_gerber/20_quick_start/02_multi_file.md → ...40_gerber/20_quick_start/02_multi_file.md
diff --git a/...r/20_quick_start/03_project_gerber_job.md → ...r/20_quick_start/03_project_gerber_job.md b/...r/20_quick_start/03_project_gerber_job.md → ...r/20_quick_start/03_project_gerber_job.md
diff --git a/...er/20_quick_start/10_custom_color_maps.md → ...er/20_quick_start/10_custom_color_maps.md b/...er/20_quick_start/10_custom_color_maps.md → ...er/20_quick_start/10_custom_color_maps.md
diff --git a/...r/20_quick_start/gbrjob_bottom_render.png → ...r/20_quick_start/gbrjob_bottom_render.png b/...r/20_quick_start/gbrjob_bottom_render.png → ...r/20_quick_start/gbrjob_bottom_render.png
diff --git a/..._gerber/20_quick_start/render_project.png → ..._gerber/20_quick_start/render_project.png b/..._gerber/20_quick_start/render_project.png → ..._gerber/20_quick_start/render_project.png
diff --git a/..._quick_start/single_file_from_project.png → ..._quick_start/single_file_from_project.png b/..._quick_start/single_file_from_project.png → ..._quick_start/single_file_from_project.png
diff --git a/...ber/20_quick_start/single_file_pillow.png → ...ber/20_quick_start/single_file_pillow.png b/...ber/20_quick_start/single_file_pillow.png → ...ber/20_quick_start/single_file_pillow.png
diff --git a/...er/20_quick_start/single_file_shapely.svg → ...er/20_quick_start/single_file_shapely.svg b/...er/20_quick_start/single_file_shapely.svg → ...er/20_quick_start/single_file_shapely.svg
diff --git a/docs/70_gerber/00_introduction.md → .../40_gerber/30_analysis/00_introduction.md b/docs/70_gerber/00_introduction.md → .../40_gerber/30_analysis/00_introduction.md
diff --git a/...anced_guide/10_stateless_introspection.md → ...30_analysis/10_stateless_introspection.md b/...anced_guide/10_stateless_introspection.md → ...30_analysis/10_stateless_introspection.md
diff --git a/...vanced_guide/20_stateful_introspection.md → .../30_analysis/20_stateful_introspection.md b/...vanced_guide/20_stateful_introspection.md → .../30_analysis/20_stateful_introspection.md
diff --git a/...rber/30_advanced_guide/00_introduction.md → ...ber/40_modify_optimize/00_introduction.md b/...rber/30_advanced_guide/00_introduction.md → ...ber/40_modify_optimize/00_introduction.md
diff --git a/...30_advanced_guide/30_code_optimization.md → ...0_modify_optimize/30_code_optimization.md b/...30_advanced_guide/30_code_optimization.md → ...0_modify_optimize/30_code_optimization.md
diff --git a/...70_gerber/60_formatter/00_introduction.md → ...40_gerber/60_formatter/00_introduction.md b/...70_gerber/60_formatter/00_introduction.md → ...40_gerber/60_formatter/00_introduction.md
diff --git a/...0_gerber/60_formatter/10_configuration.md → ...0_gerber/60_formatter/10_configuration.md b/...0_gerber/60_formatter/10_configuration.md → ...0_gerber/60_formatter/10_configuration.md
diff --git a/...er/60_formatter/20_extending_formatter.md → ...er/60_formatter/20_extending_formatter.md b/...er/60_formatter/20_extending_formatter.md → ...er/60_formatter/20_extending_formatter.md
diff --git a/docs/40_gerber/70_diagnostics/00_introduction.md b/docs/40_gerber/70_diagnostics/00_introduction.md
diff --git a/...ber/80_language_server/00_introduction.md → ...ber/80_language_server/00_introduction.md b/...ber/80_language_server/00_introduction.md → ...ber/80_language_server/00_introduction.md
diff --git a/.../80_language_server/10_vsc_integration.md → .../80_language_server/10_vsc_integration.md b/.../80_language_server/10_vsc_integration.md → .../80_language_server/10_vsc_integration.md
diff --git a/..._gerber/80_language_server/50_features.md → ..._gerber/80_language_server/50_features.md b/..._gerber/80_language_server/50_features.md → ..._gerber/80_language_server/50_features.md
diff --git a/docs/40_gerber/90_extend/00_introduction.md b/docs/40_gerber/90_extend/00_introduction.md
diff --git a/..._advanced_guide/40_extending_ast_nodes.md → ...erber/90_extend/40_extending_ast_nodes.md b/..._advanced_guide/40_extending_ast_nodes.md → ...erber/90_extend/40_extending_ast_nodes.md
diff --git a/.../30_advanced_guide/50_extending_parser.md → ...0_gerber/90_extend/50_extending_parser.md b/.../30_advanced_guide/50_extending_parser.md → ...0_gerber/90_extend/50_extending_parser.md
diff --git a/...0_advanced_guide/60_extending_compiler.md → ...gerber/90_extend/60_extending_compiler.md b/...0_advanced_guide/60_extending_compiler.md → ...gerber/90_extend/60_extending_compiler.md
diff --git a/...ed_guide/70_extending_virtual_machines.md → ...0_extend/70_extending_virtual_machines.md b/...ed_guide/70_extending_virtual_machines.md → ...0_extend/70_extending_virtual_machines.md
diff --git a/docs/60_rendering_backends/00_introduction.md b/docs/60_rendering_backends/00_introduction.md
@@ -0,0 +1 @@
+# Introduction
diff --git a/docs/90_development/00_introduction.md b/docs/90_development/00_introduction.md
@@ -0,0 +1 @@
+# Introduction
diff --git a/docs/90_development/00_setup.md b/docs/90_development/00_setup.md
diff --git a/docs/90_development/10_build.md b/docs/90_development/10_build.md
diff --git a/docs/90_development/10_env_setup.md b/docs/90_development/10_env_setup.md
@@ -0,0 +1,139 @@
+# Environment setup
+
+## Development virtual environment
+
+This project uses `Python` programming language and requires at least python `3.8` in
+development and production environment to function. During development, PyGerber relies
+on [poetr](https://pypi.org/project/poetry/) for dependency management, creation of
+development virtual environment and packaging. To work on PyGerber, you will need to
+install poetry on your system. You can find official poetry installation guidelines
+[here](https://python-poetry.org/docs/#installation). We recommend installing poetry
+globally, unless on linux distribution, in such case distribution native approach should
+be taken, usually with distribution specific package manager.
+
+To install poetry globally you can use `pip` (or any other PyPI compatible package
+manager) in terminal:
+
+```
+pip install poetry
+```
+
+!!! warning
+
+    Make sure you are installing latest version of poetry, or at least `>=1.8` as in the
+    past poetry had minor issues with stability during installation of dependencies.
+
+After installing poetry, if you haven't already, clone the repository with
+
+```
+git clone https://github.com/Argmaster/pygerber
+```
+
+and navigate to project root directory. Once there, use:
+
+```
+poetry shell
+```
+
+to create a development virtual environment and enter a shell with virtual environment
+activated. Virtual environment will be indicated by terminal prompt prefix
+`(pygerber-pyX.Y)`, (where `X.Y` is python version).
+
+Afterwards you can start process of installation of development dependencies. Use:
+
+```
+poetry install --sync --extras=all --with=docs,style
+```
+
+To install all dependencies, including optional ones. Every time you perform a
+`git pull` or change a branch, you should run this command again to make sure you have
+the correct versions of dependencies.
+
+## Git hooks
+
+PyGerber project uses [pre-commit](https://pypi.org/project/pre-commit/) to run git
+hooks to ensure high quality of code committed to the repository. You can skip
+installation of hooks if you want, but you will still have to meet their requirements
+after they are run by CI/CD pipeline. Preferably, you should install them locally and
+keep an eye on their status:
+
+```
+poe install-hooks
+```
+
+!!! warning
+
+    Don't confuse `poe` with `poetry`, `poe` is a command line tool for running simple
+    sequences of commands while `poetry` is a package manager and virtual environment
+    manager.
+
+You can check if your hooks were setup correctly by running:
+
+```
+poe test-style
+```
+
+Here is reference output of `poe test-style`:
+
+> ```
+> Poe => poetry run pre-commit run --all-files -v
+> prettier.................................................................Passed
+> - hook id: prettier
+> - duration: 0.83s
+> check illegal windows names..........................(no files to check)Skipped
+> - hook id: check-illegal-windows-names
+> check for case conflicts.................................................Passed
+> - hook id: check-case-conflict
+> - duration: 0.5s
+> check for merge conflicts................................................Passed
+> - hook id: check-merge-conflict
+> - duration: 0.38s
+> check for case conflicts.................................................Passed
+> - hook id: check-case-conflict
+> - duration: 0.48s
+> trim trailing whitespace.................................................Passed
+> - hook id: trailing-whitespace
+> - duration: 0.23s
+> debug statements (python)................................................Passed
+> - hook id: debug-statements
+> - duration: 0.23s
+> fix end of files.........................................................Passed
+> - hook id: end-of-file-fixer
+> - duration: 0.2s
+> fix utf-8 byte order marker..............................................Passed
+> - hook id: fix-byte-order-marker
+> - duration: 0.22s
+> check for added large files..............................................Passed
+> - hook id: check-added-large-files
+> - duration: 0.52s
+> check toml...............................................................Passed
+> - hook id: check-toml
+> - duration: 0.09s
+> mixed line ending........................................................Passed
+> - hook id: mixed-line-ending
+> - duration: 0.22s
+> trim trailing whitespace.................................................Passed
+> - hook id: trailing-whitespace
+> - duration: 0.22s
+> debug statements (python)................................................Passed
+> - hook id: debug-statements
+> - duration: 0.22s
+> ruff.....................................................................Passed
+> - hook id: ruff
+> - duration: 0.11s
+>
+> All checks passed!
+>
+> ruff-format..............................................................Passed
+> - hook id: ruff-format
+> - duration: 0.11s
+>
+> 284 files left unchanged
+>
+> mypy.....................................................................Passed
+> - hook id: mypy
+> - duration: 15.95s
+>
+> Poe => poetry run mypy --config-file=pyproject.toml src/pygerber/ test/
+> Success: no issues found in 279 source files
+> ```
diff --git a/docs/90_development/20_documentation.md b/docs/90_development/20_documentation.md