Decentralized 3d Neural Web Architecture Documentation

Table of Contents

1. Introduction
2. Architecture Overview
3. Key Components
4. Memory System
5. Neuron and Connection Management
6. Dynamic Parameters
7. Performance Metrics
8. Optimization
9. Adaptation
10. Usage
11. API Reference
12. Algorithm explanations and mathematics
13. Training mechanism

Introduction

Video explanation:

This documentation provides a comprehensive guide to the decentralized neural web architecture implemented in the provided code. The architecture is designed to simulate a neural network with hierarchical memory management, dynamic adaptation, and performance optimization. The goal of this architecture is to present an alternative to modern neural models, which are often complex and resource-intensive, taking inspiration from our brains, neurons are decentralized organized in layers, allowing them to interact with themselves and change theirselves overtime in more than just sates and weights, while also creating a dynamic memory system.

Requirements:

json-c library

for metal macos version metal api

for cuda version cuda

Compilation

To compile the code, run the following command in the root directory of the project:

arch64 MacOS

For metal version

clang -framework Metal -framework Foundation -I/opt/homebrew/Cellar/json-c/0.17/include -L/opt/homebrew/Cellar/json-c/0.17/lib -ljson-c -o neural_web neural_web.m

For cpu but macOS version

clang  -I/opt/homebrew/Cellar/json-c/0.17/include -L/opt/homebrew/Cellar/json-c/0.17/lib -ljson-c -o neural_web neural_webCPU.c

64/86 unix

For cpu 86/64 unix version

clang  -I/opt/homebrew/Cellar/json-c/0.17/include -L/opt/homebrew/Cellar/json-c/0.17/lib -ljson-c -o neural_web neural_web64.c

For cuda version

nvcc -o neural_web neural_web.cu -I/opt/homebrew/Cellar/json-c/0.17/include -L/opt/homebrew/Cellar/json-c/0.17/lib -ljson-c

JsonC library replace with your own imports in the command if you copied it into the project or aren't using homebrew or another version of the lib

1. Note: the perfomance of other cpu versions than metal and cuda might be off as the gpu is far more efficient than cpu
2. Note: the actual code is located in src so you should either unpack files from there into your folder or just compile there

Architecture Overview

The architecture consists of several key components:

• Neurons: The basic units of the neural network, organized into layers, each connected in a 3d like structure.
• Memory System: A hierarchical memory system to store and manage memories with varying importance.
• Dynamic Parameters: Parameters that adapt based on the network's performance and stability.
• Performance Metrics: Metrics to track the performance of the network.
• Optimization: Techniques to optimize the network's parameters for better performance.

Loss figure :

Key Components

Memory System

The memory system is designed to store and manage memories with varying importance. It consists of:

• MemoryEntry: A structure to store individual memories.
• MemoryCluster: A structure to manage a cluster of memories.
• HierarchicalMemory: A structure to manage short-term, medium-term, and long-term memories.
• MemorySystem: The main structure to manage the hierarchical memory system.

MemoryEntry

typedef struct {
  float vector[MEMORY_VECTOR_SIZE];
  float importance;
  unsigned int timestamp;
} MemoryEntry;

MemoryCluster

typedef struct MemoryCluster {
  MemoryEntry *entries;
  float importance_threshold;
  unsigned int size;
  unsigned int capacity;
} MemoryCluster;

HierarchicalMemory

typedef struct HierarchicalMemory {
  MemoryCluster short_term;
  MemoryCluster medium_term;
  MemoryCluster long_term;
  float consolidation_threshold;
  float abstraction_threshold;
  unsigned int total_capacity;
} HierarchicalMemory;

MemorySystem

typedef struct MemorySystem {
  HierarchicalMemory hierarchy;
  unsigned int head;
  unsigned int size;
  unsigned int capacity;
  MemoryEntry *entries;
} MemorySystem;

Neuron and Connection Management

Neurons are the basic units of the neural network, and connections define how neurons are interconnected.

Neuron

typedef struct {
  float state;
  float output;
  unsigned int num_connections;
  unsigned int layer_id;
} Neuron;

Dynamic Parameters

Dynamic parameters adapt based on the network's performance and stability.

DynamicParameters

typedef struct {
  float input_noise_scale;
  float weight_noise_scale;
  float base_adaptation_rate;
  float current_adaptation_rate;
  float learning_momentum;
  float stability_threshold;
  float noise_tolerance;
  float recovery_rate;
  float plasticity;
  float homeostatic_factor;
} DynamicParameters;

Performance Metrics

Performance metrics track the performance of the network.

PerformanceMetrics

typedef struct {
  double execution_time;
  float average_output;
  float error_rate;
  int batch_size;
  float learning_rate;
} PerformanceMetrics;

Optimization

Optimization techniques are used to improve the network's performance.

OptimizationState

typedef struct {
  int optimal_batch_size;
  float optimal_learning_rate;
  double best_execution_time;
  float best_performance_score;
} OptimizationState;

Adaptation

Adaptation metrics track the network's adaptation to changes.

AdaptationMetrics

typedef struct {
  float input_noise_resistance;
  float weight_noise_resistance;
  float adaptation_speed;
  float baseline_performance;
  float noisy_performance;
} AdaptationMetrics;

Usage

Initialization

To initialize the neural network and memory system, use the following functions:

MemorySystem *memorySystem = createMemorySystem(MEMORY_BUFFER_SIZE);
Neuron neurons[MAX_NEURONS];
uint connections[MAX_NEURONS * MAX_CONNECTIONS] = {0};
float weights[MAX_NEURONS * MAX_CONNECTIONS] = {0};
float input_tensor[INPUT_SIZE] = {0};
initializeNeurons(neurons, connections, weights, input_tensor);

API Reference

Memory System Functions

• MemorySystem *createMemorySystem(unsigned int capacity): Creates a new memory system.
• void freeMemorySystem(MemorySystem *system): Frees the memory system.
• void addMemory(MemorySystem *system, Neuron *neurons, float *input_tensor, unsigned int timestamp): Adds a memory to the system.
• void saveMemorySystem(MemorySystem *system, const char *filename): Saves the memory system to a file.
• MemorySystem *loadMemorySystem(const char *filename): Loads the memory system from a file.
• void saveHierarchicalMemory(MemorySystem *system, const char *filename): Saves the hierarchical memory to a file.
• void loadHierarchicalMemory(MemorySystem *system, const char *filename): Loads the hierarchical memory from a file.

Neuron and Connection Management Functions

• void initializeNeurons(Neuron *neurons, uint *connections, float *weights, float *input_tensor): Initializes the neurons.
• void updateNeuronStates(Neuron *neurons, float *recurrent_weights): Updates the neuron states.
• void updateWeights(float *weights, Neuron *neurons, uint *connections, float learning_rate): Updates the weights.

Dynamic Parameters Functions

• DynamicParameters initDynamicParameters(): Initializes the dynamic parameters.
• void updateDynamicParameters(DynamicParameters *params, float performance_delta, float stability_measure, float error_rate): Updates the dynamic parameters.

Performance Metrics Functions

• float calculatePerformanceScore(PerformanceMetrics metrics): Calculates the performance score.
• float computeAverageOutput(Neuron *neurons): Computes the average output.
• float computeErrorRate(Neuron *neurons, float *previous_outputs): Computes the error rate.

Optimization Functions

• void optimizeParameters(OptimizationState *opt_state, PerformanceMetrics *history, int history_size): Optimizes the parameters.

Adaptation Functions

• void adaptNetworkDynamic(Neuron *neurons, float *weights, DynamicParameters *params, float performance_delta, float *input_tensor): Adapts the network with dynamic parameters.

Algorithm and Mathematics

This document outlines the core algorithms and mathematical principles behind a decentralized neural network architecture. These mechanisms enable hierarchical memory management, dynamic adaptation, and optimization.

---

Key Components

1. Neuron State Update

This algorithm calculates the new state of each neuron based on its current state, inputs, and neighboring influences.

How It Works:

• State Update Formula:
  new_state = (current_state * decay_factor) + (recurrent_inputs * recurrent_weight) + (neighbor_influences * neighbor_weight)
• Activation Function: The output is scaled using a hyperbolic tangent (tanh) function:
  output = tanh(state * scale)

---

2. Weight Update

Adjusts the connections (weights) between neurons using a modified Hebbian learning rule.

How It Works:

• Weight Update Formula:
  delta_w = learning_rate * (pre_activation * post_activation - weight * decay_factor)
• Normalization: Weights are clipped to prevent unbounded growth:
  new_weight = max(-1.0, min(1.0, weight + delta_w))

---

3. Memory Management

Maintains memories in a hierarchical system, adjusting their importance dynamically.

How It Works:

• Memory Importance:
  importance = sum(abs(vector[i]) for i in range(vector_size)) / vector_size
• Decay Over Time:
  new_importance = importance * decay_factor
• Strengthening Important Memories:
  new_importance = importance * strengthen_factor

---

4. Dynamic Parameter Adaptation

Automatically tunes parameters based on performance and stability.

How It Works:

• Adaptation Rate:
  new_adaptation_rate = (momentum * adaptation_rate) + ((1 - momentum) * target_rate)
• Plasticity Adjustment:
  new_plasticity = plasticity * stability_factor
• Noise Tolerance:
  new_noise_tolerance = max(0.1, noise_tolerance * (1 - error_rate))

---

5. Performance Optimization

Optimizes learning rate and batch size based on network performance.

How It Works:

• Performance Score:
  performance_score = (time_score * 0.4) + (output_score * 0.4) + (error_penalty * 0.2)
• Batch Size Adjustment:
  new_batch_size = (current_batch_size % max_batch_size) + 1
• Learning Rate Update:
  new_learning_rate = current_learning_rate * (rand() / RAND_MAX) * 0.5 + 0.75

---

6. Pattern Matching

Finds similar memories using cosine similarity.

How It Works:

• Cosine Similarity:
  similarity = (sum(vector1[i] * vector2[i] for i in range(vector_size))) / (sqrt(sum(vector1[i]**2 for i in range(vector_size))) * sqrt(sum(vector2[i]**2 for i in range(vector_size))))

---

7. Performance Analysis

Provides insights by calculating statistics like averages and variances.

How It Works:

• Average:
  average = sum(values) / len(values)
• Variance:
  variance = sum((x - average)**2 for x in values) / len(values)

---

8. Backpropagation

Refines the network by minimizing errors via gradient descent.

How It Works:

• Loss Function (Mean Squared Error):
  loss = sum((y[i] - y_hat[i])**2 for i in range(n)) / n
• Gradient Descent:
  new_weight = weight - (learning_rate * loss_gradient)

---

9. SIMD Operations

Performs vector operations efficiently.

How It Works:

• Addition:
  result[i] = vector1[i] + vector2[i]
• Multiplication:
  result[i] = vector1[i] * vector2[i]

---

10. Memory Vector Computation

Creates a memory vector by combining various data sources:
memory_vector = [neuron_states, neuron_outputs, input_tensor]

---

11. Network Stability Measurement

Assesses stability by comparing current and previous neuron states.

How It Works:

• Stability Measure:
  stability_measure = 1 - (sum(abs(current_state[i] - previous_state[i]) for i in range(n)) / n)

---

12. Memory Merging

Combines similar memories to reduce redundancy.

How It Works:

• Weighted Merge:
  merged_vector = ((importance1 * vector1) + (importance2 * vector2)) / (importance1 + importance2)
• Merged Importance:
  merged_importance = max(importance1, importance2) * 1.1

---

Neural Web Training Mechanism

This README file explains the training mechanism of the neural web implemented in the provided main() function. The training process involves several key components, including Metal device setup, memory system management, neural network initialization, and the main simulation loop for training. Below, we delve into the design reasons behind each component.

Table of Contents

1. Metal Device Setup
2. Memory System Management
3. Neural Network Initialization
4. Main Simulation Loop
5. Performance Tracking and Optimization
6. Dynamic Parameter Adaptation
7. Overview

Metal Device Setup

Code

id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];

Design Reason

Metal is used for its high performance and low-level access to the GPU, which is crucial for the efficient computation required in neural network training. Setting up the Metal device and command queue ensures that the GPU resources are ready for compute tasks, note cuda also can work in this place.

Memory System Management

Code

MemorySystem *memorySystem = loadMemorySystem("memory_system.dat");
if (memorySystem != NULL) {
    loadHierarchicalMemory(memorySystem, "hierarchical_memory.dat");
    // Print memory system statistics and samples
} else {
    memorySystem = createMemorySystem(MEMORY_BUFFER_SIZE);
}

Design Reason

The memory system is designed to store and manage hierarchical memory structures, which are essential for retaining learned patterns and experiences. This hierarchical approach allows the system to prioritize and manage memories based on their importance and recency, mimicking human memory processes. Loading an existing memory system ensures continuity and prevents the loss of previously learned information, along with the working memory system allowing the model to have a dynamic realtime memory.

Meta controller, cognitive system, goal planning

MetaController *metaController = initializeMetaController(network_regions);
IntrinsicMotivation *motivation = initializeMotivationSystem();
GoalSystem *goalSystem = initializeGoalSystem(10);
GlobalContextManager *contextManager = initializeGlobalContextManager(MAX_NEURONS);

Design Reason

The meta controller, cognitive system, and goal planning components are initialized using the provided functions. These components are responsible for orchestrating the overall behavior of the neural web. Allowing the model to have a dynamic realtime memory, understand in a way.

Neural Network Initialization

Code

Neuron neurons[MAX_NEURONS];
uint connections[MAX_NEURONS * MAX_CONNECTIONS] = {0};
float weights[MAX_NEURONS * MAX_CONNECTIONS] = {0};
float input_tensor[INPUT_SIZE] = {0};

if (memorySystem->size > 0) {
    // Initialize neurons from memory
} else {
    initializeNeurons(neurons, connections, weights, input_tensor);
}

Design Reason

Initializing the neural network involves setting up neurons, connections, and weights. If the memory system contains existing data, neurons are initialized from the last memory state to leverage previously learned information. This approach ensures that the network can build upon past experiences, enhancing learning efficiency and effectiveness.

Main Simulation Loop

The main simulation loop is the core of the training process. It iterates over a predefined number of steps, performing various operations to train the neural network. This loop ensures that the network is continuously learning and adapting based on new inputs and feedback. Key operations include input generation, memory maintenance, forward and backward passes, memory updates, state history updates, performance metrics updates, dynamic parameter adaptation, and pattern matching.

Performance Tracking and Optimization

Performance tracking and optimization are crucial for ensuring that the neural network operates efficiently. By periodically optimizing parameters such as learning rate and batch size, the system can adapt to changing conditions and improve overall performance. This dynamic optimization helps in achieving better convergence and accuracy.

Dynamic Parameter Adaptation

Code

updateDynamicParameters(&params, performance_delta, stability, performance_history[step].error_rate);
adaptNetworkDynamic(updatedNeurons, weights, &params, performance_delta, input_tensor);

Design Reason

Dynamic parameter adaptation allows the neural network to adjust its parameters in real-time based on performance metrics and network stability. This adaptability ensures that the network can respond to varying inputs and conditions, improving its robustness and flexibility. Parameters such as adaptation rate, input noise scale, and plasticity are adjusted to optimize learning and performance.

Overview

1. Input Data:

   • Code:

     const char *text_input = "Apple, banana, cherry, date, and elderberry are fruits.";

   • Description: The input data is a text string that serves as the initial input to the neural network. This aligns with the "Input Data" node in the diagram.

2. Neurons:

   • Code:

     Neuron neurons[MAX_NEURONS];

   • Description: An array of Neuron structures is initialized to represent the neurons in the network. This corresponds to the "Neurons" node.

3. Neuron State Update:

   • Code:

     processNeurons(neurons, max_neurons, weights, connections, max_connections, 1.5f);

   • Description: The processNeurons function updates the state of each neuron based on weights and connections. This relates to the "Neuron State Update" node.

4. Activation Function (tanh):

   • Code:

     id<MTLFunction> function = [library newFunctionWithName:@"update_neurons"];

   • Description: The activation function is defined in the shader code and applied during the forward pass. This corresponds to the "Activation Function (tanh)" node.

5. Weight Update (Hebbian Learning):

   • Code:

     id<MTLComputePipelineState> weightPipelineState = [device newComputePipelineStateWithFunction:weightFunction error:&error];

   • Description: Weights are updated using a Hebbian learning approach, implemented in the shader pipeline. This aligns with the "Weight Update (Hebbian Learning)" node.

6. Memory Management:

   • Code:

     MemorySystem *memorySystem = loadMemorySystem("memory_system.dat");

   • Description: The memory system is loaded or created to manage memories, corresponding to the "Memory Management" node.

7. Generate Memory Vector:

   • Code:

     addMemory(memorySystem, working_memory, updatedNeurons, input_tensor, lastTimestamp + step + 1, feature_projection_matrix);

   • Description: New memories are generated and added to the memory system, aligning with the "Generate Memory Vector" node.

8. Hierarchical Memory:

   • Code:

     loadHierarchicalMemory(memorySystem, "hierarchical_memory.dat");

   • Description: The memory system is hierarchical, with short-term, medium-term, and long-term memories. This corresponds to the "Hierarchical Memory" node.

9. Memory Retrieval:

   • Code:

     MemoryEntry *relevantMemory = retrieveMemory(memorySystem);

   • Description: Relevant memories are retrieved from the memory system, aligning with the "Memory Retrieval" node.

10. Memory Merging:

    • Code:

      mergeSimilarMemories(memorySystem);

    • Description: Similar memories are merged to optimize storage, corresponding to the "Memory Merging" node.

11. Predictive Coding:

    • Code:

      initPredictiveCodingParams(max_neurons);

    • Description: Predictive coding parameters are initialized and used to generate predictive inputs, aligning with the "Predictive Coding" node.

12. Memory Importance Update:

    • Code:

      decayMemorySystem(memorySystem);

    • Description: The importance of memories is updated through decay and consolidation, corresponding to the "Memory Importance Update" node.

13. Working Memory:

    • Code:

      WorkingMemorySystem *working_memory = createWorkingMemorySystem(200);

    • Description: A working memory system is created to assist with temporary storage and active processing, aligning with the "Working Memory" node.

14. Global Context Manager:

    • Code:

      GlobalContextManager *contextManager = initializeGlobalContextManager(MAX_NEURONS);

    • Description: The global context manager provides context to neurons and influences memory importance, corresponding to the "Global Context Manager" node.

15. Motivation System:

    • Code:

      IntrinsicMotivation *motivation = initializeMotivationSystem();

    • Description: The motivation system drives learning and action, prioritizing memory and attention, aligning with the "Motivation System" node.

16. Dynamic Adaptation:

    • Code:

      DynamicParameters params = initDynamicParameters();

    • Description: Dynamic parameters are used to adapt the network's learning rate and stability, corresponding to the "Dynamic Adaptation" node.

17. Performance Optimization:

    • Code:

      optimizeParameters(&opt_state, performance_history, step + 1);

    • Description: Parameters are optimized to enhance performance, aligning with the "Performance Optimization" node.

18. Network Stability Measurement:

    • Code:

      float stability = measureNetworkStability(updatedNeurons, previous_states);

    • Description: Network stability is measured to monitor and improve adaptation, corresponding to the "Network Stability Measurement" node.

19. Text Output / Action:

    • Code:

      char outputText[4096];
      transformOutputsToText(previous_outputs, MAX_NEURONS, outputText, sizeof(outputText));

    • Description: The network's output is transformed into text or action, corresponding to the "Text Output / Action" node.

Structuring int main()

1. Initialization:

   • Initialize Metal device and command queue.
   • Load or create the memory system.
   • Load shader source and create pipeline states.

2. Neural Network Setup:

   • Initialize neurons, connections, and weights.
   • Create Metal buffers for neurons, connections, weights, and other parameters.

3. Main Simulation Loop:

   • Generate task prompts and manage memory system.
   • Perform forward and backward passes using Metal command encoders.
   • Compute loss, update weights, and optimize parameters periodically.

4. Cleanup and Saving State:

   • Save the final states of the neural network, memory system, and system parameters.
   • Free allocated memory.

Example

Example of the training can be seen in the MacOS/neural_web.m file in int main or if you are not familiar with metal 86\64/neural_web64CPU.c

Needed information

Note if you modify max_neurons in the example you have to also modify the input_size to be at max greater than the number of max_neurons by 1 or just lesser than the number of max_neurons or it will have an out of bounds error

The model uses reverse pathways and generally doesn't only do patterns good because it also reverses its outputs and finds more meaning in it and additional pathways to achieve what it is supposed to similar to how humans do, or as how I think humans do.

You can also see an_idea_of_a_neural_web.pdf file for more information about the reasoning behind the model's structure, mathematics, and the like. neuron update shader and the code must be in the same directory.

To modify number of neurons change MAX_NEURONS

Only for unix type systems

Cuda version might not work the best