Developer AI Rack

[flashpixx]

Hello,

I would like to get some help and/or discussion on my mini rack idea. I'm a software developer, so I'm hoping to get a lot of information about hardware and how to build my rack.

The goal of my mini-rack is to create a complete development environment for AI and/or web development for up to 5 developers who then use a laptop to connect to the cluster of the rack.
In general, I've been working with Kubernetes for years, so I generally want to run everything as containers on the system. A short list of tools I currently use:

• http://gitlab.com with Kubernetes CI/CD Executors to run any container for building.
• https://jupyter.org/ for Python development of AI and data analysis
• https://mlflow.org/ for training and executing models
• https://airflow.apache.org/ for the execution of arbitrary workflows
• https://www.dask.org/ for numerical calculations
• https://spark.apache.org/ for distributed computing

Building on this, there are some “infrastructure tools” such as:

• Kubernetes dashboard for managing all containers
• Postgres as database (required by some tools)
• Minio for S3 persistence storage for arbitrary data (required by MLFlow)
• DNS & DHCP and an LDAP server (with user logins) should be running

My first idea was Raspberry Pi's, but I think systems with 32 GB RAM or more would be better, the question would be what would be suitable?
Jetson Nano may be a nice addition for the AI calculations, so I would rather build a hybrid system.
I would like to use an external NAS, if possible with S3 protocol for any kind of user data (git repos, AI models, data), so that I have a centrally usable storage layer.
A wifi access point and possibly a router for LAN or mobile internet would be good.

In general, I would like to do everything with OpenTofu / Terraform for provisioning the system, e.g. Cisco hardware can do this so that I can quickly set up the system again depending on use, and I would also like to build my own images for the computing modules so that I can also quickly reinstall them.

The important goal for me is to optimize the costs for computing and space in the case. I don't need a high-performance cluster, but there has to be enough computing power for simple things for several users, but at the same time the system should also be easy to transport, I was thinking of something like this, for example https://gatorco.com/shop-by-category/racks/shock-racks/

For me, it would just be a case of plugging it into the socket, opening the laptop and connecting to the network in the case.

I would be very happy to have a discussion and would welcome any suggestions and questions.

[MarcAndreWyss]

In general, I've been working with Kubernetes for years, so I generally want to run everything as containers on the system.

This will not answer your questions, but have you every thought about using RedHat OpenShift Dev Spaces (https://developers.redhat.com/products/openshift-dev-spaces)? I know that this is a commercial software, but they build it based on the open source Eclipse Che (https://eclipse.dev/che). Unfortunately Eclipse Che does not run on ARM as far as I know. I just mention it, because I would be happy to use it at home as well.

Maybe also of interest:

• https://tekton.dev
• https://argo-cd.readthedocs.io

[flashpixx]

Hi,
thanks a lot for your answer, but I like this kind of input. I know OpenShift, but not the Dev Spaces, so in general I would like to build it with free open source licenses. Argo-CD is on my todo list, but Tekton and Che could be a nice idea. Thanks for that.

At the moment I try to evaluate the basic hardware first, but this is for me the difficult part. In general I would like to build a first PoC for my HomeLab (2 developers). But it would be nice if the idea work to scale it.

[MarcAndreWyss]

I somehow understand your initial focus on the hardware, but in my opinion you have a kind of chicken / egg problem. For example if you want to use Eclipse Che, you cannot use ARM (as far as I know). Therefore focusing on ARM SBCs (or other ARM systems) would not allow you to use Eclipse Che later. Another example is Pytorch for AI. If you want to have hardware acceleration here you need a CUDA capable GPU (https://www.geeksforgeeks.org/pytorch-system-requirements/). As I understand it this would not work with Google Coral AI or just with some conversions and probably backdraws.

I think there are many other such examples. Therefore your hardware selection may later restrict your software use. Just be aware of that.

[flashpixx]

I'm not strictly fixed to ARM CPUs, especially für GPU I would like to figure out Nividia Jetson Nano and PyTorch is supported.

I would like to start, so my starting point is currently nothing I would like to build 3 nodes only for developing and e.g. a little bit CI/CD, the AI training etc should be append later if the initial system works and I figure out, if the main idea is running smoothly.
Based on future work I'm thinking into the direction of hybride cluster, because I need some nodes to run e.g. Gitlab, and more the tooling stuff and other nodes e.g. for calculation / computing.

THe current scope is, to build an initial-cluster with the provisioning stack for setting OS, configuration of all network components etc. My dream is to run a Terraform script and it flashes all e.g. Pi's with OS, configure the switch and wifi access point and install Kubernetes on the Pi's and deploy the basic Helm chart with all dev tools.

[sylvaintremblay666]

Hello!

Pleased to meet you, you just got my attention! :)

The current scope is, to build an initial-cluster with the provisioning stack for setting OS, configuration of all network components etc. My dream is to run a Terraform script and it flashes all e.g. Pi's with OS, configure the switch and wifi access point and install Kubernetes on the Pi's and deploy the basic Helm chart with all dev tools.

What you're describing here is kinda the same thing as what my associate and I are currently working on!

Lets first talk about the stack. This is obviously just the main idea, things may (and will surely) change, but that's the starting point:

• RedHat OKD (OpenShift community edition) in HA configuration for k8s
  • 3 control planes
  • 3 worker / storage nodes (2 dedicated NVMe drives per node)
• Ceph for persistent storage
• OpnSense managing all the required services (firewall, dhcp, dns, load balancer, tftp, http, pxe)
• Red Hat Developer Hub as IDP (Internal Developer Portal)
• Red Hat OpenShift Dev Spaces for Development environment (web-based vscode integrated with the whole stack)
• Argo
• Gitea for local git, or github integration (using github sso if desired)
• Any other software depending on the use case, like jupyter for AI stuff...

OKD brings its operators, applications, helm integration, prometheus monitoring, alerting, automated upgrades and a lot more, everything out of the box, enterprise grade, for free.

The most interesting point : the automation. We're currently working on the core of a new infrastructure deployment tool. This won't be yet another terraform variation, we're seeing things differently. The name of the tool is Harmony.

At the heart of the tool : rust. A simple DSL, type safety, ease of use, focused on DX. I won't go into more details here, I'll leave that to my associate who designed and implemented all the base architecture of the project.

The foundations are already working well.

• To serve all the cluster's basic services : OpnSense
• You define the topology of your cluster (control planes, workers, bootstrap node, firewall) and give the MACs and desired IPs of the hosts
• Start the TUI -> run the scores (more on the product name / analogy later on)
• We currently have scores for all the required cluster services (dns, dhcp, haproxy, tftp, http, pxe, ...)
• At the end, you have an OpnSense running all the required cluster services, all configured and ready to start a User Provisioned Installation (UPI) of Red Hat OKD (OpenShift community edition) on bare metal in PXE mode (also works on VMs)

This is very basic and opinionated for now, but its only the beginning :-) The project will be open source.

For the Mini-Rack part of the story, I have everything in hand except my 3 worker nodes, but they are in the mail. I should soon be able to complete the hardware part of the build. Lots of design / 3d printing in this thing :-) I haven't opened a build showcase yet but its on my todos.

Hardware (from top to bottom) :

• DeskPi T1 10 inch / 8U rack
• 120mm top fan
• 2x HUNSN micro firewalls 4x 2.5GB each (a little too wide but I've been imaginative)
• Yuanley 13 ports POE 1GB switch
  • 8xPOE (for the vPro KVM consoles)
  • 2xUpLink Eth (one for each firewall)
  • 1xUpLink SFP (to connect to the SFP+ switch)
• RPi4 with DeskPi HDMI/USBC extension (gives access to the KVM consoles of the hosts)
  • The RPi4 could easily be replaced by a Jetson, that would be nice! :)
• Yuanley 8 ports SFP+ switch (10GB network between the hosts)
• Lenovo Tinys:
  • 3x m920q (i5 8600T, 16GB, nvme root disk, 10GB PCIe NIC) for the control planes
  • 3x p330 (i7, 8700T, 32GB for now, 2xNVMe storage disks, msata root disk, 10GB PCIe NIC) for workers (still missing, should be delivered in the next days)

3D printed stuff (my designs) :

• the top plate with 120mm fan mount and grill
• the firewalls holding system (plate screwed from the back of the rack, recessed front plate)
• the firewalls front plate top extension to display our name
• the 0.5U RPi4 holder + wire passthrough
• yuanley sfp+ switch ears
• the Lenovo Tiny support system (a little higher than 4U so I removed the bottom plate and extended by below)

If you feel like being our first beta user, that would be nice! You can come chat with us on our discord : https://discord.gg/jnCAtMYa. Its mainly in french (because we're from Quebec, and the server is super small) but no problem at all speaking in english.

But nomatter discord, lets also continue the discussion here :)

What do you think of all that?

Sylvain