Raspbian basics

Note: Raspberry Pi Foundation has now renamed their OS from Raspbian to Raspberry Pi OS, or "RaspiOS" for short. I won't be renaming this page just yet, as Raspios Armhf basics just doesn't have the same ring.

About partitions

Before we dive into the semi-automated process for writing your image to the boot media, let's quickly consider the partitions in Raspberry Pi OS images.

initial layout

Raspbian images are full disk images, with an MBR (Master Boot Record, aka 'msdos'), and a small boot partition followed by a main second partition truncated to the size required. Here are some example sizes.

Partition	Bullseye Lite Apr'22	Buster Desktop Jul'19	Notes
Initial gap	4.2 MB	4.2 MB	sectors skipped before 8192
FAT 'boot'	256 MB	256 MB	LBA, no bootable flag
Main Linux	1.6 GB	3.3 GB

expand main partition

Naturally you are likely to want to use the space on your microSD for programs and data, so you are likely to want to expand that partition to fill the free space otherwise left on the disk. And in fact, many distributions, including Raspberry Pi OS, will perform that expansion for you automatically.

Alternatively use a third partition

That expansion will make the main system partition fill ALL the free space, but sometimes you might not want that. For instance if you have a 64 GB microSD card, or you are booting on a 240 GB solid state drive (SDD), you might only need 20 or 40 GB for the main partition. If you want to limit the size to which the OS volume expands, you should create another (third) partition in the drive, as a limiting point for how far the OS expands.

Reasons to not limit
- Flexibility: to fix the sizes with partitions, you have to know how much will be taken by each - you cannot move the space very easily afterwards
Reasons to limit
- if you know that you will overwrite the OS by rebuilding it one day, it might be possible to keep a third data partition intact
  - there is a risk, as you will need to recreate the third partition manually with exactly the same details in the partition table
  - because of the risks you would be wise to back up the data on that partition before a rebuild

For more detail on different options and pros and cons of creating an additional partition to limit expansion, see MuGammaPi Disks # additional partition

Prepare install media

So let's get your boot medium (usually a MicroSD card) ready with an operating system image that we:

Download
Write
Resize
Configure

download

Raspbian.org recommends the Raspberry Pi Foundation images as the most widely supported images for Raspbian.

Editions (see details):

Filename	Edition	Use case
raspios_full_armhf_latest	Full	User desktop with apps
raspios_armhf_latest	Desktop	Kiosk / basic desktop
raspios_lite_armhf_latest	Lite	Server / headless
raspios_oldstable_lite_armhf_latest	Lite	the previous debian version

To understand the extra packages between the Lite and the basic Desktop versions, compare the lists of installed BaselinePackages in https://github.com/artmg/MuGammaPi/

If you are running a model 3 or 4 or other that you know supports 64-bit, then you can change to use the arm64 build instead of armhf (32-bit).

wget --trust-server-names -P ~/Downloads https://downloads.raspberrypi.org/raspios_lite_armhf_latest

Use sha256sum filename if you wish to validate the checksum.

write image

First set up environment variables to control the process and answer certain questions - see the section further down for a key and explanations

#### DESCRIPTION = 

NEWHOSTNAME=
 ROOT_USER=
 ROOT_INIP=
# leave leading spaces to minimise this being captured by command history

MEDIA_LABEL=
MEDIA_DEVICE=
IMAGE_FILENAME=

WIFI_SSID=
# leave SSID blank if you want Ethernet only
 WIFI_PSK=
WIFI_COUNTRY=

ENABLE_SSH=true
#HEADLESS=true

# https://github.com/artmg/MuGammaPi/wiki/Raspbian-basics#prepare-install-media

Then use these commands to write the Distro to Flash EXCEPT the last section to Power it Down.

expand secondary partition

This now uses automated expansion of partition two using fdisk.

If you need to limit the second partition as above, then you must create a New partition number 3 BEFORE deleting partition number 2, but see notes and warnings above.

For more information about expanding partitions, and how other tools and distros do it, see https://github.com/artmg/lubuild/blob/master/help/configure/write-Distro-to-flash.md#raspberry-pi

example variables required

# credit - http://elinux.org/RPi_Resize_Flash_Partitions
# the article also explains why as well as various options for how

MEDIA_DEVICE=sdX9

code to expand

#!/bin/bash

# to manipulate partitions they should not be mounted 
udisksctl unmount --block-device /dev/${MEDIA_DEVICE:0:3}1
udisksctl unmount --block-device /dev/${MEDIA_DEVICE:0:3}2

# Print the current partition table
echo "Current partition table:"
sudo fdisk -l "/dev/${MEDIA_DEVICE:0:3}"

# Read the starting sector of the second partition
START_SECTOR=$(sudo fdisk -l "/dev/${MEDIA_DEVICE:0:3}" | grep "^/dev/${MEDIA_DEVICE:0:3}[0-9]" | sed -n '2p' | awk '{print $2}')

if [[ -n "$START_SECTOR" ]]; then
	echo "Starting sector of second partition: $START_SECTOR"

	# Use fdisk in non-interactive mode to delete and recreate the partition
	# extra n and newline on the end to NOT remove signature
	echo -e "d\n2\nn\np\n2\n$START_SECTOR\n\nw" | sudo fdisk "/dev/${MEDIA_DEVICE:0:3}"
	
	# Print the new partition table
	echo "New partition table:"
	sudo fdisk -l "/dev/${MEDIA_DEVICE:0:3}"
fi

You should expect to see a red warning for

Partition #2 contains a ext4 signature and you should not be overly concerned on occasions when you see
Re-reading the partition table failed.: Device or resource busy

# do a consistency check and resize the inner volume 
sudo e2fsck -fy /dev/${MEDIA_DEVICE:0:3}2
sudo resize2fs /dev/${MEDIA_DEVICE:0:3}2

Old manual expansion

which you should now be able to skip

# sudo parted /dev/$MEDIA_DEVICE resize 
# help

# WIP #################
# note that neither command line option (parted not fdisk delete and recreate) 
# has a simple option to automatically expand to fill all available space
# 
echo "================================================
  for now use fdisk
   - Print to see current start
   - Delete partition (2)
   - create New and accepting most defaults 
   - First Sector: type value from table printed
   - do NOT remove the partition 2 signature
   - write the new table and exit
================================================"

sudo fdisk /dev/${MEDIA_DEVICE:0:3}

Set initial configuration

TTD: Check consistency of variable names with other sections

Note: this automation in the rest of this page is similar to that used in the pages below, so perhaps they could be merged at some point?

example variables required

#### variables for hostname, user and wireless

NEWHOSTNAME=MyRaspbianName
 ROOT_USER=myUser
 ROOT_INIP=Sek-yr-P455word
# leave leading spaces to minimise this being captured by command history
# note that the command-line syntax used below _might_ 
# limit your use of 'special characters' in the password

MEDIA_DEVICE=

# omit this or leave SSID blank if you want Ethernet only
WIFI_SSID=your-station-name
 WIFI_PSK=Your_wifi_password
# country should be capitals (no quotes)
WIFI_COUNTRY=GB

#### other variables

ENABLE_SSH=true  (or false or omitted)
#HEADLESS=true   (or false or omitted)

code to configure

#### set hostname before initial boot

OLDHOSTNAME=raspberrypi

# remount at a consistent point
# this assumes a first FAT boot partitions
# followed by a second main OS partition (root)
MOUNT_BOOT=/media/$USER/osboot
MOUNT_ROOT=/media/$USER/osroot

sudo umount /dev/${MEDIA_DEVICE:0:3}1
sudo umount /dev/${MEDIA_DEVICE:0:3}2
sudo partprobe

sudo mkdir -p $MOUNT_BOOT
sudo mkdir -p $MOUNT_ROOT
sudo chmod 777 $MOUNT_BOOT
sudo chmod 777 $MOUNT_ROOT
sudo mount /dev/${MEDIA_DEVICE:0:3}1 $MOUNT_BOOT
sudo mount /dev/${MEDIA_DEVICE:0:3}2 $MOUNT_ROOT

sudo sed -i "s/$OLDHOSTNAME/$NEWHOSTNAME/g" $MOUNT_ROOT/etc/hosts
sudo sed -i "s/$OLDHOSTNAME/$NEWHOSTNAME/g" $MOUNT_ROOT/etc/hostname



#### Set user and initial password

# This is new for Bullseye from Apr 2022 - see 
# https://www.raspberrypi.com/news/raspberry-pi-bullseye-update-april-2022/

if [[ $ROOT_USER ]] ; then
  echo ${ROOT_USER}:`echo ${ROOT_INIP} | openssl passwd -6 -stdin` | sudo tee $MOUNT_BOOT/userconf.txt
fi

# leading spaces to stop this being logged in the terminal history
#  echo ${ROOT_USER}:`echo ${ROOT_INIP} | openssl passwd -6 -stdin` | sudo tee /boot/userconf.txt

# If you have issues with this process then plug the Pi 
# into an HDMI screen and keyboard, and when it auto-logs on 
# use the command   sudo rename-user 
# This will restart and prompt you for the username & password. 
# You may also create a new userconf file before running that
# command and it will use those automatically when it restarts. 



# Raspberry Pi OS since bookworm uses 
# NetworkManager persistent profiles 

if [[ $WIFI_SSID ]] ; then
	sudo tee -a $MOUNT_ROOT/etc/NetworkManager/system-connections/Wifi-SSID-initial-config.nmconnection <<EOF!
[connection]
id=Wifi-SSID-initial-config
uuid=96f19af8-ec79-4752-89df-8694b37453c1
type=wifi
interface-name=wlan0

[wifi]
mode=infrastructure
ssid=$WIFI_SSID

[wifi-security]
auth-alg=open
key-mgmt=wpa-psk
psk=$WIFI_PSK

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]
EOF!

	
	sudo chmod 600 $MOUNT_ROOT/etc/NetworkManager/system-connections/Wifi-SSID-initial-config.nmconnection 

	sudo tee -a $MOUNT_ROOT/boot/firmware/firstrun.sh <<EOF!
#!/bin/bash
# Stripped back version of the Bookworm firstrun.sh generated by the Raspberry Pi Imager
# also calls raspi-config to set Wifi country, which in turn calls iw on Bookworm
# credit https://forums.raspberrypi.com/viewtopic.php?p=2148466#p2148466
set +e
rfkill unblock wifi
for filename in /var/lib/systemd/rfkill/*:wlan ; do
    echo 0 > \$filename
done
raspi-config nonint do_wifi_country ${WIFI_COUNTRY}
# edited to remove a bug in rpi-imager that is a hangover from Bullseye
# rm -f /boot/firstrun.sh
rm -f /boot/firmware/firstrun.sh
exit 0
EOF!
	sudo chmod +X $MOUNT_ROOT/boot/firmware/firstrun.sh

	# end of wifi if
fi


#### enable ssh

# SSH is disabled by default (since Nov 2016), 
# so to enable it we create an empty file called `ssh` in the boot folder. 
# See [https://www.raspberrypi.org/blog/a-security-update-for-raspbian-pixel/]
# with SSH it is even more important to have secure, unique username and password

if [[ "$ENABLE_SSH" = true ]] ; then
  sudo touch $MOUNT_BOOT/ssh
fi


#### headless

# If intend to run the device as headless, 
# in other words with no screen attached, 
# then you should not need as much 'GPU memory'.
# Some other directly-connected devices, 
# like the camera, may use graphics memory, 
# (see the [Camera](https://github.com/artmg/MuGammaPi/wiki/Camera) article) 
# but if this is mainly a server then you can 
# free up memory for the CPU processor to use:

# However, if you do need to plug the pi 
# into an HDMI screen and keyboard to 
# access the text console, 
# that will still work fine. 

if [[ "$HEADLESS" = true ]] ; then
# RasbPiOS bookworm moved from the /boot/ parent
sudo tee -a $MOUNT_BOOT/firmware/config.txt <<EOF!

# allocate less graphics memory for headless system
gpu_mem=16
EOF!
fi

# NB: because you are limiting the graphical memory, 
# certain features may be disabled, and you may see 
# warnings or errors in the system log. For instance 
# `vchi` may give errors at startup if the
# VideoCore (Broadcom's on SoC GPU) Host Interface 
# cannot access that memory. For more detail see 
# https://www.raspberrypi.org/documentation/configuration/config-txt/memory.md



### cleanup

sync
sudo umount /dev/${MEDIA_DEVICE}1
sudo umount /dev/${MEDIA_DEVICE}2
sudo rmdir $MOUNT_BOOT
sudo rmdir $MOUNT_ROOT

echo
echo Image is READY!
echo Remove ${MEDIA_LABEL} from device ${MEDIA_DEVICE} and boot it as ${NEWHOSTNAME}

first boot

obtain MAC Address

# If you want to check the MAC address of the new hardware, once booted remotely:
ping -c 1 $NEWHOSTNAME
arp -a

log on

# or use new hostname 
ssh pi@$NEWHOSTNAME
# default password is raspberry
# but as per [https://www.raspberrypi.org/blog/a-security-update-for-raspbian-pixel/]
# we recommend you CHANGE your password from the default on first boot using
# passwd


# NB: NO need to set a Root Password
# Raspbian (like Debian) leaves 
# the root password disabled by default. 
# This is to block actors who try to use
# the root account as an attack vector

SSH certificates

If you intend to continue managing this device via SSH, best practice is to use certificates instead of passwords. This also makes it quicker and easier to safely access your devices.

See the SSH certificates instructions You can define your configuration parameters in a simple block such as below. From Bullseye onwards, as the user and password are set when the image is written, you can use this as your first logon.

You must execute this from your client, not the server!

### install certificates
  
# from client
REMOTE_HOST=myserver
REMOTE_DOMAIN=.lan.local
REMOTE_USER=pi
ADMIN_GROUP=mypiservices

# https://github.com/artmg/lubuild/blob/master/help/configure/Secure-SHell-SSH.md#generate-and-install-ssh-certificate

Updates

# elevate to get password if required
sudo echo

# look for latest contents of repos
sudo apt update
# upgrade all, as you have no config yet to break
sudo apt -y dist-upgrade

# if the upgrades were extensive, including kernel updates
# you may want to reboot then
# sudo apt -y autoremove

Basic settings

You can pre-set your options in environment variables. Set false or simply omit the ones you don't want.

SYSLOG_SERVER=192.168.1.10
SERVICE_DISCOVERY=true
DISABLE_WIRELESS=false

sudo echo

#### disable wireless

# If you have a Pi 3 or 4, but wish to only use Ethernet
# then you can simply prevent the wifi stack from loading at boot time. 
# credit - https://raspberrypi.stackexchange.com/a/108553
# help - https://github.com/raspberrypi/firmware/blob/master/boot/overlays/README

if [[ ${DISABLE_WIRELESS,,} = true ]] ; then
 # was directly in Boot folder before RasbPiOS bookworm 
 sudo tee -a /boot/firmware/config.txt <<EOF!

# disable the onboard wireless network
# rfkill now default in Pi OS, but this shouldn't hurt
dtoverlay=disable-wifi
EOF!
fi


#### Network discovery

# mDNS (a.k.a. Bonjour, ZeroConf or Avahi) provides basic local network discovery
# help - https://github.com/artmg/lubuild/blob/master/help/configure/Networked-Services.md
# NB: recent Raspberry Pi OS (even Lite) may already include mDNS packages

if [[ ${SERVICE_DISCOVERY,,} = true ]] ; then
  sudo apt update
  sudo apt install -y libnss-mdns
  # credit - https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=18207

  # debug
  # http://askubuntu.com/questions/460371/trouble-using-ssh-and-avahi
fi


#### Remote logging

# If the server is defined in environment variables, 
# this will install the rsyslog client (now comes with Raspbian Lite)
# and point the logs towards the nominated server

if [[ $SYSLOG_SERVER ]] ; then
  # connect SysLog client to nominated server
  # now uses new advanced action syntax instead of old basic 
  # as recommended since v7
  #*.* @x.x.x.x
  sudo apt install -y rsyslog
  sudo tee -a /etc/rsyslog.d/00-forward-all.conf <<EOF!
*.* action(type="omfwd" 
        target="$SYSLOG_SERVER" port="514"
        protocol="tcp" action.resumeRetryCount="100"
        queue.type="linkedList" queue.size="10000")
EOF!

  sudo service rsyslog restart
fi

disk layout

The two partitions will be presented to the running OS as:

size	device	mountpoint
60M	/dev/mmcblk0p1	/boot
the rest	/dev/root	/

Before updates the Boot partition is only about 34% utilised

Automount

If you have additional disks (drives, pendrives, etc) permanently plugged in then you will probably want to automatically mount them at boot time.


# simple steps - credit [http://www.raspberrypi-spy.co.uk/2014/05/how-to-mount-a-usb-flash-disk-on-the-raspberry-pi/]
# more detail - credit [http://www.htpcguides.com/properly-mount-usb-storage-raspberry-pi/]
ls -l /dev/disk/by-uuid/

MEDIA_UUID= # copy from above
MEDIA_FSTYPE=vfat # or ext3 or ntfs-3g
MEDIA_DEVICE=sdxY
# NB: Linux FHS definition is ambiguous whether persistently mounted drives in fstab belong in /mnt or /media 
# for now we follow the 'majority' of interpretations, which seem to use /mnt 
MOUNT_POINT=/mnt/MyMountName
# leave blank if Ext filesystem
MOUNT_OPTS=
# vfat - allow all other users RO access
# MOUNT_OPTS=,uid=$USER,gid=$USER,umask=0002
# vfat - allow all other users RW access 
# MOUNT_OPTS=,uid=$USER,gid=$USER,umask=0000
MOUNT_USER_GRP=$USER:$USER

## ONLY format if required
#sudo mkfs.$MEDIA_FSTYPE /dev/$MEDIA_DEVICE
## see also partition naming code from section Partitions / Rename in
## https://github.com/artmg/lubuild/blob/master/help/configure/disks.md#rename

# Now set up the AutoMount in fstab
sudo mkdir -p $MOUNT_POINT
sudo chown -R $MOUNT_USER_GRP $MOUNT_POINT

cat <<EOF | sudo tee --append /etc/fstab
UUID=$MEDIA_UUID $MOUNT_POINT $MEDIA_FSTYPE auto,nofail,noatime,users,rw$MOUNT_OPTS 0 0
EOF

read-only filesystem

If you want to see techniques to reduce wear on your fragile microSD cards, consider making your Pi filesystem Read Only. See read-only-raspberry for a variety of techniques.

non-interactive first boot

All of the above are interactive commands to modify the configuration on first boot. If you'd like a way to chain onto script-driven configuration, consder tools like https://github.com/nmcclain/raspberian-firstboot

Alternatively if you want the same settings for a bunch of devices you can use https://github.com/gitbls/sdm to put the modifications into the image you donwnloaded from RPi foundation BEFORE you burn it onto your boot storage medium.

other config

set hostname AFTER boot if not done previously

as an alternative to sudo raspi-config

echo "Enter new hostname: "
read NEWHOSTNAME

OLDHOSTNAME=$(hostname)
export NEWHOSTNAME
export OLDHOSTNAME
sudo sed -i "s/$OLDHOSTNAME/$NEWHOSTNAME/g" /etc/hosts
sudo sed -i "s/$OLDHOSTNAME/$NEWHOSTNAME/g" /etc/hostname
# need to reboot before it is active
# sudo reboot
#
# credit - https://pricklytech.wordpress.com/2013/04/24/ubuntu-change-hostname-permanently-using-the-command-line/

# troubleshooting - https://wiki.debian.org/HowTo/ChangeHostname

configure Wifi from command line

The new mechanism, which applies since Raspberry Pi OS Bookworm (v12) is described in NetworkManager connection profiles in keyfile format but which is also covered in Debian network configuration. Thanks to pi forums post for this news.

(see extract from code to configure above /etc/NetworkManager/system-connections/Wifi-SSID-initial-config.nmconnection)

old version

sudo editor /etc/wpa_supplicant/wpa_supplicant.conf

# add the following at the end of the file (including quotes but without hashes):
#network={
#    ssid="you-station-name"
#    psk="Your_wifi_password"
#}

# Sometimes it will autodetect the file change, or you can restart services, 
# ... or you can simply reboot :)
# help - [https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md] 

# OLD

sudo cp $MOUNT_ROOT/etc/wpa_supplicant/wpa_supplicant.conf $MOUNT_BOOT

Disconnecting ethernet makes wireless drop

Note that you may percieve issues if you plug in to set up wireless, then disconnect the wired connection – it may seem that you get no response to pings on the wireless either!

This is simply because the routing table is built to prefer the ethernet connection (check with route -n, credit https://raspberrypi.stackexchange.com/a/50665 ), and might not adjust immediately when you remove that ethernet, so the ping-return can't get back.

When you restart the device with no ethernet, the wifi should come up just fine.

Inside raspi-config

There are lots of configuration options that you can tweak using raspi-config. Most people tell you to use the interactive interface (which works in Terminal) but you can script any of the options. For clues see the Command Strings in [https://github.com/raspberrypi-ui/rc_gui/blob/master/src/rc_gui.c]

If you want to see what they do under the covers, for whatever reason, check out raspi-config itself, which is just a shell script :) [https://raw.githubusercontent.com/RPi-Distro/raspi-config/master/raspi-config]

Tests / Troubleshooting

Detect OS and version

# show OS release variables
cat /etc/*-release

# show kernel details
uname -a

# detect if using raspbian
. /etc/*-release
if [[ "${ID}" == "raspbian" ]] ; then echo This is using Raspbian ; fi ;

Detect Raspberry hardware revision

cat /proc/cpuinfo | grep 'Revision' | awk '{print $3}' | sed 's/^1000//'
# now see table in 
# http://elinux.org/RPi_HardwareHistory#Which_Pi_have_I_got.3F

# to see serial number and other info simply
cat /proc/cpuinfo

Detect MAC addresses

# show the MAC address for all available network adapters
ifconfig | grep HWaddr

# alternative when wifi is disabled
ifconfig -a | grep 'flags\|ether'

Startup issues

If you are not even beginning the boot sequence then you should check the initial diagnostics, starting with the two little LEDs on the end by GPIO pin 1 (back of SD slot)

See [http://elinux.org/R-Pi_Troubleshooting]

If you are getting stuck during the boot sequence, you can note where it goes wrong using an HDMI display. If you get no display you can use the hmdi_safe=1 option in config.txt, or simply jumper GPIO pins 5 & 6.

In either case you are likely to find the easiest ways to diagnose are:

if network comes up - via ssh
else by powering off and inserting SD card into another PC

Rescue mode

If you don't even get far enough to see any dmesg or logs on the SD card, then you will have to use Rescue Mode

insert the SD card to another PC
browse to the OS / Boot partition
edit the file cmdline.txt
go right to the very end, after rootwait
add space then the digit 1 (or single)
save, reinsert card and reboot
credit [http://raspberrypi.stackexchange.com/a/61515]

example

Supposing my Pi stops during boot on Starting NTP server. The most likely candidate is the NEXT service after that so using rescue mode I check the services for runlevel 2:

ls /etc/rc2.d -1
# shows me that samba was next after ntp
systemctl disable smbd

unfortunately order is not used as Debian does concurrent startup. To disable concurrent booting add CONCURRENCY=none to /etc/default/rcS credit [http://serverfault.com/a/245342]

Alternatively use sudo raspi-config nonint do_boot_wait 1 which turns off "Wait for Network at boot" by creating /etc/systemd/system/dhcpcd.service.d/wait.conf with -w option

systemd.debug-shell

credit [https://www.raspberrypi.org/forums/viewtopic.php?t=104314]

By adding the following to your cmdline.txt systemd.debug-shell you will be able to have a command prompt available on Ctrl-Alt-F9 for debugging systemd.

alternatively revert to SysV init

try using init=/lib/sysvinit/init in /boot/cmdline.txt instead of init=/bin/systemd

Raspbian basics

About partitions

initial layout

expand main partition

Alternatively use a third partition

Prepare install media

download

write image

expand secondary partition

example variables required

code to expand

Old manual expansion

Set initial configuration

example variables required

code to configure

first boot

obtain MAC Address

log on

SSH certificates

Updates

Basic settings

disk layout

Automount

read-only filesystem

non-interactive first boot

other config

set hostname AFTER boot if not done previously

configure Wifi from command line

old version

Disconnecting ethernet makes wireless drop

Inside raspi-config

Tests / Troubleshooting

Detect OS and version

Detect Raspberry hardware revision

Detect MAC addresses

Startup issues

Rescue mode

example

systemd.debug-shell

alternatively revert to SysV init

more

Clone this wiki locally