tqma6-yocto-mirror/docs/02-mirror-creation/step-by-step-mirror-creation.md

# Step-by-Step Yocto Mirror Creation with ci-meta-tq

**Document ID:** PROC-MIRROR-001-REV3
**Date:** 2026-03-01
**Verified:** 2026-03-01 (Debian 13 + Ubuntu 22.04 Docker, incl. air-gapped deployment)
**System:** Ubuntu 22.04 LTS (recommended), Debian 12/13 (works with warning)
**Target:** TQMa6UL Yocto Scarthgap mirror (UT build)

---

## Hardware Target

| Field | Value |
|-------|-------|
| **SoM** | TQMa6UL (sticker: `TQMA6U-AB`) |
| **SoC** | NXP i.MX6 UltraLite (Cortex-A7, single core, 32-bit) |
| **Carrier** | MBa6ULx |
| **Machine** | `tqma6ul-multi-mba6ulx` |
| **BSP Config** | `mainline` (NOT `imx`!) |

> **Warning:** Do NOT use `tqma6qdl-multi-mba6x` — that is for the Quad/Dual
> variant on a different carrier board. The UltraLite is ONLY in `mainline`.

---

## Prerequisites

### 1.1 Host System

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 4 cores | 8+ cores |
| RAM | 8 GB | 16+ GB |
| Disk | 100 GB | 200+ GB |
| OS | Ubuntu 22.04 LTS | Ubuntu 22.04 LTS |

> **Note:** Debian 12/13 also works. BitBake will show a warning
> ("Host distribution has not been validated") but builds succeed.

### 1.2 Required Packages (Ubuntu 22.04)

```bash
sudo apt update
sudo apt install -y \
    git python3 jq bash grep gawk wget diffstat \
    chrpath cpio texinfo gcc g++ make file tar \
    bzip2 gzip xz-utils zstd lz4 patch perl \
    python3-pexpect socat unzip rsync bc \
    libsdl1.2-dev xterm
```

### 1.3 Locale

```bash
sudo locale-gen en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
```

---

## Step 1: Clone ci-meta-tq

```bash
export UT_BASE=~/UT
mkdir -p ${UT_BASE} && cd ${UT_BASE}

# Clone at the Scarthgap release tag
git clone --branch scarthgap.TQ.ARM.BSP.0006 \
    --recurse-submodules \
    https://github.com/tq-systems/ci-meta-tq.git

cd ci-meta-tq
git submodule sync && git submodule update --init
```

**Verify:**
```bash
./ci/ls-configs --file
# Expected: imx, ls, mainline, ti

./ci/ls-machines --file --config=mainline | grep tqma6ul
# Expected: tqma6ul-multi-mba6ulx
```

---

## Step 2: Configure Downloads Directory

```bash
# Create shared download directory (outside build tree)
sudo mkdir -p /srv/yocto/downloads /srv/yocto/sstate-cache
sudo chown -R "$USER:$USER" /srv/yocto

# Create site.conf (global Yocto config)
mkdir -p ~/.yocto
cat > ~/.yocto/site.conf << 'EOF'
# UT Project - Download Directories
DL_DIR ?= "/srv/yocto/downloads"
SSTATE_DIR ?= "/srv/yocto/sstate-cache"
EOF
```

> **Important:** Do NOT add premirror/own-mirrors config yet!
> That is only for the air-gapped target machine (Step 6).

---

## Step 3: Initialize Build Environment

```bash
cd ${UT_BASE}/ci-meta-tq

# CRITICAL: Accept the NXP/FSL EULA (required for firmware-imx)
export ACCEPT_FSL_EULA=1

# Set machine
export MACHINE=tqma6ul-multi-mba6ulx

# Initialize (creates build_ut/ directory)
. ./setup-environment build_ut mainline
```

> **Note:** You will see a Qt6 GPL Exception EULA prompt.
> The `ACCEPT_FSL_EULA=1` environment variable auto-accepts it.
> Without this variable, the setup script blocks waiting for input.

**CRITICAL:** Some recipes (firmware-imx) check local.conf, not just the
environment variable. Add this to your local.conf:

```bash
echo 'ACCEPT_FSL_EULA = "1"' >> conf/local.conf
```

**Verify:**
```bash
bitbake -e | grep -E "^MACHINE=|^DISTRO=|^DL_DIR="
# Expected:
#   MACHINE="tqma6ul-multi-mba6ulx"
#   DISTRO="spaetzle"
#   DL_DIR="/srv/yocto/downloads"
```

---

## Step 4: Download All Sources (Fetch Only)

For mirror creation, you do NOT need a full build. Fetch-only is faster:

```bash
# Fetch all sources for the target image (no compilation)
bitbake tq-image-small-debug --runall=fetch
```

**Expected:** ~510 fetch tasks, takes 30-60 minutes depending on bandwidth.

**Verify:**
```bash
du -sh /srv/yocto/downloads/
# Expected: ~5 GB

ls /srv/yocto/downloads/*.done | wc -l
# Expected: ~138 completed downloads
```

> **Alternative:** For a full build (also populates sstate-cache):
> ```bash
> bitbake tq-image-small-debug
> ```
> This takes 4-8 hours but also verifies the build actually succeeds.

---

## Step 5: Generate License Table

The license table is generated from BitBake metadata, NOT from build output:

```bash
cd ${UT_BASE}/ci-meta-tq
export ACCEPT_FSL_EULA=1
export MACHINE=tqma6ul-multi-mba6ulx
. ./setup-environment build_ut mainline

# Generate the build list (all 264 recipes)
bitbake tq-image-small-debug -g
# Creates: pn-buildlist, task-depends.dot

# Extract license + SRC_URI for every recipe
echo "Package,Version,License,SRC_URI,Recipe_File" > license-table.csv

while IFS= read -r recipe; do
    output=$(bitbake -e "$recipe" 2>/dev/null)
    pv=$(echo "$output" | grep '^PV=' | head -1 | sed 's/^PV="//' | sed 's/"$//')
    license=$(echo "$output" | grep '^LICENSE=' | head -1 | sed 's/^LICENSE="//' | sed 's/"$//')
    src_uri=$(echo "$output" | grep '^SRC_URI=' | head -1 | sed 's/^SRC_URI="//' | sed 's/"$//')
    recipe_file=$(echo "$output" | grep '^FILE=' | head -1 | sed 's/^FILE="//' | sed 's/"$//')
    first_url=$(echo "$src_uri" | grep -oP '(https?|ftp|git)://[^\s]+' | head -1 | sed 's/;.*//')
    recipe_file=$(echo "$recipe_file" | sed "s|.*/sources/||")
    license=$(echo "$license" | tr ',' ';')
    echo "\"$recipe\",\"$pv\",\"$license\",\"$first_url\",\"$recipe_file\"" >> license-table.csv
done < pn-buildlist

echo "Done: $(wc -l license-table.csv) entries"
```

> **Note:** This takes ~30 minutes (one `bitbake -e` call per recipe).
> All Yocto variables (`${GNU_MIRROR}`, `${PV}`, etc.) are fully resolved.

---

## Step 6: Package the Mirror

```bash
cd ${UT_BASE}

TIMESTAMP=$(date +%Y%m%d)
ARCHIVE="UT-tqma6ul-yocto-scarthgap-${TIMESTAMP}"

mkdir -p ${ARCHIVE}/{sources,downloads,configs,docs}

# Copy repository (with submodules)
cp -r ci-meta-tq ${ARCHIVE}/sources/

# Copy all downloaded sources
cp -r /srv/yocto/downloads/* ${ARCHIVE}/downloads/

# Copy configs
cp ~/.yocto/site.conf ${ARCHIVE}/configs/
cp ci-meta-tq/build_ut/conf/local.conf ${ARCHIVE}/configs/ 2>/dev/null
cp ci-meta-tq/build_ut/conf/bblayers.conf ${ARCHIVE}/configs/ 2>/dev/null

# Copy license table
cp ci-meta-tq/build_ut/license-table.csv ${ARCHIVE}/docs/

# Create archive
tar czf ${ARCHIVE}.tar.gz ${ARCHIVE}/
sha256sum ${ARCHIVE}.tar.gz > ${ARCHIVE}.tar.gz.sha256

echo "Archive: ${ARCHIVE}.tar.gz ($(du -h ${ARCHIVE}.tar.gz | cut -f1))"
```

---

## Step 7: Air-Gapped Deployment

On the target machine (no internet):

```bash
# Extract
tar xzf UT-tqma6ul-yocto-scarthgap-*.tar.gz
cd UT-tqma6ul-yocto-scarthgap-*/

# Configure premirror (point to local downloads)
mkdir -p ~/.yocto
cat > ~/.yocto/site.conf << EOF
DL_DIR = "$(pwd)/downloads"
SSTATE_DIR = "/tmp/sstate-cache"
SOURCE_MIRROR_URL = "file://$(pwd)/downloads/"
INHERIT += "own-mirrors"
BB_NO_NETWORK = "1"
EOF

# Build
cd sources/ci-meta-tq
export ACCEPT_FSL_EULA=1

# CRITICAL: Also add to local.conf
echo 'ACCEPT_FSL_EULA = "1"' >> conf/local.conf
export MACHINE=tqma6ul-multi-mba6ulx
. ./setup-environment build_ut mainline

bitbake tq-image-small-debug
```

---

## Verified Results (2026-03-01)

| Step | Status | Notes |
|------|--------|-------|
| Clone ci-meta-tq | ✅ | Tag scarthgap.TQ.ARM.BSP.0006 |
| setup-environment | ✅ | Requires ACCEPT_FSL_EULA=1 |
| Fetch all sources | ✅ | 510/510 tasks, ~5 GB |
| License extraction | ✅ | 264/264 recipes, 224 URLs |
| ci/ls-machines | ✅ | tqma6ul-multi-mba6ulx found |
| Debian 13 host | ✅ | Warning but functional |
| Ubuntu 22.04 host | ✅ | Tested in Docker, all steps pass |
| Air-gapped fetch (BB_NO_NETWORK=1) | ✅ | 510/510 from local mirror |
| Full build | ✅ | Air-gap verified 2026-03-02 (4098/4098 tasks) |

---

**Author:** Siggi ⚙️
**Revision:** 3.1 (Ubuntu 22.04 + air-gap verified)