tqma6-yocto-mirror/docs/02-mirror-creation/step-by-step-mirror-creation.md

# Step-by-Step Yocto Mirror Creation

**Document ID:** PROC-MIRROR-001  
**Date:** 2026-03-01  
**System:** Ubuntu 22.04 LTS  
**Environment:** Outside corporate network (Internet access required)  
**Target:** TQMa6UL Yocto Scarthgap mirror for offline deployment

---

## Prerequisites

### 1.1 Host System Requirements

| Resource | Minimum | Recommended | Notes |
|----------|---------|-------------|-------|
| CPU | 4 cores | 8+ cores | Parallel builds |
| RAM | 8 GB | 16+ GB | BitBake is memory-intensive |
| Disk Space | 100 GB | 200+ GB | Sources + build artifacts |
| OS | Ubuntu 22.04 | Ubuntu 22.04 LTS | Verified platform |
| Internet | Required | Broadband | For initial download |

### 1.2 Required Packages

Install on Ubuntu 22.04:

```bash
sudo apt update
sudo apt install -y \
    gawk wget git diffstat unzip texinfo gcc build-essential \
    chrpath socat cpio python3 python3-pip python3-pexpect \
    xz-utils debianutils iputils-ping python3-git \
    python3-jinja2 libegl1-mesa libsdl1.2-dev xterm \
    locales lz4 zstd rpcsvc-proto \
    file bsdmainutils curl

# Ensure Python 3 is default
sudo apt install -y python3-distutils
```

### 1.3 Locale Configuration

```bash
sudo locale-gen en_US.UTF-8
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
```

---

## Step 1: Create Working Directory

```bash
# Create base directory for mirror creation
mkdir -p ~/tqma6-yocto-mirror
export MIRROR_BASE=~/tqma6-yocto-mirror
cd $MIRROR_BASE

# Create subdirectories
mkdir -p sources layers downloads sstate-cache
```

---

## Step 2: Clone Yocto Poky (Reference Distribution)

```bash
cd $MIRROR_BASE/sources

# Clone Poky (Yocto reference) - Scarthgap branch
git clone -b scarthgap --depth 1 \
    https://git.yoctoproject.org/git/poky.git \
    poky-scarthgap

cd poky-scarthgap
# Full clone for complete history (optional but recommended for approval)
git fetch --unshallow

cd $MIRROR_BASE/sources
```

**Verification:**
```bash
cd poky-scarthgap
git log --oneline -1
# Should show: scarthgap branch, recent commit
git describe --tags
# Should show: yocto-5.0.x or similar
```

---

## Step 3: Clone OpenEmbedded Meta-Layer

```bash
cd $MIRROR_BASE/sources

# Clone meta-openembedded (contains essential recipes)
git clone -b scarthgap --depth 1 \
    https://git.openembedded.org/meta-openembedded \
    meta-openembedded

cd meta-openembedded
git fetch --unshallow

cd $MIRROR_BASE/sources
```

**Included sub-layers:**
- `meta-oe` - Base additional recipes
- `meta-python` - Python packages
- `meta-networking` - Network tools
- `meta-filesystems` - Filesystem tools
- `meta-perl` - Perl modules

---

## Step 4: Clone TQ BSP Layer

```bash
cd $MIRROR_BASE/sources

# Clone TQ BSP layer - specific version for TQMa6UL
git clone -b scarthgap.TQ.ARM.BSP.0006 --depth 1 \
    https://github.com/tq-systems/meta-tq.git \
    meta-tq

cd meta-tq
git fetch --unshallow

# Verify the branch
git branch -a | grep scarthgap

cd $MIRROR_BASE/sources
```

**Critical:** Ensure you use the exact tag `scarthgap.TQ.ARM.BSP.0006` for reproducibility.

---

## Step 5: Initialize Build Environment

```bash
cd $MIRROR_BASE/sources/poky-scarthgap

# Source the build environment
source oe-init-build-env ../build

# This creates the build directory and sets up environment
# You should now be in: ~/tqma6-yocto-mirror/build
```

---

## Step 6: Configure Build for TQMa6UL

### 6.1 Edit `conf/local.conf`

```bash
# Edit the local.conf file
vim conf/local.conf
```

**Add/modify these settings:**

```conf
# Machine selection for TQMa6UL on MBa6x
MACHINE ??= "tqma6ulx-mba6ulx"

# Parallelism (adjust to your CPU)
BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j8"

# Download directory (for mirroring)
DL_DIR ?= "${MIRROR_BASE}/downloads"

# Shared state cache
SSTATE_DIR ?= "${MIRROR_BASE}/sstate-cache"

# Keep source code for license compliance
RM_WORK_EXCLUDE += "*"
INHERIT += "archiver"
ARCHIVER_MODE[src] = "original"
ARCHIVER_MODE[diff] = "1"

# Generate license information
COPY_LIC_MANIFEST = "1"
COPY_LIC_DIRS = "1"

# Create source mirror
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL ?= "file://${MIRROR_BASE}/mirror/"
BB_GENERATE_MIRROR_TARBALLS = "1"
```

### 6.2 Add Layers to `conf/bblayers.conf`

```bash
vim conf/bblayers.conf
```

**Content:**
```conf
BBLAYERS ?= " \
  ${MIRROR_BASE}/sources/poky-scarthgap/meta \
  ${MIRROR_BASE}/sources/poky-scarthgap/meta-poky \
  ${MIRROR_BASE}/sources/poky-scarthgap/meta-yocto-bsp \
  ${MIRROR_BASE}/sources/meta-openembedded/meta-oe \
  ${MIRROR_BASE}/sources/meta-openembedded/meta-python \
  ${MIRROR_BASE}/sources/meta-openembedded/meta-networking \
  ${MIRROR_BASE}/sources/meta-openembedded/meta-filesystems \
  ${MIRROR_BASE}/sources/meta-tq/meta-tq \
  "
```

---

## Step 7: First Build (Downloads All Sources)

```bash
# Still in build directory
# This will download all required source packages

bitbake core-image-minimal
```

**Expected duration:** 2-6 hours (first build, depending on hardware)

**What happens:**
1. BitBake parses all recipes
2. Downloads source code for every package
3. Builds toolchain, kernel, rootfs
4. Creates deployment images

---

## Step 8: Create Source Mirror

### 8.1 Generate Mirror Tarballs

```bash
# Create mirror directory
mkdir -p ${MIRROR_BASE}/mirror

# Copy downloaded sources
cp -r ${MIRROR_BASE}/downloads/* ${MIRROR_BASE}/mirror/ 2>/dev/null || true

# Create source archive with archiver
bitbake core-image-minimal -c do_populate_lic
```

### 8.2 Collect License Information

```bash
# Generate license manifest
bitbake core-image-minimal -c do_populate_lic

# Copy license files
mkdir -p ${MIRROR_BASE}/licenses
cp -r ${MIRROR_BASE}/build/tmp/deploy/licenses/* ${MIRROR_BASE}/licenses/

# Create license CSV (see separate script)
```

---

## Step 9: Package the Mirror

### 9.1 Create Archive Structure

```bash
cd ${MIRROR_BASE}

# Create organized structure for approval
mkdir -p mirror-package/{sources,licenses,build-instructions}

# Copy source layers (git repositories)
cp -r sources/* mirror-package/sources/

# Copy license information
cp -r licenses/* mirror-package/licenses/ 2>/dev/null || true

# Copy downloads (source tarballs)
mkdir -p mirror-package/downloads
cp -r downloads/* mirror-package/downloads/ 2>/dev/null || true

# Document the configuration
cp build/conf/local.conf mirror-package/build-instructions/
cp build/conf/bblayers.conf mirror-package/build-instructions/
```

### 9.2 Create Archive

```bash
cd ${MIRROR_BASE}

# Create timestamped archive
export TIMESTAMP=$(date +%Y%m%d-%H%M%S)
export ARCHIVE_NAME="tqma6-yocto-mirror-scarthgap-${TIMESTAMP}.tar.gz"

tar czf ${ARCHIVE_NAME} \
    --exclude='.git/objects/pack/*' \
    mirror-package/

# Calculate checksum
sha256sum ${ARCHIVE_NAME} > ${ARCHIVE_NAME}.sha256

echo "Archive created: ${ARCHIVE_NAME}"
echo "Size: $(du -h ${ARCHIVE_NAME} | cut -f1)"
```

---

## Step 10: Generate License Table

**See:** [License Table Generation Script](../03-license-analysis/generate-license-table.sh)

The license table must include:
- Package name
- Version
- License (SPDX identifier)
- License file location
- Source location
- Military use approval (Yes/No/Conditional)

---

## Verification Checklist

Before delivering mirror:

- [ ] All git repositories cloned (poky, meta-openembedded, meta-tq)
- [ ] Complete source downloads in `downloads/` directory
- [ ] Build successful: `core-image-minimal` completed
- [ ] License information generated in `licenses/`
- [ ] Archive created with timestamp
- [ ] SHA256 checksum calculated
- [ ] License compliance table generated

---

## Troubleshooting

### Problem: Fetch failures during build

**Solution:** Check internet connectivity, proxy settings. Some corporate networks block git:// protocol:
```bash
git config --global url."https://".insteadOf git://
```

### Problem: Disk space exhausted

**Solution:** Ensure at least 100GB free. Clean build if needed:
```bash
bitbake -c cleanall core-image-minimal
```

### Problem: Permission denied

**Solution:** Ensure proper user permissions, avoid building as root.

---

## Next Steps

1. **Transfer archive** to corporate network (USB, secure transfer)
2. **Submit for software approval** with license table
3. **Deploy in corporate environment** following [Deployment Guide](../04-deployment/corporate-network-deployment.md)

---

**Document Version:** 1.0  
**Author:** Siggi ⚙️  
**Last Updated:** 2026-03-01