mirror/sysmngr
1
0
Fork 0
mirror of https://dev.iopsys.eu/system/sysmngr.git synced 2025-12-10 08:14:38 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
sysmngr/docs/guide/FirmwareUpgrade.md
Sukru Senli fbd5fead54 add dmcli as well
2025-11-26 22:51:40 +01:00

1725 lines
49 KiB
Markdown
Raw Blame History

# Firmware Upgrade System - Complete Guide
## Table of Contents
1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Configuration Parameters](#configuration-parameters)
4. [Upgrade Methods](#upgrade-methods)
5. [Configuration Preservation](#configuration-preservation)
6. [Operator Configuration Guide](#operator-configuration-guide)
7. [Troubleshooting](#troubleshooting)
8. [API Reference](#api-reference)
---
## Overview
The iopsys firmware upgrade system provides a robust, dual-bank firmware management solution with flexible configuration preservation options. The system supports multiple upgrade methods and protocols while maintaining backward compatibility.
### Key Features
- **Dual-bank architecture** - Safe firmware upgrades with automatic fallback
- **Multiple upgrade protocols** - USP/TR-369 (via GUI or controller), TR-069/CWMP, DHCP onboarding, CLI
- **Selective configuration preservation** - Separate control for operator and user settings
- **Async operations** - Non-blocking upgrades with progress tracking
- **Scheduled activations** - Time-window based firmware activation
- **Auto-rollback** - Automatic fallback on boot failure
### System Components
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Upgrade Interfaces │
├──────────┬──────────────────────────┬──────────────────┬────────────────────┤
│ TR-069 │ USP/TR-369 │ DHCP │ fwbank CLI │
│ (CWMP) │ │ Onboarding │ │
│ │ │ │ │
│ icwmp │ ┌─────────┬─────────┐ │ udhcpc hooks │ /etc/sysmngr/ │
│ Agent │ │ Sulu │ dmcli │ │ │ fwbank │
│ │ │ GUI │ CLI │ │ │ │
│ │ └────┬────┴────┬────┘ │ │ │
│ │ │ │ │ │ │
│ │ └────┬────┘ │ │ │
│ │ │ │ │ │
│ │ ┌─────▼──────┐ │ │ │
│ │ │ OB-USPa │ │ │ │
│ │ │ Agent │ │ │ │
└────┬─────┴──────┴────┬───────┴──────┴────┬─────────────┴──────┬─────────────┘
│ │ │ │
├─────────────────┤ │ │
│ BBF Datamodel │ │ │
│ (sysmngr) │ │ │
│ Device.DeviceInfo │ │
│ .FirmwareImage.{i} │ │
│ Download()/Activate() │ │
└──────────┬──────┘ │ │
│ │ │
└────────────┬──────────────┴────────────────────┘
│
┌───────▼────────┐
│ fwbank │ (UBUS: fwbank.upgrade)
│ (Core logic) │
└───────┬────────┘
│
┌────────────────┼────────────────┐
│ │ │
┌─────▼─────┐ ┌──────▼──────┐ ┌─────▼─────┐
│ opconf │ │ sysupgrade │ │ ubus │
│ (config) │ │ (OpenWrt) │ │ (events) │
└───────────┘ └─────────────┘ └───────────┘
```
**Protocol Paths:**
- **TR-069/USP:** Use BBF datamodel → sysmngr → fwbank UBUS
- **DHCP/fwbank CLI:** Call fwbank UBUS directly (bypass BBF datamodel)
- **Sulu:** Web GUI using USP/TR-369 protocol via OB-USPa agent (WebSocket)
- **dmcli:** CLI using BBF datamodel via local UBUS or remote USP (MQTT/WebSocket)
---
## Architecture
### Dual-Bank System
The firmware is stored in two independent banks, allowing safe upgrades with automatic fallback.
```
┌──────────────────────────────────────────────────┐
│ Flash Memory Layout │
├────────────┬────────────┬─────────────────────────┤
│ Bank 1 │ Bank 2 │ /usr_data │
│ (Firmware) │ (Firmware) │ (Persistent Storage) │
├────────────┼────────────┼─────────────────────────┤
│ Active │ Inactive │ - opconf.json │
│ Boot │ Upgrade │ - userconf.json │
│ │ │ - userconf/keep.d/ │
└────────────┴────────────┴─────────────────────────┘
```
#### Bank States
Each bank can have one of the following states:
| State | Description | Can Upgrade | Can Activate |
|-------|-------------|-------------|--------------|
| **Active** | Currently running firmware | ❌ | N/A (already active) |
| **Boot** | Will boot on next reboot | ❌ | N/A (will boot) |
| **Upgrade** | Available for new firmware | ✅ | ✅ |
| **Invalid** | Corrupted or unavailable | ❌ | ❌ |
**Example Status:**
```json
{
"bank": [
{
"id": 1,
"fwver": "iop-v5.14.0",
"swver": "5.14.0",
"active": true,
"boot": true,
"upgrade": false,
"status": "Active"
},
{
"id": 2,
"fwver": "",
"swver": "",
"active": false,
"boot": false,
"upgrade": true,
"status": "Available"
}
]
}
```
### Core Components
#### 1. fwbank (Shell Script)
**Location:** `/etc/sysmngr/fwbank`
**Purpose:** Core firmware bank management
**Methods:**
- `dump` - Get bank information
- `upgrade` - Write firmware to bank
- `set_bootbank` - Set which bank boots next
- `copy_config` - Preserve configuration
**UBUS Interface:**
```bash
ubus call fwbank dump
ubus call fwbank upgrade '{"path": "/tmp/fw.bin", "bank": 2, "auto_activate": true, "keep_settings": true}'
ubus call fwbank set_bootbank '{"bank": 2}'
ubus call fwbank copy_config '{"keep_settings": true, "keep_opconf": true, "config_scope": "UserOnly"}'
```
#### 2. sysmngr (C Daemon)
**Location:** `/usr/sbin/sysmngr`
**Purpose:** BBF TR-181 datamodel implementation
**Key Files:**
- `src/fw_images.c` - FirmwareImage object implementation
- `src/fwbank.c` - fwbank integration
- `src/utils.c` - Download/validation utilities
**Datamodel Paths:**
```
Device.DeviceInfo.
├── ActiveFirmwareImage (Reference to active bank)
├── BootFirmwareImage (Reference to boot bank)
├── MaxNumberOfActivateTimeWindows (5)
└── FirmwareImage.{i}.
├── Alias (e.g., "cpe-1")
├── Name (Firmware name)
├── Version (Software version)
├── Available (true/false)
├── Status (Active/Available/Invalid)
├── Download() (Async operation)
└── Activate() (Async operation)
```
#### 3. opconf (Configuration Manager)
**Location:** `/usr/sbin/opconf_conf_handler`
**Purpose:** Manage operator and user configuration separation
**Storage:**
```
/usr_data/
├── opconf/
│ └── opconf.json # Operator configuration (ISP settings)
└── userconf/
├── userconf.json # User configuration (end-user settings)
└── keep.d/ # Backup of specific files
└── etc/config/
├── wifi
├── firewall
└── ...
```
---
## Configuration Parameters
### 1. keep_config (KeepConfig)
**Type:** Boolean (0/1 or "0"/"1")
**Default:** `1` (preserve settings)
**Purpose:** Master switch for configuration preservation
**Behavior:**
- `1` - Preserve configurations according to `config_scope`
- `0` - Factory reset, discard ALL configurations
**Usage:**
```json
{
"keep_settings": 1
}
```
or in BBF operations:
```json
{
"X_IOWRT_EU_KeepConfig": "1"
}
```
### 2. keep_opconf (KeepOpConf)
**Type:** Boolean (0/1 or "0"/"1")
**Default:** `1` (preserve operator config)
**Purpose:** Control operator configuration preservation
**Behavior:**
- `1` - Keep `/usr_data/opconf/opconf.json`
- `0` - Delete operator configuration (ISP reset)
**Only effective when:** `keep_config=1`
**Usage:**
```json
{
"keep_opconf": 1
}
```
or in BBF operations:
```json
{
"X_IOWRT_EU_KeepOpConf": "1"
}
```
**Operator Configuration Contains:**
- ACS/USP controller credentials
- WAN interface settings
- VLANs and firewall rules
- Management protocol settings
- Default WiFi credentials
### 3. config_scope (ConfigScope)
**Type:** String enum
**Default:** `"UserOnly"`
**Possible Values:**
- `"All"` - Full backup (sysupgrade.tgz + userconf.json)
- `"UserOnly"` - Selective backup (userconf.json only)
**Purpose:** Define granularity of configuration preservation
**Only effective when:** `keep_config=1`
**Usage:**
```json
{
"config_scope": "UserOnly"
}
```
or in BBF operations:
```json
{
"X_IOWRT_EU_ConfigScope": "UserOnly"
}
```
### Configuration Matrix
| keep_config | keep_opconf | config_scope | Result |
|-------------|-------------|--------------|--------|
| 0 | - | - | **Factory Reset** - All configs deleted |
| 1 | 0 | UserOnly | **User Only** - Operator config deleted, user settings preserved via userconf.json |
| 1 | 1 | UserOnly | **User+Operator (Lightweight)** - Both preserved, selective backup |
| 1 | 0 | All | **User+System** - Operator config deleted, full system backup + userconf.json |
| 1 | 1 | All | **Everything** - Full preservation, both userconf.json and sysupgrade.tgz created |
### User Configuration (userconf.json)
**Generated by:** `/usr/sbin/genuserconf`
**Applied on boot:** Via `/etc/init.d/opconf` init script
**Contains:**
- LAN network settings (IP, netmask, DHCP pool)
- UPnP enable/disable
- Firewall port forwarding rules
- User account passwords
- WiFi AP configuration (SSID, password, encryption)
- VoIP SIP client settings
- LED PWM settings
**Example:**
```json
{
"network": {
"lan": {
"ipaddr": "192.168.1.1",
"netmask": "255.255.255.0"
}
},
"wireless": {
"wifi0": {
"ssid": "MyHomeNetwork",
"key": "SecurePassword123"
}
}
}
```
### System Backup (sysupgrade.tgz)
**Created by:** `sysupgrade -b /tmp/sysupgrade.tgz`
**Copied by:** `platform_copy_config`
**Contains:** ALL UCI configs from `/etc/config/*`
**Size:** Typically 500KB - 2MB
**Only created when:** `config_scope="All"`
---
## Upgrade Methods
The firmware upgrade system supports **four upgrade methods**:
1. **USP/TR-369** - Modern BBF protocol (via Sulu GUI or external controller) → Uses BBF datamodel
2. **TR-069/CWMP** - Legacy ACS protocol → Uses BBF datamodel
3. **DHCP Onboarding** - Zero-touch provisioning → Direct UBUS to fwbank
4. **CLI** - Direct command-line access → Direct UBUS to fwbank
> **Important Notes:**
> - **Sulu and dmcli** are clients that use **USP/TR-369 protocol** to access the BBF datamodel. They are not separate upgrade methods but rather different interfaces for USP operations.
> - **Sulu:** Browser-based GUI with WebSocket to local OB-USPa agent
> - **dmcli:** Command-line interface that can connect locally (UBUS) or remotely (USP)
> - **DHCP and fwbank CLI** bypass the BBF datamodel layer and call fwbank UBUS methods directly, providing lower-level access but with limited protocol features (no async operations, no transfer complete events).
### Method 1: USP/TR-369 (Recommended)
**Protocol:** BBF USP (User Services Platform) / TR-369
**Agent:** OB-USPa (Open Broadband - USP Agent)
**Clients:**
- **Sulu Web GUI** - Browser-based interface (WebSocket connection to OB-USPa)
- **dmcli** - Command-line interface for BBF datamodel (local UBUS or remote USP)
- **External USP Controller** - Remote management platform (STOMP/MQTT/WebSocket)
**Configuration:** Per-operation parameters (highest flexibility)
**Datamodel:** `Device.DeviceInfo.FirmwareImage.{i}.Download()` / `Activate()`
#### USP Client 1: Sulu Web GUI
**Interface:** React-based web application
**Communication:** WebSocket to local OB-USPa agent
**Configuration Control:**
- **File Upload Mode:** `KeepConfig` toggle available
- **URL Download Mode:** Uses system defaults (no config control in GUI)
**GUI Configuration Options:**
| Feature | File Upload | URL Download |
|---------|-------------|--------------|
| KeepConfig | ✅ User toggle | ❌ System default |
| KeepOpConf | ❌ System default | ❌ System default |
| ConfigScope | ❌ System default | ❌ System default |
| Authentication | ❌ None | ✅ Username/Password |
| Progress Tracking | ✅ Upload + Install | ❌ No |
> **Note:** The Sulu GUI currently only exposes the `KeepConfig` toggle in file upload mode. To control `KeepOpConf` and `ConfigScope`, operators must use system defaults (configured in firmware) or use an external USP controller with full parameter support.
##### Example: Sulu GUI Firmware Upload
```javascript
// Sulu GUI uploads firmware file in chunks
await uploadFileChunked({
Filename: "firmware.itb",
Content: "<base64-encoded-firmware>",
Filesize: "52428800",
Checksum: "a3f5e1d2..."
});
// Trigger upgrade with config control
await downloadToDevice({
URL: "file:///tmp/obuspa/firmware.itb",
X_IOWRT_EU_KeepConfig: "1", // User toggle
X_IOWRT_EU_KeepOpConf: "1", // Default
X_IOWRT_EU_ConfigScope: "UserOnly", // Default
AutoActivate: "1"
});
```
#### USP Client 2: External USP Controller
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Download()",
"input": {
"URL": "https://firmware.isp.com/cpe/v5.15.0.itb",
"Username": "fwuser",
"Password": "secret123",
"FileSize": "52428800",
"CheckSumAlgorithm": "SHA-256",
"CheckSum": "a3f5e1d2c4b6...",
"AutoActivate": "1",
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "0",
"X_IOWRT_EU_ConfigScope": "UserOnly"
}
}
```
**Result:** Operator config reset, user WiFi/network settings preserved
#### USP Client 3: dmcli (Data Model CLI)
**Interface:** Interactive command-line interface for BBF data models
**Modes:**
- **Local Mode:** Run directly on device via SSH/console (uses local UBUS to sysmngr)
- **Remote Mode:** Run from computer and connect via USP (MQTT/WebSocket to OB-USPa)
**Configuration:** Full control via operation parameters
**Installation:**
```bash
# On device (pre-installed)
ssh operator@192.168.1.1
dmcli
# On host computer (requires NodeJS)
npm install -g @gnx/dmcli@latest
```
**Remote Connection Setup:**
```bash
# Configure USP connection profile
dmcli
# set Settings.USP.ActiveConnectionProfile iowrt-gnx-operator
# do Settings.USP.Connect()
```
##### Example: Firmware Upgrade via dmcli
**Interactive Mode:**
```bash
# Start dmcli
dmcli
# Navigate to firmware object
# cd Device.DeviceInfo.FirmwareImage.2
# Check current status
# get Status Version Name
# Download firmware
# do Download() URL=https://firmware.isp.com/v5.15.0.itb AutoActivate=1 X_IOWRT_EU_KeepConfig=1 X_IOWRT_EU_KeepOpConf=0 X_IOWRT_EU_ConfigScope=UserOnly
# For async operations, use -n flag to not wait
# do -n Download() URL=https://firmware.isp.com/v5.15.0.itb AutoActivate=1
```
**Non-Interactive Mode:**
```bash
# Single command execution
dmcli do Device.DeviceInfo.FirmwareImage.2.Download() \
URL=https://firmware.isp.com/v5.15.0.itb \
AutoActivate=1 \
X_IOWRT_EU_KeepConfig=1 \
X_IOWRT_EU_KeepOpConf=0 \
X_IOWRT_EU_ConfigScope=UserOnly
# Activate previously downloaded firmware
dmcli do Device.DeviceInfo.FirmwareImage.2.Activate() \
X_IOWRT_EU_KeepConfig=1 \
X_IOWRT_EU_KeepOpConf=1 \
X_IOWRT_EU_ConfigScope=All
```
**Configuration Control:**
| Feature | dmcli Support |
|---------|---------------|
| KeepConfig | ✅ Full control via parameter |
| KeepOpConf | ✅ Full control via parameter |
| ConfigScope | ✅ Full control via parameter |
| Authentication | ✅ HTTP auth in URL parameter |
| Progress Tracking | ✅ Async mode with `-n` flag |
| Auto-completion | ✅ Tab completion for paths and parameters |
**Advantages:**
- Full access to BBF datamodel with tab completion
- Can be scripted for automation
- Supports both local and remote operation
- Comprehensive parameter control
- Interactive exploration of data model
#### Example: Scheduled Activation
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Activate()",
"input": {
"TimeWindow.1.Start": "3600", // 1 hour from now
"TimeWindow.1.End": "7200", // 2 hours from now
"TimeWindow.1.Mode": "AnyTime",
"TimeWindow.1.MaxRetries": "3",
"TimeWindow.2.Start": "86400", // 24 hours from now
"TimeWindow.2.End": "90000", // 25 hours from now
"TimeWindow.2.Mode": "Immediately",
"TimeWindow.2.MaxRetries": "-1",
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "1",
"X_IOWRT_EU_ConfigScope": "All"
}
}
```
**Cron Jobs Created:**
```cron
# Window 1: Between 14:00-16:00 today, up to 3 retries
0 14 26 11 * sh /usr/share/bbfdm/scripts/bbf_activate_handler.sh 'AnyTime' '2' '7200' 'false' '3' '1' '1' 'All' ''
# Window 2: Between 14:00-15:00 tomorrow, infinite retries
0 14 27 11 * sh /usr/share/bbfdm/scripts/bbf_activate_handler.sh 'Immediately' '2' '3600' 'true' '-1' '1' '1' 'All' ''
```
### Method 2: TR-069/CWMP
**Controller:** ACS (Auto Configuration Server)
**Configuration:** UCI defaults in `/etc/config/cwmp`
**RPC:** `Download(1 Firmware Upgrade Image)`
#### UCI Configuration
**File:** `/etc/config/cwmp`
```uci
config cpe 'cpe'
option acs_url 'https://acs.isp.com/cwmp'
option acs_user 'cpe-user'
option acs_passwd 'secret'
# Firmware upgrade config defaults
option KeepConfig '1'
option KeepOpConf '1'
option ConfigScope 'UserOnly'
```
#### ACS Download RPC Flow
1. **ACS sends Download RPC:**
```xml
<Download>
<CommandKey>fw-upgrade-20251126</CommandKey>
<FileType>1 Firmware Upgrade Image</FileType>
<URL>https://firmware.isp.com/cpe/v5.15.0.itb</URL>
<Username>fwuser</Username>
<Password>secret123</Password>
<FileSize>52428800</FileSize>
</Download>
```
2. **icwmp processes download:**
```c
// src/download.c:445-497
cwmp_apply_multiple_firmware(true) {
// Read config from UCI
keep_config = cwmp_ctx.conf.cpe_keep_config;
keep_opconf = cwmp_ctx.conf.cpe_keep_opconf;
config_scope = cwmp_ctx.conf.cpe_config_scope;
// Call BBF datamodel via UBUS
ubus_call("bbfdm", "operate", {
"path": "Device.DeviceInfo.FirmwareImage.2.Download()",
"input": {
"URL": "file:///tmp/firmware.img",
"AutoActivate": true,
"X_IOWRT_EU_KeepConfig": keep_config,
"X_IOWRT_EU_KeepOpConf": keep_opconf,
"X_IOWRT_EU_ConfigScope": config_scope
}
});
}
```
3. **CPE sends TransferComplete event back to ACS**
#### Dynamic Configuration Override
To change config behavior before upgrade:
```bash
# ISP wants to reset operator config on next upgrade
ubus call cwmp set '{"cpe.KeepOpConf": "0"}'
uci set cwmp.cpe.KeepOpConf='0'
uci commit cwmp
# Then ACS initiates Download RPC
```
### Method 3: DHCP Onboarding
**Protocol:** Direct UBUS call to fwbank (does NOT use USP/TR-369 or BBF datamodel)
**Server:** DHCP server (ISP infrastructure)
**Client:** udhcpc hooks (`/etc/udhcpc.user.d/dhcp-on-boarding`)
**Configuration:** `reset_mode` flag only (limited control)
**DHCP Options:** Option 125 (sub-option 2) or Option 224
#### DHCP Option Format
**Option 125 (Vendor-Specific):**
```
Enterprise-ID: 25167 (Genexis)
Sub-option 2: Firmware upgrade URL
Hex format:
00006250 # Genexis Enterprise ID
14 # Length
02 # Sub-option code (firmware)
12 # URL length
687474703a2f2f... # http://192.168.1.5/fw.bin
```
**Option 224 (Legacy):**
```
Raw hex-encoded URL:
687474703a2f2f3139322e3136382e312e352f66772e62696e
```
#### URL Syntax
**Immediate upgrade, keep config:**
```
http://server.com/firmware.bin
```
**Immediate upgrade, factory reset:**
```
http://server.com/firmware.bin,reset_mode=full
```
**Scheduled upgrade:**
```
http://server.com/firmware.bin,20251215,02,4
# Upgrade on Dec 15, 2025, random time between 02:00-06:00
```
**Scheduled with reset:**
```
http://server.com/firmware.bin,reset_mode=full,14,2
# Today at random time between 14:00-16:00, factory reset
```
#### Configuration Mapping
```bash
# In dhcp_upgrade_handling.sh
if [ "$reset_mode" = "full" ]; then
keep_settings="0"
else
keep_settings="1"
fi
# Call fwbank
ubus call fwbank upgrade '{
"path": "/tmp/firmware.bin",
"bank": 2,
"auto_activate": true,
"reboot": true,
"keep_settings": '$keep_settings'
# Note: keep_opconf and config_scope NOT passed, use system defaults
}'
```
**Limitation:** Cannot control `keep_opconf` or `config_scope` via DHCP
**Workaround:** Pre-configure defaults in firmware
```uci
# /rom/etc/config/cwmp (in firmware image)
config cpe 'cpe'
option KeepConfig '1'
option KeepOpConf '1'
option ConfigScope 'UserOnly'
```
#### DHCP Server Example (Kea)
```json
{
"subnet4": [{
"subnet": "192.168.100.0/24",
"pools": [{"pool": "192.168.100.10 - 192.168.100.250"}],
"option-data": [
{
"name": "vendor-encapsulated-options",
"code": 125,
"data": "00006250 1E 02 1C 687474703a2f2f3139322e3136382e3130302e312f66772e69746220",
"always-send": true
}
]
}]
}
```
Decodes to: `http://192.168.100.1/fw.itb`
### Method 4: CLI (fwbank)
**Protocol:** Direct UBUS call to fwbank (does NOT use USP/TR-369 or BBF datamodel)
**Interface:** Direct shell script invocation
**Script:** `/etc/sysmngr/fwbank`
**Configuration:** Full control via parameters
**Usage:**
```bash
# Dump bank information
/etc/sysmngr/fwbank call dump
# Upgrade with full config control
/etc/sysmngr/fwbank call upgrade '{
"path": "/tmp/firmware.itb",
"bank": 2,
"auto_activate": true,
"reboot": false,
"keep_settings": true,
"keep_opconf": false,
"config_scope": "UserOnly"
}'
# Copy config separately
/etc/sysmngr/fwbank call copy_config '{
"keep_settings": true,
"keep_opconf": true,
"config_scope": "All"
}'
# Set boot bank
/etc/sysmngr/fwbank call set_bootbank '{"bank": 2}'
# Manual reboot
reboot
```
---
## Configuration Preservation
### Process Flow
```
┌────────────────────────────────────────────────────────────┐
│ Upgrade Initiated │
└────────────────────┬───────────────────────────────────────┘
│
▼
┌───────────────────────┐
│ Parameters Received │
│ - keep_settings │
│ - keep_opconf │
│ - config_scope │
└───────────┬───────────┘
│
▼
┌──────────────┐
│ keep_config │
│ = 1? │
└──┬───────┬───┘
│ │
NO ──┘ └── YES
│ │
▼ ▼
┌────────────┐ ┌──────────────────┐
│ Skip all │ │ opconf_conf_ │
│ config │ │ handler │
│ handling │ └────────┬─────────┘
└─────┬──────┘ │
│ ▼
│ ┌──────────────────┐
│ │ keep_opconf = 0?│
│ └────┬──────┬──────┘
│ │ │
│ YES ─┘ └─ NO
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Delete │ │
│ │ opconf.json │ │
│ └──────┬──────┘ │
│ │ │
│ └─────┬───────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ config_scope = │
│ │ "All" / "UserOnly"? │
│ └────┬────────────┬────┘
│ │ │
│ All ─┘ └─ UserOnly
│ │ │
│ ▼ ▼
│ ┌──────────┐ ┌─────────────┐
│ │ Generate │ │ Generate │
│ │userconf +│ │ userconf. │
│ │sysupgrade│ │ json only │
│ │ .tgz │ └──────┬──────┘
│ └────┬─────┘ │
│ └─────────┬─────────┘
│ │
└────────┬────────┘
│
▼
┌──────────────┐
│ sysupgrade │
│ executes │
└──────┬───────┘
│
▼
┌──────────────┐
│ Reboot │
└──────┬───────┘
│
▼
┌────────────────┐
│ Apply opconf │
│ (if exists) │
└────────┬───────┘
│
▼
┌────────────────┐
│ Apply userconf │
│ (if exists) │
└────────┬───────┘
│
▼
┌────────────────┐
│ Delete userconf│
│ .json │
└────────────────┘
```
### opconf_conf_handler Details
**Location:** `/usr/sbin/opconf_conf_handler`
**Called by:** fwbank script before sysupgrade
**Parameters:**
```bash
opconf_conf_handler -k <keep_config> -o <keep_opconf> -s <config_scope>
```
**Logic:**
```bash
#!/bin/sh
KEEP_CONFIG="$1" # 0 or 1
KEEP_OPCONF="$2" # 0 or 1
CONFIG_SCOPE="$3" # "All" or "UserOnly"
# Step 1: Handle opconf.json
if [ "$KEEP_OPCONF" -eq "0" ]; then
rm -f /usr_data/opconf/opconf.json
fi
# Step 2: Handle userconf
if [ "$KEEP_CONFIG" -eq "0" ]; then
rm -f /usr_data/userconf/userconf.json
rm -rf /usr_data/userconf/keep.d/
else
if [ "$CONFIG_SCOPE" = "All" ] || [ "$CONFIG_SCOPE" = "UserOnly" ]; then
# Generate userconf.json from current system state
genuserconf > /usr_data/userconf/userconf.json
# Backup specific config files
copy_config_over_upgrade
fi
fi
```
### genuserconf Script
**Purpose:** Extract user-configurable settings into JSON
**Source Files:**
```bash
/etc/config/network → LAN settings
/etc/config/wireless → WiFi settings
/etc/config/firewall → Port forwarding
/etc/config/upnpd → UPnP state
/etc/config/voice → VoIP settings
/etc/config/system → LED settings
```
**Example Output:**
```json
{
"network": {
"lan": {
"proto": "static",
"ipaddr": "192.168.100.1",
"netmask": "255.255.255.0",
"ip6assign": "60"
},
"lan_dhcp": {
"start": "100",
"limit": "150",
"leasetime": "12h"
}
},
"wireless": {
"radio0_ap0": {
"ssid": "MyNetwork",
"encryption": "psk2",
"key": "MyPassword123"
}
},
"firewall": {
"portforward_ssh": {
"src": "wan",
"dest": "lan",
"proto": "tcp",
"src_dport": "2222",
"dest_ip": "192.168.100.50",
"dest_port": "22"
}
}
}
```
### On-Boot Application
**Init Script:** `/etc/init.d/opconf`
**Boot Sequence:**
```bash
#!/bin/sh /etc/rc.common
START=19 # Early boot, before network
start() {
# Check if /usr_data is mounted
if ! mountpoint -q /usr_data; then
return 0
fi
# Step 1: Apply opconf.json (if exists)
if [ -f /usr_data/opconf/opconf.json ]; then
opconf_apply /usr_data/opconf/opconf.json
# Note: opconf.json is NOT deleted, persists across reboots
fi
# Step 2: Apply userconf.json (if exists)
if [ -f /usr_data/userconf/userconf.json ]; then
userconf_apply /usr_data/userconf/userconf.json
# Delete after applying (one-time use)
rm -f /usr_data/userconf/userconf.json
fi
# Step 3: Restore files from keep.d/
if [ -d /usr_data/userconf/keep.d ]; then
cp -a /usr_data/userconf/keep.d/* /
fi
}
```
---
## Operator Configuration Guide
### Scenario 1: ISP wants full control over CPE, no user customization preserved
**Method:** TR-069 or USP Controller
**Configuration:**
**/etc/config/cwmp (firmware defaults):**
```uci
config cpe 'cpe'
option KeepConfig '0'
# KeepOpConf and ConfigScope ignored when KeepConfig=0
```
**Result:** Every firmware upgrade = factory reset
**Use Case:** Managed CPE, ISP owns all settings
### Scenario 2: ISP wants to preserve user WiFi passwords but reset ISP settings
**Method:** USP Controller (per-upgrade control)
**Download RPC:**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Download()",
"input": {
"URL": "https://firmware.isp.com/v5.15.0.itb",
"AutoActivate": "1",
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "0",
"X_IOWRT_EU_ConfigScope": "UserOnly"
}
}
```
**Result:**
- ✅ User WiFi SSID/password preserved
- ✅ User LAN IP preserved
- ✅ User port forwards preserved
- ❌ ACS URL reset to firmware default
- ❌ WAN settings reset to firmware default
- ❌ VLAN settings reset to firmware default
**Use Case:** ISP changing backend infrastructure, wants fresh provisioning
### Scenario 3: Home user upgrading via GUI, wants to keep everything
**Method:** USP/TR-369 (Sulu Web GUI)
**User Action:**
1. Navigate to System → Firmware
2. Upload firmware.itb
3. ✅ Enable "Keep Settings" toggle
4. Click "Start Upgrade"
**Sulu sends USP Download() operation:**
```json
{
"URL": "file:///tmp/obuspa/firmware.itb",
"X_IOWRT_EU_KeepConfig": "1",
"AutoActivate": "1"
}
```
**Result:**
- ✅ All settings preserved (defaults to keep_opconf=1, config_scope="UserOnly")
- ✅ WiFi, LAN, port forwards preserved
- ✅ ACS/ISP settings preserved
**Limitation:** Current Sulu GUI cannot selectively reset operator config. To control `KeepOpConf` and `ConfigScope`, use an external USP controller or configure system defaults in firmware.
### Scenario 4: DHCP onboarding in new deployment, zero-touch provisioning
**Method:** DHCP Option 125
**DHCP Server sends:**
```
Option 125: http://bootstrap.isp.com/fw-v5.14.0.itb,reset_mode=full
Option 125 Sub-3: http://bootstrap.isp.com/config-$MAC$.uci.enc
```
**CPE boots with factory firmware:**
1. Gets DHCP lease
2. Downloads firmware → factory reset (reset_mode=full)
3. Downloads encrypted config → applies ISP settings
4. Reboots into new firmware with ISP provisioning
**Configuration:** Entirely driven by DHCP, no pre-config needed
### Scenario 5: Scheduled maintenance window upgrade
**Method:** USP Controller with Activate()
**Controller sends Download() first:**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Download()",
"input": {
"URL": "https://firmware.isp.com/v5.15.0.itb",
"AutoActivate": "0" // Don't activate yet
}
}
```
**Then sends Activate() with time windows:**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Activate()",
"input": {
"TimeWindow.1.Start": "3600", // 1 hour from now
"TimeWindow.1.End": "10800", // 3 hours from now
"TimeWindow.1.Mode": "WhenIdle",
"TimeWindow.1.MaxRetries": "5",
"TimeWindow.2.Start": "86400", // 24 hours (fallback)
"TimeWindow.2.End": "93600",
"TimeWindow.2.Mode": "Immediately",
"TimeWindow.2.MaxRetries": "-1",
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "1",
"X_IOWRT_EU_ConfigScope": "All"
}
}
```
**Cron Jobs:**
```cron
# Primary window: Tonight 01:00-04:00, activate when idle, up to 5 retries
0 1 27 11 * sh /usr/share/bbfdm/scripts/bbf_activate_handler.sh 'WhenIdle' '2' '10800' 'false' '5' '1' '1' 'All' ''
# Fallback: Tomorrow night, force activation, infinite retries
0 1 28 11 * sh /usr/share/bbfdm/scripts/bbf_activate_handler.sh 'Immediately' '2' '7200' 'true' '-1' '1' '1' 'All' ''
```
**Use Case:** ISP wants to upgrade during low-traffic hours, with automatic fallback
---
## Troubleshooting
### Issue: Configuration not preserved after upgrade
**Symptoms:**
- WiFi password reset to default
- LAN IP back to 192.168.1.1
- Port forwarding rules gone
**Diagnosis:**
1. Check if `/usr_data` partition exists:
```bash
mount | grep usr_data
df -h | grep usr_data
```
2. Check opconf_conf_handler was called:
```bash
logread | grep opconf_conf_handler
```
3. Check for userconf.json:
```bash
ls -la /usr_data/userconf/
```
4. Check config parameters used:
```bash
logread | grep -E "keep_settings|keep_opconf|config_scope"
```
**Solutions:**
- **Missing /usr_data:** Partition not created, config preservation disabled
```bash
# Check flash layout
cat /proc/mtd
```
- **opconf not available:** System falls back to legacy mode
```bash
which opconf_conf_handler
# If missing: config_scope forced to "All"
```
- **Parameters not passed:** Check upgrade method's config support
```bash
# DHCP upgrade - add reset_mode parameter
# GUI upgrade - toggle "Keep Settings"
# TR-069 - check /etc/config/cwmp defaults
```
### Issue: Operator config deleted unexpectedly
**Symptoms:**
- ACS URL reset to default
- WAN interface unconfigured
- Need to re-provision device
**Diagnosis:**
```bash
# Check if opconf.json exists
ls -la /usr_data/opconf/opconf.json
# Check upgrade parameters
logread | grep keep_opconf
```
**Possible Causes:**
1. **USP controller sent KeepOpConf=0:**
- Check USP controller logs for Download() parameters
2. **TR-069 UCI default was 0:**
```bash
uci show cwmp.cpe.KeepOpConf
# Should be '1', if '0' → operator config deleted
```
3. **DHCP upgrade with no opconf support:**
- DHCP onboarding doesn't support keep_opconf
- Pre-configure firmware with ISP settings baked in
**Prevention:**
```uci
# Set system-wide defaults in firmware
cat > /rom/etc/config/cwmp << EOF
config cpe 'cpe'
option KeepConfig '1'
option KeepOpConf '1'
option ConfigScope 'UserOnly'
EOF
```
### Issue: Firmware upgrade stuck at "Downloading"
**Symptoms:**
- Bank status shows "Downloading" indefinitely
- No "Available" status after 10+ minutes
- Device not rebooted
**Diagnosis:**
```bash
# Check sysupgrade events
ubus listen sysupgrade &
# Check fwbank status
ubus call fwbank dump
# Check for hung sysupgrade process
ps | grep sysupgrade
```
**Possible Causes:**
1. **Invalid firmware image:**
```bash
sysupgrade -T /tmp/firmware.itb
# Should output: Image check succeeded
```
2. **Insufficient flash space:**
```bash
df -h /tmp
df -h /overlay
```
3. **sysupgrade process crashed:**
```bash
logread | grep sysupgrade | tail -50
```
**Recovery:**
```bash
# Kill hung sysupgrade
killall sysupgrade
# Clean up temp files
rm -f /tmp/firmware-*
# Reset bank status manually (if needed)
ubus call fwbank set_bootbank '{"bank": 1}' # Back to current bank
reboot
```
### Issue: Device boots into wrong bank after upgrade
**Symptoms:**
- Upgraded bank 2, but still running bank 1
- Boot bank doesn't match expectation
**Diagnosis:**
```bash
# Check current running bank
ubus call fwbank dump | jsonfilter -e '@.bank[@.active=true]'
# Check boot bank
ubus call fwbank dump | jsonfilter -e '@.bank[@.boot=true]'
# Check bootloader environment
fw_printenv boot_bank
```
**Possible Causes:**
1. **AutoActivate was 0:**
- Download() completed but didn't set boot bank
- Need to manually call Activate()
2. **Bootloader failed to switch:**
- U-Boot issue
- Environment not writable
**Solution:**
```bash
# Manually set boot bank
ubus call fwbank set_bootbank '{"bank": 2}'
# Copy config
ubus call fwbank copy_config '{
"keep_settings": true,
"keep_opconf": true,
"config_scope": "All"
}'
# Reboot
reboot
```
---
## API Reference
### fwbank UBUS Methods
#### fwbank.dump
**Description:** Get firmware bank information
**Parameters:** None
**Returns:**
```json
{
"bank": [
{
"id": 1,
"fwver": "iop-v5.14.0",
"swver": "5.14.0",
"omci_swver": "5.14",
"active": true,
"boot": true,
"upgrade": false,
"status": "Active"
},
{
"id": 2,
"fwver": "iop-v5.15.0",
"swver": "5.15.0",
"omci_swver": "5.15",
"active": false,
"boot": false,
"upgrade": true,
"status": "Available"
}
]
}
```
**Example:**
```bash
ubus call fwbank dump
```
#### fwbank.upgrade
**Description:** Upgrade firmware to specified bank
**Parameters:**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| path | string | ✅ | - | Path to firmware file |
| bank | integer | ✅ | - | Target bank ID (1 or 2) |
| auto_activate | boolean | ❌ | false | Set as boot bank after upgrade |
| reboot | boolean | ❌ | false | Reboot after upgrade |
| keep_settings | boolean | ❌ | true | Preserve configurations |
| keep_opconf | boolean | ❌ | true | Preserve operator config |
| config_scope | string | ❌ | "UserOnly" | Config scope ("All" or "UserOnly") |
**Returns:**
```json
{
"result": "ok"
}
```
or
```json
{
"result": "error",
"message": "Invalid firmware image"
}
```
**Example:**
```bash
ubus call fwbank upgrade '{
"path": "/tmp/firmware.itb",
"bank": 2,
"auto_activate": true,
"reboot": true,
"keep_settings": true,
"keep_opconf": false,
"config_scope": "UserOnly"
}'
```
#### fwbank.set_bootbank
**Description:** Set which bank will boot on next reboot
**Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| bank | integer | ✅ | Bank ID to boot (1 or 2) |
**Returns:**
```json
{
"result": "ok"
}
```
**Example:**
```bash
ubus call fwbank set_bootbank '{"bank": 2}'
```
#### fwbank.copy_config
**Description:** Copy configuration from active bank to boot bank
**Parameters:**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| keep_settings | boolean | ❌ | true | Copy user configs |
| keep_opconf | boolean | ❌ | true | Copy operator config |
| config_scope | string | ❌ | "UserOnly" | Scope ("All" or "UserOnly") |
**Returns:**
```json
{
"result": "ok"
}
```
**Example:**
```bash
ubus call fwbank copy_config '{
"keep_settings": true,
"keep_opconf": true,
"config_scope": "All"
}'
```
### BBF Datamodel Operations
#### Device.DeviceInfo.FirmwareImage.{i}.Download()
**Description:** Download firmware to specified bank
**Input Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| URL | string | ✅ | Firmware download URL |
| AutoActivate | boolean | ❌ | Set as boot bank after download |
| Username | string | ❌ | HTTP authentication username |
| Password | string | ❌ | HTTP authentication password |
| FileSize | unsignedInt | ❌ | Expected file size (bytes) |
| CheckSumAlgorithm | string | ❌ | "MD5", "SHA-1", "SHA-256" |
| CheckSum | string | ❌ | Expected checksum value |
| X_IOWRT_EU_KeepConfig | boolean | ❌ | Preserve configurations |
| X_IOWRT_EU_KeepOpConf | boolean | ❌ | Preserve operator config |
| X_IOWRT_EU_ConfigScope | string | ❌ | "All" or "UserOnly" |
**Returns:** Async operation, sends TransferComplete event when done
**USP Example:**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Download()",
"input": {
"URL": "https://fw.example.com/firmware.itb",
"AutoActivate": "1",
"FileSize": "52428800",
"CheckSumAlgorithm": "SHA-256",
"CheckSum": "a3f5e1d2c4b6...",
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "0",
"X_IOWRT_EU_ConfigScope": "UserOnly"
}
}
```
#### Device.DeviceInfo.FirmwareImage.{i}.Activate()
**Description:** Activate downloaded firmware (optionally scheduled)
**Input Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| TimeWindow.{i}.Start | unsignedInt | ❌ | Seconds from now to start |
| TimeWindow.{i}.End | unsignedInt | ❌ | Seconds from now to end |
| TimeWindow.{i}.Mode | string | ❌ | "AnyTime", "Immediately", "WhenIdle", "ConfirmationNeeded" |
| TimeWindow.{i}.UserMessage | string | ❌ | Message to display to user |
| TimeWindow.{i}.MaxRetries | int | ❌ | -1 (infinite) to 10 |
| X_IOWRT_EU_KeepConfig | boolean | ❌ | Preserve configurations |
| X_IOWRT_EU_KeepOpConf | boolean | ❌ | Preserve operator config |
| X_IOWRT_EU_ConfigScope | string | ❌ | "All" or "UserOnly" |
**Returns:** Immediate success, activation occurs at scheduled time
**USP Example (immediate):**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Activate()",
"input": {
"X_IOWRT_EU_KeepConfig": "1",
"X_IOWRT_EU_KeepOpConf": "1",
"X_IOWRT_EU_ConfigScope": "All"
}
}
```
**USP Example (scheduled):**
```json
{
"command": "Device.DeviceInfo.FirmwareImage.2.Activate()",
"input": {
"TimeWindow.1.Start": "3600",
"TimeWindow.1.End": "7200",
"TimeWindow.1.Mode": "WhenIdle",
"TimeWindow.1.MaxRetries": "3",
"X_IOWRT_EU_KeepConfig": "1"
}
}
```
---
## Appendices
### Appendix A: Configuration Files Reference
| Path | Purpose | Format |
|------|---------|--------|
| `/etc/sysmngr/fwbank` | Bank management script | Shell script |
| `/etc/config/cwmp` | TR-069 config defaults | UCI |
| `/etc/config/dhcp-on-boarding` | DHCP upgrade settings | UCI |
| `/usr_data/opconf/opconf.json` | Operator configuration | JSON |
| `/usr_data/userconf/userconf.json` | User configuration (temp) | JSON |
| `/usr_data/userconf/keep.d/` | Backed up config files | Directory tree |
| `/tmp/sysupgrade.tgz` | Full config backup | Tarball |
| `/tmp/firmware-XXXXXX` | Downloaded firmware | Binary |
| `/etc/crontabs/root` | Scheduled activations | Crontab |
### Appendix B: Event Types
| Event | Source | Payload | Purpose |
|-------|--------|---------|---------|
| `sysupgrade` | fwbank | `{"bank_id": "2", "status": "Available"}` | Upgrade progress |
| `transfer_complete` | sysmngr | BBF TransferComplete format | Notify controller |
**Example sysupgrade event:**
```bash
ubus listen sysupgrade
# Output during upgrade:
{ "sysupgrade": { "bank_id": "2", "status": "Downloading" } }
{ "sysupgrade": { "bank_id": "2", "status": "Available" } }
```
### Appendix C: Error Codes
| Code | Message | Cause | Solution |
|------|---------|-------|----------|
| 9001 | Command failure | Generic operation error | Check logs, retry |
| 9007 | Invalid arguments | Bad parameter value | Validate input |
| 9010 | Resources exceeded | Out of memory/storage | Free space, retry |
| - | "Invalid firmware image" | sysupgrade -T failed | Check image format |
| - | "Checksum mismatch" | File corrupted | Re-download |
| - | "Download failed" | HTTP error | Check URL/credentials |
### Appendix D: Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2024-01 | Initial dual-bank support |
| 1.1 | 2024-06 | Added opconf/userconf separation |
| 1.1.3 | 2024-11 | Enhanced BBF USP support, scheduled activation |
---
## Quick Reference
### Default Configuration Values
```
keep_config: 1 (preserve)
keep_opconf: 1 (preserve)
config_scope: "UserOnly"
auto_activate: 0 (manual)
reboot: 0 (manual)
```
### Common Commands
#### Using dmcli (BBF Datamodel - Recommended)
```bash
# Check firmware status
dmcli get Device.DeviceInfo.FirmwareImage.
# Upgrade with full config preservation
dmcli do Device.DeviceInfo.FirmwareImage.2.Download() \
URL=https://fw.example.com/firmware.itb \
AutoActivate=1 \
X_IOWRT_EU_KeepConfig=1 \
X_IOWRT_EU_KeepOpConf=1 \
X_IOWRT_EU_ConfigScope=All
# Upgrade with operator reset, keep user settings
dmcli do Device.DeviceInfo.FirmwareImage.2.Download() \
URL=https://fw.example.com/firmware.itb \
AutoActivate=1 \
X_IOWRT_EU_KeepConfig=1 \
X_IOWRT_EU_KeepOpConf=0 \
X_IOWRT_EU_ConfigScope=UserOnly
# Factory reset upgrade
dmcli do Device.DeviceInfo.FirmwareImage.2.Download() \
URL=https://fw.example.com/firmware.itb \
AutoActivate=1 \
X_IOWRT_EU_KeepConfig=0
```
#### Using fwbank UBUS (Direct Access)
```bash
# Check current status
ubus call fwbank dump
# Upgrade with full config preservation
ubus call fwbank upgrade '{"path":"/tmp/fw.itb","bank":2,"auto_activate":true,"reboot":true,"keep_settings":true,"keep_opconf":true,"config_scope":"All"}'
# Upgrade with operator reset, keep user settings
ubus call fwbank upgrade '{"path":"/tmp/fw.itb","bank":2,"auto_activate":true,"keep_settings":true,"keep_opconf":false,"config_scope":"UserOnly"}'
# Factory reset upgrade
ubus call fwbank upgrade '{"path":"/tmp/fw.itb","bank":2,"auto_activate":true,"keep_settings":false}'
# Check upgrade events
logread -f | grep -E "sysupgrade|fwbank|opconf"
```
### Decision Tree: Which Config Parameters?
```
Are you upgrading via...
├─ USP/TR-369 Protocol
│ ├─ Sulu Web GUI
│ │ └─ Toggle: "Keep Settings" checkbox in File Upload mode
│ │ Basic control, KeepConfig only (URL download uses defaults)
│ │
│ ├─ dmcli (Data Model CLI)
│ │ └─ Use: X_IOWRT_EU_KeepConfig, X_IOWRT_EU_KeepOpConf, X_IOWRT_EU_ConfigScope
│ │ Full control via operation parameters
│ │ Works locally (UBUS) or remotely (USP)
│ │ Example: dmcli do Device.DeviceInfo.FirmwareImage.2.Download() URL=... AutoActivate=1 X_IOWRT_EU_KeepConfig=1
│ │
│ └─ External USP Controller
│ └─ Use: X_IOWRT_EU_KeepConfig, X_IOWRT_EU_KeepOpConf, X_IOWRT_EU_ConfigScope
│ Per-operation control, full flexibility
│
├─ TR-069 ACS (CWMP)
│ └─ Configure: /etc/config/cwmp (KeepConfig, KeepOpConf, ConfigScope)
│ Firmware defaults, applies to all upgrades
│
├─ DHCP Onboarding
│ └─ Use: reset_mode=full (or omit)
│ Limited control, only factory reset vs preserve
│
└─ CLI (fwbank)
└─ Use: keep_settings, keep_opconf, config_scope parameters
Direct control, all options available
```
---
**Document Version:** 1.0
**Last Updated:** 2025-11-26
**Author:** iopsys Technical Documentation
**Contact:** support@iopsys.eu
Reference in a new issue View git blame Copy permalink