BroadBand Forum Data Models (BBFDM)

bbfdm is a data model library implementation which includes a list of objects, parameters and operates used for CPE management through remote control protocols such as TR-069/CWMP or TR-369/USP.

Design of bbfdm

bbfdm library is structred as follow :

├── dm...(.c and .h)
├── dmtree
│   ├── json
│   ├── tr104
│   ├── tr143
│   ├── tr181
│   └── vendor
│       ├── iopsys
│       ├── openwrt
│       └── vendor.h
├── libbbf_api
├── scripts
└── tools

• dmtree folder which includes all supported Data Models. It contains 5 folders:

  • tr181 folder : TR-181 Data Model files

  • tr104 folder : Voice Services Data Model files

  • tr143 folder : Diagnostics Data Model files

  • vendor folder : Vendor Data Model files

  • json folder : TR-181 and TR-104 JSON files

• libbbf_api folder which contains the source code of all API functions (UCI, Ubus, JSON, CLI and memory management)

• scripts folder which contains the Diagnostics scripts

• tools folder which contains some tools to generate Data Model in C, JSON, XML and Excel format

• dm...(.c and .h) files which contains the bbfdm engine (operate, diagnostics) functions

How to add support for a new Object/Parameter

As mentioned above, all Data Models are stored in the 'dmtree' folder. In order to implement a new object/parameter, you need to expand its get/set/add/delete functions and then save them in the rigth folder.

bbfdm library offers a tool to generate templates of the source code from json files placed under 'dmtree/json'. So, any developer can fill these json files (tr181 or tr104) with mapping field according to UCI, UBUS or CLI commands then generate the source code in C.

$ python generate_dm_c.py
Usage: generate_dm_c.py <data model name> [Object path]
data model name:   The data model(s) to be used, for ex: tr181 or tr181,tr104
Examples:
  - generate_dm_c.py tr181
    ==> Generate the C code of tr181 data model in datamodel/ folder
  - generate_dm_c.py tr104
    ==> Generate the C code of tr104 data model in datamodel/ folder
  - generate_dm_c.py tr181,tr104
    ==> Generate the C code of tr181 and tr104 data model in datamodel/ folder
  - generate_dm_c.py tr181 Device.DeviceInfo.
    ==> Generate the C code of Device.DeviceInfo object in datamodel/ folder
  - generate_dm_c.py tr104 Device.Services.VoiceService.{i}.Capabilities.
    ==> Generate the C code of Device.Services.VoiceService.{i}.Capabilities. object in datamodel/ folder

Below some examples of UCI, UBUS or CLI mappings:

UCI command

• @Name: the section name of paraent object

• @i: is the number of instance object

	"mapping": [
		{
			"type": "uci",
			"uci": {
				"file": "wireless",
				"section": {
					"type": "wifi-device",
					"name": "@Name",
					"index": "@i-1"
				},
				"option": {
					"name": "disabled"
				}
			}
		}
	]

UBUS command

• @Name: the section name of paraent object

	"mapping": [
		{
			"type": "ubus",
			"ubus": {
				"object": "network.device",
				"method": "status",
				"args": {
					"name": "@Name"
				},
				"key": "statistics.rx_bytes"
			}
		}
	]

CLI command:

• @Name: the section name of paraent object

• -i: is the number of arguments command

	"mapping": [
		{
			"type" : "cli",
			"cli" : {
				"command" : "wlctl",
				"args" : [
					"-i",
					"@Name",
					"bands"
				]
			}
		}
	]

After building the templates of C source code, a datamodel folder will be generated under 'tools' folder that contains all files related to each object under root "Device."

Note: You can generate the source code without filling out the mapping field in the JSON file

Object definition

Each object in the DMOBJ table contains the following arguments:

Argument	Description
OBJ	A string of the object name. Example “Bridging”, “IP”, “DeviceInfo”, “WiFi”
permission	The permission of the object. Could be &DMREAD or &DMWRITE. If it's &DMWRITE then we can add/delete instances of this object
addobj	The function to add new instance under this object. This function will be triggered when the ACS/Controller call AddObject of this object
delobj	The function to delete instance under this object. This function will be triggered when the ACS/Controller call DeleteObject of an instance of this object
checkdep	A string of the object dependency, it can be a file("file:/etc/config/network") or an ubus object,method("ubus:network.interface->status"). If it's NULL then the object has always appeared in the tree
browseinstobj	This function allow to browse all instances under this object
nextdynamicobj	Pointer to the next of DMOBJ which contains a list of the child objects using json files, shared libraries and vendor extension
dynamicleaf	Pointer to the next of DMLEAF which contains a list of the child parameters using json files, shared libraries and vendor extension
nextobj	Pointer to a DMOBJ array which contains a list of the child objects
leaf	Pointer to a DMLEAF array which contains a list of the child parameters
linker	This argument is used for LowerLayer parameters or to make reference to other instance object in the tree
bbfdm_type	The bbfdm type of the object. Could be BBFDM_CWMP, BBFDM_USP, BBFDM_BOTH or BBFDM_NONE.If it's BBFDM_BOTH then we can see this object in all protocols (CWMP, USP,...)

Parameter definition

Each parameter in the DMLEAF table contains the following arguments:

Argument	Description
PARAM	A string of the parameter name. Example “Enable”, “Status”, “Name”
permission	The permission of the parameter. Could be &DMREAD or &DMWRITE.If it's &DMWRITE then we can set a value for this parameter
type	Type of the parameter: DM_STRING, DM_BOOL, DM_UNINT,...
getvalue	The function which return the value of this parameter
setvalue	The function which set the value of this parameter
bbfdm_type	The bbfdm type of the parameter. Could be BBFDM_CWMP, BBFDM_USP, BBFDM_BOTH or BBFDM_NONE.If it's BBFDM_BOTH then we can see this parameter in all protocols (CWMP, USP,...)

BBF API

libbbf_api is a library which contains the source code of all API functions (UCI, Ubus, JSON, CLI and memory management). these API are used for GET/SET/ADD/Delete/Operate calls which can be called in internal or external packages.

The most used one are as follow:

1. dmuci_get_option_value_string: execute the uci get value

int dmuci_get_option_value_string(char *package, char *section, char *option, char **value)

Argument:

• package: package name
• section: section name
• option: option name
• value: the value of the returned option

2. dmuci_get_value_by_section_string: execute the uci get value

int dmuci_get_value_by_section_string(struct uci_section *s, char *option, char **value)

Argument:

• section: section name
• option: option name
• value: the value of the returned option

3. uci_foreach_sections: browse all sections by package and section type

#define uci_foreach_sections(package, stype, section)

Argument:

• package: package name
• stype: section type to browse
• section: return section pointer for each loop iteration

4. dmubus_call: execute the ubus call

int dmubus_call(char *obj, char *method, struct ubus_arg u_args[], int u_args_size, json_object **req_res)

Argument:

• obj: ubus obj
• method: ubus method
• u_args: ubus arguments
• u_args_size: number of ubus arguments
• req_res: the json message of the ubus call

5. dmubus_call_set: set the ubus call

int dmubus_call_set(char *obj, char *method, struct ubus_arg u_args[], int u_args_size);

Argument:

• obj: ubus obj
• method: ubus method
• u_args: ubus arguments
• u_args_size: number of ubus arguments

Note1: For other funtions, please refer to dmuci, dmubus, dmjson, dmcommon and dmmem (.c and .h) files in the link

Note2: When developing a new parameters/features in the Data Model, it's highly recommended to use the memory management functions of libbbf_api allocate and free because it's freed at the end of each RPCs.

The list of memory management functions of libbbf_api are:

dmmalloc(x)
dmcalloc(n, x)
dmrealloc(x, n)
dmstrdup(x)
dmasprintf(s, format, ...)
dmastrcat(s, b, m)
dmfree(x)

BBFDM Vendor

bbfdm library can be used to Extend the Data Model with new objects/parameters, to Overwrite existing objects/parameters with new ones and Exclude some objects/parameters from Data Model tree.

How to add new vendor

1. Create a vendor folder

Create a new folder under 'dmtree/vendor/' which contains all files related to the vendor

2. Fill Extend, Overwrite and Exclude tables with objects/parameters

Create the first vendor C file which contains new Extend, Overwrite and Exclude tables of objects/parameters.

Extend and Overwrite table

The Extend and Overwrite tables contain entries of DM_MAP_OBJ structure.

The DM_MAP_OBJ structure contains three arguments:

Argument	Description
parentobj	A string of the parent object name. Example “Device.IP.Diagnostics.”, “Device.DeviceInfo”, “Device.WiFi.Radio.”
nextobject	Pointer to a DMOBJ array which contains a list of the child objects
parameter	Pointer to a DMLEAF array which contains a list of the child parameters

Exclude table

Each entry in the exclude table is a string which could be a path of object or parameter that need to be excluded from the tree

The following link contains example of Extend, Overwrite and Exclude table.

3. Adding vendor and standard objects/Parameters

Implement the new vendor/standard objects and parameters as defined above in the first section.

Example: Custom Vendor Object Dropbear

4. link vendor tables to the main tree

To register the new vendor tables, you need to link them in the main three tables:

• tVendorExtension

• tVendorExtensionOverwrite

• tVendorExtensionExclude

These tables are defined in the file 'dmtree/vendor/vendor.c'.

Example: Link vendor tables to the main tree

5. Enable vendor

To enable the new vendor

• Define BBF_VENDOR_EXTENSION macro

• Add the new vendor in the list BBF_VENDOR_LIST macro

• Define the vendor prefix using BBF_VENDOR_PREFIX macro

Example of Config Options:

BBF_VENDOR_EXTENSION=y
BBF_VENDOR_LIST="iopsys,test"
BBF_VENDOR_PREFIX="X_TEST_COM_"

Note1: The libbbfdm vendor list can support multi-vendor with comma seperated.

Note2: If multi vendors are supported and there is a object/parameter that is implmented by multi customers in different way, the implemented object/parameter of the first vendor name in the BBF_VENDOR_LIST will be considered.

Note3: Overwrite and Exclude are only considered in dmtree/vendor/<vendor>/

• The directory 'dmtree/vendor/test/' contains an example of test vendor implementation

BBFDM Dynamic Object/Parameter/Operate

bbfdm library allows all applications installed on the box to import its own Data Model parameters at run time in two formats:

• Shared library

• JSON files

1. Shared library via external package

The application should bring its shared library under '/usr/lib/bbfdm/' path that contains the sub tree of Objects/Parameters and the related functions Get/Set/Add/Delete/Operate. The new added objects, parameters and operates will be automatically shown by icwmpd and uspd/obuspa.

Each library should contains two Root tables:

• “tDynamicObj”

• “tDynamicOperate”

DynamicObject definition

The “tDynamicObj” table contains entries of DM_MAP_OBJ structure.

The DM_MAP_OBJ structure contains three arguments:

Argument	Description
parentobj	A string of the parent object name. Example “Device.IP.Diagnostics.”, “Device.DeviceInfo”, “Device.WiFi.Radio.”
nextobject	Pointer to a DMOBJ array which contains a list of the child objects
parameter	Pointer to a DMLEAF array which contains a list of the child parameters

DynamicOperate definition

The “tDynamicOperate” table contains entries of DM_MAP_OPERATE structure.

The DM_MAP_OPERATE structure contains two arguments:

Argument	Description
pathname	A string of the path name operate. Example “Device.BBKSpeedTest”, “Device.WiFi.AccessPoint.*.X_IOPSYS_EU_Reset”
operation	The function to be executed in the operation
type	The type of the operation. Could be sync or async
args	The input and output arguments of the operation

For the other tables, they are defined in the same way as the Object and Parameter definition described above.

Note1: Shared library can only add vendor or standard objects that are not implemented by libbbfdm

Note2: Shared library is not allowed to overwrite objects/parameters

• For more examples on the external packages, you can see these links: BulkData, XMPP

2. JSON File via external package

The application should bring its JSON file under '/etc/bbfdm/json/' path with UCI and UBUS mappings. The new added parameters will be automatically shown by icwmpd and uspd/obuspa.

Some examples on JSON Definition

1. Object without instance:

"Device.CWMP.": {
    "type": "object", 
    "protocols": [
        "cwmp", 
        "usp"
    ], 
    "array": false
}

2. Object with instace:

• UCI command: uci show wireless | grep wifi-device

"Device.X_IOPSYS_EU_Radio.{i}.": {
    "type": "object", 
    "protocols": [
        "cwmp", 
        "usp"
    ], 
    "array": true,
    "mapping": {
        "type": "uci", 
        "uci": {
            "file": "wireless", 
            "section": {
                "type": "wifi-device"
            }, 
            "dmmapfile": "dmmap_wireless"
        }
    }
}

• UBUS command: ubus call dsl status | jsonfilter -e @.line

"Device.DSL.Line.{i}.": {
	"type": "object", 
	"protocols": [
		"cwmp", 
		"usp"
	], 
	"array": true,
	"mapping": {
		"type": "ubus", 
		"ubus": {
			"object": "dsl", 
			"method": "status", 
			"args": {}, 
			"key": "line"
		}
	}
}

3. Parameter under object with instance:

• UCI command: uci get wireless.@wifi-device[0].country

• @i: is the number of instance object

"Country": {
	"type": "string", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": true, 
	"mapping": [
		{
			"type" : "uci",
			"uci" : {
				"file" : "wireless",
				"section" : {
					"type": "wifi-device",
					"index": "@i-1"
				},
				"option" : {
					"name" : "country"
				}
			}
		}
	]
}

• UBUS command: ubus call network.interface status '{"interface":"lan"}' | jsonfilter -e @.device

• @Name: the section name of paraent object, in this example, the section name is "lan"

"SSID": {
	"type": "string", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": false,
	"mapping": [
		{
			"type" : "ubus",
			"ubus" : {
				"object" : "network.interface",
				"method" : "status",
				"args" : {
					"interface" : "@Name"
				},
				"key" : "device"
			}
		}
	]
}

• UBUS command: ubus call wifi status | jsonfilter -e @.radios[0].noise

"Noise": {
	"type": "int", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": false,
	"mapping": [
		{
			"type" : "ubus",
			"ubus" : {
				"object" : "wifi",
				"method" : "status",
				"args" : {},
				"key" : "radios[i-1].noise"
			}
		}
	]
}

4. Parameter without instance:

• UCI command: uci get cwmp.cpe.userid

"Username": {
	"type": "string", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": true, 
	"mapping": [
		{
			"type" : "uci",
			"uci" : {
				"file" : "cwmp",
				"section" : {
					"type": "cwmp",
	      				"name": "cpe"
	       			},
				"option" : {
					"name" : "userid"
				}
			}
		}
	]
}

• UBUS command: ubus call system info | jsonfilter -e @.uptime

"Uptime": {
	"type": "unsignedInt", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": false,
	"mapping": [
		{
			"type" : "ubus",
			"ubus" : {
				"object" : "system",
				"method" : "info",
				"args" : {},
				"key" : "uptime"
			}
		}
	]
}

• UBUS command: ubus call system info | jsonfilter -e @.memory.total

"Total": {
	"type": "unsignedInt", 
	"protocols": [
		"cwmp", 
		"usp"
	],
	"read": true, 
	"write": false,
	"mapping": [
		{
			"type" : "ubus",
			"ubus" : {
				"object" : "system",
				"method" : "info",
				"args" : {},
				"key" : "memory.total"
			}
		}
	]
}

Note1: JSON File can only add vendor or standard objects that are not implemented by libbbfdm

Note2: JSON File is not allowed to overwrite objects/parameters

• For more examples on JSON files, you can see these links: X_IOPSYS_EU_MCPD, UserInterface, X_IOPSYS_EU_Dropbear

BBFDM Tools

XML generator

It is a generator of Data Model tree in XML with two format: Broadband Forum schema and HDM.

$ python generate_dm_xml.py -h
Usage: generate_dm_xml.py [options...] <urls>
Options: 
 -r, --remote-dm              Check OBJ/PARAM under these repositories if it is not found under bbf repo
 -v, --vendor-list            Generate data model tree with vendor extension OBJ/PARAM
 -p, --vendor-prefix          Generate data model tree using this vendor prefix. Default vendor prefix: X_IOPSYS_EU_
 -f, --format                 Generate data model tree with HDM format. Default format: BBF
 -d, --device-protocol        Generate data model tree using this device protocol. Default device protocol: DEVICE_PROTOCOL_DSLFTR069v1
 -m, --manufacturer           Generate data model tree using this manufacturer. Default manufacturer: iopsys
 -o, --manufacturer-oui       Generate data model tree using this manufacturer oui. Default manufacturer oui: 002207
 -c, --product-class          Generate data model tree using this product class. Default product class: DG400PRIME
 -n, --model-name             Generate data model tree using this model name. Default model name: DG400PRIME-A
 -s, --software-version       Generate data model tree using this software version. Default software version: 1.2.3.4
 -h, --help                   This help text
Urls: 
 url^(branch,hash,tag)        The url with branch, hash or tag to be used

Examples: 
  - python generate_dm_xml.py
    ==> Generate xml file in datamodel.xml
  - python generate_dm_xml.py -f HDM
    ==> Generate xml file with HDM format in datamodel.xml
  - python generate_dm_xml.py -v iopsys
    ==> Generate xml file using iopsys extension in datamodel.xml
  - python generate_dm_xml.py -r https://dev.iopsys.eu/feed/iopsys.git^devel,https://dev.iopsys.eu/iopsys/mydatamodel.git^5c8e7cb740dc5e425adf53ea574fb529d2823f88
    ==> Generate xml file in datamodel.xml
  - python generate_dm_xml.py -v iopsys,openwrt,test -r https://dev.iopsys.eu/feed/iopsys.git^6.0.0ALPHA1 -p X_TEST_COM_
    ==> Generate xml file in datamodel.xml

JSON generator

It is a generator of Data Model JSON format

$ python generate_dm_json.py 
Usage: generate_dm_json.py <tr-xxx cwmp xml data model> <tr-xxx usp xml data model> [Object path]
Examples:
  - generate_dm_json.py tr-181-2-14-1-cwmp-full.xml tr-181-2-14-1-usp-full.xml Device.
    ==> Generate the json file of the sub tree Device. in tr181.json
  - generate_dm_json.py tr-104-2-0-2-cwmp-full.xml tr-104-2-0-2-usp-full.xml Device.Services.VoiceService.
    ==> Generate the json file of the sub tree Device.Services.VoiceService. in tr104.json
  - generate_dm_json.py tr-106-1-2-0-full.xml Device.
    ==> Generate the json file of the sub tree Device. in tr106.json

Example of xml data model file: https://www.broadband-forum.org/cwmp/tr-181-2-14-1-cwmp-full.xml

Excel generator

It is a generator of excel sheet with supported and unsupported Data Model tree.

$ python generate_dm_excel.py
Usage: generate_dm_excel.py <data model name> [options...] <urls>
data model name:              The data model(s) to be used, for ex: tr181 or tr181,tr104
Options: 
 -r, --remote-dm              Check OBJ/PARAM under these repositories if it is not found under bbf repo
 -v, --vendor-list            Generate data model tree with vendor extension OBJ/PARAM
 -p, --vendor-prefix          Generate data model tree using this vendor prefix. Default vendor prefix: X_IOPSYS_EU_
 -h, --help                   This help text
Urls: 
 url^(branch,hash,tag)        The url with branch, hash or tag to be used

Examples: 
  - python generate_dm_excel.py tr181
    ==> Generate excel file in datamodel.xls
  - python generate_dm_excel.py tr104
    ==> Generate excel file in datamodel.xls
  - python generate_dm_excel.py tr181,tr104 -r https://dev.iopsys.eu/feed/iopsys.git^release-5.3,https://dev.iopsys.eu/iopsys/mydatamodel.git^5c8e7cb740dc5e425adf53ea574fb529d2823f88
    ==> Generate excel file in datamodel.xls
  - python generate_dm_excel.py tr181,tr104 -v iopsys,openwrt,test -r https://dev.iopsys.eu/feed/iopsys.git^6.0.0ALPHA1 -p X_TEST_COM_
    ==> Generate excel file in datamodel.xls

Dependencies

To successfully build libbbfdm, the following libraries are needed:

Dependency	Link	License
libuci	https://git.openwrt.org/project/uci.git	LGPL 2.1
libubox	https://git.openwrt.org/project/libubox.git	BSD
libubus	https://git.openwrt.org/project/ubus.git	LGPL 2.1
libjson-c	https://s3.amazonaws.com/json-c_releases	MIT
libcurl	https://dl.uxnr.de/mirror/curl	MIT
libtrace	https://github.com/apietila/libtrace.git	GPLv2
libbbf_api	https://dev.iopsys.eu/iopsys/bbf.git	LGPL 2.1