# Datamodel diagnostics and support tools bbfdm offers several tools/utilities to facilitate(s) - Generation of json based datamodel definition from broadband forum xml based definition - Generation of "C" code templates based on json definition - Generate xml definition of supported datamodel for Nokia HDM ACS - Generate xml definition of supported datamodel for other ACS - Generate list of supported data models in XML format for both USP and CWMP variants - Provide tools to validate JSON based datamodel plugins ## Dependencies Tools are mostly written in python and shell script, some requires docker images - python3-pip - libxml2-utils - docker.io - jsonschema - xlwt - ubus System utilities: python3-pip, libxml2-utils docker.io ```bash $ sudo apt install -y python3-pip $ sudo apt install -y libxml2-utils ``` Python utilities: jsonschema, xlwt, ubus ```bash $ pip3 install jsonschema xlwt ubus ``` To install docker follow [external link](https://docs.docker.com/engine/install) ## Tools Below are the list of tools | Tools | Description | | ------------------------- | ------------------------------------------------------------ | | convert_dm_xml_to_json.py | Tool to convert Broadband forum's xml based datamodel definition to JSON based datamodel Definition | | convert_dm_json_to_c.py | Tool to generate json based datamodel definition with ubus/uci mappings to C code | | validate_json_plugin.py | Validate json based datamodel plugin files | | generate_dm.sh | Generate list of supported/un-supported parameters based of json input| ### convert_dm_xml_to_json.py [Broadband Forum](https://www.broadband-forum.org/) provides TR181 and other datamodel definitions in two formats xml(machine friendly) and html(User friendly), - [CWMP Specific datamodels](https://cwmp-data-models.broadband-forum.org/#sec:current-data-models) - [USP specific datamodels](https://usp-data-models.broadband-forum.org/#sec:current-data-models) In bbfdm, we needed a unified file which can be used for machine translations as well as at the same time readable to humans, so we provide a this tools to convert Data Model from Broadband Forum XML format to JSON format. This tools can be used as shown below ```bash $ ./tools/convert_dm_xml_to_json.py Usage: python convert_dm_xml_to_json -d Options: -d, --directory : Directory containing XML files to convert to JSON Example: ./tools/convert_dm_xml_to_json.py -d test/tools/ ==> Generate the JSON file containing of all XML files defined under test/tools/ directory in datamodel.json Example of xml data model file: https://www.broadband-forum.org/cwmp/tr-181-2-*-cwmp-full.xml ``` ### convert_dm_json_to_c.py To add the datamodel via bbfdm, it is required to follow [datamodel guide](https://dev.iopsys.eu/bbf/bbfdm/-/blob/devel/docs/guide/datamodel_as_microservice.md), which allows to add the datamodel with json plugins, or with DotSO plugins. This tool can generate template "C" code from JSON datamodel definitions. ```bash $ ./tools/convert_dm_json_to_c.py Usage: ./tools/convert_dm_json_to_c.py [Object path] Examples: - ./tools/convert_dm_json_to_c.py ==> Generate the C code of full data model in datamodel/ folder - ./tools/convert_dm_json_to_c.py Device.DeviceInfo. ==> Generate the C code of Device.DeviceInfo object in datamodel/ folder ``` ### validate_json_plugin.py This tool helps in validating the json schema, which is very helpful in the development of a JSON based plugins. ```bash $ ./tools/validate_json_plugin.py test/files/etc/bbfdm/json/X_IOPSYS_EU_TEST.json $ ./tools/validate_json_plugin.py tools/datamodel.json ``` More examples available in [this path](../test/files/usr/share/bbfdm/plugins). ### generate_dm.sh This tool generates the list of supported datamodel objects/parameters in xml and xls format, based on the input. Historically bbfdm tools used to do text parsing to provide list of supported datamodel parameters, which has many limitations: - Strict binding of datamodel definitions - Need to maintain specific sequence in definition This improved tool usages an docker image to get the list of supported datamodel to provide the accurate output. #### How this works Based on plugins listed in tools_input.json file, it simulates a runtime environment with docker image and get the datamodel from `bbfdm` ubus object(exposed by bbfdmd) which gets the data from all supported plugins and microservices. ```bash Usage: ./tools/generate_dm.sh [OPTIONS]... -I -i json input file path relative to top directory -h help examples: ~/git/bbfdm$ ./tools/generate_dm.sh -i tools/tools_input.json ``` The parameters/keys used in tools_input.json file are mostly self-explanatory but few parameters are required a bit more details. | Key | Description | |-----|-------------| | manufacturer | The manufacturer's name, e.g., "IOPSYS" | | protocol | The device protocol, e.g., "DEVICE_PROTOCOL_DSLFTR069v1 | | manufacturer_oui | The Manufacturer's Organizationally Unique Identifier (OUI) in hexadecimal format, e.g., "002207" | | product_class" | The product class, e.g., "DG400PRIME" | | model_name | The model name, e.g., "DG400PRIME-A" | | software_version | The software version, e.g., "1.2.3.4" | | dm_json_files | This should contain the list of json file path, where each file contains the definition of DM objects/parameters | | vendor_prefix | The prefix used by vendor for vendor extension in DM objects/parameters, e.g., "X_IOPSYS_EU_" | | plugins | A list of plugins with associated repositories and data model files | | | repo: The path of the plugin repository. Could be 'URL' or 'folder_path' | | | proto: The protocol of the plugin repository. Could be 'git' or 'local' | | | version: (optional): The version of the git plugin | | | dm_files: A list of data model files associated with the plugin | | | extra_dependencies: (optional): Extra dependencies for the plugin, if any | | output.acs | Currently the tool support two variants of xml definitions of DM objects/parameters | | | hdm: This variant of xml is compatible with Nokia HDM ACS | | | default: This contains the generic definition which has the capability to define more descriptive DM objects/parameters | | output.file_format | Output file formats, e.g., ["xls", "xml"] | | output.output_dir | The output directory for generated files, e.g., "./out" | | output.output_file_prefix | The prefix for output file names, e.g., "datamodel" | > Note: > To add more description about the vendor extended DM objects/parameters, it is required to add the definition of the required/related DM objects/parameters in a json file (The json structure should follow same format as given in [datamodel.json](datamodel.json)), The same json file need to be defined in dm_json_files list. The input json file should be defined as follow: ```bash { "manufacturer": "iopsys", "protocol": "DEVICE_PROTOCOL_DSLFTR069v1", "manufacturer_oui": "002207", "product_class": "DG400PRIME", "model_name": "DG400PRIME-A", "software_version": "1.2.3.4", "dm_json_files": [ "tools/datamodel.json" ] "vendor_prefix": "X_IOPSYS_EU_", "plugins": [ { "repo": "https://dev.iopsys.eu/bbf/mydatamodel.git", "proto": "git", "version": "tag/hash/branch", "dm_files": [ "src/datamodel.c", "src/additional_datamodel.c" ] }, { "repo": "https://dev.iopsys.eu/bbf/mybbfplugin.git", "proto": "git", "version": "tag/hash/branch", "dm_files": [ "dm.c" ] }, { "repo": "https://dev.iopsys.eu/bbf/mydatamodeljson.git", "proto": "git", "version": "tag/hash/branch", "dm_files": [ "src/plugin/testdm.json" ] }, { "repo": "/home/iopsys/sdk/mypackage/", "proto": "local", "dm_files": [ "src/datamodel.c", "additional_datamodel.c" ] }, { "repo": "/src/feeds/mypackage/", "proto": "local", "dm_files": [ "datamodel.c", "src/datamodel.json" ] } ], "output": { "acs": [ "hdm", "default" ], "file_format": [ "xml", "xls" ], "output_dir": "./out", "output_file_prefix": "datamodel" } } ``` --- **NOTE** > All the tools need to be executed from the top directory. ---