mirror/bbfdm
1
0
Fork 0
mirror of https://dev.iopsys.eu/bbf/bbfdm.git synced 2025-12-10 07:44:39 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Improve bbf tools

Browse source
This commit is contained in:
Amin Ben Ramdhane 2021-10-11 18:07:54 +01:00
parent 9c27d482a6
commit 7f6f4e36a0
6 changed files with 175 additions and 80 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
README.md
View file

@ -735,14 +735,14 @@ Example of xml data model file: https://www.broadband-forum.org/cwmp/tr-181-2-14
```bash
$ ./generate_dm_xml.py -h
usage: generate_dm_xml.py [-h] [-r https://dev.iopsys.eu/iopsys/stunc.git^devel] [-v iopsys] [-p X_IOPSYS_EU_] [-d DEVICE_PROTOCOL_DSLFTR069v1] [-m iopsys] [-u 002207] [-c DG400PRIME] [-n DG400PRIME-A]
usage: generate_dm_xml.py [-h] [-r git^https://dev.iopsys.eu/iopsys/stunc.git^devel] [-v iopsys] [-p X_IOPSYS_EU_] [-d DEVICE_PROTOCOL_DSLFTR069v1] [-m iopsys] [-u 002207] [-c DG400PRIME] [-n DG400PRIME-A]
[-s 1.2.3.4] [-f BBF] [-o datamodel.xml]
Script to generate list of supported and non-supported parameter in xml format
optional arguments:
-h, --help show this help message and exit
-r https://dev.iopsys.eu/iopsys/stunc.git^devel, --remote-dm https://dev.iopsys.eu/iopsys/stunc.git^devel
-r git^https://dev.iopsys.eu/iopsys/stunc.git^devel, --remote-dm git^https://dev.iopsys.eu/iopsys/stunc.git^devel
Includes OBJ/PARAM defined under remote repositories defined as bbf plugin
-v iopsys, --vendor-list iopsys
Generate data model tree with vendor extension OBJ/PARAM.
@ -770,21 +770,25 @@ Part of BBF-tools, refer Readme for more examples
More examples:
```bash
$ ./generate_dm_xml.py -v iopsys -v openwrt
$ ./generate_dm_xml.py -v iopsys -p X_IOPSYS_EU_ -r git^https://dev.iopsys.eu/iopsys/stunc.git^devel
$ ./generate_dm_xml.py -f HDM -v iopsys -p X_IOPSYS_EU_ -o iopsys.xml
```
> Note: For the remote data model, *git* is the only proto allowed to use in the *generate_dm_xml.py* script. Therefore, if you want to use vendor extensions from a local repository, you must use the *generate_dm.py* script.
### Excel generator
[Python script](./tools/generate_dm_excel.py) to generat list of supported and un-supported parameters in excel sheet.
```bash
$ ./generate_dm_excel.py -h
usage: generate_dm_excel.py [-h] -d tr181 [-r https://dev.iopsys.eu/iopsys/stunc.git^devel] [-v iopsys] [-p X_IOPSYS_EU_] [-o supported_datamodel.xls]
usage: generate_dm_excel.py [-h] -d tr181 [-r git^https://dev.iopsys.eu/iopsys/stunc.git^devel] [-v iopsys] [-p X_IOPSYS_EU_] [-o supported_datamodel.xls]
Script to generate list of supported and non-supported parameter in xls format
optional arguments:
-h, --help show this help message and exit
-d tr181, --datamodel tr181
-r https://dev.iopsys.eu/iopsys/stunc.git^devel, --remote-dm https://dev.iopsys.eu/iopsys/stunc.git^devel
-r git^https://dev.iopsys.eu/iopsys/stunc.git^devel, --remote-dm git^https://dev.iopsys.eu/iopsys/stunc.git^devel
Includes OBJ/PARAM defined under remote repositories defined as bbf plugin
-v iopsys, --vendor-list iopsys
Generate data model tree with vendor extension OBJ/PARAM
@ -799,7 +803,8 @@ Part of BBF-tools, refer Readme for more examples
More examples:
```bash
$ ./generate_dm_excel.py -d tr181 -v iopsys -v openwrt -o datamodel.xls
$ ./generate_dm_excel.py -d tr181 -d tr104 -v iopsys -v openwrt -o datamodel.xls
$ ./generate_dm_excel.py -d tr181 -d tr104 -v iopsys -o datamodel.xls
$ ./generate_dm_excel.py -d tr181 -v iopsys -p X_IOPSYS_EU_ -r git^https://dev.iopsys.eu/iopsys/xmppc.git^devel -o datamodel_iopsys.xls
```
### Data Model generator
@ -832,6 +837,7 @@ The input json file should be defined as follow:
"plugins": [
{
"repo": "https://dev.iopsys.eu/iopsys/mydatamodel.git",
"proto": "git",
"version": "tag/hash/branch",
"dm_files": [
"src/datamodel.c",
@ -840,6 +846,7 @@ The input json file should be defined as follow:
},
{
"repo": "https://dev.iopsys.eu/iopsys/mybbfplugin.git",
"proto": "git",
"version": "tag/hash/branch",
"dm_files": [
"dm.c"
@ -847,10 +854,27 @@ The input json file should be defined as follow:
},
{
"repo": "https://dev.iopsys.eu/iopsys/mydatamodeljson.git",
"proto": "git",
"version": "tag/hash/branch",
"dm_files": [
"src/plugin/datamodel.json"
]
},
{
"repo": "/home/iopsys/sdk/mypackage/",
"proto": "local",
"dm_files": [
"src/datamodel.c",
"additional_datamodel.c"
]
},
{
"repo": "/src/feeds/mypackage/",
"proto": "local",
"dm_files": [
"datamodel.c",
"src/datamodel.json"
]
}
],
"output": {
@ -868,6 +892,10 @@ The input json file should be defined as follow:
}
```
> Note1: For the local repository, you must use an absolute path as repo option.
> Note2: If proto is not defined in the json config file, then git is used by default as proto option.
- For more examples of tools input json file, you can see this link: [tools_input.json](./devel/tools/tools_input.json)
## Dependencies

152
tools/bbf_common.py
View file

@ -10,6 +10,7 @@ import json
from collections import OrderedDict
CURRENT_PATH = os.getcwd()
BBF_ERROR_CODE = 0
BBF_TR181_ROOT_FILE = "device.c"
BBF_TR104_ROOT_FILE = "servicesvoiceservice.c"
BBF_VENDOR_ROOT_FILE = "vendor.c"
@ -65,7 +66,7 @@ def create_folder(folder_name):
# rmtree exception handler
def rmtree_handler(_func, path, _exc_info):
print("Failed to remove %s" % path)
print(f'Failed to remove {path}')
def remove_folder(folder_name):
if os.path.isdir(folder_name):
@ -124,7 +125,7 @@ def fill_list_supported_dm():
def fill_data_model_file():
fp = open(DATA_MODEL_FILE, 'a', encoding='utf-8')
for value in LIST_SUPPORTED_DM:
print("%s" % value, file=fp)
print(f'{value}', file=fp)
fp.close()
@ -140,6 +141,10 @@ def generate_datamodel_tree(filename):
fp = open(filename, 'r', encoding='utf-8')
for line in fp:
line = line.lstrip()
if line.startswith(tuple(LIST_IGNORED_LINE)) is True:
continue
if "DMOBJ" in line:
table_name = line[:line.index('[]')].rstrip(
@ -156,10 +161,7 @@ def generate_datamodel_tree(filename):
if obj_found == 0 and param_found == 0:
continue
if line.startswith(tuple(LIST_IGNORED_LINE)) is True:
continue
if "{0}" in line:
if "{0}" in line.replace(" ", ""):
obj_found = 0
param_found = 0
obj_found_in_list = 0
@ -207,7 +209,7 @@ def generate_datamodel_tree(filename):
obj_found_in_list = 1
LIST_PARAM.remove(value)
param = line.rstrip('\n').split(", ")
param = line.rstrip('\n').split(",")
param_name = parent_obj + param[0].replace("{", "").replace(
"\"", "").replace("BBF_VENDOR_PREFIX", BBF_VENDOR_PREFIX).replace(" ", "")
param_permission = param[1].replace("&", "").replace(" ", "")
@ -227,6 +229,10 @@ def generate_dynamic_datamodel_tree(filename):
fp = open(filename, 'r', encoding='utf-8')
for line in fp:
line = line.lstrip()
if line.startswith(tuple(LIST_IGNORED_LINE)) is True:
continue
if "DM_MAP_OBJ" in line:
obj_found = 1
@ -234,24 +240,21 @@ def generate_dynamic_datamodel_tree(filename):
if obj_found == 0:
continue
if line.startswith(tuple(LIST_IGNORED_LINE)) is True:
continue
if "{0}" in line:
if "{0}" in line.replace(" ", ""):
obj_found = 0
continue
# Object Table
if obj_found == 1:
obj = line.rstrip('\n').split(", ")
obj_name = obj[0][1:].replace("\"", "")
obj_name = obj[0][1:].replace("\"", "").replace(" ", "").replace("BBF_VENDOR_PREFIX", BBF_VENDOR_PREFIX)
if obj[1] != "NULL":
LIST_OBJ.append(obj_name + ":" + obj[1])
if obj[2] != "NULL":
LIST_PARAM.append(obj_name + ":" + obj[2].replace("},", ""))
LIST_PARAM.append(obj_name + ":" + obj[2].replace("},", "").replace(" ", ""))
fp.close()
@ -335,7 +338,7 @@ def generate_supported_dm(vendor_prefix=None, vendor_list=None, plugins=None):
if vendor_list is not None and isinstance(vendor_list, list) and vendor_list:
cd_dir(BBF_DMTREE_PATH)
for vendor in vendor_list:
vendor_dir = "vendor/" + vendor + "/tr181"
vendor_dir = f'vendor/{vendor}/tr181'
if os.path.isdir(vendor_dir):
cd_dir(vendor_dir)
@ -352,7 +355,7 @@ def generate_supported_dm(vendor_prefix=None, vendor_list=None, plugins=None):
cd_dir(BBF_DMTREE_PATH)
vendor_dir = "vendor/" + vendor + "/tr104"
vendor_dir = f'vendor/{vendor}/tr104'
if os.path.isdir(vendor_dir):
cd_dir(vendor_dir)
@ -366,54 +369,103 @@ def generate_supported_dm(vendor_prefix=None, vendor_list=None, plugins=None):
cd_dir(BBF_DMTREE_PATH)
############## Download && Generate Plugins Data Models ##############
global BBF_ERROR_CODE
if plugins is not None and isinstance(plugins, list) and plugins:
print("Generating datamodels from defined plugins...")
cd_dir(CURRENT_PATH)
if isinstance(plugins, list):
for plugin in plugins:
proto = get_option_value(plugin, "proto")
repo = get_option_value(plugin, "repo")
version = get_option_value(plugin, "version")
remove_folder(".repo")
try:
subprocess.run(["git", "clone", repo, ".repo"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check = True)
except (OSError, subprocess.SubprocessError) as _e:
print("Failed to clone %s" % repo)
if repo is None:
BBF_ERROR_CODE += 1
continue
if version is not None:
try:
subprocess.run(["git", "-C", ".repo", "checkout", version],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True)
except (OSError, subprocess.SubprocessError) as _e:
print("Failed to checkout git version %s" % version)
if proto is not None and proto == "local":
print(f' - Processing plugin: {plugin} at {repo}')
if os.path.isdir(".repo"):
if version is None:
print(' - Processing ' + repo)
if os.path.isdir(f"{repo}"):
print(f' Processing {repo}')
dm_files = get_option_value(plugin, "dm_files")
if dm_files is not None and isinstance(dm_files, list):
for dm_file in dm_files:
if os.path.isfile(f"{repo}/{dm_file}"):
generate_dynamic_datamodel_tree(f"{repo}/{dm_file}")
generate_datamodel_tree(f"{repo}/{dm_file}")
generate_dynamic_json_datamodel_tree(f"{repo}/{dm_file}")
else:
BBF_ERROR_CODE += 1
else:
files = os.popen(f'find {repo}/ -name datamodel.c').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_datamodel_tree(file)
generate_datamodel_tree(file)
files = os.popen(f'find {repo}/ -name "*.json"').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_json_datamodel_tree(file)
else:
print(' - Processing ' + repo + '^' + version)
print(f' {repo} is not a directory !!!!!')
BBF_ERROR_CODE += 1
dm_files = get_option_value(plugin, "dm_files")
if dm_files is not None and isinstance(dm_files, list):
for dm_file in dm_files:
generate_dynamic_datamodel_tree(".repo/" + dm_file)
generate_datamodel_tree(".repo/" + dm_file)
generate_dynamic_json_datamodel_tree(".repo/" + dm_file)
else:
files = os.popen('find .repo/ -name datamodel.c').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_datamodel_tree(file)
generate_datamodel_tree(file)
files = os.popen('find .repo/ -name "*.json"').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_json_datamodel_tree(file)
else:
print(f' - Processing plugin: {plugin}')
version = get_option_value(plugin, "version")
remove_folder(".repo")
try:
subprocess.run(["git", "clone", repo, ".repo"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check = True)
except (OSError, subprocess.SubprocessError) as _e:
print(f' Failed to clone {repo} !!!!!')
BBF_ERROR_CODE += 1
if version is not None:
try:
subprocess.run(["git", "-C", ".repo", "checkout", version],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=True)
except (OSError, subprocess.SubprocessError) as _e:
print(f' Failed to checkout git version {version} !!!!!')
BBF_ERROR_CODE += 1
if os.path.isdir(".repo"):
if version is None:
print(f' Processing {repo}')
else:
print(f' Processing {repo}^{version}')
dm_files = get_option_value(plugin, "dm_files")
if dm_files is not None and isinstance(dm_files, list):
for dm_file in dm_files:
if os.path.isfile(".repo/" + dm_file):
generate_dynamic_datamodel_tree(".repo/" + dm_file)
generate_datamodel_tree(".repo/" + dm_file)
generate_dynamic_json_datamodel_tree(".repo/" + dm_file)
else:
BBF_ERROR_CODE += 1
else:
files = os.popen('find .repo/ -name datamodel.c').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_datamodel_tree(file)
generate_datamodel_tree(file)
files = os.popen('find .repo/ -name "*.json"').read()
for file in files.split('\n'):
if os.path.isfile(file):
generate_dynamic_json_datamodel_tree(file)
remove_folder(".repo")
else:
BBF_ERROR_CODE += 1
print('Generating of plugins done')
############## Remove Duplicated Element from List ##############
global LIST_SUPPORTED_DM

1
tools/generate_dm.py
View file

@ -117,3 +117,4 @@ if isinstance(file_format, list):
bbf.remove_file(bbf.DATA_MODEL_FILE)
print("Datamodel generation completed, aritifacts shall be available in out directory or as per input json configuration")
sys.exit(bbf.BBF_ERROR_CODE)

24
tools/generate_dm_excel.py
View file

@ -6,6 +6,7 @@
from collections import OrderedDict
import os
import sys
import json
import argparse
import xlwt
@ -109,12 +110,12 @@ def parse_object_tree(dm_name_list):
for obj, value in data.items():
if obj is None:
print("!!!! %s : Wrong JSON Data model format!" % dm)
print(f'!!!! {dm} : Wrong JSON Data model format!')
continue
parse_standard_object(obj, value)
else:
print("!!!! %s : Data Model doesn't exist" % dm)
print(f"!!!! {dm} : Data Model doesn't exist")
parse_dynamic_object(dm_name_list)
@ -183,9 +184,9 @@ def generate_excel(dm_name_list, output_file="datamodel.xml"):
generate_excel_file(output_file)
if os.path.isfile(output_file):
print(" - Excel file generated: %s" % output_file)
print(f' - Excel file generated: {output_file}')
else:
print(" - Error in excel file generation %s" % output_file)
print(f' - Error in excel file generation {output_file}')
### main ###
@ -206,7 +207,7 @@ if __name__ == '__main__':
parser.add_argument(
'-r', '--remote-dm',
action='append',
metavar = 'https://dev.iopsys.eu/iopsys/stunc.git^devel',
metavar = 'git^https://dev.iopsys.eu/iopsys/stunc.git^devel',
help= 'Includes OBJ/PARAM defined under remote repositories defined as bbf plugin'
)
@ -219,7 +220,7 @@ if __name__ == '__main__':
parser.add_argument(
'-p', '--vendor-prefix',
default = 'iopsys',
default = 'X_IOPSYS_EU_',
metavar = 'X_IOPSYS_EU_',
help = 'Generate data model tree using provided vendor prefix for vendor defined objects'
)
@ -238,13 +239,16 @@ if __name__ == '__main__':
for f in args.remote_dm:
x = f.split('^')
r = {}
r["repo"] = x[0]
if len(x) == 2:
r["version"] = x[1]
r["proto"] = x[0]
if len(x) > 1:
r["repo"] = x[1]
if len(x) == 3:
r["version"] = x[2]
plugins.append(r)
bbf.generate_supported_dm(args.vendor_prefix, args.vendor_list, plugins)
bbf.clean_supported_dm_list()
generate_excel(args.datamodel, args.output)
print("Datamodel generation completed, aritifacts available in %s" %args.output)
print(f'Datamodel generation completed, aritifacts available in {args.output}')
sys.exit(bbf.BBF_ERROR_CODE)

30
tools/generate_dm_xml.py
View file

@ -4,6 +4,7 @@
# Author: Amin Ben Ramdhane <amin.benramdhane@pivasoftware.com>
import os
import sys
import argparse
import xml.etree.ElementTree as ET
import xml.dom.minidom as MD
@ -189,7 +190,7 @@ def generate_xml(acs = 'default', output_file="datamodel.xml"):
DM_OBJ_COUNT = 0
DM_PARAM_COUNT = 0
print("Generating BBF Data Models in xml format for %s acs..." % acs)
print(f'Generating BBF Data Models in xml format for {acs} acs...')
bbf.fill_list_supported_dm()
if acs == "HDM":
@ -198,12 +199,12 @@ def generate_xml(acs = 'default', output_file="datamodel.xml"):
generate_bbf_xml_file(output_file)
if os.path.isfile(output_file):
print(" - XML file generated: %s" % output_file)
print(f' - XML file generated: {output_file}')
else:
print(" - Error in generating xml file")
print(' - Error in generating xml file')
print(" - Number of BBF Data Models objects is %d" % DM_OBJ_COUNT)
print(" - Number of BBF Data Models parameters is %d" % DM_PARAM_COUNT)
print(f' - Number of BBF Data Models objects is {DM_OBJ_COUNT}')
print(f' - Number of BBF Data Models parameters is {DM_PARAM_COUNT}')
### main ###
if __name__ == '__main__':
@ -215,7 +216,7 @@ if __name__ == '__main__':
parser.add_argument(
'-r', '--remote-dm',
action='append',
metavar = 'https://dev.iopsys.eu/iopsys/stunc.git^devel',
metavar = 'git^https://dev.iopsys.eu/iopsys/stunc.git^devel',
help= 'Includes OBJ/PARAM defined under remote repositories defined as bbf plugin'
)
@ -228,7 +229,7 @@ if __name__ == '__main__':
parser.add_argument(
'-p', '--vendor-prefix',
default = 'iopsys',
default = 'X_IOPSYS_EU_',
metavar = 'X_IOPSYS_EU_',
help = 'Generate data model tree using provided vendor prefix for vendor defined objects.'
)
@ -242,8 +243,8 @@ if __name__ == '__main__':
parser.add_argument(
"-m", "--manufacturer",
default = 'iopsys',
metavar = 'iopsys',
default = 'IOPSYS',
metavar = 'IOPSYS',
help = 'Generate data model tree using this manufacturer.'
)
@ -304,13 +305,16 @@ if __name__ == '__main__':
for f in args.remote_dm:
x = f.split('^')
r = {}
r["repo"] = x[0]
if len(x) == 2:
r["version"] = x[1]
r["proto"] = x[0]
if len(x) > 1:
r["repo"] = x[1]
if len(x) == 3:
r["version"] = x[2]
plugins.append(r)
bbf.generate_supported_dm(args.vendor_prefix, args.vendor_list, plugins)
bbf.clean_supported_dm_list()
generate_xml(args.format, args.output)
print("Datamodel generation completed, aritifacts available in %s" %args.output)
print(f'Datamodel generation completed, aritifacts available in {args.output}')
sys.exit(bbf.BBF_ERROR_CODE)

10
tools/tools_input.json
View file

@ -1,5 +1,5 @@
{
"manufacturer": "iopsys",
"manufacturer": "IOPSYS",
"protocol": "DEVICE_PROTOCOL_DSLFTR069v1",
"manufacturer_oui": "002207",
"product_class": "DG400PRIME",
@ -13,15 +13,18 @@
"plugins": [
{
"repo": "https://dev.iopsys.eu/iopsys/bulkdata.git",
"proto": "git",
"dm_files": [
"datamodel.c"
]
},
{
"repo": "https://dev.iopsys.eu/iopsys/xmppc.git"
"repo": "https://dev.iopsys.eu/iopsys/xmppc.git",
"proto": "git"
},
{
"repo": "https://dev.iopsys.eu/iopsys/stunc.git",
"proto": "git",
"version": "devel",
"dm_files": [
"datamodel.c"
@ -29,6 +32,7 @@
},
{
"repo": "https://dev.iopsys.eu/iopsys/udpechoserver.git",
"proto": "git",
"version": "master",
"dm_files": [
"datamodel.c"
@ -36,6 +40,7 @@
},
{
"repo": "https://dev.iopsys.eu/iopsys/twamp.git",
"proto": "git",
"version": "master",
"dm_files": [
"datamodel.c"
@ -43,6 +48,7 @@
},
{
"repo": "https://dev.iopsys.eu/iopsys/periodicstats.git",
"proto": "git",
"version": "devel",
"dm_files": [
"bbf_plugin/bbf_plugin.c"