DMPACK User Guide

dmapi is an HTTP-RPC API service for remote DMPACK database access. The web application has to be executed through a FastCGI-compatible web server. It is recommended to use lighttpd(1). The service is configured through environment variables. The web server or FastCGI spawner must be able to pass environment variables to dmapi.

The dmapi service offers endpoints for clients to insert beats, logs, and observations into the local SQLite database, and to request data in CSV or JSON format. Only HTTP GET and POST requests are accepted. All POST data has to be serialised in Fortran 95 Namelist format, with optional deflate or zstd compression. Section RPC API gives an overview of the available endpoints.

Authentication and encryption are independent from dmapi and have to be provided by the web server. If HTTP Basic Auth is enabled, the sensor id of each beat, log, node, sensor, and observation sent to the HTTP-RPC service must match the name of the authenticated user. For example, to store an observation of a node with the id node-1, the user name of the client must be node-1 as well. If the observation is sent by any other user, it will be rejected (HTTP 401).

The response format depends on the MIME type set in the HTTP Accept header of the request, either:

By default, responses are in CSV format. The Namelist format is available only for single records. Status messages are returned as key–value pairs, indicated by content type text/plain.

See section RPC Server for a basic lighttpd(1) configuration.

The dmbackup utility creates an online backup of a running SQLite database. By default, the SQLite backup API is used. The program is functional equivalent to running the sqlite3(1) command-line interface:

dmbackup does not replace existing backup databases.

Create an online backup of an observation database:

The dmbeat program is a heartbeat emitter that sends handshake messages via HTTP POST to a remote dmapi service. Heartbeats include the following attributes:

The server may inspect the data to check if a client is still running and has network access. The RPC endpoint on the server is expected at URL [http|https]://<host>:<port>/api/v1/beat.

Send a single heartbeat to a dmapi service on localhost:

A sensor node with id dummy-node must exist in the server database. The web application dmweb lists the beats received by the server.

The dmbot program is an XMPP bot that accepts commands via chat. Access to the bot is limited to the JIDs added to table group in the configuration file. Requests from clients whose JID is not in the table will be rejected. If table group is empty, all clients are allowed to send commands to the bot.

The XMPP resource is automatically set to the name of the bot instance. If the JID of the bot account is bot@example.com and the bot name is dmbot, the full JID will be set to bot@example.com/dmbot.

All commands start with prefix !. For an overview, send chat command !help to the bot. The bot understands the following commands:

Passing the XMPP credentials via the command-line arguments --jid and --password is insecure on multi-user operating systems and only recommended for testing.

Connect with JID bot@example.com to an XMPP server on port 5223 and wait for commands:

If no configuration file is used, any client may send commands to the bot without authorisation. Start a chat with the bot JID and send a command. For instance, on command !uptime the bot sends a reply like the following:

The dmdb program collects observations from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmdb name and leading /. The IPC option enables process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 if a new observation has been received. Only a single process shall wait for the semaphore.

Only observation types in binary format are accepted. Log messages are stored to database by the distinct dmlogger program.

Create a message queue /dmdb, wait for incoming observations, and store them in the given database:

Log messages and observation ids are printed to stdout if argument --verbose is set.

The dmdbctl utility program performs create, read, update, or delete operations (CRUD) on the observation database.

Only nodes, sensors, and targets are supported. All data attributes are passed through command-line arguments.

Add node, sensor, and target to observation database:

Delete a target from the database:

Read attributes of sensor sensor-1:

The dmexport program writes beats, logs, nodes, sensors, targets, observations, and data points from database to file, in ASCII block, CSV, JSON, or JSON Lines format. The ASCII block format is only available for X/Y data points. The types data point, log, and observation require a sensor id, a target id, and a time range in ISO 8601 format.

If no output file is given, the data is printed to standard output. The output file will be overwritten if it already exists. If no records are found, an empty file will be created.

Export log messages from database to JSON file:

Export observations from database to CSV file:

The dmfeed program creates a web feed from log messages in Atom Syndication Format. The log messages are read from database and written as XML to standard output or file.

The feed id has to be a 36 characters long UUID with hyphens. News aggregators will use the id to identify the feed. Therefore, the id should not be reused among different feeds. Run dmuuid to generate a valid UUIDv4.

The time stamp of the feed in element updated is set to the date and time of the last log message. If no logs have been added to the database since the last file modification of the feed, the output file is not updated, unless argument --force is passed. To update the feed periodically, add dmfeed to crontab.

If an XSLT style sheet is given, web browsers may be able to display the Atom feed in HTML format. Set the option to the (relative) path of the public XSL on the web server. An example style sheet feed.xsl is located in /usr/local/share/dmpack/.

First, generate a unique feed id:

Then, write the last 50 log messages in Atom format to file feed.xml, and include a link to the XSLT style sheet feed.xsl:

Copy the XSLT style sheet to the directory of the Atom feed:

If /var/www/ is served by a web server, feed readers can subscribe to the feed. Additionally, we may translate feed and style sheet into a single HTML document feed.html, using an arbitrary XSLT processor, for instance:

The dmfs program reads observations from file system, virtual file, or named pipe. The program can be used to read sensor data from the 1-Wire File System (OWFS).

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. dmfs can act as a sole data logger if output and format are set. If the output path is set to -, observations are written to stdout instead of file.

The requests of each observation have to contain the path of the (virtual) file in attribute request. Response values are extracted by named group from the raw response using the given regular expression pattern. Afterwards, the observation is forwarded to the next receiver via POSIX message queue.

A configuration file is mandatory to describe the jobs to perform. Each observation must have a valid target id. Node, sensor, and target have to be present in the database.

Start dmfs to execute the jobs in the configuration file:

The dmgrc program creates log messages from Leica GeoCOM return codes. Observations received by POSIX message queue are searched for a GeoCOM return code (GRC) response. If the code does not equal GRC_OK, a log message is sent to the configured logger instance.

By default, observation responses of name grc are verified. For each GeoCOM error code, a custom log level may be specified in the configuration file. Otherwise, the default log level is used instead.

A configuration file is not required, but allows to specifiy the log level of certain GeoCOM return codes. In the following example configuration, the default log level for all return codes other than GRC_OK is set to LL_WARNING. The level is further refined for specific GeoCOM codes:

See section GeoCOM API for a table of all supported return codes. Pass the path of the configuration file through the command-line argument:

The name argument must match the name of the configuration table. A logger process of name dmlogger must be running to process the generated log messages.

The dminfo utility program prints build, database, and system information to standard output. The path to the beat, log, or observation database is passed through command-line argument --database. Only one database can be specified.

The output contains compiler version and options; database PRAGMAs, tables, and number of rows; as well as system name, version, and host name.

Print build, database, and system information:

The dmimport program reads logs, nodes, sensors, targets, and observations in CSV format from file and imports them into the database. The database inserts are transaction-based. If an error occurs, the transaction is rolled back, and no records are written to the database at all.

The database has to be a valid DMPACK database and must contain the tables required for the input records. The nodes, sensors, and targets referenced by input observations must exist in the database. The nodes referenced by input sensors must exist as well.

Import observations from CSV file observ.csv into database observ.sqlite:

The dminit utility program creates beat, log, and observation databases. No action is performed if the specified database already exists. A synchronisation table is required for observation and log synchronisation with an dmapi server. The argument can be omitted if this feature is not needed. The journal mode Write-Ahead Logging (WAL) should be enabled for databases with multiple readers.

Create an observation database with remote synchronisation tables (WAL):

Create a log database with remote synchronisation tables (WAL):

Create a heartbeat database (WAL):

The dmlog utility forwards a log message to the message queue of a dmlogger or dmrecv instance. The program may be executed through a shell script to add logs to the DMPACK database. The argument --message is mandatory. The default log level is info. Pass the name of the dmlogger or dmrecv instance to send the log to through command-line argument --logger.

Logs are sent in binary format. The program terminates after log transmission. The log level may be one of the following:

Both, parameter strings and literal log level values, are accepted as command-line arguments. For level warning, set argument --level to 3 or warning.

Send a log message to the message queue of logger dmlogger:

The dmlogger process will receive the log message in real-time and store it in the log database (if the log level is ≥ the configured minimum log level):

The dmlogger program collects log messages from a POSIX message queue and writes them to a SQLite database. The name of the message queue will equal the given dmlogger name with leading /, by default /dmlogger.

If a minimum log level is selected, only logs of a level greater or equal the minimum are stored in the database. Log messages with a lower level are printed to standard output before being discarded (only if the verbose flag is enabled).

The IPC option allows an optional process synchronisation via a named POSIX semaphores. The value of the semaphore is changed from 0 to 1 whenever a new log was received. The name of the semaphore will equal the dmlogger name with leading /.

Only a single process should wait for the semaphore unless round-robin passing is desired. This feature may be used to automatically synchronise incoming log messages with a remote HTTP-RPC API server. dmsync will wait for new logs before starting synchronisation if the dmlogger instance name has been passed through command-line argument --wait.

The following log levels are accepted:

Create a message queue /dmlogger, wait for incoming logs, and store them in the given database if logs are of level error (4) or higher:

Push semaphore /dmlogger each time a log has been received:

Let dmsync wait for semaphore /dmlogger before synchronising the log database with host 192.168.1.100, then repeat:

The dmlua program runs a custom Lua script to process observations received from message queue. Each observation is passed as a Lua table to the function of the name given in option procedure. If the option is not set, function name process is assumed by default. The Lua function must return the (modified) observation table on exit.

The observation returned from the Lua function is forwarded to the next receiver specified in the receivers list of the observation. If no receivers are left, the observation will be discarded.

The following Lua script script.lua just prints observation table observ to standard output, before returning it to dmlua unmodified:

Any observation sent to receiver dmlua will be passed to the Lua function process() in script.lua, then forwarded to the next receiver (if any):

The dmmb program reads values from or writes values to Modbus RTU/TCP registers by sequentially processing the job list loaded from a configuration file. Each request of an observation must contain the Modbus register parameters in the request string. The value of the first response is set to the result of the read operation. Up to 8 requests to read and/or write values are permitted. Integers read from a register may be scaled using an optional scale denominator.

For example, to read a 2-byte unsigned integer from holding register 40050 of slave device 2 with a scale factor of 1/10, the attribute request of a request must be set to:

Or, to read a 4-byte floating-point value in ABCD byte order from register 40060:

Only integer values may be written to an input register, for instance:

The value is converted to uint16 automatically. The command string can be in lower or upper case, white spaces are optional.

The following fields are supported in the command string:

Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout, else to file.

A configuration file is required to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets if observations will be forwarded to dmdb.

The following example can be used as a starting point for a custom configuration file. The job list contains one observation with two requests to read temperature and humidity values from the holding register. The temperature is provided as a 4-byte float in ABCD byte order in register 40060, the humidity as a 2-byte unsigned integer in register 40050. The humidity will be scaled automatically by 1/10, i.e., an integer value of 600 is converted to the real value 60.0.

The dmmb program opens a Modbus RTU connection to /dev/ttyUSB0 (19200 baud, 8N2), then reads temperature and humidity from slave device 1 every 60 seconds. The observations are printed to stdout in JSONL format:

The dmmbctl command-line program reads a value from or writes a value to a register of a connected Modbus RTU/TCP device. Modbus RTU requires the command-line arguments --path, --baudrate, --bytesize, --parity, and --stopbits. For Modbus TCP, only --address and --port must be passed.

The following data types are supported:

In order to read floating-point values, set --type to float and --order to the byte order used by the Modbus device, either abcd, badc, cdab, or dcba. Only integer values may be written to a register.

Read the current temperature in °C measured by a Pt100 RTD that is connected to an I/O module with Modbus RTU interface:

The I/O module is attached through an RS-485 adapter on /dev/ttyUSB (19200 baud, 8E1) and configured to use slave id 1. The value is read from register 40050 and converted to float in abcd byte order.

The dmpipe program reads responses from processes connected through a pipe to read sensor data from a third-party program. Requests of an observation have to contain the process to call in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern.

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. Node id, sensor id, and observation id are added by dmpipe. If the observation will be stored in a database, the node, sensor and target ids have to exist in the database.

The example reads the remaining battery life returned by the sysctl(8) tool (available on FreeBSD):

On Linux, the battery life can be read with dmfs from /sys/class/power_supply/BAT0/capacity instead.

The regular expression pattern describes the response and defines the group battery for extraction. The name of one of the responses in the responses table must equal the group name. The observation will be forwarded to the message queue of a dmdb process. Backslash characters in the string values have to be escaped with \.

Pass the path of the configuration file to dmpipe:

The result returned by sysctl(8) will be formatted according to the current locale (decimal separator). You may have to change the locale first to match the regular expression pattern:

The dmplot program is a front-end to gnuplot(1) that creates plots of observations read from database. Plots are either written to file or displayed in terminal or X11 window.

Depending on the selected terminal back-end, you may have to set the environment variable GDFONTPATH to the path of the local font directory first:

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile:

The output file is ignored when using the terminals sixelgd and x11. Plotting parameters passed via command-line have priority over those from configuration file.

Create a plot of observations selected from database observ.sqlite in PNG format, and write the file to /tmp/plot.png:

Output the plot directly to terminal, using the configuration in dmplot.conf:

The sixelgd format requires a terminal emulator with Sixel support, such as xterm(1) or mlterm(1).

The dmrecv program listens to the POSIX message queue of its name and writes received logs or observations to stdout, file, or named pipe; in CSV, JSON Lines, or Namelist format. By default, the serialised data is appended to the end of the output file. If argument --replace is passed, the file will be replaced consecutively.

Received observations are not forwarded to the next specified receiver unless argument --forward is set. If no receivers are defined or left, the observation will be discarded after output. If the JSON Lines output format is selected, logs and observations are written as JSON objects to file or stdout, separated by new line (\n). Use jq(1) to convert records in JSON Lines file input.jsonl into a valid JSON array in output.json:

The output format block is only available for observation data and requires a response name to be set. Observations will be searched for this response name and converted to data point type if found. The data point is printed in ASCII block format.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Write log messages received from POSIX message queue /dmrecv to file /tmp/logs.csv in CSV format:

Output observations in JSON Lines format to stdout:

Write the observations serialised in JSON Lines format to named pipe /tmp/fifo_dmrecv:

Another process can now read the observations from /tmp/fifo_dmrecv:

Responses in block format can also be piped to a graph tool like trend to update a chart in real-time. For instance, to pipe the responses of name tz0 in observations received through message queue /dmrecv to the trend graph program, run:

GNU awk is used to extract the response value from the stream, before it is piped to trend(1).

The dmreport program creates reports in HTML5 format, containing plots of observations and/or log messages selected from database. Plots are created by calling gnuplot(1) and inlining the returned image (GIF, PNG, SVG) as a base64-encoded data URI. Any style sheet file with classless CSS can be included to alter the presentation of the report. A basic style sheet dmreport.css and its minified version dmreport.min.css are provided in /usr/local/share/dmpack/dmreport/. The output of dmreport is a single HTML file with inlined CSS. Use a command-line tool like wkhtmltopdf to convert the HTML report to PDF format.

Depending on the selected plot format, the environment variable GDFONTPATH may have to be set to the local font directory containing the TrueType fonts first, for example:

Add the export statement to the global profile /etc/profile. If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to /etc/profile:

A configuration file is mandatory to create reports. Only a few parameters can be set through command-line arguments. Passed command-line arguments have priority over settings in the configuration file.

The settings are stored in Lua table dmreport in the configuration file. The observations are read from database observ.sqlite, the log messages from log.sqlite. You might want to use absolute paths for the databases.

The sensor node dummy-node, the sensor dummy-sensor, and the target dummy-target must exist in the database, and the observations to plot need to have responses of name tz0. Write a report to file report.html based on settings in dmreport.conf. The command-line arguments overwrite the settings of the configuration file:

In order to update reports periodically, we can customise the shell script mkreport.sh in /usr/local/share/dmpack/dmreport/. The script determines the timestamps of the last and the current month (to allow observations to arrived late), which will then be passed to dmreport to create monthly reports. Modify the script according to your set-up:

The shell script writes two reports to /var/www/reports/.

The directory may be served by lighttpd(1). Add the script to your crontab to run the report generation periodically.

The dmsend program reads observations or logs in CSV and Fortran 95 Namelist format, and sends them sequentially to the POSIX message queue of a given receiver. The data is either read from file or standard input. If the input data is of type observ and the argument --forward is passed, each observation will be sent to its next specified receiver in the receivers list instead of the receiver given through argument --receiver. If no receivers are set, or if the end of the receivers list is reached, the observation will be discarded.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Read a single observation from Namelist file observ.nml and send it to the next receiver specified by attribute next:

Send multiple logs in CSV file logs.csv sequentially to process dmrecv:

The dmserial program sends requests to a sensor or actor connected via USB/RS-232/RS-422/RS-485. Sensor commands and responses are sent/received through a teletype (TTY) device provided by the operating system. A pseudo-terminal (PTY) may be used to connect a virtual sensor.

Each request of an observation must contains the raw request intended for the sensor in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern. Each group name must match a response name. Response names are limited to 32 characters. Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output file and format are set. If the output is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets. Parameters and functions of the Lua API may be used in the configuration file. The following baud rates are supported: 50, 75, 110, 134, 150, 200, 300, 600, 1200, 1800, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 230400, 460800, 921600.

Read the jobs to perform from configuration file and execute them sequentially:

The dmsync program synchronises logs, nodes, observations, sensors, and targets from local databases concurrently with a remote dmapi server. The synchronisation may be started only once if no interval is set (to transfer nodes, sensors, and targets initially from client to server), periodically as a cron job, or by waiting for a POSIX semaphore.

The nodes, sensors, and targets referenced by observations in the local database must also exist in the remote server database. They can be created on the server with dmdbctl or dmweb, or sent from client to server with dmsync. Logs and targets do not require any additional database entries on the server-side.

The client databases must contain synchronisation tables. The tables are created automatically by dminit if command-line argument --sync is passed. Otherweise, start dmsync with argument --create once to add the missing tables.

If the RPC server uses HTTP Basic Auth for authentication, the RPC user name must match the node id of the transmitted node, sensor, observation, log, or beat records, or the server will reject the requests and return HTTP 401 (Unauthorized).

The database records are serialised in Fortran 95 Namelist format and optionally compressed before being sent to the server. The program uses libcurl for data transfer, and deflate or zstd for compression. The RPC API endpoints to post records to are expected at URL [http|https]://<host>:<port>/api/v1/<endpoint>.

The result of each synchronisation attempt is stored in the local database. Records are marked as synchronised only if the server returns HTTP 201 (Created).

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

Initially synchronise nodes, sensors, and targets in the local observation database with an HTTP-RPC server (without authentication):

Synchronise observations:

Synchronise log messages:

The dmuuid program is a command-line tool to generate pseudo-random UUIDs. By default, DMPACK uses 32 characters long UUIDv4 identifiers in hexadecimal format (without hyphens). Hyphens can be added by a command-line flag. The option --convert expects UUIDv4 identifiers to be passed via standard input. Invalid identifiers will be replaced with the default UUID. The program may be used to create a feed id for dmfeed.

Create three identifiers:

Create a UUIDv4 with hyphens:

Add hyphens to a hexadecimal UUID:

The dmved program captures VE.Direct status data received from a connected Victron Energy Maximum Power Point Tracking (MPPT) solar charge controller or battery monitor, either:

An official Victron Energy USB cable or TTL adapter with JST PH connector is required for data link. The TTY will be configured to 19200 baud (8N1). Values are captured once per second. An observation containing the responses of the device is sent in the specified interval to the configured receiver.

The following VE.Direct fields are supported, depending on the device:

For a connected SmartSolar MPPT charger, create a configuration file dmved.conf, set the device to mppt and the path to the TTY path, for example /dev/ttyUSB0 if a Victron Energy USB adapter cable is used. Change node id, sensor id, and target id to your setup:

Start dmved to read and forward status data from the connected MPPT every 60 seconds:

Start dmrecv to receive observations and output them to stdout in JSONL format:

dmweb is a CGI-based web user interface for DMPACK database access on client and server. The web application has to be executed through a CGI-compatible web server. It is recommended to run lighttpd(1). Any other server must be able to pass environment variables to the CGI application. gnuplot(1) is required for the plotting back-end (no-X11 flavour is sufficient).

The web application provides the following pages:

The style sheet of dmweb is based on missing.css. It can be replaced with any other classless CSS theme. For best experience, the IBM Plex font family should be installed locally.

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile /etc/profile:

On FreeBSD, it might be necessary to add the environment variable GDFONTPATH to the path of the font directory:

The map view requires a URL to the tile server in environment variable DM_TILE_URL. For example, set the variable to https://tile.openstreetmap.org/{z}/{x}/{y}.png to use OpenStreetMap as the backend.

Copy the directory /usr/local/share/dmpack/dmweb manually to the WWW root directory, or create a symlink. Environment variables are used to configure dmweb. Transport security and authentication have to be managed by the web server. See section Web UI for an example configuration.