DMPACK User’s Guide

First, install the build and run-time dependencies:

Instead of math/gnuplot, you may want to install math/gnuplot-lite which does not depend on X11 (but neither includes raster graphic terminals).

Optionally, install Pygments and AsciiDoctor to generate the man pages and the User’s Guide:

Then, clone the repository recursively. Run the provided POSIX Makefile to build from source:

Install the library and all programs system-wide to /usr/local:

You can change the installation prefix with argument PREFIX. To install to a custom directory, run:

The DMPACK programs require the shared library libgfortran.so if they have been compiled with GNU Fortran.

On Debian, install GCC, GNU Fortran, and the build environment:

The third-party dependencies have to be installed with development headers:

Instead of package gnuplot, you can install the no-X11 flavour gnuplot-nox, if raster image formats are not needed (SVG output only). Depending on the package repository, the names of the Lua packages may differ.

Clone the DMPACK repository, and then execute the Makefile with build target linux:

Install the library and all programs system-wide to /usr:

To install to a custom directory, run:

The local time zone of the sensor client should be set to a zone without summer daylight-saving. For instance, time zone Europe/Berlin implies Central European Summer Time (CEST), which is usually not desired for long-term observations, as it leads to time jumps. Instead, use time zone GMT+1 or UTC in this case.

The system time should be updated periodically by synchronising it with network time servers. A Network Time Protocol (NTP) client has to be installed and configured to enable the synchronisation.

On Linux, power saving for USB devices may be enabled by default. This can cause issues if sensors are attached through an USB adapter. USB power saving is enabled if the kernel boot parameter usbcore.autosuspend is not equal -1:

We can update the boot loader to turn auto-suspend off. Edit /etc/default/grub and change GRUB_CMDLINE_LINUX_DEFAULT to:

Then, update the boot loader:

The system has to be rebooted for the changes to take effect.

The operating system must have POSIX message queues enabled to run DMPACK programs on sensor nodes.

On Unix-like operating system, cron is usually used to run jobs periodically. For example, in order to update an XML feed, or to generate HTML reports, add a schedule of the task to perform to the local crontab(5) file.

For example, we can edit the cron jobs of user www with crontab(1):

The following crontab(5) file adds a task to generate reports every hour:

The shell script mkreport.sh must have the execution bits set. Update the script to your configuration.

dmapi is an HTTP-RPC API service for remote DMPACK database access. The web application has to be executed through a FastCGI-compatible web server or a FastCGI spawner. It is recommended to run lighttpd(1).

The dmapi service offers endpoints for clients to insert beats, logs, and observations into the local SQLite database, and to request data in CSV or JSON format. Authentication and encryption are independent from dmapi and have to be provided by the web server.

All POST data has to be serialised in Fortran 95 Namelist format, with optional deflate compression.

If HTTP Basic Auth is enabled, the sensor id of each beat, log, node, sensor, and observation sent to the RPC service must match the name of the authenticated user. For example, to store an observation of a node with the id node-1, the HTTP Basic Auth user name must equal the node id. If the observation is sent by any other user, it will be rejected (HTTP 401).

The web application is configured through environment variables. The web server or FastCGI spawner must be able to pass environment variables to dmapi. See RPC Server for an example configuration.

The service accepts HTTP GET and POST requests. Section RPC API gives an overview of the available endpoints. The response format depends on the MIME type set in the HTTP Accept header of the request, either:

By default, responses are in CSV format. The Namelist format is available only for single records. Status messages are returned as key–value pairs, signaled by MIME type text/plain.

The dmbackup utility creates an online backup of a running SQLite database. By default, the SQLite backup API is used. The program is functional equivalent to running the sqlite3(1) command-line interface:

dmbackup does not replace existing backup databases.

The dmbeat program is a heartbeat emitter that sends status messages via HTTP POST to a remote dmapi service. The status messages include timestamp, system uptime, and last connection error. The server may inspect this data to check if a client is still running and has network access. The RPC endpoint is expected at [http|https]://<host>:<port>/api/v1/beat.

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

The dmdb program collects observations from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmdb name, by default dmdb. The IPC option enables process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 if a new observation has been received. The name of the semaphore equals the dmdb name. Only a single process may wait for the semaphore.

The dmdbctl utility program performs create, read, update, or delete operations (CRUD) on the observation database. Only nodes, sensors, and targets are supported. Data attributes are passed through command-line arguments.

The dmexport program writes beats, logs, nodes, sensors, targets, observations, and data points from database to file, in ASCII block, CSV, JSON, or JSON Lines format. The ASCII block format is only available for X/Y data points. The types data point, log, and observation require a sensor id, a target id, and a time range in ISO 8601 format.

If no output file is given, the data is printed to standard output. The output file will be overwritten if it already exists. If no records are found, an empty file will be created.

This program creates a web feed from log messages in Atom Syndication Format. The log messages are read from database and written as XML to standard output or file.

The feed id has to be a 36 characters long UUID with hyphens. News aggregators use the id to identify the feed. Therefore, the id should not be reused among different feeds. Run dmuuid to generate a valid UUID4.

If an XSLT style sheet is given, web browsers may be able to display the Atom feed in HTML format. Set the option to the (relative) path of the public XSL on the web server. An example style sheet feed.xsl is located in /usr/local/share/dmpack/.

The dmfs program reads observations from file system, virtual file, or named pipe. The program can be used to read sensor data from the 1-Wire File System (OWFS).

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. dmfs can act as a sole data logger if output and format are set. If the output path is set to -, observations are written to stdout instead of file.

The requests of each observation have to contain the path of the (virtual) file in attribute request. Response values are extracted by named group from the raw response using the given regular expression pattern. Afterwards, the observation is forwarded to the next receiver via POSIX message queue.

A configuration file is mandatory to describe the jobs to perform. Each observation must have a valid target id. Node, sensor, and target have to be present in the database.

The dmgraph program is a front-end to gnuplot(1) that creates plots of observations read from database. Plots are either written to file or displayed in terminal or X11 window.

Depending on the selected terminal backend, you may have to set the environment variable GDFONTPATH to the local font directory first:

The output file is ignored when using the terminals sixelgd and x11. Plotting parameters passed via command-line have priority over those from configuration file.

The dminfo utility program prints build, database, and system information to standard output. The path to the beat, log, or observation database is passed through command-line argument --database.

The output contains compiler version and options; database PRAGMAs, tables, and number of rows; as well as system name, version, and host name.

The dmimport program reads logs, nodes, sensors, targets, and observations in CSV format from file and imports them into the database. The database inserts are transaction-based. If an error occurs, the transaction is rolled back, and no records are written into the database at all.

The database has to be a valid DMPACK database and must contain the tables required for the input records. The nodes, sensors, and targets referenced by input observations must exist in the database. The nodes referenced by input sensors must exist as well.

The dminit utility program creates beat, log, and observation databases. No action is performed if the specified database already exists.

A synchronisation table is required for observation and log synchronisation with an dmapi server. The argument can be omitted if this functionality is not used.

The journal mode Write-Ahead Logging (WAL) should be enabled for databases with multiple readers.

The dmlog utility forwards a log message to the message queue of a dmlogger instance. The argument --message is mandatory. The default log level is INFO. Pass the name of the dmlogger instance through argument --logger. The program terminates after log transmission.

The following log levels are accepted:

The dmlogger program collects log messages from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmlogger name with leading /, by default /dmlogger.

If a minimum log level is selected, only logs of a level greater equal the minimum are stored in the database. Log messages with lower level are printed to standard output before being discarded (if verbose mode is enabled).

The IPC option allows process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 whenever a new log was received. The name of the semaphore equals the dmlogger name with leading /. Only a single process should wait for the semaphore unless round-robin passing is desired.

This feature may be used to automatically synchronise incoming log messages with a remote HTTP-RPC API server. dmsync will wait for new logs before starting synchronisation if the dmlogger instance name has been passed through command-line argument --wait.

The following log levels are accepted:

The dmlua program runs a custom Lua script to process observations received from message queue. Each observation is passed as a Lua table to the function of the name given in option procedure. If the option is not set, function name process is assumed by default. The Lua function must return the (modified) observation table on exit.

The observation returned from the Lua function is forwarded to the next receiver specified in the receivers list of the observation. If no receivers are left, the observation will be discarded.

The dmpipe program reads responses from processes connected via pipe.

All requests of an observation have to contain the process in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern.

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. Node id, sensor id, and observation id are added by dmpipe. Node, sensor, and target have to be present in the database for the observation to be stored.

The dmrecv program listens to the POSIX message queue of its name and writes received logs or observations to stdout, file, or named pipe; in CSV, JSON Lines, or Namelist format. By default, the serialised data is appended to the end of the output file. If argument --replace is passed, the file will be replaced consecutively.

Received observations are not forwarded to the next specified receiver unless argument --forward is set. If no receivers are defined or left, the observation will be discarded after output.

The output format block is only available for observation data and requires a response name to be set. Observations will be searched for this response name and converted to data point type if found. The data point is printed in ASCII block format.

If the JSON Lines output format is selected, logs and observations are written as JSON objects to file or stdout, separated by new line (\n). Use jq(1) to convert records in JSON Lines file input.jsonl into a valid JSON array in output.json:

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

The dmreport program creates reports in HTML5 format, containing plots of observations and/or log messages selected from database. Plots are created by calling gnuplot(1) and inlining the returned image (GIF, PNG, SVG) as a base64-encoded data URI. Any style sheet file with classless CSS can be included to alter the presentation of the report. The output of dmreport is a single HTML file.

A configuration file is mandatory to create reports. Only a few parameters can be set through command-line arguments. Passed command-line arguments have priority over settings in the configuration file.

The dmsend program reads observations or logs in CSV or Fortran 95 Namelist format, and sends them sequentially to the POSIX message queue of the given receiver. The data is either read from file or from standard input. If the input data is of type observ and the argument --forward is passed, each observation will be sent to its next specified receiver in the receivers list. If no receivers are declared, or if the end of the receivers list is reached, the observation will not be forwarded.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

The dmserial program sends requests to a sensor or actor connected via USB/RS-232/RS-422/RS-485. Sensor commands and responses are sent/received through a teletype (TTY) device provided by the operating system. A pseudo-terminal (PTY) may be used to connect a virtual sensor.

Each request of an observation must contains the raw request intended for the sensor in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern. Each group name must match a response name. Response names are limited to eight characters.

Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout, else to file.

A configuration file is required to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets.

The following baud rates are supported: 50, 75, 110, 134, 150, 200, 300, 600, 1200, 1800, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 230400, 460800, 921600.

The dmsync program synchronises logs, nodes, observations, sensors, and targets from local database concurrently with a remote dmapi server. The synchronisation may be started only once if no interval is set (to transfer nodes, sensors, and targets from client to server), periodically as a cron job, or by waiting for a POSIX semaphore.

The nodes, sensors, and targets referenced by observations in the local database must also exist in the remote server database. They can be created either with dmdbctl or dmweb, but also synchronised with dmsync. Logs and targets do not require any additional database entries on server-side.

The client databases must contain synchronisation tables. The tables are created automatically by dminit if command-line argument --sync is passed. Alternatively, start dmsync with argument --create once.

If the RPC server uses HTTP Basic Auth for authentication, the RPC user name must match the node id of the transmitted node, sensor, observation, log, or beat record. Otherwise, it will be rejected by the RPC server (HTTP 401).

The database records are send in compressed Fortran 95 Namelist format via HTTP to the server. The program uses libcurl for data transfer. The accessed RPC API endpoints are expected under URL [http|https]://<host>:<port>/api/v1/<endpoint>.

The result of each synchronisation attempt is stored in the local database. Records are marked as synchronised only if the server returns HTTP 201 (Created).

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

The dmuuid program is a command-line tool to generate pseudo-random UUID4s. By default, DMPACK uses 32 characters long UUID4s in hexadecimal format (without hyphens). Hyphens can be added by a command-line flag. The option --convert expects UUID4s to be passed via standard input. Invalid UUID4s will be replaced with the default UUID4.

dmweb is a CGI-based web user interface for DMPACK database access on client and server. The web application has to be executed through a CGI-compatible web server. It is recommended to run lighttpd(1). Any other server must be able to pass environment variables to the CGI application. gnuplot(1) is required for the plotting backend (no-X11 flavour is sufficient).

The web application provides the following pages:

The style sheet of dmweb is based on missing.css. It may be replaced with any other classless CSS theme. For best experience, the IBM Plex font family should be installed locally.

Environment variables are used to configure dmweb. Transport security and authentication have to be provided by the web server.

See section Web UI for an example configuration.

Set auth.backend.htpasswd.userfile to the path of the file that contains the HTTP Basic Auth credentials, or remove the related lines from the configuration if authentication is not desired. You can run openssl(1) to add credentials to the htpasswd user file:

Replace <user> and <password> with real values. Instead of a htpasswd file for, we may select a different authentication backend, for example, LDAP, MySQL/MariaDB, PostgreSQL, or SQLite 3. See the lighttpd(1) auth module documentation for further instructions.

If the HTTP-RPC API will be accessed by a client-side application running in the browser, the web server has to be configured to send the appropriate Cross-Origin Resource Sharing (CORS) headers. By default, asynchronous JavaScript requests are forbidden by the same-origin security policy. Refer to the documentation of the web server on how to set the Access-Control-* headers. For lighttpd(1), load the module mod_setenv and add response headers for OPTION requests:

If the web server is behind a reverse proxy, CORS headers should be set by the proxy instead.

The databases are expected to be in directory /var/dmpack/. Change the environment variables in the web server configuration to the actual paths. The observation, log, and beat databases the web applications will access must be created and initialised beforehand:

Make sure the web server has read and write access to the directory:

Change www:www to the user and the group the web server is running as.

The snippet in this section may be added to the lighttpd(1) configuration to run the dmapi service. The lighttpd(1) web server does not require an additional FastCGI spawner. The following server modules have to be imported:

Add the IP address of the proxy server to the list of trusted forwarders to have access to the real IP of a client.

The FastCGI socket will be written to /var/run/lighttpd/sockets/dmapi.sock. Change max-procs to the desired number of FastCGI processes. Set the environment variables to the locations of the databases. The databases must exist prior start.

On FreeBSD, add the service to the system rc file /etc/rc.conf and start the server manually:

If served locally, access the RPC API at http://127.0.0.1/api/v1/.

The lighttpd(1) web server has to be configured to run the CGI application under base path /dmpack/. The following server modules are required:

The example configuration may be appended to your lighttpd.conf:

Copy the CSS file from /usr/local/share/dmpack/dmpack.min.css to the WWW root directory, in this case, /var/www/, or just create a symlink. If the style sheet has to be served from a path different from the root path, add a rewrite rule or alias to the web server configuration.

On FreeBSD, add the service to the system rc file /etc/rc.conf and start the server manually:

If served locally, access the web application at http://127.0.0.1/dmpack/.

Returns service status in API status format as text/plain.

Returns all heartbeats in CSV, JSON, or JSON Lines format from database.

Returns logs of a given node and time range in CSV, JSON, or JSON Lines format from database. Node id and time range are mandatory.

Returns all nodes in CSV, JSON, or JSON Lines format from database.

Returns observations of given node, sensor, target, and time range from database, in CSV, JSON, or JSON Lines format.

Returns all sensors in CSV, JSON, or JSON Lines format from database.

Returns all targets in CSV, JSON, or JSON Lines format from database.

Returns time series as observation views or data points (X/Y records) in CSV format from database. In comparison to the observation endpoint, the time series include only a single response, selected by name.

Returns heartbeat of a given node in CSV, JSON, or Namelist format from database.

On POST, adds or updates heartbeat given in Namelist format. Optionally, the payload may be deflate compressed. The API returns HTTP 201 Created if the beat was accepted.

If HTTP Basic Auth is used, the user name must match the node_id attribute of the beat, otherwise, the request will be rejected as unauthorised (HTTP 401).

Returns single log of passed id in CSV, JSON, or Namelist format from database.

On POST, adds log in Namelist format to database. Optionally, the payload may be deflate compressed. The API returns HTTP 201 Created if the log was accepted.

If HTTP Basic Auth is used, the user name must match the node_id attribute of the log, otherwise, the request will be rejected as unauthorised (HTTP 401).