Project Description

The Deformation Monitoring Package (DMPACK) is a free and open source software package for sensor control and automated time series processing in engineering geodesy and geotechnics. The package consists of a library libdmpack and additional programs based on it which serve as a reference implementation of solutions to various problems in deformation monitoring, such as:

  • sensor control,

  • sensor data parsing and processing,

  • database access,

  • remote procedure calls,

  • data synchronisation and export,

  • spatial transformations,

  • time series analysis,

  • client status messages,

  • distributed logging,

  • plotting and reporting,

  • web-based data access,

  • MQTT and XMPP connectivity,

  • Leica GeoCOM API,

  • scripting,

  • e-mail.

DMPACK is a scientific monitoring system developed for automated control measurements of buildings, infrastructure, terrain, geodetic nets, and other objects through autonomous sensor networks in the IoT. The programs for sensor data collection are intended to be run on client hardware connected to the Internet through LTE/5G, usually industrial embedded systems or single-board computers.

Observation data is periodically collected by the clients from arbitrary sensors, like total stations, digital levels, inclinometers, weather stations, or GNSS receivers. The raw sensor responses are structured, post-processed, locally stored, and transmitted to a central monitoring server that provides an HTTP-RPC API for client–server communication.

The software package can be used to monitor objects like:

  • bridges, tunnels, dams,

  • roads, railways,

  • construction sites, mining areas,

  • slopes, landslides, cliffs, glaciers,

  • churches, monasteries, and other heritage buildings.

DMPACK is written in Fortran 2018 and integrates the relational SQLite database for time series and log storage on client and server. The server component is optional. If preferred, the data distribution may be omitted for local monitoring only.

The software package relies on POSIX standards for system calls and process management. The client-side message passing is based on POSIX message queues and POSIX semaphores. Currently, only 64-bit Linux (glibc) and FreeBSD are supported as operating systems.

The source code of DMPACK is released under the ISC licence that is functionally equivalent to the BSD 2-Clause and MIT licences. The source code and the documentation are available online. See the project website for further information:

Software Architecture

schema
Figure 1. Schematic view of the DMPACK client–server architecture

The DMPACK library is based on a modular architecture that loosely follows the Unix philosophy, i. e., distinct programs cover well-defined tasks. The scheduling, process management and synchronisation, as well as the inter-process communication between programs are handled by the operating system, using POSIX system calls.

The DMPACK programs can be deployed either on a single sensor node with no or only limited network access, or within a sensor network of one or more nodes connected to a central monitoring server:

Data Logger

In the most basic use case, the DMPACK programs dmserial, dmfs, and dmpipe act as data loggers without any database connectivity, by writing the observations to flat files in CSV or JSON Lines format. Any further data processing or analytics is subject to third-party programs.

Local Monitoring

The majority of the DMPACK programs depend on inter-process communication and database access. For instance, a dmserial process controlling an attached sensor may transmit observations and logs in real-time to the message queues of dmdb and dmlogger to be stored in the local databases. The database records could then be exported periodically to flat files. Synchronisation with a remote server is optional.

Distributed Monitoring

Sensor nodes connected to a central monitoring server can run dmsync to send observations and log messages automatically to the HTTP-RPC interface of dmapi to be stored in the server databases. Additionally, the server may collect status messages to monitor the condition of the sensor nodes.

The programs are conceived to run headless on embedded devices and servers. Some of them, such as dmapi, dmreport, or dmweb, can be deployed on sensor nodes and monitoring servers alike.

Similar Software

There are similar open source projects that provide middleware for autonomous sensor networks:

52°North Sensor Observation Service

The reference implementation of the OGC Sensor Observation Service (SOS) in Java, by 52°North Spatial Information Research GmbH. Offers an interoperable interface for publishing and querying sensor data and meta data. Additional client applications enable analysis and visualisation of the measurement data. The project is mostly inactive. (GPLv2)

Argus

A non-geodetic sensor data monitoring and alerting solution built with Node.js, MariaDB, and React. (MIT)

FROST

Fraunhofer Open Source SensorThings (FROST) is the reference implementation of the OGC SensorThings API in Java. The project provides an HTTP- and MQTT-based message bus for data transmission between client and server. Developed by Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung (IOSB), Karlsruhe. (LGPLv3)

Global Sensor Networks

A Java-based software middleware designed to facilitate the deployment and programming of sensor networks, by Distributed Information Systems Laboratory (EPFL), Switzerland. The project appears to be abandoned. (GPLv3)

istSOS2

A server implementation of the OGC Sensor Observation Service in Python, for managing and dispatching observations from monitoring sensors. The project also provides a graphical user interface and a RESTful web API to automate administration procedures. Developed by Istituto Scienze della Terra, University of Applied Sciences and Arts of Southern Switzerland. The software seems not to be actively maintained anymore. (GPLv2)

Kotori

A multi-channel, multi-protocol, telemetry data acquisition and graphing toolkit for time-series data processing in Python. It supports scientific environmental monitoring projects, distributed sensor networks, and likewise scenarios. (AGPLv3)

OpenADMS

The Open Automatic Deformation Monitoring software is an IoT sensor network middleware in Python 3. The system was developed as a prototype of DMPACK and includes client and server programs. (BSD)

OpenSensorHub

Java-based middleware for building Sensor Webs in the Internet of Things. Based on OGC standards from the Sensor Web Enablement (SWE) initiative. (MPLv2)

Project Mjolnir

An open source client–server IoT architecture for scientific sensor networks written in Python, by University of Alabama in Huntsville and NASA. Includes a sensor client for data logging, uplink and control, as well as a server component to store, serve/display, and monitor data from remote sensors. Further development of the software has been stopped. (MIT)

TSDSystem

The Time Series Database management System (TSDSystem) is a framework to collect, archive and share time series of volcanological observatories from sensor networks at Istituto Nazionale di Geofisica e Vulcanologia, Osservatorio Etneo (Italy). It provides a web service to perform writing and reading data via standard web communication protocols. Basic web applications are supplied for joint and synchronised time series data visualisation as well as representation of stations on a geographical map. (MIT)

Ulyxes

An open source project in Python to control robotic total stations (RTS) and other sensors, and to publish observation results on web based maps. Developed at the Department of Geodesy and Surveying of the Budapest University of Technology and Economics. (GPLv2)

Requirements

DMPACK has the following requirements:

  • Linux or FreeBSD operating system

  • 64-bit platform (x86-64, AArch64)

  • Fortran 2018 and ANSI C compiler (GNU, Intel)

Additional dependencies have to be present to build and run the software of this package:

Optionally, for client-side camera access:

The web applications require a compatible web server, like:

DMPACK depends on additional interface libraries. If the repository is cloned recursively with Git, or if the project is built using FPM, the submodules will be downloaded automatically. Otherwise, run script fetchvendor.sh first.

Paths used by default
Path Description

/usr/local/bin/

DMPACK programs.

/usr/local/etc/dmpack/

DMPACK configuration files.

/usr/local/include/dmpack/

DMPACK module files.

/usr/local/lib/

DMPACK libraries.

/usr/local/man/man1/

DMPACK man pages.

/usr/local/share/dmpack/

DMPACK examples, scripts, style sheets.

/var/dmpack/

DMPACK databases.

/var/www/

WWW root directory.

Installation

This section describes the steps to build the DMPACK library and programs from source, either with POSIX make or the Fortran Package Manager (FPM). At the moment, support for the Fortran Package Manager is experimental, and using GNU/BSD Make is the recommended way. Display the available build targets of the Makefile:

$ make help

Or, output the selected build options:

$ make options PREFIX=/opt

See section System Configuration on how to configure the operating system after the installation. The shared libraries libgcc.so, libgfortran.so, and libquadmath.so must be present on the target system if the DMPACK programs have been compiled with GNU Fortran.

Linux

On Debian, install the compilers and the build environment first:

$ sudo apt-get install gcc gfortran git make pkg-config

The third-party dependencies have to be installed with development headers:

$ sudo apt-get install --no-install-recommends curl ffmpeg gnuplot graphicsmagick \
  libblas-dev libcurl4 libcurl4-openssl-dev libfcgi-bin libfcgi-dev libhdf5-103-1 \
  libhdf5-dev liblapack-dev liblua5.4 liblua5.4-dev libmodbus5 libmodbus-dev \
  libpcre2-8-0 libpcre2-dev libsqlite3-0 libsqlite3-dev libstrophe0 libstrophe-dev \
  libzstd1 libzstd-dev lua5.4 sqlite3 zlib1g zlib1g-dev

Instead of package gnuplot, you may prefer the no-X11 flavour gnuplot-nox if raster graphic formats are not required (limiting the output formats essentially to SVG only). The packages ffmpeg and graphicsmagick are only required for client-side camera access and may be omitted. The SQLite 3 package version must be ≥ 3.39.0. If the version in the package repository is too old, like on Ubuntu 22.04 LTS, you can also build the library from source. Depending on the Linux distribution, the names of the HDF5 and Lua packages may differ.

Note

If Intel oneAPI is used, it is necessary to build HDF5 from source, as the versions in the Linux package repositories have been compiled with GNU Fortran and are therefore incompatible. See section HDFView for hints regarding the build process.

Make

Clone the DMPACK repository with Git, using command-line argument --recursive:

$ git clone --depth 1 --recursive https://github.com/dabamos/dmpack
$ cd dmpack/

If Git is not available, download the archive of the master branch instead and run the shell script fetchvendor.sh to fetch the missing submodules:

$ curl -L -s -o dmpack-master.zip https://github.com/dabamos/dmpack/archive/refs/heads/master.zip
$ unzip dmpack-master.zip
$ cd dmpack-master/
$ sh fetchvendor.sh

Then, execute build target linux of the Makefile to compile the source:

$ make linux

On 64-bit ARM platforms, like those of the Raspberry Pi 3/4/5, select build target linux_aarch64 instead:

$ make linux_aarch64

Install the DMPACK libraries and programs system-wide to /usr/local:

$ sudo make install

Or, to install to directory /opt, run:

$ sudo make install PREFIX=/opt

Path /opt/bin/ must be added to the global PATH environment variable to run DMPACK programs from the command-line.

Note
Custom SQLite 3

If the SQLite 3 library has been built from source and installed to /usr/local/lib, overwrite the variable LIBSQLITE3 to pass the path of shared library libsqlite3.so:

$ make build OS=linux LIBSQLITE3="-L/usr/local/lib -lsqlite3"

If more than one library is installed, additionally specify the path with linker flag -Wl,-rpath=/usr/local/lib.

Note
Intel oneAPI Compilers

If Intel oneAPI is used instead of the GNU Compiler Collection, run:

$ make build OS=linux CC=icx FC=ifx PPFLAGS= \
  CFLAGS="-mtune=native -O2 -fpic" FFLAGS="-mtune=native -O2 -fpic" \
  LDFLAGS="-module ./include -I./include" \
  INCHDF5="-I/opt/include" \
  LIBHDF5="-Wl,-rpath=/opt/lib -L/opt/lib -lhdf5_fortran -lhdf5"

In this particular case, the HDF5 libraries are installed to /opt/lib/, and the HDF5 modules files to /opt/include/. Change the paths to the actual locations.

FPM

To build DMPACK using the Fortran Package Manager, change to the cloned or downloaded repository, and run:

$ export FFLAGS="-D__linux__ `pkg-config --cflags hdf5` -ffree-line-length-0"
$ fpm test --flag "$FFLAGS"
$ fpm build --profile release --flag "$FFLAGS"
$ fpm install

The library and programs will be installed to directory ~/.local by default. If the compilation fails with an error message stating that -llua-5.4 cannot be found, update the library names in the build manifests:

$ sed -i "s/lua-5/lua5/g" fpm.toml
$ sed -i "s/lua-5/lua5/g" build/dependencies/fortran-lua54/fpm.toml

FreeBSD

First, install the build and run-time dependencies:

$ doas pkg install archivers/zstd comms/libmodbus databases/sqlite3 devel/git \
  devel/pcre2 devel/pkgconf ftp/curl graphics/GraphicMagick lang/gcc \
  lang/lua54 math/gnuplot math/lapack multimedia/ffmpeg net-im/libstrophe \
  science/hdf5 www/fcgi

Instead of math/gnuplot, you may want to install package math/gnuplot-lite which does not depend on X11 (but lacks the raster graphic terminals). The web applications additionally require a web server:

$ doas pkg install www/lighttpd

Optionally, install Pygments and AsciiDoctor to generate the man pages and the User Guide from source:

$ doas pkg install devel/rubygem-pygments.rb textproc/rubygem-asciidoctor

Make

The repository has to be cloned recursively using command-line argument --recursive:

$ git clone --depth 1 --recursive https://github.com/dabamos/dmpack
$ cd dmpack/

If Git is not available, download the archive of the master branch and run the shell script fetchvendor.sh to fetch the submodules:

$ curl -L -s -o dmpack.zip https://github.com/dabamos/dmpack/archive/refs/heads/master.zip
$ unzip dmpack.zip
$ cd dmpack-master/
$ sh fetchvendor.sh

Execute the Makefile with build target freebsd:

$ make freebsd

Install the library and all programs system-wide to /usr/local:

$ doas make install

You can change the installation prefix with argument PREFIX. To install to directory /opt instead, run:

$ doas make install PREFIX=/opt

In this case, path /opt/bin must be included in the global PATH environment variable.

FPM

Either clone the repository with Git, or download the archive of the master branch. Then, run:

$ export FFLAGS="-D__FreeBSD__ -I/usr/local/include -ffree-line-length-0"
$ fpm test --flag "$FFLAGS"
$ fpm build --profile release --flag "$FFLAGS"
$ fpm install

The Fortran Package Manager will fetch all third-party dependencies automatically, but the configuration and shared files have to be installed manually. The library and programs will be installed to ~/.local by default.

Updates

Update the cloned source code repository and its submodules with Git:

$ git pull
$ git submodule update --remote
$ make purge
$ make [freebsd|linux|linux_aarch64]
$ sudo make install PREFIX=/opt

Deformation Monitoring Entities

The data structures of DMPACK are based on the following entities. The date and time format used internally is a 32-characters long ISO 8601 time stamp in microsecond resolution, with time separator T and mandatory GMT offset, for example, 1970-01-01T00:00:00.000000+00:00. The human-readable output format 1970-01-01 00:00:00 +00:00 is used where reasonable. Global coordinates are in longitude/latitude order (east-west/north-south order).

Observation Entities

Node

A unique sensor node within a sensor network. Contains id, name, description, and optional position.

Sensor

A unique sensor attached to a node, with id, name, description, and optional position.

Target

A unique measurement target (point of interest, location) with id, name, description, and optional position. Multiple nodes and sensors may share a single target.

Observation

A single measurement identified by name and unique UUID identifier that contains requests to and responses from a sensor, referencing a node, a sensor, and a target. An observation can contain up to 8 requests which will be sent to the sensor in sequential order.

Request

Command to send to the sensor, referencing an observation and ordered by index. A request can contain up to 16 responses.

Response

Floating-point values in the raw response of a sensor can be matched by regular expression groups. Each matched group is stored as a response. Responses reference a request, and are ordered by index. They contain name, type, value, unit, and an optional error code.

Log Entities

Log

Log message of a sensor node, either of level debug, info, warning, error, critical, or user, and optionally related to a sensor, a target, and an observation.

Beat Entities

Beat

Short status message (heartbeat, handshake) that contains node id, client address, client version, time stamp, system uptime, and last connection error, sent periodically from client to server.

RPC Entities

API Status

Short key–value response of the HTTP-RPC API service in plain-text format.

Program Overview

DMPACK includes programs for sensor I/O, database management, observation processing, and other tasks related to automated control measurements. The programs may be classified into the following categories.

Databases

dmbackup

Creates an online backup of a database by either using the SQLite backup API or VACUUM INTO.

dmdb

Stores observations received from POSIX message queue in a SQLite database.

dmdbctl

A command-line interface to the DMPACK observation database, to read, add, update, or delete nodes, sensors, and targets.

dmexport

Exports beats, nodes, sensors, targets, observations, and logs from database to file, either in CSV, JSON, or JSON Lines format.

dmimport

Imports nodes, sensors, targets, observations, and logs from CSV file into database.

dminit

Creates and initialises observation, log, and beat databases.

dmlogger

Stores logs received from POSIX message queue in a SQLite database.

Message Passing

dmlog

A utility program to send log messages from command-line or shell script to the POSIX message queue of a dmlogger process, to be stored in the log database.

dmrecv

Receives logs or observations from POSIX message queue and writes them to stdout, file, or named pipe.

dmsend

Sends observations or logs from file to a DMPACK application via POSIX message queue.

Observation Processing

dmgrc

Inspects received observations and creates log messages from GeoCOM return codes.

dmlua

Runs a custom Lua script to process observations and forward them to the next specified receiver.

Plots & Reports

dmplot

Creates line plots of time series read from database, with output to file, terminal, or X11 window. Uses gnuplot(1) internally as plotting back-end.

dmreport

Creates HTML reports containing plots and optionally log messages.

Remote Procedure Calls

dmapi

A FastCGI-based HTTP-RPC service that provides an API for node, sensor, target, observation, and log synchronisation, as well as heartbeat transmission. Clients may either send records to be stored in the server database, or request data of a given time range. Depending on the HTTP Accept header, the server returns data in CSV, JSON, JSON Lines or Namelist format. Requires a FastCGI-compatible web server, such as lighttpd(1).

dmbeat

Sends short status messages (heartbeats) periodically to a remote dmapi instance.

dmbot

Accepts commands sent via XMPP/Jabber to monitor the status of sensor nodes.

dmsync

Synchronises nodes, sensors, targets, observations, and log messages between client and dmapi server. Only uni-directional synchronisation from client to server is supported.

Sensor Control

dmfs

Reads sensor data from virtual file system, file, or named pipe. The program be used to read values from sensors connected via 1-Wire (OWFS). Observations are forwarded via POSIX message queue and/or written to file.

dmpipe

Executes a program as a sub-process connected through an anonymous pipe and forwards the output via POSIX message queue. Optionally, observations are written to file or stdout.

dmserial

Connects to a TTY/PTY serial port for sensor communication. The program sends requests to a connected sensor to receive responses. The program pre-processes the response data using regular expressions and forwards observations via POSIX message queue.

Utilities

dminfo

Prints system and database information as key–value pairs to standard output.

dmuuid

A command-line tool to generate unique UUID identifiers (by default in hexadecimal format without hyphens).

Web

dmfeed

Creates an Atom syndication feed in XML format (RFC 4287) from logs of given sensor node and log level. If the feed is served by a web server, clients can subscribe to it by using a feed reader or news aggregator. The program may be executed periodically as a cron job.

dmweb

A CGI-based web user interface for DMPACK database access on client and server. Requires a web server and gnuplot(1).

Programs

This section contains descriptions of all DMPACK programs with their respective command-line arguments. Some programs read settings from an optional or mandatory configuration file. Example configuration files are provided in directory /usr/local/etc/dmpack/.

The files are ordinary Lua scripts, i.e., you can add Lua control structures for complex tables or access the Lua API of DMPACK. In your editor, set the language to Lua to enable syntax highlighting (for instance, set syntax=lua in Vim), or use file ending .lua instead of .conf. The set-up of the web applications is outlined in the next section.

dmapi

dmapi is an HTTP-RPC API service for remote DMPACK database access. The web application has to be executed through a FastCGI-compatible web server. It is recommended to use lighttpd(1). The service is configured through environment variables. The web server or FastCGI spawner must be able to pass environment variables to dmapi.

The dmapi service offers endpoints for clients to insert beats, logs, and observations into the local SQLite database, and to request data in CSV or JSON format. Only HTTP GET and POST requests are accepted. All POST data has to be serialised in Fortran 95 Namelist format, with optional deflate or zstd compression. Section RPC API gives an overview of the available endpoints.

Authentication and encryption are independent from dmapi and have to be provided by the web server. If HTTP Basic Auth is enabled, the sensor id of each beat, log, node, sensor, and observation sent to the HTTP-RPC service must match the name of the authenticated user. For example, to store an observation of a node with the id node-1, the user name of the client must be node-1 as well. If the observation is sent by any other user, it will be rejected (HTTP 401).

Environment variables of dmapi(1)
Environment Variable Description

DM_DB_BEAT

Path to heartbeat database (required).

DM_DB_LOG

Path to log database (required).

DM_DB_OBSERV

Path to observation database (required).

DM_READ_ONLY

Set to 1 to enable read-only database access.

The response format depends on the MIME type set in the HTTP Accept header of the request, either:

  • application/json (JSON)

  • application/jsonl (JSON Lines)

  • application/namelist (Fortran 95 Namelist)

  • text/comma-separated-values (CSV)

  • text/plain (plain text)

By default, responses are in CSV format. The Namelist format is available only for single records. Status messages are returned as key–value pairs, indicated by content type text/plain.

See section RPC Server for a basic lighttpd(1) configuration.

dmbackup

The dmbackup utility creates an online backup of a running SQLite database. By default, the SQLite backup API is used. The program is functional equivalent to running the sqlite3(1) command-line interface:

$ sqlite3 <database> ".backup '<output>'"

dmbackup does not replace existing backup databases.

Command-Line Options

Option Short Default Description

--backup file

-b

–

Path of the backup database.

--database file

-d

–

Path of the SQLite database to backup.

--help

-h

–

Output available command-line arguments and quit.

--vacuum

-U

off

Use VACUUM INTO instead of the SQLite backup API.

--verbose

-V

off

Print backup progess (not in vacuum mode).

--version

-v

–

Output version information and quit.

--wal

-W

off

Enable WAL journal for backup database.

Examples

Create an online backup of an observation database:

$ dmbackup --database /var/dmpack/observ.sqlite --backup /tmp/observ.sqlite

dmbeat

The dmbeat program is a heartbeat emitter that sends handshake messages via HTTP POST to a remote dmapi service. Heartbeats include the following attributes:

Attribute Description

node_id

Node id.

address

IPv4/IPv6 address of client.

client

Client software name and version.

time_sent

Date and time heartbeat was sent (ISO 8601).

time_recv

Date and time heartbeat was received (ISO 8601).

error

Last client connection error.

interval

Emit interval in seconds.

uptime

Client uptime in seconds.

The server may inspect the data to check if a client is still running and has network access. The RPC endpoint on the server is expected at URL [http|https]://<host>:<port>/api/v1/beat.

Command-Line Options

Option Short Default Description

--compression name

-x

zstd

Compression library to use (none, zlib, zstd).

--config file

-c

–

Path to configuration file.

--count n

-C

0

Number of heartbeats to send (unlimited if 0).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--host host

-H

–

IP or FQDN of HTTP-RPC API host (for instance, 127.0.0.1 or iot.example.com).

--interval seconds

-I

0

Emit interval in seconds.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmbeat

Optional name of instance and table in configuration.

--node id

-N

–

Node id.

--password string

-P

–

API password.

--port port

-q

0

Port of HTTP-RPC API server (0 for automatic).

--tls

-E

off

Use TLS encryption.

--username string

-U

–

API user name. If set, implies HTTP Basic Auth.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Send a single heartbeat to a dmapi service on localhost:

$ dmbeat --node dummy-node --host 127.0.0.1 --count 1 --verbose

A sensor node with id dummy-node must exist in the server database. The web application dmweb lists the beats received by the server.

dmbot

The dmbot program is an XMPP bot that accepts commands via chat. Access to the bot is limited to the JIDs added to table group in the configuration file. Requests from clients whose JID is not in the table will be rejected. If table group is empty, all clients are allowed to send commands to the bot.

The XMPP resource is automatically set to the name of the bot instance. If the JID of the bot account is bot@example.com, the full JID will be set to bot@example.com/dmbot if the bot name is dmbot.

All commands start with prefix !. For an overview, send chat command !help to the bot. The following commands are supported:

!beats

Return current time of the sensor node in Swatch Internet Time (.beats).

!date

Return date and time of the sensor node in ISO 8601.

!help

Return help text.

!jid

Return full JID of bot.

!log <level> "<message>"

Send log message of given level to logger. The argument level must be a valid log level name or numeric log level. The argument message must be in quotes if it contains spaces.

!node

Return node id of bot.

!poke

Return a message if the bot is online.

!reconnect

Reconnect bot and update local roster list.

!uname

Return name and version of the operating system.

!uptime

Return uptime of the operating system.

!version

Return bot version.

Passing the XMPP credentials via the command-line arguments --jid and --password is insecure on multi-user operating systems and only recommended for testing.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file.

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--host host

-H

–

FQDN of XMPP server (for instance, example.com).

--jid string

-J

–

Bot Jabber id (for example, bot@example.com).

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmbot

Optional name of instance, XMPP resource, and table in configuration.

--node id

-N

–

Node id.

--password string

-P

–

Bot password.

--port port

-q

5222

Port of XMPP server.

--reconnect

-R

off

Reconnect on error.

--tls

-E

off

Force TLS encryption.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Connect with JID bot@example.com to an XMPP server on port 5223 and wait for commands:

$ dmbot --node dummy-node --jid bot@example.com --password secret \
  --host example.com --port 5223 --tls --verbose

If no configuration file is used, any client may send commands to the bot without authorisation. Start a chat with the bot JID and send a command. For instance, on command !uptime the bot sends a reply like the following:

uptime: 0 days 23 hours 57 mins 32 secs

dmdb

The dmdb program collects observations from a POSIX message queue and stores them in a SQLite database. The name of the message queue equals the given dmdb name and leading /. The IPC option enables process synchronisation via POSIX semaphores. The value of the semaphore is changed from 0 to 1 if a new observation has been received. Only a single process shall wait for the semaphore.

Only observation types in binary format are accepted. Log messages are stored to database by the distinct dmlogger program.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file.

--database file

-d

–

Path to SQLite observation database.

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--ipc

-Q

off

Uses a POSIX semaphore for process synchronisation. The name of the semaphore matches the instance name (with leading /). The semaphore is set to 1 whenever a new observation was received. Only a single process may wait for this semaphore, otherwise, reading occurs in round-robin fashion.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmdb

Optional name of program instance, configuration, POSIX message queue, and POSIX semaphore.

--node id

-N

–

Node id.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Create a message queue /dmdb, wait for incoming observations, and store them in the given database:

$ dmdb --name dmdb --node dummy-node --database /var/dmpack/observ.sqlite --verbose

Log messages and observation ids are printed to stdout if argument --verbose is set.

dmdbctl

The dmdbctl utility program performs create, read, update, or delete operations (CRUD) on the observation database.

Create

Add nodes, sensors, and targets to the database.

Read

Read nodes, sensors, and targets from database. Print the records to standard output.

Update

Update nodes, sensors, and targets in the database.

Delete

Delete nodes, sensors, and targets from the database.

Only nodes, sensors, and targets are supported. All data attributes are passed through command-line arguments.

Command-Line Options

Option Short Default Description

--alt alt

-A

–

Node, sensor, or target altitude (optional).

--create type

-C

–

Create record of given type (node, sensor, or target).

--database file

-d

–

Path to SQLite observation database (required).

--delete type

-D

–

Delete record of given type (node, sensor, or target).

--help

-h

–

Output available command-line arguments and quit.

--id id

-I

–

Node, sensor, or target id (required).

--lat lat

-L

–

Node, sensor, or target latitude (optional).

--lon lon

-G

–

Node, sensor, or target longitude (optional).

--meta meta

-M

–

Node, sensor, or target meta description (optional).

--name name

-n

–

Node, sensor, or target name.

--node id

-N

–

Id of node the sensor is associated with.

--read type

-R

–

Read record of given type (node, sensor, or target).

--sn sn

-Q

–

Serial number of sensor (optional).

--state n

-S

–

Target state (optional).

--type name

-t

none

Sensor type (none, rts, gnss, …).

--update type

-U

–

Updates record of given type (node, sensor, or target).

--verbose

-V

off

Print additional log messages to stderr.

--version

-v

–

Output version information and quit.

--x x

-X

–

Local node, sensor, or target x (optional).

--y y

-Y

–

Local node, sensor, or target y (optional).

--z z

-Z

–

Local node, sensor, or target z (optional).

Examples

Add node, sensor, and target to observation database:

$ dmdbctl -d observ.sqlite -C node --id node-1 --name "Node 1"
$ dmdbctl -d observ.sqlite -C sensor --id sensor-1 --name "Sensor 1" --node node-1
$ dmdbctl -d observ.sqlite -C target --id target-1 --name "Target 1"

Delete a target from the database:

$ dmdbctl -d observ.sqlite -D target --id target-1

Read attributes of sensor sensor-1:

$ dmdbctl -d observ.sqlite -R sensor --id sensor-1
sensor.id: sensor-1
sensor.node_id: node-1
sensor.type: virtual
sensor.name: Sensor 1
sensor.sn: 12345
sensor.meta: dummy sensor
sensor.x: 0.000000000000
sensor.y: 0.000000000000
sensor.z: 0.000000000000
sensor.lon: 0.000000000000
sensor.lat: 0.000000000000
sensor.alt: 0.000000000000

dmexport

The dmexport program writes beats, logs, nodes, sensors, targets, observations, and data points from database to file, in ASCII block, CSV, JSON, or JSON Lines format. The ASCII block format is only available for X/Y data points. The types data point, log, and observation require a sensor id, a target id, and a time range in ISO 8601 format.

If no output file is given, the data is printed to standard output. The output file will be overwritten if it already exists. If no records are found, an empty file will be created.

Output file formats
Type Block CSV JSON JSONL

beat

✓

✓

✓

dp

✓

✓

✓

✓

log

✓

✓

✓

node

✓

✓

✓

observ

✓

✓

✓

sensor

✓

✓

✓

target

✓

✓

✓

Command-Line Options

Option Short Default Description

--database file

-d

–

Path to SQLite database (required).

--format format

-f

–

Output file format (block, csv, json, jsonl).

--from timestamp

-B

–

Start of time range in ISO 8601 (required for types dp, log, and observ).

--header

-H

off

Add CSV header.

--help

-h

–

Output available command-line arguments and quit.

--node id

-N

–

Node id (required).

--output file

-o

–

Path of output file.

--response name

-R

–

Response name for type dp.

--sensor id

-S

–

Sensor id (requied for types dp and observ).

--separator char

-s

,

CSV field separator.

--target id

-T

–

Target id (required for types dp and observ).

--to timestamp

-E

–

End of time range in ISO 8601 (required for types dp, log, observ).

--type type

-t

–

Type of record to export: beat, dp, log, node, observ, sensor, target (required).

--version

-v

–

Output version information and quit.

Examples

Export log messages from database to JSON file:

$ dmexport --database log.sqlite --type log --format json --node dummy-node \
  --from 2020-01-01 --to 2023-01-01 --output /tmp/log.json

Export observations from database to CSV file:

$ dmexport --database observ.sqlite --type observ --format csv --node dummy-node \
  --sensor dummy-sensor --target dummy-target --from 2020-01-01 --to 2025-01-01 \
  --output /tmp/observ.csv

dmfeed

The dmfeed program creates a web feed from log messages in Atom Syndication Format. The log messages are read from database and written as XML to standard output or file.

The feed id has to be a 36 characters long UUID with hyphens. News aggregators will use the id to identify the feed. Therefore, the id should not be reused among different feeds. Run dmuuid to generate a valid UUIDv4.

The time stamp of the feed in element updated is set to the date and time of the last log message. If no logs have been added to the database since the last file modification of the feed, the output file is not updated, unless argument --force is passed. To update the feed periodically, add dmfeed to crontab.

If an XSLT style sheet is given, web browsers may be able to display the Atom feed in HTML format. Set the option to the (relative) path of the public XSL on the web server. An example style sheet feed.xsl is located in /usr/local/share/dmpack/.

Command-Line Options

Option Short Default Description

--author name

-A

–

Name of feed author or organisation.

--config file

-c

–

Path to configuration file.

--database file

-d

–

Path to SQLite log database.

--email address

-M

–

E-mail address of feed author (optional).

--entries count

-E

50

Maximum number of entries in feed (max. 500).

--force

-F

–

Force file output even if no new log records are available.

--help

-h

–

Output available command-line arguments and quit.

--id uuid

-I

–

UUID of the feed, 36 characters long with hyphens.

--maxlevel level

-K

critical

Select log messages of the given maximum log level (from debug or 1 to user or 6). Must be greater or equal the minimum level.

--minlevel level

-L

debug

Select log messages of the given minimum log level (from debug or 1 to user or 6).

--name name

-n

dmfeed

Name of instance and table in configuration.

--node id

-N

–

Select log messages of the given node id.

--output file

-o

stdout

Path of the output file. If empty or -, the Atom feed will be printed to standard output.

--subtitle string

-G

–

Sub-title of feed.

--title string

-C

–

Title of feed.

--url url

-U

–

Public URL of the feed.

--version

-v

–

Output version information and quit.

--xsl

-x

–

Path to XSLT style sheet.

Examples

First, generate a unique feed id:

$ dmuuid --hyphens
19c12109-3e1c-422c-ae36-3ba19281f2e

Then, write the last 50 log messages in Atom format to file feed.xml, and include a link to the XSLT style sheet feed.xsl:

$ dmfeed --database /var/dmpack/log.sqlite --output /var/www/feed.xml \
  --id 19c12109-3e1c-422c-ae36-3ba19281f2e --xsl feed.xsl

Copy the XSLT style sheet to the directory of the Atom feed:

$ cp /usr/local/share/dmpack/feed.xsl /var/www/

If /var/www/ is served by a web server, feed readers can subscribe to the feed. Additionally, we may translate feed and style sheet into a single HTML document feed.html, using an arbitrary XSLT processor, for instance:

$ xsltproc --output feed.html /var/www/feed.xsl /var/www/feed.xml

dmfs

The dmfs program reads observations from file system, virtual file, or named pipe. The program can be used to read sensor data from the 1-Wire File System (OWFS).

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. dmfs can act as a sole data logger if output and format are set. If the output path is set to -, observations are written to stdout instead of file.

The requests of each observation have to contain the path of the (virtual) file in attribute request. Response values are extracted by named group from the raw response using the given regular expression pattern. Afterwards, the observation is forwarded to the next receiver via POSIX message queue.

A configuration file is mandatory to describe the jobs to perform. Each observation must have a valid target id. Node, sensor, and target have to be present in the database.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file (required).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--format format

-f

–

Output format, either csv or jsonl.

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmfs

Name of instance and table in configuration.

--node id

-N

–

Node id.

--output file

-o

–

Output file to append observations to (- for stdout).

--sensor id

-S

–

Sensor id.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Start dmfs to execute the jobs in the configuration file:

$ dmfs --name dmfs --config /usr/local/etc/dmpack/dmfs.conf --verbose

dmgrc

The dmgrc program creates log messages from Leica GeoCOM return codes. Observations received by POSIX message queue are searched for a GeoCOM return code (GRC) response. If the code does not equal GRC_OK, a log message is sent to the configured logger instance.

By default, observation responses of name grc are verified. For each GeoCOM error code, a custom log level may be specified in the configuration file. Otherwise, the default log level is used instead.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file (required).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--level level

-L

3

Default log level (from debug or 1 to user or 6).

--logger name

-l

–

Name of dmlogger process to send logs to.

--name name

-n

dmgrc

Name of instance and table in configuration.

--node id

-N

–

Node id.

--response name

-R

grc

Response name of the GeoCOM return code.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

A configuration file is not required, but allows to specifiy the log level of certain GeoCOM return codes. In the following example configuration, the default log level for all return codes other than GRC_OK is set to LL_WARNING. The level is further refined for specific GeoCOM codes:

-- dmgrc.conf
dmgrc = {
  logger = "dmlogger",
  node = "dummy-node",
  response = "grc",
  level = LL_WARNING,
  levels = {
    debug = { GRC_ABORT, GRC_SHUT_DOWN, GRC_NO_EVENT },
    info = { GRC_SLEEP_NODE, GRC_NA, GRC_STOPPED },
    warning = { GRC_TMC_ACCURACY_GUARANTEE, GRC_AUT_NO_TARGET },
    error = { GRC_FATAL },
    critical = {},
    user = {}
  },
  debug = false,
  verbose = true
}

See section GeoCOM API for a table of all supported return codes. Pass the path of the configuration file through the command-line argument:

$ dmgrc --name dmgrc --config /usr/local/etc/dmpack/dmgrc.conf

The name argument must match the name of the configuration table. A logger process of name dmlogger must be running to process the generated log messages.

dminfo

The dminfo utility program prints build, database, and system information to standard output. The path to the beat, log, or observation database is passed through command-line argument --database. Only one database can be specified.

The output contains compiler version and options; database PRAGMAs, tables, and number of rows; as well as system name, version, and host name.

Command-Line Options

Option Short Default Description

--database file

-d

–

Path to SQLite database.

--help

-h

–

Output available command-line arguments and quit.

--version

-v

–

Output version information and quit.

Examples

Print build, database, and system information:

$ dminfo --database /var/dmpack/observ.sqlite
build.compiler: GCC version 13.2.0
build.options: -mtune=generic -march=x86-64 -std=f2018
db.application_id: 444D31
db.foreign_keys: T
db.journal_mode: wal
db.library: libsqlite3/3.46.0
db.path: /var/dmpack/observ.sqlite
db.schema_version: 1
db.size: 286720
db.table.nodes.rows: 1
db.table.observs.rows: 202
...

dmimport

The dmimport program reads logs, nodes, sensors, targets, and observations in CSV format from file and imports them into the database. The database inserts are transaction-based. If an error occurs, the transaction is rolled back, and no records are written to the database at all.

The database has to be a valid DMPACK database and must contain the tables required for the input records. The nodes, sensors, and targets referenced by input observations must exist in the database. The nodes referenced by input sensors must exist as well.

Command-Line Options

Option Short Default Description

--database file

-d

–

Path to SQLite database (required, unless in dry mode).

--dry

-D

off

Dry mode. Reads and validates records from file but skips database import.

--help

-h

–

Output available command-line arguments and quit.

--input file

-i

–

Path to input file in CSV format (required).

--quote char

-q

–

CSV quote character.

--separator char

-s

,

CSV field separator.

--type type

-t

–

Type of record to import, either log, node, observ, sensor, target (required).

--verbose

-V

off

Print progress to stdout.

--version

-v

–

Output version information and quit.

Examples

Import observations from CSV file observ.csv into database observ.sqlite:

$ dmimport --type observ --input observ.csv --database observ.sqlite --verbose

dminit

The dminit utility program creates beat, log, and observation databases. No action is performed if the specified database already exists. A synchronisation table is required for observation and log synchronisation with an dmapi server. The argument can be omitted if this feature is not needed. The journal mode Write-Ahead Logging (WAL) should be enabled for databases with multiple readers.

Command-Line Options

Option Short Default Description

--database file

-d

–

Path of the new SQLite database (required).

--force

-F

off

Force the table creation even if the database already exists.

--help

-h

–

Output available command-line arguments and quit.

--sync

-s

off

Add synchronisation tables. Enable for data synchronisation between client and server.

--type type

-t

–

Type of database, either beat, log, or observ (required).

--version

-v

–

Output version information and quit.

--wal

-W

off

Enable journal mode Write-Ahead Logging (WAL).

Examples

Create an observation database with remote synchronisation tables (WAL):

$ dminit --database /var/dmpack/observ.sqlite --type observ --sync --wal

Create a log database with remote synchronisation tables (WAL):

$ dminit --database /var/dmpack/log.sqlite --type log --sync --wal

Create a heartbeat database (WAL):

$ dminit --database /var/dmpack/beat.sqlite --type beat --wal

dmlog

The dmlog utility forwards a log message to the message queue of a dmlogger or dmrecv instance. The program may be executed through a shell script to add logs to the DMPACK database. The argument --message is mandatory. The default log level is info. Pass the name of the dmlogger or dmrecv instance to send the log to through command-line argument --logger.

Logs are sent in binary format. The program terminates after log transmission. The log level may be one of the following:

Level Parameter String Description

1

debug

Debug message.

2

info

Hint or info message.

3

warning

Warning message.

4

error

Non-critical error message.

5

critical

Critical error message.

6

user

User-defined log level.

Both, parameter strings and literal log level values, are accepted as command-line arguments. For level warning, set argument --level to 3 or warning.

Command-Line Options

Option Short Default Description

--error n

-e

0

DMPACK error code (optional).

--help

-h

–

Output available command-line arguments and quit.

--level level

-L

info

Log level, from debug or 1 to user or 6.

--logger name

-l

dmlogger

Name of logger instance and POSIX message queue.

--message string

-m

–

Log message (max. 512 characters).

--node id

-N

–

Node id (optional).

--observ id

-O

–

Observation id (optional).

--sensor id

-S

–

Sensor id (optional).

--source source

-Z

–

Source of the log message (optional).

--target id

-T

–

Target id (optional).

--verbose

-V

off

Print log to stderr.

--version

-v

–

Output version information and quit.

Examples

Send a log message to the message queue of logger dmlogger:

$ dmlog --level warning --message "low battery" --source dmlog --verbose
2022-12-09T22:50:44.161000+01:00 [WARNING] dmlog - low battery

The dmlogger process will receive the log message in real-time and store it in the log database (if the log level is ≥ the configured minimum log level):

$ dmlogger --node dummy-node --database /var/dmpack/log.sqlite --verbose
2022-12-09T22:50:44.161000+01:00 [WARNING] dmlog - low battery

dmlogger

The dmlogger program collects log messages from a POSIX message queue and writes them to a SQLite database. The name of the message queue will equal the given dmlogger name with leading /, by default /dmlogger.

If a minimum log level is selected, only logs of a level greater or equal the minimum are stored in the database. Log messages with a lower level are printed to standard output before being discarded (only if the verbose flag is enabled).

The IPC option allows an optional process synchronisation via a named POSIX semaphores. The value of the semaphore is changed from 0 to 1 whenever a new log was received. The name of the semaphore will equal the dmlogger name with leading /.

Only a single process should wait for the semaphore unless round-robin passing is desired. This feature may be used to automatically synchronise incoming log messages with a remote HTTP-RPC API server. dmsync will wait for new logs before starting synchronisation if the dmlogger instance name has been passed through command-line argument --wait.

The following log levels are accepted:

Level Parameter String Description

1

debug

Debug message.

2

info

Hint or info message.

3

warning

Warning message.

4

error

Non-critical error message.

5

critical

Critical error message.

6

user

User-defined log level.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file.

--database file

-d

–

Path to SQLite log database.

--help

-h

–

Output available command-line arguments and quit.

--ipc

-Q

off

Use POSIX semaphore for process synchronisation. The name of the semaphore matches the instance name (with leading slash). The semaphore is set to 1 whenever a new log message is received. Only a single process may wait for this semaphore.

--minlevel level

-L

info

Minimum level for a log to be stored in the database, from debug or 1 to user or 6.

--name name

-n

dmlogger

Name of logger instance, configuration, POSIX message queue, and POSIX semaphore.

--node id

-N

–

Node id.

--verbose

-V

off

Print received logs to stderr.

--version

-v

–

Output version information and quit.

Examples

Create a message queue /dmlogger, wait for incoming logs, and store them in the given database if logs are of level error (4) or higher:

$ dmlogger --node dummy-node --database log.sqlite --minlevel warning

Push semaphore /dmlogger each time a log has been received:

$ dmlogger --node dummy-node --database log.sqlite --ipc

Let dmsync wait for semaphore /dmlogger before synchronising the log database with host 192.168.1.100, then repeat:

$ dmsync --type log --database log.sqlite --host 192.168.1.100 --wait dmlogger

dmlua

The dmlua program runs a custom Lua script to process observations received from message queue. Each observation is passed as a Lua table to the function of the name given in option procedure. If the option is not set, function name process is assumed by default. The Lua function must return the (modified) observation table on exit.

The observation returned from the Lua function is forwarded to the next receiver specified in the receivers list of the observation. If no receivers are left, the observation will be discarded.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file (optional).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmlua

Name of instance and table in configuration.

--node id

-N

–

Node id.

--procedure name

-p

process

Name of Lua function to call.

--script file

-s

–

Path to Lua script to run.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

The following Lua script script.lua just prints observation table observ to standard output, before returning it to dmlua unmodified:

-- script.lua
function process(observ)
    print(dump(observ))
    return observ
end

function dump(o)
   if type(o) == 'table' then
      local s = '{\n'
      for k, v in pairs(o) do
         if type(k) ~= 'number' then k = '"' .. k .. '"' end
         s = s .. '[' .. k .. '] = ' .. dump(v) .. ',\n'
      end
      return s .. '}'
   else
      return tostring(o)
   end
end

Any observation sent to receiver dmlua will be passed to the Lua function process() in script.lua, then forwarded to the next receiver (if any):

$ dmlua --name dmlua --node dummy-node --script script.lua --verbose

dmpipe

The dmpipe program reads responses from processes connected through a pipe to read sensor data from a third-party program. Requests of an observation have to contain the process to call in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern.

If any receivers are specified, observations are forwarded to the next receiver via POSIX message queue. The program can act as a sole data logger if output and format are set. If the output path is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. Node id, sensor id, and observation id are added by dmpipe. If the observation will be stored in a database, the node, sensor and target ids have to exist in the database.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file (required).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--format format

-f

–

Output format, either csv or jsonl.

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmpipe

Name of instance and table in configuration.

--node id

-N

–

Node id.

--output file

-o

–

Output file to append observations to (- for stdout).

--sensor id

-S

–

Sensor id.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

The example reads the remaining battery life returned by the sysctl(8) tool (available on FreeBSD):

$ sysctl hw.acpi.battery.life
hw.acpi.battery.life: 100

On Linux, the battery life can be read with dmfs from /sys/class/power_supply/BAT0/capacity instead.

The regular expression pattern describes the response and defines the group battery for extraction. The name of one of the responses in the responses table must equal the group name. The observation will be forwarded to the message queue of a dmdb process. Backslash characters in the string values have to be escaped with \.

-- dmpipe.conf
dmpipe = {
  logger = "dmlogger",              -- Logger to send logs to.
  node = "dummy-node",              -- Node id (required).
  sensor = "dummy-sensor",          -- Sensor id (required).
  output = "",                      -- Path to output file, `-` for stdout.
  format = "none",                  -- Output format (`csv` or `jsonl`).
  jobs = {                          -- Jobs to perform.
    {
      disabled = false,             -- Skip job.
      onetime = false,              -- Run job only once.
      observation = {               -- Observation to execute.
        name = "dummy-observ",      -- Observation name (required).
        target_id = "dummy-target", -- Target id (required).
        receivers = { "dmdb" },     -- List of receivers (up to 16).
        requests = {                -- Pipes to open.
          {
            request = "sysctl hw.acpi.battery.life", -- Command to execute.
            pattern = "[.a-z]+: (?<battery>[0-9]+)", -- RegEx pattern.
            delay = 0,              -- Delay in mseconds.
            responses = {
              {
                name = "battery",   -- RegEx group name (max. 8 characters).
                unit = "%"          -- Response unit (max. 8 characters).
                type = RESPONSE_TYPE_REAL64 -- Response value type.
              }
            }
          }
        }
      },
      delay = 60 * 1000,            -- Delay to wait afterwards in mseconds.
    }
  },
  debug = false,                    -- Forward logs of level DEBUG via IPC.
  verbose = true                    -- Print messages to standard output.
}

Pass the path of the configuration file to dmpipe:

$ dmpipe --name dmpipe --config /usr/local/etc/dmpipe.conf

The result returned by sysctl(8) will be formatted according to the current locale (decimal separator). You may have to change the locale first to match the regular expression pattern:

$ export LANG=C
$ dmpipe --name dmpipe --config /usr/local/etc/dmpipe.conf

dmplot

The dmplot program is a front-end to gnuplot(1) that creates plots of observations read from database. Plots are either written to file or displayed in terminal or X11 window.

Depending on the selected terminal back-end, you may have to set the environment variable GDFONTPATH to the path of the local font directory first:

$ export GDFONTPATH="/usr/local/share/fonts/webfonts/"

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile:

alias gnuplot="gnuplot-nox"

The output file is ignored when using the terminals sixelgd and x11. Plotting parameters passed via command-line have priority over those from configuration file.

Terminals supported by dmplot
Terminal Description

ansi

ASCII format, in ANSI colours.

ascii

ASCII format.

gif

GIF format (libgd).

png

PNG format (libgd).

pngcairo

PNG format (libcairo), created from vector graphics.

sixelgd

Sixel format (libgd), originally for DEC terminals.

svg

W3C Scalable Vector Graphics (SVG) format.

x11

Persistent X11 window (libX11).

Format descriptors allowed in the output file name
Descriptor Description (Format)

%Y

year (YYYY)

%M

month (MM)

%D

day of month (DD)

%h

hour (hh)

%m

minute (mm)

%s

second (ss)

Command-Line Options

Option Short Default Description

--background color

-G

–

Background colour (for example, #ffffff or white).

--config file

-c

–

Path to configuration file.

--database file

-d

–

Path to SQLite observation database.

--font name

-A

–

Font name or file path (for example, Open Sans, arial.ttf, monospace).

--foreground color

-P

#3b4cc0

Foreground colour (for example, #ff0000 or red).

--from timestamp

-B

–

Start of time range in ISO 8601.

--height n

-H

400

Plot height.

--help

-h

–

Output available command-line arguments and quit.

--name name

-n

dmplot

Name of table in configuration.

--node id

-N

–

Node id.

--output file

-o

–

File path of plot image. May include format descriptors.

--response name

-R

–

Response name.

--sensor id

-S

–

Sensor id.

--target id

-T

–

Target id.

--terminal terminal

-m

–

Plot format.

--title title

-C

–

Plot title.

--to timestamp

-E

–

End of time range in ISO 8601.

--version

-v

–

Output version information and quit.

--width n

-W

1000

Plot width.

Examples

Create a plot of observations selected from database observ.sqlite in PNG format, and write the file to /tmp/plot.png:

$ dmplot --database /var/dmpack/observ.sqlite --terminal pngcairo --output /tmp/plot.png \
  --node dummy-node --sensor dummy-sensor --target dummy-target --response dummy \
  --from 2024 --to 2025

Output the plot directly to terminal, using the configuration in dmplot.conf:

$ dmplot --name dmplot --config dmplot.conf --terminal sixelgd

The sixelgd format requires a terminal emulator with Sixel support, such as xterm(1) or mlterm(1).

dmplot
Figure 2. Plotting time series directly in XTerm

dmrecv

The dmrecv program listens to the POSIX message queue of its name and writes received logs or observations to stdout, file, or named pipe; in CSV, JSON Lines, or Namelist format. By default, the serialised data is appended to the end of the output file. If argument --replace is passed, the file will be replaced consecutively.

Received observations are not forwarded to the next specified receiver unless argument --forward is set. If no receivers are defined or left, the observation will be discarded after output. If the JSON Lines output format is selected, logs and observations are written as JSON objects to file or stdout, separated by new line (\n). Use jq(1) to convert records in JSON Lines file input.jsonl into a valid JSON array in output.json:

$ jq -s '.' input.jsonl > output.json

The output format block is only available for observation data and requires a response name to be set. Observations will be searched for this response name and converted to data point type if found. The data point is printed in ASCII block format.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Output formats of logs and observations
Type Block CSV JSONL NML

log

✓

✓

✓

observ

✓

✓

✓

✓

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file.

--debug

-D

off

Forward log messages of level debug (if logger is set).

--format format

-f

–

Output format (block, csv, jsonl, nml).

--forward

-F

off

Forward observations to the next specified receiver.

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmrecv

Name of table in configuration and POSIX message queue to subscribe to.

--node id

-N

–

Optional node id.

--output file

-o

stdout

Output file to append observations to (- for stdout).

--replace

-r

off

Replace output file instead of appending data.

--response name

-R

–

Name of observation response to output (required for format block).

--type type

-t

–

Data type to receive: log or observ.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Write log messages received from POSIX message queue /dmrecv to file /tmp/logs.csv in CSV format:

$ dmrecv --name dmrecv --type log --format csv --output /tmp/logs.csv

Output observations in JSON Lines format to stdout:

$ dmrecv --name dmrecv --type observ --format jsonl

Write the observations serialised in JSON Lines format to named pipe /tmp/fifo_dmrecv:

$ mkfifo /tmp/fifo_dmrecv
$ dmrecv --name dmrecv --type observ --format jsonl --output /tmp/fifo_dmrecv

Another process can now read the observations from /tmp/fifo_dmrecv:

$ tail -f /tmp/fifo_dmrecv

Responses in block format can also be piped to a graph tool like trend to update a chart in real-time. For instance, to pipe the responses of name tz0 for observations received through message queue /dmrecv to trend(1), run:

$ dmrecv --name dmrecv --type observ --format block --response tz0 \
  | awk '{ print $2 | "trend - 60" }'

dmreport

The dmreport program creates reports in HTML5 format, containing plots of observations and/or log messages selected from database. Plots are created by calling gnuplot(1) and inlining the returned image (GIF, PNG, SVG) as a base64-encoded data URI. Any style sheet file with classless CSS can be included to alter the presentation of the report. A basic style sheet dmreport.css and its minified version dmreport.min.css are provided in /usr/local/share/dmpack/dmreport/. The output of dmreport is a single HTML file with inlined CSS. Use a command-line tool like wkhtmltopdf to convert the HTML report to PDF format.

Depending on the selected plot format, the environment variable GDFONTPATH may have to be set to the local font directory containing the TrueType fonts first, for example:

$ export GDFONTPATH="/usr/local/share/fonts/webfonts/"

Add the export statement to the global profile /etc/profile. If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to /etc/profile:

alias gnuplot="gnuplot-nox"

A configuration file is mandatory to create reports. Only a few parameters can be set through command-line arguments. Passed command-line arguments have priority over settings in the configuration file.

Format descriptors allowed in the output file name
Descriptor Description (Format)

%Y

year (YYYY)

%M

month (MM)

%D

day of month (DD)

%h

hour (hh)

%m

minute (mm)

%s

second (ss)

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file (required).

--from timestamp

-B

–

Start of time range in ISO 8601.

--help

-h

–

Output available command-line arguments and quit.

--name name

-n

dmreport

Name of program instance and configuration.

--node id

-N

–

Sensor node id.

--output path

-o

–

Path of the HTML output file. May include format descriptors.

--style path

-C

–

Path to the CSS file to inline.

--to timestamp

-E

–

End of time range in ISO 8601.

--version

-v

–

Output version information and quit.

Examples

The settings are stored in Lua table dmreport in the configuration file. The observations are read from database observ.sqlite, the log messages from log.sqlite. You might want to use absolute paths for the databases.

-- dmreport.conf
dmreport = {
  node = "dummy-node",
  from = "1970-01-01T00:00:00.000000+00:00",
  to = "2070-01-01T00:00:00.000000+00:00",
  output = "%Y-%M-%D_dummy-report.html",
  style = "/usr/local/share/dmpack/dmreport/dmreport.min.css",
  title = "Monitoring Report",
  subtitle = "Project",
  meta = "",
  plots = {
    disabled = false,            -- Disable plots.
    database = "observ.sqlite",  -- Path to observation database.
    title = "Plots",             -- Overwrite default heading.
    meta = "",                   -- Optional description.
    observations = {             -- List of plots to generate.
      {
        sensor = "dummy-sensor", -- Sensor id (required).
        target = "dummy-target", -- Target id (required).
        response = "tz0",        -- Response name (required).
        unit = "deg C",          -- Response unit.
        format = "svg",          -- Plot format (gif, png, pngcairo, svg).
        title = "Temperature",   -- Plot title.
        subtitle = "tz0",        -- Plot sub-title.
        meta = "",               -- Optional description.
        color = "#ff0000",       -- Graph colour.
        width = 1000,            -- Plot width.
        height = 300,            -- Plot height.
      }
    }
  },
  logs = {
    disabled = false,            -- Disable logs.
    database = "log.sqlite",     -- Path to log database.
    minlevel = LL_WARNING,       -- Minimum log level (default: LL_WARNING).
    maxlevel = LL_CRITICAL,      -- Maximum log level (default: LL_CRITICAL).
    title = "Logs",              -- Overwrite default heading.
    meta = "",                   -- Optional description.
  }
}

The sensor node dummy-node, the sensor dummy-sensor, and the target dummy-target must exist in the database, and the observations to plot need to have responses of name tz0. Write a report to file report.html based on settings in dmreport.conf. The command-line arguments overwrite the settings of the configuration file:

$ dmreport --name dmreport --config dmreport.conf --output report.html

In order to update reports periodically, we can customise the shell script mkreport.sh in /usr/local/share/dmpack/dmreport/. The script determines the timestamps of the last and the current month (to allow observations to arrived late), which will then be passed to dmreport to create monthly reports. Modify the script according to your set-up:

dmreport="/usr/local/bin/dmreport"
name="dmreport"
config="/usr/local/etc/dmpack/dmreport.conf"
output="/var/www/reports/"

The shell script writes two reports to /var/www/reports/.

$ sh /usr/local/share/dmpack/dmreport/mkreport.sh
--- Writing report of 2023-08 to file /var/www/reports/2023-08_report.html ...
--- Writing report of 2023-09 to file /var/www/reports/2023-09_report.html ...

The directory may be served by lighttpd(1). Add the script to your crontab to run the report generation periodically.

dmsend

The dmsend program reads observations or logs in CSV and Fortran 95 Namelist format, and sends them sequentially to the POSIX message queue of a given receiver. The data is either read from file or standard input. If the input data is of type observ and the argument --forward is passed, each observation will be sent to its next specified receiver in the receivers list instead of the receiver given through argument --receiver. If no receivers are set, or if the end of the receivers list is reached, the observation will be discarded.

The program settings are passed through command-line arguments or an optional configuration file. The arguments overwrite settings from file.

Command-Line Options

Option Short Default Description

--config file

-c

–

Path to configuration file.

--debug

-D

off

Forward log messages of level debug (if logger is set).

--format format

-f

–

Input format: csv or nml.

--input file

-i

stdin

Path to input file (empty or - for stdin).

--forward

-F

off

Forward observations to the next specified receiver.

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmsend

Name of instance and table in configuration.

--node id

-N

–

Optional node id.

--receiver name

-r

–

Name of receiver/message queue.

--type type

-t

–

Input data type: log or observ.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Read a single observation from Namelist file observ.nml and send it to the next receiver specified by attribute next:

$ dmsend --type observ --format nml --input observ.nml --forward

Send multiple logs in CSV file logs.csv sequentially to process dmrecv:

$ dmsend --receiver dmrecv --type log --format csv --input logs.csv

dmserial

The dmserial program sends requests to a sensor or actor connected via USB/RS-232/RS-422/RS-485. Sensor commands and responses are sent/received through a teletype (TTY) device provided by the operating system. A pseudo-terminal (PTY) may be used to connect a virtual sensor.

Each request of an observation must contains the raw request intended for the sensor in attribute request. Response values are extracted by group from the raw response using the given regular expression pattern. Each group name must match a response name. Response names are limited to eight characters. Observations will be forwarded to the next receiver via POSIX message queue if any receiver is specified. The program can act as a sole data logger if output file and format are set. If the output is set to -, observations are printed to stdout.

A configuration file is mandatory to configure the jobs to perform. Each observation must have a valid target id. The database must contain the specified node, sensor, and targets. Parameters and functions of the Lua API may be used in the configuration file. The following baud rates are supported: 50, 75, 110, 134, 150, 200, 300, 600, 1200, 1800, 2400, 4800, 9600, 19200, 38400, 57600, 115200, 230400, 460800, 921600.

Command-Line Options

Option Short Default Description

--baudrate n

-B

9600

Number of symbols transmitted per second.

--bytesize n

-Z

8

Byte size (5, 6, 7, 8).

--config file

-c

–

Path to configuration file (required).

--debug

-D

off

Forward log messages of level debug (if logger is set).

--dtr

-Q

off

Enable Data Terminal Ready (DTR).

--format format

-f

–

Output format, either csv or jsonl.

--help

-h

–

Output available command-line arguments and quit.

--logger name

-l

–

Optional name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmserial

Name of instance and table in configuration.

--node id

-N

–

Node id.

--output file

-o

–

Output file to append observations to (- for stdout).

--parity name

-P

none

Parity bits (none, even, or odd).

--rts

-R

off

Enable Request To Send (RTS).

--sensor id

-S

–

Sensor id.

--stopbits n

-O

1

Number of stop bits (1, 2).

--timeout n

-T

0

Connection timeout in seconds (max. 25).

--path path

-p

–

Path to TTY/PTY device (for example, /dev/ttyU0).

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

Examples

Read the jobs to perform from configuration file and execute them sequentially:

$ dmserial --name dmserial --config /usr/local/etc/dmpack/dmserial.conf --verbose

dmsync

The dmsync program synchronises logs, nodes, observations, sensors, and targets from local databases concurrently with a remote dmapi server. The synchronisation may be started only once if no interval is set (to transfer nodes, sensors, and targets initially from client to server), periodically as a cron job, or by waiting for a POSIX semaphore.

The nodes, sensors, and targets referenced by observations in the local database must also exist in the remote server database. They can be created on the server with dmdbctl or dmweb, or sent from client to server with dmsync. Logs and targets do not require any additional database entries on the server-side.

The client databases must contain synchronisation tables. The tables are created automatically by dminit if command-line argument --sync is passed. Otherweise, start dmsync with argument --create once to add the missing tables.

If the RPC server uses HTTP Basic Auth for authentication, the RPC user name must match the node id of the transmitted node, sensor, observation, log, or beat records, or the server will reject the requests and return HTTP 401 (Unauthorized).

The database records are serialised in Fortran 95 Namelist format and optionally compressed before being sent to the server. The program uses libcurl for data transfer, and deflate or zstd for compression. The RPC API endpoints to post records to are expected at URL [http|https]://<host>:<port>/api/v1/<endpoint>.

The result of each synchronisation attempt is stored in the local database. Records are marked as synchronised only if the server returns HTTP 201 (Created).

Passing the server credentials via the command-line arguments --username and --password is insecure on multi-user operating systems and only recommended for testing.

Command-Line Options

Option Short Default Description

--compression name

-x

zstd

Compression library to use (none, zlib, zstd).

--config file

-c

–

Path to configuration file.

--create

-C

off

Create missing database synchronisation tables.

--database file

-d

–

Path to log or observation database.

--debug

-D

off

Forward log messages of level debug (if logger is set).

--help

-h

–

Output available command-line arguments and quit.

--host host

-H

–

IP address or FQDN of HTTP-RPC API host (for instance, 127.0.0.1 or iot.example.com).

--interval seconds

-I

60

Synchronisation interval in seconds. If set to 0, synchronisation is executed only once.

--logger name

-l

–

Name of logger. If set, sends logs to dmlogger process of given name.

--name name

-n

dmsync

Name of program instance and configuration.

--node id

-N

–

Node id, required for types sensor and observ.

--password string

-P

–

API password.

--port port

-q

0

Port of HTTP-RPC API server (0 for automatic).

--tls

-E

off

Use TLS-encrypted connection.

--type type

-t

–

Type of data to sychronise, either log, node, observ, sensor, or target. Type log requires a log database, all other an observation database.

--username string

-U

–

API user name. If set, implies HTTP Basic Auth.

--verbose

-V

off

Print log messages to stderr.

--version

-v

–

Output version information and quit.

--wait name

-w

–

Name of POSIX semaphore to wait for. Synchronises databases if semaphore is > 0.

Examples

Initially synchronise nodes, sensors, and targets in the local observation database with an HTTP-RPC server (without authentication):

$ dmsync --database observ.sqlite --type node --host 192.168.1.100
$ dmsync --database observ.sqlite --type sensor --node dummy-node --host 192.168.1.100
$ dmsync --database observ.sqlite --type target --host 102.168.1.100

Synchronise observations:

$ dmsync --database observ.sqlite --type observ --host 192.168.1.100

Synchronise log messages:

$ dmsync --database log.sqlite --type log --host 192.168.1.100

dmuuid

The dmuuid program is a command-line tool to generate pseudo-random UUIDs. By default, DMPACK uses 32 characters long UUIDv4 identifiers in hexadecimal format (without hyphens). Hyphens can be added by a command-line flag. The option --convert expects UUIDv4 identifiers to be passed via standard input. Invalid identifiers will be replaced with the default UUID. The program may be used to create a feed id for dmfeed.

Command-Line Options

Option Short Default Description

--convert

-C

off

Add hyphens to 32 characters long hexadecimal UUIDs passed via stdin.

--count n

-n

1

Number of identifiers to generate.

--help

-h

–

Output available command-line arguments and quit.

--hyphens

-H

off

Return 36 characters long UUIDv4 with hyphens.

--version

-v

–

Output version information and quit.

Examples

Create three identifiers:

$ dmuuid --count 3
6827049760c545ad80d4082cc50203e8
ad488d0b8edd4c6c94582e702a810ada
3d3eee7ae1fb4259b5df72f854aaa369

Create a UUIDv4 with hyphens:

$ dmuuid --hyphens
d498f067-d14a-4f98-a9d8-777a3a131d12

Add hyphens to a hexadecimal UUID:

$ echo "3d3eee7ae1fb4259b5df72f854aaa369" | dmuuid --convert
3d3eee7a-e1fb-4259-b5df-72f854aaa369

dmweb

dmweb is a CGI-based web user interface for DMPACK database access on client and server. The web application has to be executed through a CGI-compatible web server. It is recommended to run lighttpd(1). Any other server must be able to pass environment variables to the CGI application. gnuplot(1) is required for the plotting back-end (no-X11 flavour is sufficient).

The web application provides the following pages:

Dashboard

Lists heartbeats, logs, and observations that have been added to the databases most recently.

Nodes

Lists all sensor nodes, and allows to add new ones.

Sensors

Lists all sensors, and allows to add new ones.

Targets

Lists all targets, and allows to add new ones.

Observations

Lists observations in database, selected by filter.

Plots

Creates plots in SVG format from observation responses in database.

Logs

Lists log messages stored in database, with optional filter.

Beats

Lists received heartbeat messages, sorted by node id. The beat view shows the time the heartbeat was sent and received, as well as the time passed since then, additionally in Swatch Internet Time.

Map

Displays nodes, sensors, and targets inside an interactive map.

The style sheet of dmweb is based on missing.css. It can be replaced with any other classless CSS theme. For best experience, the IBM Plex font family should be installed locally.

If gnuplot(1) is installed under a name other than gnuplot, for example, gnuplot-nox, create a symbolic link or add an alias to the global profile /etc/profile:

alias gnuplot="gnuplot-nox"

On FreeBSD, it might be necessary to add the environment variable GDFONTPATH to the path of the font directory:

export GDFONTPATH="/usr/local/share/fonts/webfonts/"
Environment variables of dmweb(1)
Environment Variable Description

DM_DB_BEAT

Path to heartbeat database (server).

DM_DB_LOG

Path to log database (client, server).

DM_DB_OBSERV

Path to observation database (client, server).

DM_READ_ONLY

Set to 1 to enable read-only database access.

DM_TILE_URL

URL of tile server.

The map view requires a URL to the tile server in environment variable DM_TILE_URL. For example, set the variable to https://tile.openstreetmap.org/{z}/{x}/{y}.png to use OpenStreetMap as the backend.

Copy the directory /usr/local/share/dmpack/dmweb manually to the WWW root directory, or create a symlink. Environment variables are used to configure dmweb. Transport security and authentication have to be managed by the web server. See section Web UI for an example configuration.

dmweb
Figure 3. Plotting of time series through the dmweb user interface

Web Applications

Comparision between DMPACK web applications
dmapi dmweb

Description

HTTP-RPC API

Web UI

Base Path

/api/v1/

/dmpack/

Protocol

FastCGI

CGI

Location

server

client, server

Configuration

environment variables

environment variables

Authentication

HTTP Basic Auth

HTTP Basic Auth

Content Types

CSV, JSON, JSON Lines, Namelist, Text

HTML5

HTTP Methods

GET, POST

GET, POST

Database

SQLite 3

SQLite 3

Read-Only Mode

✓

✓

The following web applications are part of DMPACK:

dmapi

HTTP-RPC API for data synchronisation, time series access, and heartbeat collection.

dmweb

Web user interface for database configuration, data access, and plotting.

Both applications may be served by the same web server. It is recommended to run them in lighttpd(1). On FreeBSD, install the package with:

# pkg install www/lighttpd

The web server is configured through /usr/local/etc/lighttpd/lighttpd.conf. See the lighttpd wiki on how to configure the web server. In the listed examples, the DMPACK executables are assumend to be in /usr/local/bin/, but you may copy the programs to /var/www/cgi-bin/ or any other directory. Set an appropriate owner, such as the one the server is running as.

Authentication

The HTTP-RPC API and the web interface will be publicly accessible if the web server is not configured to manage user authentication. HTTP Basic Auth is a simple method to authenticate users by name and password. The lighttpd(1) web server includes an auth module with various back-ends. In the web server configuration, set auth.backend.htpasswd.userfile to the path of the file that contains the credentials. You can run openssl(1) to add one or more user accounts with hashed password (SHA-512) to the htpasswd file, in this case /usr/local/etc/lighttpd/htpasswd:

# read AUTH_USR
# read AUTH_PWD
# printf "%s:%s\n" $AUTH_USR `openssl passwd -6 "$AUTH_PWD"` \
  >> /usr/local/etc/lighttpd/htpasswd

Enter the user name and the associated password after the read commands. As an alternative to storing the credentials in a flat file, we can select a different authentication back-end, for example, LDAP, PAM, or database. See the documentation of the module for further instructions. In the web server configuration, load module mod_authn_file, select the back-end, and enable authentication globally or for specific routes:

# Load authentication module.
server.modules += ( "mod_authn_file" )

# Authentication back-end and path of user file.
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/usr/local/etc/lighttpd/htpasswd"

# Protected routes.
$HTTP["url"] =^ "/api/v1" {
  auth.require = ( "" => (
    "method"  => "basic",
    "realm"   => "dmpack",
    "require" => "valid-user"
  ))
}

Cross-Origin Resource Sharing

If the HTTP-RPC API will be accessed by a client-side application running in the browser, the web server has to be configured to send the appropriate Cross-Origin Resource Sharing (CORS) headers. By default, asynchronous JavaScript requests are forbidden by the same-origin security policy. Refer to the documentation of the web server on how to set the Access-Control-* headers. For lighttpd(1), load the module mod_setenv and add response headers for OPTION requests:

$HTTP["request-method"] =~ "^(OPTIONS)$" {
  setenv.add-response-header = (
    "Access-Control-Allow-Origin"   => "*",
    "Access-Control-Allow-Headers"  =>
        "accept, origin, x-requested-with, content-type, x-transmission-session-id",
    "Access-Control-Expose-Headers" => "X-Transmission-Session-Id",
    "Access-Control-Allow-Methods"  => "GET, POST, OPTIONS"
  )
}

If the web server is behind a reverse proxy, CORS headers should be set by the proxy instead.

Databases

The databases are expected to be in directory /var/dmpack/. Change the environment variables in the web server configuration to the actual paths. The observation, log, and beat databases the web applications will access must be created and initialised beforehand:

# dminit --type observ --database /var/dmpack/observ.sqlite --wal
# dminit --type log --database /var/dmpack/log.sqlite --wal
# dminit --type beat --database /var/dmpack/beat.sqlite --wal

Make sure the web server has read and write access to the directory and all databases inside:

# chown -R www:www /var/dmpack

Change www:www to the user and the group the web server is running as.

RPC Server

The snippet in this section may be added to the lighttpd(1) configuration to run the dmapi service. The lighttpd(1) web server does not require an additional FastCGI spawner. The following server modules have to be imported:

  • mod_authn_file (HTTP Basic Auth)

  • mod_extforward (real IP, only if the server is behind a reverse proxy)

  • mod_fastcgi (FastCGI)

Add the IP address of the proxy server to the list of trusted forwarders to have access to the real IP of a client.

# Listen on all network interfaces.
$SERVER["socket"] == "0.0.0.0:80" { }

# Load lighttpd modules.
server.modules += (
  "mod_authn_file",
  "mod_extforward",
  "mod_fastcgi"
)

# Set authentication back-end and path of user file.
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/usr/local/etc/lighttpd/htpasswd"

# Real IP of client in case the server is behind a reverse proxy. Set one or
# more trusted proxies.
# extforward.headers = ( "X-Real-IP" )
# extforward.forwarder = ( "<PROXY IP>" => "trust" )

# FastCGI configuration. Run 4 worker processes, and pass the database paths
# through environment variables.
fastcgi.server = (
  "/api/v1" => ((
    "socket"      => "/var/lighttpd/sockets/dmapi.sock",
    "bin-path"    => "/usr/local/bin/dmapi",
    "max-procs"   => 4,
    "check-local" => "disable",
    "bin-environment" => (
      "DM_DB_BEAT"   => "/var/dmpack/beat.sqlite",
      "DM_DB_LOG"    => "/var/dmpack/log.sqlite",
      "DM_DB_OBSERV" => "/var/dmpack/observ.sqlite",
      "DM_READ_ONLY" => "0"
    )
  ))
)

# URL routing.
$HTTP["url"] =^ "/api/v1" {
  # Enable HTTP Basic Auth.
  auth.require = ( "" => (
    "method"  => "basic",
    "realm"   => "dmpack",
    "require" => "valid-user"
  ))
}

The FastCGI socket will be written to /var/run/lighttpd/sockets/dmapi.sock. Change max-procs to the desired number of FastCGI processes. Set the environment variables to the locations of the databases. The databases must exist prior start. On FreeBSD, add the service to the system rc file /etc/rc.conf and start the server manually:

# sysrc lighttpd_enable="YES"
# service lighttpd start

If served locally, access the RPC API at http://127.0.0.1/api/v1/.

Web UI

The lighttpd(1) web server has to be configured to run the CGI application under base path /dmpack/. The following server modules are required:

  • mod_alias (URL rewrites)

  • mod_authn_file (HTTP Basic Auth)

  • mod_cgi (Common Gateway Interface)

  • mod_setenv (CGI environment variables)

The example configuration may be appended to your lighttpd.conf:

# Listen on all network interfaces.
$SERVER["socket"] == "0.0.0.0:80" { }

# Load lighttpd modules.
server.modules += (
  "mod_alias",
  "mod_authn_file",
  "mod_cgi",
  "mod_setenv"
)

# Set maximum number of concurrent connections and maximum
# HTTP request size of 8192 KiB (optional).
server.max-connections  = 32
server.max-request-size = 8192

# Pass the database paths through environment variables.
setenv.add-environment = (
  "DM_DB_BEAT"   => "/var/dmpack/beat.sqlite",
  "DM_DB_LOG"    => "/var/dmpack/log.sqlite",
  "DM_DB_OBSERV" => "/var/dmpack/observ.sqlite",
  "DM_READ_ONLY" => "0"
  "DM_TILE_URL"  => "https://tile.openstreetmap.org/{z}/{x}/{y}.png",
)

# Set authentication back-end and path of user file.
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/usr/local/etc/lighttpd/htpasswd"

# URL routing.
$HTTP["url"] =^ "/dmpack/" {
  # Map URL to CGI executable.
  alias.url += ( "/dmpack" => "/usr/local/bin/dmweb" )

  # Enable HTTP Basic Auth for all paths.
  auth.require = ( "" => (
    "method"  => "basic",
    "realm"   => "dmpack",
    "require" => "valid-user"
  ))

  # CGI settings. Do not assign file endings to script interpreters,
  # execute only applications with execute bit set, enable write and
  # read timeouts of 30 seconds.
  cgi.assign = ( "" => "" )
  cgi.execute-x-only = "enable"
  cgi.limits = (
    "write-timeout"     => 30,
    "read-timeout"      => 30,
    "tcp-fin-propagate" => "SIGTERM"
  )
}

Copy the directory dmweb from /usr/local/share/dmpack/ (or /opt/share/dmpack/) to the WWW root directory, in this case, /var/www/, or simply create a symlink:

# ln -s /usr/local/share/dmpack/dmweb /var/www/dmweb

If the files have to be served from a path other than the root path, add a rewrite rule or alias to the web server configuration. On FreeBSD, add the service to the system rc file /etc/rc.conf and start the web server manually:

# sysrc lighttpd_enable="YES"
# service lighttpd start

If served locally, access the web application at http://127.0.0.1/dmpack/.

Databases

The DMPACK programs use three distinct databases to store deformation monitoring entity records:

Observation Database

Stores nodes, sensors, targets, observations, observation receivers, observation requests, and observation responses, with optional synchronisation tables for all record types.

Log Database

Stores all log messages in single table.

Beat Database

Stores heartbeat messages by unique node id.

The databases are usually located in directory /var/dmpack/ or /opt/var/dmpack/.

Administration

The sqlite3(1) program is stand-alone command-line shell for SQLite database access that allows the user to execute arbitrary SQL statements. Third-party programs provide an additional graphical user interface:

DB Browser for SQLite (DB4S)

A spreadsheet-like visual interface for Linux, Unix, macOS, and Windows. (MPLv2, GPLv3)

HeidiSQL

A free database administration tool for MariaDB, MySQL, MS SQL Server, PostgreSQL, and SQLite. For Windows only. (GPLv2)

phpLiteAdmin

A web front-end for SQLite database administration written in PHP. (GPLv3)

SQLiteStudio

A free, open source, and multi-platform SQLite database manager written in C++ that is based on the Qt framework. (GPLv3)

SQLite Web

A web-based SQLite database browser in Python. (MIT)

Entity–Relationship Model

UML
Figure 4. Beat database