How we connected hundreds of end devices across Slovakia to monitoring
Managing an extensive network of geographically distributed devices presents both a technical and organizational challenge. In our case, we are dealing with over 700 independently functioning devices spread across the entire territory of Slovakia. These devices run on the Linux operating system and are deployed in various conditions – whether in urban areas with standard infrastructure or in more remote locations with limited internet connectivity.
Our goal is to ensure smooth operation and timely identification of potential problems. That is why we built a solution that allows us to actively and centrally monitor the status of each device, regardless of its current network conditions.
The importance of monitoring
The goal of monitoring is not just to reactively record faults but, above all, to proactively prevent them. Early detection of symptoms such as increasing CPU temperature, excessive RAM usage, or gradual failure of an SD card can prevent unexpected downtime and reduced service availability. Equally important is the fast detection of non-functional devices that have stopped communicating with the server, which allows us to respond and plan service interventions.
What equipment do the devices have?
Each device is distributed with its own software stack, which, in addition to the application core, includes a component that ensures “call home” functionality. This component regularly checks if there is a request to establish a connection with the server and, if necessary, initiates an SSH connection with a central “jump host.” This allows us to access the device without needing public IP addresses or port forwarding, as if it were directly in the local network.
The software is designed to work reliably even in unstable networks, with support for repeated connection attempts and fallback logic in case the primary endpoint fails.
Implementation of centralized monitoring
On the server side, we have developed several components that work together within the monitoring and interaction process:
- The monitoring tool is a central web application that allows administrators to define monitoring requirements for individual devices. Here, we specify which metrics we want to track (e.g., CPU load, temperature, SD card capacity, uptime, etc.) and which devices should be assigned these purposes.
- These requirements are recorded in the Call Home database, which individual devices periodically check. Upon detecting a new request, the device initiates an SSH connection through a secure tunnel to the SSH Jump Host.
- Once the device connects, the onboarding process follows, during which scripts for collecting metric data and settings needed for regular reporting are loaded.
- The device then periodically sends the collected data to the central monitoring tool through a secure channel. This data is stored in a database and visualized in the monitoring tool, which also evaluates it and creates events used in incident management processes.
- These requirements are recorded in the Call Home database, which individual devices periodically check. Upon detecting a new request, the device initiates an SSH connection through a secure tunnel to the SSH Jump Host.
Collected metrics and analysis
The most important parameters we evaluate include:
- SD card health and lifespan (using SMART attributes and monitoring IO errors)
- CPU overheating (e.g., a regular increase above 80°C)
- Unscheduled downtime or repeated reboots outside of defined service windows
- Excessive RAM and SWAP memory usage, which could indicate issues with running processes
In addition, we log network connection history, ping/HTTP health check response times, and the availability of services provided by the device.
Architecture
The diagram illustrates the individual components and their interconnection. From the central tool through the database to the remote devices, each step ensures a reliable flow of information.
Experiences and future development
In practice, the system has proven to be a robust solution that saves us not only costs but also the time of our technical staff. Thanks to it, we can effectively identify devices in critical condition and intervene before problems escalate.
Overall, this solution is flexible and scalable, meaning it can be extended with minimal adjustments to thousands of devices across various domains – from IoT to industrial monitoring.
More about this topic also in our blog: How monitoring helps keep systems running.