Skip to main content

Data Model Guide

ThingConnect Pulse collects and stores monitoring data in a structured way to provide both real-time status and historical analysis. This guide explains how your monitoring data is organized and stored.

Data Collection Overview​

Pulse continuously monitors your configured endpoints and stores the results in a local SQLite database. The system collects three types of data:

  1. Raw Check Data - Individual probe results
  2. 15-Minute Rollups - Aggregated quarter-hour statistics
  3. Daily Rollups - Daily summary statistics

Raw Check Data​

Every time Pulse probes an endpoint, it creates a raw check record.

Data Structure​

{
  "timestamp": "2024-12-01T14:30:45Z",
  "status": "UP",
  "responseTimeMs": 12.5,
  "errorMessage": null
}

Fields Explained​

Timestamp

  • Exact time when the probe was executed
  • Stored in UTC timezone
  • Used for all time-based analysis

Status

  • UP: Probe succeeded
  • DOWN: Probe failed
  • Based on individual probe result, not overall endpoint status

Response Time (RTT)

  • Round-trip time in milliseconds
  • null for failed probes
  • Includes network latency and service response time

Error Message

  • Details about failure (when status is DOWN)
  • Examples: "Connection timeout", "Host unreachable", "HTTP 500"
  • null for successful probes

Raw Data Characteristics​

Collection Frequency

  • Collected at your configured interval (typically 30-300 seconds)
  • Every endpoint probe generates one raw check record
  • No aggregation or filtering applied

Retention Policy

  • 60 days of raw check data
  • Automatically pruned after retention period
  • Most detailed data available for recent analysis

Use Cases

  • Troubleshooting specific incidents
  • Minute-by-minute analysis
  • Identifying exact timing of outages
  • Performance spike investigation
Raw Data Best Practices

Raw data is perfect for troubleshooting recent issues, but use rollups for longer-term analysis to improve performance and reduce data size.

15-Minute Rollup Data​

Every 15 minutes, Pulse automatically calculates aggregated statistics from raw check data.

Data Structure​

{
  "bucketTimestamp": "2024-12-01T14:30:00Z",
  "uptimePercentage": 95.5,
  "averageResponseTimeMs": 23.7,
  "downEvents": 1
}

Fields Explained​

Bucket Timestamp

  • Start time of the 15-minute window
  • Always aligned to quarter-hour boundaries (00, 15, 30, 45 minutes)
  • Example: 14:30:00 covers 14:30:00 to 14:44:59

Uptime Percentage

  • Percentage of successful probes in the 15-minute window
  • Calculated as: (successful probes / total probes) × 100
  • Range: 0.0 to 100.0

Average Response Time

  • Mean response time of successful probes only
  • Failed probes are excluded from calculation
  • null if no successful probes in the window

Down Events

  • Number of times status changed from UP to DOWN
  • Indicates instability even with good uptime percentage
  • Used to detect flapping conditions

Rollup Calculation Process​

Data Aggregation

  1. Collect all raw checks in 15-minute window
  2. Calculate uptime percentage from status values
  3. Average response times from successful probes
  4. Count status transitions from UP to DOWN

Automatic Processing

  • Runs every 5 minutes in background
  • Processes completed 15-minute windows
  • No manual intervention required

15-Minute Rollup Characteristics​

Retention Policy

  • Unlimited retention - never automatically deleted
  • Provides long-term trending capability
  • Balances detail with storage efficiency

Use Cases

  • Daily and weekly trend analysis
  • Shift-based reporting for manufacturing
  • Performance baseline establishment
  • Capacity planning over months

Daily Rollup Data​

Each day, Pulse creates summary statistics for the entire 24-hour period.

Data Structure​

{
  "bucketDate": "2024-12-01",
  "uptimePercentage": 98.2,
  "averageResponseTimeMs": 19.4,
  "downEvents": 3
}

Fields Explained​

Bucket Date

  • Date for the 24-hour period
  • Always in YYYY-MM-DD format
  • Based on local server time, not UTC

Daily Metrics

  • Same calculation methods as 15-minute rollups
  • Aggregated across entire day
  • Provides high-level daily health summary

Daily Rollup Characteristics​

Retention Policy

  • Unlimited retention - permanent storage
  • Enables years of historical analysis
  • Minimal storage impact

Use Cases

  • Monthly and quarterly reporting
  • Long-term availability trends
  • Management dashboards and KPIs
  • Service level agreement (SLA) tracking

Outage Tracking​

Pulse automatically detects and records outages based on consecutive probe failures.

Outage Definition​

Outage Start

  • Triggered after 2 consecutive failed probes
  • Prevents brief network hiccups from triggering alerts
  • Start timestamp from first failed probe

Outage End

  • Recorded after 2 consecutive successful probes
  • Ensures stable recovery before declaring outage over
  • End timestamp from first successful probe

Outage Data Structure​

{
  "startTimestamp": "2024-12-01T14:30:45Z",
  "endTimestamp": "2024-12-01T14:45:12Z",
  "durationSeconds": 867,
  "lastErrorMessage": "Connection timeout"
}

Outage Characteristics​

Flap Damping

  • 2/2 threshold prevents false outage alerts
  • Brief interruptions don't create outage records
  • Focuses on sustained availability issues

Duration Calculation

  • Measured from first failure to first recovery
  • Includes the stabilization period
  • Reported in seconds for precision

Sparkline Data​

The dashboard shows mini-charts (sparklines) for recent endpoint status.

Data Structure​

{
  "timestamp": "2024-12-01T14:30:00Z",
  "status": "UP"
}

Sparkline Characteristics​

Time Window

  • Last 24 hours of status data
  • Sampled at regular intervals for chart display
  • Based on actual probe results, not rollups

Visual Representation

  • Green segments: UP status periods
  • Red segments: DOWN status periods
  • Quick visual health indicator
Data Relationships

All data types are linked by endpoint ID and timestamp, allowing you to drill down from daily summaries to individual probe results for detailed analysis.

Data Storage and Performance​

Database Technology​

SQLite Database

  • Local file-based storage
  • Located in C:\ProgramData\ThingConnect.Pulse\data
  • No external database server required

Storage Considerations​

Database Growth

  • Raw data: ~1KB per endpoint per hour
  • Rollups: ~100 bytes per endpoint per day
  • Automatic pruning keeps database size manageable

Query Performance

  • Indexed by timestamp and endpoint ID
  • 15-minute and daily rollups optimize long-term queries
  • Raw data queries limited to recent time periods

Backup Recommendations​

Data Protection

  • Database file can be copied while service is running
  • Configuration and data stored separately
  • Include in regular backup procedures

Disaster Recovery

  • Database rebuild possible from configuration
  • Historical data requires backup restoration
  • Rolling restart preserves data integrity

Data Access Methods​

Web Interface​

  • Dashboard for real-time and recent data
  • History page for trend analysis
  • CSV export for external processing

API Access​

  • REST endpoints for programmatic access
  • Same data available as web interface
  • Authentication required for all access

Direct Database Access​

  • SQLite database accessible with standard tools
  • Read-only access recommended
  • Schema knowledge required for queries

Next Steps​