Plotly miscalculating time increments

General
1

Heatmap Y-Axis and Timestamp Issues - Investigation Report

Executive Summary

This document provides a comprehensive analysis of persistent issues with the financial heatmap visualization system, specifically focusing on Y-axis metric label display problems and 30-minute timeframe timestamp collapse issues. Multiple implementation approaches were attempted, with varying degrees of success, but fundamental issues remain unresolved.

Problem Statement

Primary Issues Identified

  1. Y-Axis Metric Label Display: Metric names (RSI, GarchVolatility, HestonVolatility) not displaying properly on the Y-axis
  2. 30-Minute Timeframe Single Timestamp: Heatmap displays as a single time increment instead of proper 30-minute intervals
  3. HTTP 500 Server Errors: Backend API returning internal server errors during data requests

User Reports

  • "The heatmap y-axis was measured in minute increments, not 30-minutes, indicating incorrect x-axis time formatting"
  • "The heatmap was incorrectly displaying time increments and there was no measure of time on either axis"
  • "The heatmap for the 30-minute timeframe is displayed as though it is a single individual increment of time"

Technical Analysis

Root Cause Investigation

Issue 1: Y-Axis Configuration Problem

Root Cause: Plotly.js heatmap implementation was passing categorical metric names directly as Y-axis values, which conflicts with Plotly's expectation for numeric coordinates.

Evidence:

// PROBLEMATIC (Original Implementation):
const trace = {
    z: zValues,
    x: timestamps,
    y: metricNames, // ["RSI", "GarchVolatility"] - String array causes issues
    type: 'heatmap'
};

Issue 2: Timestamp Filtering Collapse

Root Cause: Frontend timestamp filtering logic was aggressively reducing multiple timestamps to a single point when data intervals didn't match expected timeframe intervals.

Evidence:

Generated 30m tick positions: {
    timeframe: '30m', 
    tickCount: 1, 
    minTime: '2025-10-18T15:52:43.000Z', 
    maxTime: '2025-10-18T15:52:43.000Z'
}

Implementation History

Attempt 1: Direct Y-Axis Fix (Initial)

Approach: Enhanced tick configuration with explicit tickvals and ticktext mapping
Status: :cross_mark: Ineffective

Implementation:

yaxis: {
    tickmode: 'array',
    tickvals: metricNames.map((_, index) => index),
    ticktext: metricNames,
    autorange: false,
    range: [-0.5, metricNames.length - 0.5]
}

Attempt 2: Alternative Numeric Y-Axis Implementation

Approach: Transform categorical labels to numeric coordinates with separate tick labeling
Status: :cross_mark: Ineffective

Implementation:

// Data transformation for proper Plotly.js structure
const numericYValues = metricNames.map((_, index) => index); // [0, 1, 2, ...]
const yAxisConfig = {
    tickmode: 'array',
    tickvals: numericYValues,
    ticktext: metricNames
};

const trace = {
    z: zValues,
    x: timestamps,
    y: numericYValues, // Numeric coordinates
    text: textMatrix,  // Metric names for hover
    type: 'heatmap'
};

Verification System Added:

verifyHeatmapConfiguration(traces, layout, metricNames, yAxisConfig) {
    // Comprehensive validation of:
    // - Y-axis numeric values
    // - Tick mapping accuracy
    // - Z-matrix structure integrity
    // - Text matrix for hover tooltips
}

Attempt 3: Timestamp Range Preservation

Approach: Use original timestamps for X-axis display while filtering only data values
Status: :cross_mark: Ineffective

Implementation:

// Use original timestamps for x-axis to prevent collapse
const xAxisConfig = this.getXAxisConfiguration(timeframe, rawTimestamps);
finalXData = rawTimestamps; // Preserve full time range

Attempt 4: Enhanced Tick Generation

Approach: Generate ticks around single timestamps for proper X-axis display
Status: :cross_mark: Ineffective

Implementation:

if (isSingleTimestamp || tickCount <= 1) {
    // Generate ticks around central time
    tickCount = Math.min(6, 8);
    const centerTime = minTime;
    
    for (let i = -Math.floor(tickCount / 2); i < Math.ceil(tickCount / 2); i++) {
        const tickTime = centerTime + (i * tickInterval);
        tickvals.push(new Date(tickTime));
    }
}

Attempt 5: No Frontend Filtering Architecture

Approach: Eliminate all frontend timestamp filtering, use simple automatic tick generation
Status: :cross_mark: Ineffective

Implementation:

// CRITICAL FIX: Always use original timestamps for display
const timestamps = rawTimestamps; // Use ALL original timestamps

// Simple x-axis configuration without complex tick generation
const xAxisConfig = this.getSimpleXAxisConfiguration(timeframe);

getSimpleXAxisConfiguration(timeframe) {
    switch (timeframe) {
        case '30m':
            return {
                tickmode: 'auto',
                tickformat: '%H:%M',
                type: 'date',
                dtick: 1800000 // 30 minutes hint
            };
    }
}

Current Technical State

File Modifications Made

Primary File: static/heatmap.js

Key Methods Modified:

  1. renderHeatmap() - Core rendering logic (lines 552-850)
  2. getYAxisConfiguration() - Y-axis setup (lines 1198-1224)
  3. transformDataForHeatmap() - Data structure transformation (lines 1239-1302)
  4. verifyHeatmapConfiguration() - Pre-render validation (lines 1315-1407)
  5. generateTickPositions() - Tick generation logic (lines 1123-1254)
  6. getSimpleXAxisConfiguration() - Simplified axis config (lines 1282-1329)

Current Implementation Features

  • :white_check_mark: Comprehensive error handling and diagnostics
  • :white_check_mark: Multi-layer verification system
  • :white_check_mark: Enhanced logging throughout data flow
  • :white_check_mark: Y-axis numeric coordinate transformation
  • :white_check_mark: Automatic vs manual tick generation options
  • :cross_mark: Y-axis metric labels still not displaying correctly
  • :cross_mark: 30-minute timeframe still showing single timestamp

Diagnostic Information

Console Output Analysis

Expected Diagnostic Patterns:

🔍 ALTERNATIVE Y-AXIS DIAGNOSTICS: {
    implementation: 'Alternative Numeric Y-Axis with Tick Labels',
    metricNames: ["RSI", "GarchVolatility", "HestonVolatility"],
    transformedYValues: [0, 1, 2],
    yAxisTickMapping: { tickvals: [0,1,2], ticktext: ["RSI","GarchVolatility","HestonVolatility"] }
}

🔧 SIMPLE X-AXIS: Using automatic tick generation for 30m
🔧 DATA VERIFICATION: Checking alignment between timestamps and z-values

Error Handling Enhancement

if (!verificationResult.success) {
    console.error('❌ ALTERNATIVE IMPLEMENTATION: Verification failed:', verificationResult.errors);
    this.showError(`Heatmap configuration error: ${verificationResult.errors.join(', ')}`);
    return; // Prevent rendering with invalid data
}

Identified Technical Challenges

Challenge 1: Plotly.js Heatmap Architecture

Issue: Fundamental misunderstanding of Plotly.js heatmap requirements for multi-layer data visualization.

Attempted Solutions:

  • Single trace with 2D Z-matrix vs multiple traces
  • Numeric vs categorical Y-axis coordinates
  • Explicit vs automatic tick configuration

Challenge 2: Timestamp Processing

Issue: Backend data intervals (15-second) vs frontend timeframe expectations (30-minute) causing data collapse.

Attempted Solutions:

  • Frontend timestamp filtering
  • Data alignment corrections
  • Tick generation around single timestamps

Challenge 3: Data Flow Architecture

Issue: Unclear separation between backend aggregation and frontend display responsibilities.

Current Understanding: Backend should handle timeframe aggregation, frontend should only handle display formatting.

Recommendations for Resolution

Immediate Actions Required

  1. Backend Investigation

    • Verify that the backend API is correctly aggregating data for 30-minute timeframes
    • Check if the 15-second raw data is being properly aggregated to 30-minute intervals
    • Validate API response structure for timeframe-specific requests

  2. Plotly.js Architecture Review

    • Research proper multi-layer heatmap implementation in Plotly.js
    • Consider alternative visualization libraries if Plotly.js limitations are identified
    • Test with minimal data sets to isolate the issue

  3. Data Structure Validation

    • Add comprehensive data validation at the API response level
    • Verify that metric names are being properly passed from backend
    • Ensure timestamp arrays maintain proper time intervals

Long-term Architectural Changes

  1. Separation of Concerns

    • Backend: Data aggregation, timeframe processing, metric calculation
    • Frontend: Display formatting, user interaction, visualization rendering

  2. Enhanced Error Handling

    • API response validation before rendering attempts
    • Graceful degradation when data is insufficient
    • User-friendly error messages for different failure modes

  3. Testing Framework

    • Unit tests for individual rendering components
    • Integration tests for complete data flow
    • Visual regression tests for rendering accuracy

Current Status

Overall Status: :red_circle: UNRESOLVED

Working Components:

  • :white_check_mark: Raw tick data and 3-minute timeframes display correctly
  • :white_check_mark: Data loading and processing pipeline functional
  • :white_check_mark: Error handling and diagnostics comprehensive

Persistent Issues:

  • :cross_mark: Y-axis metric labels not displaying properly across all timeframes
  • :cross_mark: 30-minute timeframe collapses to single timestamp display
  • :cross_mark: HTTP 500 errors occurring intermittently

Next Steps: Backend investigation and alternative visualization approach research required.