Plot History Management With File Persistence

by Alex Johnson 46 views

In this article, we delve into the critical aspects of implementing a robust plot history management system using file-based persistence. This feature is essential for iterative data analysis workflows, enabling users to navigate, compare, and export plot variations efficiently. Inspired by RStudio's proven architecture, we will explore the architecture, core components, implementation plan, and design decisions involved in building such a system.

Understanding the Need for Plot History Management

Plot history management is crucial in modern data analysis. The absence of a systematic plot history management system can significantly hinder the user experience. Without it, users face challenges in revisiting, comparing, and managing plots generated during different stages of analysis. Specifically, users cannot easily:

  • Navigate through past plots, making it difficult to compare different versions or variations.
  • Persist plots across sessions, leading to the loss of valuable work when the application is closed.
  • View plot thumbnails in a timeline, which provides a quick overview of the analysis process.
  • Export individual plots from history, limiting the ability to share or further process specific visualizations.

These limitations underscore the need for a comprehensive plot history management system that addresses these shortcomings and enhances the overall data analysis workflow. This article outlines a solution inspired by the architecture of RStudio's plot history, leveraging file-based persistence for robust and efficient plot management.

Solution: A File-Based Plot History Management System

To address these challenges, we propose a file-based plot history management system. This system will allow users to navigate through past plots, persist plots across sessions, view plot thumbnails in a timeline, and export individual plots from history. The solution is inspired by RStudio's proven architecture, which has demonstrated its effectiveness in managing plot history efficiently.

The system's architecture is designed around several core principles, including efficient storage, reliable identification, and seamless integration with the user interface. The following sections will delve into the key components and design decisions that underpin this system.

Architecture Overview

The architecture of our file-based plot history management system comprises several key components that work together to provide a seamless user experience. The core of the system is the PlotHistoryManager, a Rust-based module responsible for managing plot data. This manager utilizes a circular buffer to store a limited number of plots (e.g., 50) and employs UUIDs for unique plot identification. File-based persistence ensures that plots are saved across sessions, and an event-driven mechanism keeps the user interface synchronized with changes in the plot history.

PlotHistoryManager (Rust)
├── Circular buffer (max 50 plots)
├── UUID-based identification
├── File-based persistence
└── Event-driven updates

Plot Storage Structure:
{project_dir}/.reprod/plots/
├── plots.json          # Metadata (active plot, UUIDs, timestamps)
├── {uuid1}.png        # Plot image
├── {uuid2}.png        # Plot image
└── ...

The plot storage structure is organized within a dedicated directory, typically named .reprod/plots/, inside the project directory. This directory contains a plots.json file that stores metadata about the plots, such as their UUIDs, timestamps, and the active plot index. Additionally, each plot is saved as a PNG image file, named using its UUID. This structure ensures that plot data is easily accessible and well-organized.

Core Components

The file-based plot history management system consists of three core components: the backend (Rust), the frontend (TypeScript/React), and the WebSocket protocol. Each component plays a crucial role in the overall functionality of the system.

1. Backend (Rust)

The backend is implemented in Rust to ensure performance and reliability. The core of the backend is the PlotHistoryManager, which handles plot storage, retrieval, and management. This module is located in the core/src/plot_history/ directory.

pub struct PlotHistoryManager {
    plots: VecDeque<Plot>,      // Circular buffer (max 50)
    active_index: usize,
    storage_path: PathBuf,
    max_plots: usize,
}

pub struct Plot {
    id: String,                  // UUID
    timestamp: i64,
    width: u32,
    height: u32,
    image_path: PathBuf,
    code: Option<String>,        // R code that generated the plot
}

impl PlotHistoryManager {
    pub fn new(storage_path: PathBuf, max_plots: usize) -> Self;
    pub fn add_plot(&mut self, plot_data: Vec<u8>, code: Option<String>) -> Result<String>;
    pub fn get_plot(&self, id: &str) -> Option<&Plot>;
    pub fn set_active(&mut self, index: usize) -> Result<()>;
    pub fn remove_plot(&mut self, id: &str) -> Result<()>;
    pub fn list_plots(&self) -> Vec<PlotInfo>;
    pub fn save_state(&self) -> Result<()>;
    pub fn restore_state(&mut self) -> Result<()>;
}

The PlotHistoryManager uses a circular buffer (VecDeque) to store plots, ensuring that the system does not consume excessive memory. Each plot is represented by a Plot struct, which includes a unique ID (UUID), timestamp, dimensions, image path, and the R code used to generate the plot. The PlotHistoryManager provides methods for adding plots, retrieving plots, setting the active plot, removing plots, listing plots, saving the state, and restoring the state.

2. Frontend (TypeScript/React)

The frontend, built using TypeScript and React, provides the user interface for interacting with the plot history. A new component, PlotHistoryPanel.tsx, is responsible for displaying the plot history and providing navigation controls. This component includes a timeline view of plot thumbnails, navigation buttons (prev/next/jump), a plot export button, and a delete plot action.

State Management: client/src/core/state/slices/plotHistorySlice.ts

The state of the plot history is managed using a dedicated slice, plotHistorySlice.ts, which defines the state structure and reducers for updating the plot history.

interface PlotHistoryState {
    plots: PlotInfo[];
    activeIndex: number;
    loading: boolean;
}

interface PlotInfo {
    id: string;
    timestamp: number;
    width: number;
    height: number;
    thumbnailUrl: string;
    code?: string;
}

The PlotHistoryState interface includes an array of PlotInfo objects, the index of the active plot, and a loading flag. The PlotInfo interface contains the plot's ID, timestamp, dimensions, thumbnail URL, and the code used to generate the plot. This structured state management ensures that the frontend can efficiently display and interact with the plot history.

3. WebSocket Protocol

The WebSocket protocol facilitates communication between the backend and frontend, enabling real-time updates to the plot history. New messages are defined in shared/src/ws.ts to handle plot creation, updates, and actions.

// Server -> Client
| { type: 'plot_created'; plot: PlotInfo }
| { type: 'plot_history_updated'; plots: PlotInfo[]; activeIndex: number }

// Client -> Server
| { type: 'plot_history_list' }
| { type: 'plot_set_active'; index: number }
| { type: 'plot_remove'; id: string }
| { type: 'plot_export'; id: string; format: 'png' | 'pdf' }

Messages from the server to the client include plot_created and plot_history_updated, which inform the frontend about new plots and updates to the plot history. Client-to-server messages include plot_history_list, plot_set_active, plot_remove, and plot_export, which allow the frontend to request plot history data, set the active plot, remove a plot, and export a plot.

Implementation Plan

The implementation of the file-based plot history management system is divided into three phases, each focusing on a specific set of features and functionalities. This phased approach allows for incremental development and testing, ensuring a robust and reliable system.

Phase 1: Basic Plot History (MVP) - 6-8 hours

Phase 1 focuses on implementing the core functionality of the plot history system. This includes the backend components for managing plots and the frontend components for displaying the plot history.

Backend:

  • [ ] Create plot_history module with PlotHistoryManager
  • [ ] Implement circular buffer (VecDeque with max 50 plots)
  • [ ] UUID generation for each plot
  • [ ] Save plots as PNG files
  • [ ] Create plots.json metadata file
  • [ ] Add WebSocket handlers for plot history operations

Frontend:

  • [ ] Create PlotHistoryPanel component
  • [ ] Implement plot timeline view
  • [ ] Add navigation controls (prev/next buttons)
  • [ ] Display active plot indicator
  • [ ] Create plotHistorySlice for state management

Integration:

  • [ ] Hook plot creation event to auto-add to history
  • [ ] Update Console panel to show plot history
  • [ ] Add plot navigation hotkeys (Ctrl+Left/Right)

Phase 2: Persistence & Export - 4-6 hours

Phase 2 extends the functionality of the system by adding persistence and export capabilities. This ensures that plot history is maintained across sessions and that users can export plots for further use.

Backend:

  • [ ] Implement save_state() / restore_state()
  • [ ] Integrate with project save/load
  • [ ] Add plot export functionality (PNG, PDF via R)
  • [ ] Handle plot deletion with file cleanup

Frontend:

  • [ ] Add "Export Plot" button in history panel
  • [ ] Implement export dialog with format selection
  • [ ] Show plot metadata (timestamp, dimensions, code)
  • [ ] Add "Delete Plot" action

Phase 3: Advanced Features - 4-6 hours

Phase 3 introduces advanced features that enhance the user experience and provide additional functionality. These features include thumbnail generation, plot comparison, and search capabilities.

Backend:

  • [ ] Generate plot thumbnails for timeline
  • [ ] Add plot comparison feature
  • [ ] Implement plot search by code/timestamp
  • [ ] Memory optimization with lazy loading

Frontend:

  • [ ] Thumbnail grid view
  • [ ] Plot comparison side-by-side view
  • [ ] Search/filter plots
  • [ ] Drag-to-reorder plots (optional)

File Structure

The file structure of the project is organized to separate the backend and frontend components, making the codebase maintainable and scalable. The following structure outlines the key directories and files involved in the plot history management system.

core/src/plot_history/
├── mod.rs                   # Module exports
├── manager.rs               # PlotHistoryManager
├── plot.rs                  # Plot struct
└── persistence.rs           # Save/restore logic

client/src/components/plot-history/
├── PlotHistoryPanel.tsx     # Main component
├── PlotTimeline.tsx         # Timeline view
├── PlotCard.tsx             # Individual plot card
├── PlotNavigator.tsx         # Navigation controls
└── ExportDialog.tsx         # Export UI

client/src/core/state/slices/
└── plotHistorySlice.ts      # State management

Metadata Format (plots.json)

The plots.json file stores metadata about the plots, including their UUIDs, timestamps, dimensions, and the code used to generate them. This metadata is used to reconstruct the plot history when the application is restarted.

{
  "version": 1,
  "active_index": 0,
  "max_plots": 50,
  "plots": [
    {
      "id": "a7f3c2d1-4b5e-6789-abcd-ef0123456789",
      "timestamp": 1732567890123,
      "width": 800,
      "height": 600,
      "image_file": "a7f3c2d1-4b5e-6789-abcd-ef0123456789.png",
      "code": "plot(1:10, main='My Plot')"
    }
  ]
}

The JSON structure includes a version number, the index of the active plot, the maximum number of plots to store, and an array of plot objects. Each plot object contains the plot's ID, timestamp, dimensions, image file name, and the code used to generate the plot.

Design Decisions

Several key design decisions were made to ensure the efficiency, reliability, and scalability of the plot history management system. These decisions are based on best practices and lessons learned from existing implementations, such as RStudio's plot history.

  1. Circular Buffer: A circular buffer is used to limit the number of plots stored in memory, automatically evicting the oldest plots when the limit (50) is reached. This prevents excessive memory consumption and ensures that the system remains responsive.
  2. UUID Identification: UUIDs (Universally Unique Identifiers) are used to identify each plot uniquely. This prevents filename conflicts and ensures reliable cross-referencing between the metadata and the image files.
  3. JSON Metadata: Metadata about the plots is stored in a JSON file. JSON is a simple, debuggable, and easily extendable format, making it ideal for storing plot metadata.
  4. PNG Storage: Plots are saved as PNG images. PNG is a fast, widely supported, and high-quality image format, making it suitable for storing plot data.
  5. Lazy Loading: Plot images are loaded only when needed for display. This improves performance by reducing the memory footprint and load times.
  6. Event-Driven: The system automatically captures plots on R execution completion, ensuring that the plot history is always up-to-date.

Key Learnings from RStudio

Analyzing RStudio's implementation of plot history management provides valuable insights and best practices that can be applied to our system. RStudio's approach has been carefully studied to ensure that our implementation leverages the most effective techniques.

Based on analysis of RStudio's implementation (docs/.obsidian/design/rstudio-plot-history-management.md):

  1. ✅ UUID-based identification for reliable tracking
  2. ✅ File-based persistence for cross-session state
  3. ✅ Circular buffer for automatic memory management
  4. ✅ Lazy loading for efficiency
  5. ✅ Metadata separation from image data
  6. ✅ Event-driven capture on graphics device events

Testing Requirements

Comprehensive testing is essential to ensure the reliability and correctness of the plot history management system. The following testing requirements outline the key areas that need to be tested.

  • [ ] Plot history persists across app restarts
  • [ ] Circular buffer correctly evicts oldest plots
  • [ ] Navigation updates active plot correctly
  • [ ] Export generates valid PNG/PDF files
  • [ ] Plot deletion removes files from disk
  • [ ] WebSocket messages sync state correctly
  • [ ] Performance with 50+ plots in history
  • [ ] Concurrent plot creation handling

Estimated Time

The estimated time for implementing the file-based plot history management system is broken down by phase.

  • Phase 1 (MVP): 6-8 hours
  • Phase 2 (Persistence): 4-6 hours
  • Phase 3 (Advanced): 4-6 hours
  • Total: 14-20 hours

Difficulty

The implementation of this system is considered to be of high difficulty due to the complexity involved in state synchronization, file I/O, event-driven architecture, and memory management.

⭐⭐⭐⭐ Difficult

  • Complex state synchronization (Rust ↔ TypeScript)
  • File I/O and persistence
  • Event-driven architecture
  • Memory management with circular buffer
  • WebSocket protocol design

Priority

The plot history management system is a P1 (High priority) feature, as it is core to the data analysis workflow.

Dependencies

The implementation of this system relies on several dependencies, including Rust libraries and existing infrastructure components.

  • Rust: uuid, serde, serde_json
  • Plot rendering already implemented in execution system
  • WebSocket infrastructure in place

Related Issues

This feature is related to other issues in the project, such as PDF export and execution history management.

  • #136 - Generate PDF from R Markdown Export (related to PDF export)
  • Execution history management (similar pattern)

References

The design and implementation of this system are informed by RStudio's plot history management and the design document.

  • Design doc: docs/.obsidian/design/rstudio-plot-history-management.md
  • RStudio source: /Users/shinyayoshida/Job/self/YC/rstudio/src/cpp/r/session/graphics/

Conclusion

Implementing a file-based plot history management system is crucial for enhancing the user experience in data analysis workflows. By leveraging a circular buffer, UUID identification, JSON metadata, and file-based persistence, we can create a robust and efficient system. This system will enable users to navigate, compare, and export plots seamlessly, thereby improving their productivity and the quality of their analysis. The phased implementation plan, along with comprehensive testing, will ensure the successful deployment of this critical feature. To further explore the topic, consider checking out the official documentation and resources on data visualization techniques.