Building DataDiluvium: A Data Generation Tool – Part 3: Data Generation and Final Implementation

In Parts 1 and 2, I set up the development environment and implemented the schema parsing functionality. Now, I’ll explore the data generation system and final implementation details that make DataDiluvium a complete solution.

Data Generation System

1. Generator Registry

I’ve implemented the generator system in src/app/lib/generators/registry.ts. This registry manages different types of data generators:

import { Generator } from './types';
import { SequentialNumberGenerator } from './basic';
import { UsernameGenerator } from './basic';
import { DateGenerator } from './basic';
import { ForeignKeyGenerator } from './basic';

export const generatorRegistry = new Map<string, Generator>();

// Register built-in generators
generatorRegistry.set('Sequential Number', new SequentialNumberGenerator());
generatorRegistry.set('Username', new UsernameGenerator());
generatorRegistry.set('Date', new DateGenerator());
generatorRegistry.set('Foreign Key', new ForeignKeyGenerator());

2. Generator Types

I’ve defined the Generator interface in src/app/lib/generators/types.ts:

export interface Generator {
  name: string;
  generate(count: number): Promise<any[]>;
  validate(value: any): boolean;
}

3. Built-in Generators

I’ve implemented several built-in generators in src/app/lib/generators/basic.ts:

export class SequentialNumberGenerator implements Generator {
  private nextNumber: number = 1;

  async generate(count: number): Promise<number[]> {
    const numbers = Array.from(
      { length: count },
      () => this.nextNumber++
    );
    return numbers;
  }
}

export class UsernameGenerator implements Generator {
  async generate(count: number): Promise<string[]> {
    return Array.from({ length: count }, () =>
      faker.internet.userName()
    );
  }
}

Export Functionality

1. Export Implementation

I’ve implemented the export functionality in src/app/generated/page.tsx. The export system supports multiple formats:

const exportData = () => {
  if (!generatedData) return;

  Object.entries(generatedData).forEach(([tableName, tableData]) => {
    let content: string;
    let extension: string;
    let mimeType: string;

    switch (selectedExportFormat) {
      case 'json':
        content = JSON.stringify(tableData, null, 2);
        extension = '.json';
        mimeType = 'application/json';
        break;
      case 'json-rich':
        const richData = tableData.rows.map(row => {
          const obj: { [key: string]: any } = {};
          tableData.columns.forEach((col, index) => {
            obj[col] = row[index];
          });
          return obj;
        });
        content = JSON.stringify(richData, null, 2);
        extension = '.json';
        mimeType = 'application/json';
        break;
      case 'csv':
        // CSV implementation
        break;
      case 'xml':
        // XML implementation
        break;
      case 'txt':
        // Plain text implementation
        break;
      case 'sql':
        // SQL inserts implementation
        break;
    }
  });
};

Dark Mode Support

1. Theme Implementation

I’ve implemented dark mode using Tailwind CSS and Next.js. The theme configuration is in tailwind.config.ts:

module.exports = {
  darkMode: 'class',
  content: [
    './src/pages/**/*.{js,ts,jsx,tsx,mdx}',
    './src/components/**/*.{js,ts,jsx,tsx,mdx}',
    './src/app/**/*.{js,ts,jsx,tsx,mdx}',
  ],
  theme: {
    extend: {
      colors: {
        // Custom color palette
      },
    },
  },
  plugins: [],
}

2. Theme Toggle

I’ve implemented the theme toggle in src/app/components/ThemeToggle.tsx:

export function ThemeToggle() {
  const [theme, setTheme] = useState<'light' | 'dark'>('light');

  useEffect(() => {
    const savedTheme = localStorage.getItem('theme') as 'light' | 'dark';
    if (savedTheme) {
      setTheme(savedTheme);
      document.documentElement.classList.toggle('dark', savedTheme === 'dark');
    }
  }, []);

  const toggleTheme = () => {
    const newTheme = theme === 'light' ? 'dark' : 'light';
    setTheme(newTheme);
    localStorage.setItem('theme', newTheme);
    document.documentElement.classList.toggle('dark');
  };

  return (
    <button
      onClick={toggleTheme}
      className="p-2 rounded-md hover:bg-gray-100 dark:hover:bg-gray-800"
    >
      {theme === 'light' ? '🌙' : '☀️'}
    </button>
  );
}

Responsive Design

1. Layout Components

I’ve implemented the responsive layout across multiple components:

  1. Navigation (src/app/components/Navigation.tsx):
<nav className="fixed left-0 top-0 h-full w-48 bg-white dark:bg-gray-900 border-r border-gray-200 dark:border-gray-800">
  {/* Navigation content */}
</nav>
  1. Main Content (src/app/layout.tsx):
<div className="min-h-screen p-8 sm:p-20 ml-48">
  {children}
</div>

Error Boundaries and Loading States

1. Error Boundary

I’ve implemented the error boundary in src/app/components/ErrorBoundary.tsx:

export class ErrorBoundary extends React.Component<
  { children: React.ReactNode },
  { hasError: boolean }
> {
  constructor(props: { children: React.ReactNode }) {
    super(props);
    this.state = { hasError: false };
  }

  static getDerivedStateFromError(error: any) {
    return { hasError: true };
  }

  render() {
    if (this.state.hasError) {
      return (
        <div className="p-4 bg-red-50 dark:bg-red-900/20 text-red-700 dark:text-red-300 rounded-md">
          <h2 className="text-lg font-semibold">Something went wrong</h2>
          <p>Please try refreshing the page</p>
        </div>
      );
    }

    return this.props.children;
  }
}

2. Loading States

I’ve implemented loading states using React Suspense and loading.tsx files in the app directory:

// src/app/loading.tsx
export default function Loading() {
  return (
    <div className="flex items-center justify-center min-h-screen">
      <div className="animate-spin rounded-full h-12 w-12 border-b-2 border-blue-500"></div>
    </div>
  );
}

Conclusion

I’ve now completed DataDiluvium as a full solution for generating test data from SQL schemas. The application features:

  • Robust SQL schema parsing
  • Flexible data generation system
  • Multiple export formats
  • Dark mode support
  • Responsive design
  • Error handling and loading states

I’ve organized the code following Next.js 13+ best practices and used modern React patterns throughout. The application is ready for deployment and can be extended with additional generators and export formats as needed.

Building DataDiluvium: A Data Generation Tool – Part 2: Core Implementation and Schema Handling

In Part 1, we set up our development environment and outlined the project structure. Now, we’ll dive into implementing the core functionality of DataDiluvium, starting with SQL schema parsing and the user interface for schema input.

SQL Schema Parsing Implementation

1. SQL Parser Setup

The SQL parser is implemented in src/app/lib/parsers/sqlParser.ts. This module is responsible for:

  1. Parsing SQL schema definitions using the node-sql-parser library
  2. Extracting table and column information
  3. Mapping SQL data types to appropriate data generators

The parser works by:

  1. Taking a SQL string input
  2. Converting it to an Abstract Syntax Tree (AST)
  3. Traversing the AST to extract:
    • Table names
    • Column names
    • Data types
    • Default values
    • Foreign key relationships

The SchemaColumn type defines the structure of our parsed columns:

export type SchemaColumn = {
  tableName: string;
  columnName: string;
  dataType: string;
  defaultValue: string | null;
  generator?: string;
  referencedTable?: string;
  referencedColumn?: string;
};

2. Schema Validation

The validation system is implemented in src/app/lib/validators/schemaValidator.ts. This module performs several important validations:

  1. Duplicate Column Check:

    • Ensures no duplicate column names exist within the same table
    • Uses a Map to track columns per table
  2. Foreign Key Validation:

    • Verifies that referenced tables and columns exist
    • Checks for circular references
    • Validates data type compatibility

The validation results are structured as:

type ValidationResult = {
  errors: ValidationError[];
  warnings: ValidationWarning[];
};

User Interface Implementation

1. Schema Input Component

The main schema input interface is implemented in src/app/components/SchemaInput.tsx. This component provides:

  1. A textarea for SQL schema input
  2. Real-time validation feedback
  3. A preview of the parsed schema
  4. Error and warning displays

The component uses Tailwind CSS for styling and is structured as:

<div className="space-y-4">
  {/* SQL Input Section */}
  <div className="flex flex-col space-y-2">
    <label>SQL Schema</label>
    <textarea
      value={schemaText}
      onChange={handleSchemaChange}
      className="w-full h-64 p-4 font-mono text-sm border rounded-md"
    />
  </div>

  {/* Validation Results Section */}
  <div className="space-y-2">
    {/* Error and Warning Displays */}
  </div>

  {/* Schema Preview Section */}
  <div className="mt-4">
    {/* Table Preview */}
  </div>
</div>

2. Real-time Validation and Error Handling

The debounced validation system is implemented in src/app/lib/hooks/useDebounce.ts. This custom hook:

  1. Prevents excessive validation calls while typing
  2. Waits for a specified delay before updating
  3. Cleans up timeouts on unmount

Usage example:

const debouncedSchema = useDebounce(schemaText, 500);

Building the Foundation

1. State Management

The state management system is implemented in src/app/lib/context/SchemaContext.tsx. This context provides:

  1. Global schema state
  2. Validation results
  3. State update methods

The context is wrapped around the application in src/app/layout.tsx:

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body>
        <SchemaProvider>
          {children}
        </SchemaProvider>
      </body>
    </html>
  );
}

2. Navigation and Layout

The navigation component is implemented in src/app/components/Navigation.tsx. This component:

  1. Provides consistent navigation across pages
  2. Implements dark mode support
  3. Uses Next.js Link components for client-side navigation

The layout structure follows Next.js 13+ app router conventions:

src/app/
├── layout.tsx        # Root layout with providers
├── page.tsx          # Home page
├── schema/
│   └── page.tsx      # Schema input page
├── generate/
│   └── page.tsx      # Data generation page
└── generated/
    └── page.tsx      # Generated data display page

Next Steps

In Part 3, we’ll explore:

  • Implementing the data generation system
  • Creating the export functionality
  • Adding dark mode support
  • Implementing responsive design
  • Adding error boundaries and loading states

Stay tuned for the final part of our series, where we’ll complete the DataDiluvium implementation!

Building DataDiluvium: A Data Generation Tool – Part 1: Prerequisites and Project Overview

(To read up on how to use the site, check out the previous post Effortless Data Generation for Developers.)

DataDiluvium is a web-based tool I’ve built designed to help developers, database administrators, and data engineers generate realistic test data based on SQL schema definitions. The tool takes SQL table definitions as input and produces sample data in various formats, making it easier to populate development and testing environments with meaningful data.

Project Overview

The core functionality of DataDiluvium includes:

  • SQL schema parsing and validation
  • Customizable data generation rules per column
  • Support for foreign key relationships
  • Multiple export formats (JSON, CSV, XML, Plain Text, SQL Inserts)
  • Real-time preview of generated data
  • Dark mode support
  • Responsive design

Prerequisites and Setup

Development Environment

  1. Node.js and npm

  2. Git

  3. Code Editor

    • VS Code (or Cursor?)
    • Extensions:
      • ESLint
      • Prettier
      • Tailwind CSS IntelliSense
      • TypeScript and JavaScript Language Features

Core Technologies

  1. Next.js 15.1.3

    • React framework for production
    • App Router architecture
    • Server-side rendering capabilities
    • Installation: npx create-next-app@latest
  2. TypeScript

    • For type-safe JavaScript
    • Included with Next.js setup
    • Version: 5.x
  3. Tailwind CSS

    • Utility-first CSS framework
    • For responsive design
    • Dark mode support
    • Installation: Included with Next.js setup

Additional Dependencies

  1. SQL Parser

    • For parsing SQL schema definitions
    • We’ll use node-sql-parser for SQL parsing capabilities
  2. Data Generation Libraries

    • faker for generating realistic data
    • uuid for unique identifier generation
    • date-fns for date manipulation

Project Structure

datadiluvium/
├── src/
│   ├── app/                    # Next.js app router pages
│   ├── components/             # Reusable React components
│   ├── lib/                    # Core functionality
│   │   ├── generators/        # Data generation logic
│   │   ├── parsers/          # SQL parsing logic
│   │   └── types/            # TypeScript type definitions
│   └── styles/                # Global styles
├── public/                    # Static assets
└── package.json              # Project dependencies

Initial Setup Steps

  1. Create a new Next.js project:

    npx create-next-app@latest datadiluvium --typescript --tailwind --eslint
    
  2. Navigate to the project directory:

    cd datadiluvium
    
  3. Install additional dependencies:

    npm install node-sql-parser @faker-js/faker uuid date-fns
    
  4. Configure TypeScript and ESLint:

    • The Next.js setup will create these configurations
    • Additional rules may be added as needed
  5. Set up Tailwind CSS:

    • Configuration will be created by Next.js
    • Custom theme settings can be added later

Development Guidelines

  1. Code Style

    • Use TypeScript for all new files
    • Follow ESLint and Prettier configurations
    • Use functional components with hooks
    • Implement proper error handling
  2. Git Workflow

    • Use feature branches
    • Write meaningful commit messages
    • Follow conventional commits format
  3. Testing Strategy

    • Unit tests for data generation logic
    • Integration tests for SQL parsing
    • Component tests for UI elements

NOTE: for each of these I’m going to try to follow them. It’s always tough to get a 1000% consistency across a code base!

Next Steps

In the next part of this series, we’ll dive into implementing the core SQL parsing functionality and the schema input interface. We’ll explore how to:

  • Parse SQL schema definitions
  • Validate table structures
  • Create a user-friendly interface for schema input
  • Implement real-time validation and error handling

Stay tuned for Part 2, where we’ll begin building the foundation of the DataDiluvium App!

Effortless Data Generation for Developers

DataDiluvium is a web-based tool available at datadiluvium.com that helps developers, database administrators, and data engineers generate realistic test data from SQL schema definitions. Whether you’re setting up a development environment, creating test scenarios, or preparing data for demonstrations, DataDiluvium streamlines the process of data generation.

What is DataDiluvium?

Purpose

DataDiluvium serves several key purposes:

  1. Development Environment Setup: Quickly populate development databases with meaningful test data
  2. Testing: Generate consistent test data for automated testing scenarios
  3. Demonstrations: Create realistic data sets for product demonstrations
  4. Data Migration Testing: Validate data migration scripts with generated test data
  5. Schema Validation: Test database schema designs with realistic data

Key Features

  • SQL schema parsing and validation
  • Customizable data generation rules
  • Support for foreign key relationships
  • Multiple export formats (JSON, CSV, XML, Plain Text, SQL Inserts)
  • Real-time preview of generated data
  • Dark mode support
  • Responsive design

How to Use DataDiluvium

1. Accessing the Application

  1. Visit datadiluvium.com
  2. No account required – start using immediately
  3. Your data is processed locally in your browser

2. Defining Your Schema

  1. Navigate to the Schema page
  2. Enter your SQL schema definition in the text area Example:

    CREATE TABLE users (
        id INT PRIMARY KEY,
        username VARCHAR(50) NOT NULL,
        email VARCHAR(100) NOT NULL,
        created_at DATETIME DEFAULT CURRENT_TIMESTAMP
    );
    
    CREATE TABLE orders (
        id INT PRIMARY KEY,
        user_id INT,
        total_amount DECIMAL(10,2),
        created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
        FOREIGN KEY (user_id) REFERENCES users(id)
    );
    
  3. The application will automatically:

    • Parse your schema
    • Validate the structure
    • Suggest appropriate data generators
    • Show a preview of the parsed schema

3. Configuring Data Generation

  1. For each column, you can:

    • Select a data generator
    • Set custom parameters
    • Define relationships
  2. Available generators include:

    • Sequential Numbers
    • Usernames
    • Email addresses
    • Dates
    • Foreign Keys
    • Custom text
    • And more…
  3. Set the number of rows to generate:

    • Global row count for all tables
    • Table-specific row counts
    • Preview sample data before generation

4. Generating Data

  1. Click the “Generate” button
  2. Review the generation summary
  3. Confirm the generation
  4. Wait for the process to complete

5. Exporting Data

  1. Choose your preferred export format:

    • JSON: Standard JSON format with columns and rows
    • JSON (rich): Array of objects with column names as keys
    • CSV: Comma-separated values with headers
    • XML: Structured XML format
    • Plain Text: Human-readable format with numbered rows
    • SQL Inserts: Ready-to-use SQL INSERT statements
  2. Click the “Export” button
  3. Files will be downloaded automatically:

    • One file per table
    • Named according to the table name
    • Appropriate file extension based on format

Best Practices

1. Schema Design

  • Use clear, descriptive table and column names
  • Include appropriate constraints
  • Define foreign key relationships
  • Use appropriate data types

2. Data Generation

  • Start with a small number of rows for testing
  • Use appropriate generators for each column type
  • Consider data relationships when setting up foreign keys
  • Preview data before generating large sets

3. Export Selection

  • Choose JSON for application development
  • Use CSV for spreadsheet applications
  • Select SQL Inserts for direct database population
  • Consider Plain Text for human review

Example Workflow

Scenario: Setting up a Development Environment

  1. Define Schema

    CREATE TABLE products (
        id INT PRIMARY KEY,
        name VARCHAR(100) NOT NULL,
        price DECIMAL(10,2),
        category_id INT,
        created_at DATETIME DEFAULT CURRENT_TIMESTAMP
    );
    
    CREATE TABLE categories (
        id INT PRIMARY KEY,
        name VARCHAR(50) NOT NULL
    );
    
  2. Configure Generators

    • id: Sequential Number
    • name: Product Name
    • price: Random Decimal (10-1000)
    • category_id: Foreign Key to categories
    • created_at: Current Date
  3. Generate Data

    • Set 100 rows for products
    • Set 10 rows for categories
    • Generate and review
  4. Export

    • Choose SQL Inserts format
    • Download and execute in your development database

Tips and Tricks

1. Performance

  • Generate data in smaller batches for large schemas
  • Use appropriate generators for better performance
  • Preview data before large generations

2. Data Quality

  • Use meaningful generators for each column type
  • Consider data relationships
  • Validate generated data before use

3. Export Formats

  • JSON (rich) for application development
  • CSV for data analysis
  • SQL Inserts for database population
  • Plain Text for quick review

Support and Resources

  • Visit datadiluvium.com for the latest version
  • Check the documentation for detailed guides
  • Review sample schemas in the SQL samples section
  • Contact support for questions or feedback

Conclusion

DataDiluvium provides a user-friendly and powerful solution for generating test data from SQL schemas. Whether you’re a developer setting up a new project or a database administrator preparing test environments, DataDiluvium streamlines the process of data generation and helps ensure data quality and consistency.

MongoDB and CAP Theorem: Key Insights

When you first dive into distributed systems, the CAP theorem feels like an unavoidable pop quiz. A pop quiz that forces you to choose between Consistency, Availability, and Partition Tolerance. Traditionally, many have painted MongoDB as a system that prioritizes Availability and Partition Tolerance, placing it squarely in the AP camp. However, there’s a compelling argument that MongoDB can also be seen as a CP system in certain scenarios, especially when compared to systems like Cassandra, which is widely categorized as AP.

Rethinking MongoDB: CP or AP?

The debate often centers on how MongoDB handles consistency. In its default setup, MongoDB opts for high availability, ensuring that your application stays up even when parts of the network go dark. This has led many to view it as an AP system. However, MongoDB also offers robust consistency guarantees, especially with its replica set configurations and tunable write concerns, that can push it toward the CP corner under specific conditions. In essence, MongoDB gives you the flexibility to dial up consistency when your application demands it, blurring the traditional AP versus CP lines.

Apache Cassandra, on the other hand, is designed to be AP by default. It emphasizes continuous availability and partition tolerance at the cost of immediate consistency, relying on eventual consistency as its safety net. This distinction is important when architecting systems because it underscores the need to choose the right tool based on your application’s tolerance for stale data versus downtime.

Embracing CAP in MongoDB

MongoDB’s architecture is built with the CAP theorem in mind. It ensures that, even during network partitions, your system remains operational. By favoring Availability and Partition Tolerance in many configurations, MongoDB allows applications to keep running even when data isn’t instantly consistent across every node. Yet, thanks to configurable read preferences and write concerns, you can pivot toward stronger consistency if your use case requires it. This makes MongoDB a versatile option in the distributed database arena.

Architectural Deep Dive: Enabling MongoDB’s CAP Capabilities

The resilience of MongoDB under the ole’ CAP theorem is the result of carefully designed architectural elements:

  • Replica Sets: At the heart of MongoDB’s fault tolerance are replica sets. A replica set consists of multiple nodes holding the same data, with one primary handling writes and several secondaries replicating that data. If the primary node fails, one of the secondaries is quickly promoted, ensuring that the system remains available. This design not only underpins availability but also provides the option to enforce stronger consistency when needed.
  • Sharding: To manage massive workloads, MongoDB distributes data across shards. Each shard is a replica set in its own right, allowing the database to scale horizontally while maintaining resilience. Sharding ensures that even if one segment of the system is under heavy load or experiences issues, the rest of the system can continue to function smoothly.
  • Configurable Consistency: MongoDB’s flexible read preferences and write concerns empower developers to adjust the balance between consistency and performance. Direct your reads to the primary when absolute consistency is required, or lean on secondary reads to optimize for speed. These settings allow you to tailor the database behavior to the specific needs of your application.

ACID Compliance in a Distributed World

Despite the inherent trade-offs of CAP, MongoDB doesn’t sideline the importance of ACID (Atomicity, Consistency, Isolation, Durability) properties, especially when data integrity is a must.

  • Atomicity and Consistency: With the introduction of multi-document ACID transactions in MongoDB 4.0, operations across multiple documents or collections are treated as a single, atomic unit. This means that either all operations succeed together, or none do. This ensures data isn’t left in a partial state.
  • Isolation: Leveraging snapshot isolation, MongoDB offers a consistent view of data throughout the duration of a transaction. This isolation prevents concurrent operations from stepping on each other’s toes, preserving transactional integrity even in high-concurrency environments.
  • Durability: Thanks to replica set architecture, once a transaction is committed, the data is safely replicated across nodes. This redundancy ensures that your data remains intact even in the face of node failures.

Expanding the Conversation: From CAP to PACELC

Some have posited the position that traditional CAP theorem paints an incomplete picture. Enter the PACELC theorem, coined by Daniel J. Abadi, which extends the discussion beyond just partitions. PACELC states that if there is a Partition (P), you must choose between Availability (A) and Consistency (C); Else (E), when the system is running normally, you still face a trade-off between Latency (L) and Consistency (C).

This broader perspective is crucial for understanding systems like MongoDB. Even when partitions aren’t an issue, the latency inherent in ensuring strict consistency can impact performance. MongoDB’s configurable consistency settings let you optimize for lower latency when absolute consistency isn’t mission-critical, and vice versa. In doing so, MongoDB embodies the PACELC trade-offs. Which helps balance the needs of an application in both partitioned and normal operating conditions.

Ensuring Balance for the Desired Outcome

Striking the right balance between CAP (or PACELC) trade-offs and ACID compliance is essential for designing systems that meet your application’s unique demands. Here’s how you can ensure that balance:

  • Define Your Priorities: Evaluate whether your application can handle eventual consistency for improved availability, or if strict consistency is vital. This will inform your choice between MongoDB’s flexible settings.
  • Tune Your Settings: Adjust your write concerns and read preferences based on the desired outcome. If durability and consistency are paramount, opt for a “majority” write concern. Even if that introduces a slight latency hit.
  • Monitor and Iterate: Use MongoDB’s robust telemetry and monitoring tools to understand how your system behaves under different configurations. Regularly review and adjust your settings as your application scales and requirements evolve.
  • Consider the Trade-offs: Embrace the broader PACELC framework to evaluate your system’s performance. Whether you’re managing latency in normal operations or choosing between availability and consistency during partitions, keep your application’s goals at the forefront of your design decisions.

Conclusion

The world of distributed systems is rife with trade-offs, and the CAP theorem ** expanded by PACELC ** provides a powerful framework for navigating them. MongoDB’s architecture, with its smart use of replica sets, sharding, and configurable consistency, offers a flexible platform that can be tuned to act as either CP or AP, depending on your needs. By supporting ACID transactions and allowing you to balance latency, consistency, availability, and durability, MongoDB empowers you to design systems that not only meet the technical challenges of distributed computing but also align perfectly with your desired outcomes.