markdown_ld

# Changelog

All notable changes to MarkdownLd will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.0] - 2025-08-25

### Added
- Diff/merge foundations: Change, Patch, Conflict, MergeResult, StreamEvent
- Block-level diff with inline token ops and similarity-based updates
- Combined diff API: `MarkdownLd.diff/3`
- Streaming diff with paragraph/heading chunking, stable IDs, deletions
- High-level merge API: `MarkdownLd.Merge.merge_texts/4`
- JSON-LD extraction (code fences, frontmatter) with basic context expansion
- Conflict formatter (text + map) for UI
- SPEC.md draft (Markdown‑LD Profile v0.1)
 - Rename-aware heading alignment in streaming (fuzzy match)
 - Inline preview renderer for ops (`MarkdownLd.Diff.Preview`)
 - QCPrompt parser and stream helpers (`QCP.parse/1`, `QCP.Stream.process/1`) with tests

### Changed
- README updates and internal refactors for patch application

### Notes
- JSON-LD handling is pragmatic; full 1.1 (remote contexts, expansion/flattening) is future work.

## [0.3.0] - 2025-01-20

### Added
- **High-performance Rust NIF implementation** with SIMD optimizations
- **Zero-copy binary processing** for maximum efficiency
- **Memory pool management** with bumpalo for reduced allocations
- **Pattern caching** with LRU cache for repeated markdown structures
- **Parallel batch processing** with both Elixir and Rust-side concurrency
- **Comprehensive parsing support**:
  - Headings (H1-H6) with line number tracking
  - Markdown links with text and URL extraction
  - Fenced and indented code blocks with language detection
  - Task lists (GitHub-style checkboxes)
  - SIMD-optimized word counting
- **Performance tracking and metrics**:
  - Operation counters and timing
  - Cache hit rate monitoring
  - Memory usage tracking
  - Pattern cache statistics
- **Advanced build system**:
  - Multiple compilation profiles (dev, prod, bench, PGO)
  - SIMD feature detection (Apple Silicon NEON, x86 AVX2)
  - Profile-Guided Optimization support
  - Comprehensive CI/CD pipeline
- **Streaming API** with backpressure control
- **Comprehensive benchmark suite** with performance comparisons
- **Full API documentation** with HexDocs integration
- **Configuration system** for runtime and compile-time options

### Performance
- **10-50x faster** than pure Elixir markdown parsers
- **Sub-millisecond processing** for typical documents
- **Gigabyte-per-second throughput** for large documents
- **Memory-efficient** with 50-80% reduction in allocations
- **Scalable** to thousands of documents per second

### Technical
- **Rust backend** with pulldown-cmark parser integration
- **SIMD vectorization** for string operations and pattern matching  
- **Advanced memory management** with arena allocation
- **Multi-threaded processing** with rayon parallel processing
- **Comprehensive error handling** with graceful fallbacks
- **Production-ready** architecture with monitoring

### Documentation
- Complete API documentation with examples
- Performance benchmarking reports
- Architecture and design documentation
- Build system documentation
- Contributing guidelines

## [0.2.0] - 2025-01-19

### Added
- Basic Elixir project structure
- Core API design and interfaces
- Initial Rust NIF skeleton
- Basic markdown parsing functionality
- Test suite foundation

### Technical
- Mix project configuration
- Rustler integration setup
- Basic CI/CD pipeline
- Development tooling

## [0.1.0] - 2025-01-18

### Added
- Initial project creation
- Basic project structure
- Core concepts and architecture planning

### Technical
- Repository setup
- License selection (MIT)
- Initial development environment

---

## Upgrade Guide

### From 0.2.x to 0.3.x

The 0.3.0 release introduces the high-performance Rust implementation with significant API enhancements:

#### New Features
- All parsing functions now return structured data with line numbers
- Added batch processing functions: `parse_batch/2` and `parse_batch_rust/2`
- Added streaming API: `parse_stream/2`
- Added performance tracking functions: `get_performance_stats/0`, `reset_performance_stats/0`
- Added individual extraction functions: `extract_links/1`, `extract_headings/1`, etc.

#### Configuration Changes
- New configuration options for SIMD, memory pools, and parallel processing
- Runtime configuration support for per-operation settings

#### Performance Improvements
- 10-50x performance improvements across all operations
- Zero-copy binary processing option
- Memory pool optimization reduces allocation overhead
- SIMD acceleration for string operations

#### Breaking Changes
- None - the 0.3.0 API is fully backwards compatible with 0.2.x
- All existing code will continue to work without changes

## Migration Examples

### Basic Usage (No Changes Required)
```elixir
# This continues to work exactly as before
{:ok, result} = MarkdownLd.parse("# Hello World")
```

### New Performance Features
```elixir
# New: Zero-copy processing
{:ok, result} = MarkdownLd.parse_binary(binary_content)

# New: Batch processing
{:ok, results} = MarkdownLd.parse_batch(document_list)

# New: Performance metrics
{:ok, stats} = MarkdownLd.get_performance_stats()
```

### New Configuration Options
```elixir
# config/config.exs
config :markdown_ld,
  parallel: true,
  max_workers: System.schedulers_online(),
  cache_patterns: true,
  simd_enabled: true
```

## Planned Features

### 0.4.0 (Q2 2025)
- **JSON-LD metadata extraction** from markdown frontmatter
- **Schema.org structured data** generation
- **Semantic HTML output** with microdata
- **Plugin system** for custom processors
- **Advanced caching** with persistence options

### 0.5.0 (Q3 2025)
- **Real-time processing** with file system watching
- **Web API** for HTTP-based markdown processing
- **GraphQL integration** for metadata queries
- **Advanced analytics** and content insights

### 1.0.0 (Q4 2025)
- **Production stability** guarantees
- **Enterprise features** (clustering, monitoring)
- **Full semantic web** integration
- **Complete JSON-LD 1.1** specification support

## Support

- **Issues**: https://github.com/nocsi/markdown-ld/issues
- **Discussions**: https://github.com/nocsi/markdown-ld/discussions
- **Documentation**: https://hexdocs.pm/markdown_ld
- **Performance Reports**: See PERFORMANCE_REPORT.md

---

*MarkdownLd - High-performance markdown processing for production systems*