Skip to main content

Document Versioning & RAG Processing

Pidima's advanced document management system provides version control, intelligent processing, and AI-powered document understanding through RAG (Retrieval-Augmented Generation) technology. This ensures your AI features always work with the most relevant and up-to-date documentation.

Overview​

When you attach documents to your project, Pidima:

  1. Stores Multiple Versions: Maintains complete version history
  2. Processes for AI: Extracts and indexes content using RAG
  3. Monitors Status: Tracks document processing and availability
  4. Enables Context-Aware AI: Powers chatbot and Requirements Assist with document knowledge

Document Versions​

Version Management​

Each document attachment supports multiple versions:

  • Version Numbers: Automatic versioning (v1.0, v1.1, v2.0, etc.)
  • Version Status: Active, Obsolete, or Processing
  • Upload Tracking: Timestamp and uploader for each version
  • Size Monitoring: File size for each version

Document versioning showing multiple versions with RAG monitoring

Version Indicators​

The document list displays key version information:

  • Badge Count: Shows "2 versions" or number of versions available
  • Active Version: Highlighted version currently in use (green "Active" badge)
  • Obsolete Versions: Previous versions marked as "Obsolete" (gray badge)
  • Latest Version: The most recently uploaded version

Uploading New Versions​

To add a new version of an existing document:

  1. Click "Attach New Document"
  2. Upload a file with the same name as an existing document
  3. System automatically creates a new version
  4. Previous version is marked as obsolete by default
  5. New version enters processing queue

Alternatively, use the Edit action next to a document to:

  • Replace with a new version
  • Update description
  • Modify metadata

RAG Processing​

What is RAG?​

RAG (Retrieval-Augmented Generation) is an advanced AI technique that powers Pidima's intelligent document understanding:

  • Extracts Content: Parses text from PDFs, Word docs, and other formats
  • Chunks Information: Breaks documents into semantic chunks for efficient processing
  • Creates Embeddings: Generates AI-understandable vector representations
  • Enables Retrieval: Allows AI to find relevant sections quickly and accurately
  • Enhances Responses: Grounds AI answers in your actual documents, not generic knowledge
  • Version-Aware: Tracks which document version was used for each AI interaction

How RAG Powers Pidima's AI Features​

RAG processing enables several intelligent features:

1. AI Chatbot Enhancement

  • Answers questions by referencing your specific documents
  • Cites exact page numbers and sections
  • Compares multiple documents simultaneously
  • Provides document-grounded responses, not generic AI knowledge

2. Requirements Assist

  • Uses document content for intelligent autofill suggestions
  • References specifications when rewriting requirements
  • Derives child requirements from parent specifications
  • Validates requirements against documented standards

3. Test Generation

  • Creates test cases based on specification details
  • Includes acceptance criteria from uploaded documents
  • References test procedures in your documentation
  • Ensures coverage of all documented requirements

4. Compliance Checking

  • Compares requirements against regulatory standards
  • Identifies gaps in compliance documentation
  • Suggests additions based on standard requirements
  • Provides traceability to specific regulation clauses

Processing Status​

Each document version shows processing status:

Processed (Green)

  • ✓ Document fully processed and indexed
  • ✓ Available for AI features
  • ✓ Content searchable by chatbot
  • ✓ Ready for Requirements Assist operations

Processing (Yellow/Orange)

  • ⏳ Document being processed
  • ⏳ AI features may have limited access
  • ⏳ Typically completes within 1-5 minutes
  • ⏳ Depends on document size and complexity

Failed (Red)

  • ✗ Processing encountered an error
  • ✗ Document may be corrupted or unsupported
  • ✗ Not available for AI features
  • ✗ Requires re-upload or format conversion

Processing Time​

Processing duration varies by:

  • Document Size: Larger files take longer

    • Small (< 1 MB): 1-2 minutes
    • Medium (1-10 MB): 2-5 minutes
    • Large (> 10 MB): 5-15 minutes

  • Document Type: Some formats require more processing

    • Text/PDF: Fastest
    • Scanned PDFs: Requires OCR, slower
    • Complex layouts: Additional processing time

  • System Load: Peak usage may increase processing time

Applied Levels and Modules​

Configuring Document Scope​

Control which parts of your project use each document:

Applied Levels Specify which requirement levels should reference this document:

  • Customer Requirements
  • System Requirements
  • Subsystem Requirements
  • Component Requirements

Applied Modules Select functional modules that use this document:

  • Software
  • Hardware
  • Mechanical
  • Electrical
  • Safety
  • Security

Benefits of Scoping​

Properly configured scope ensures:

  • Relevant AI Responses: Chatbot references appropriate documents
  • Faster Processing: AI searches smaller, relevant document sets
  • Better Context: Requirements Assist uses domain-specific documents
  • Cleaner Results: Reduces irrelevant information in AI outputs

Configuration Example​

For a document "EV_Battery_Requirements.pdf":

Applied Levels: 
  ✓ System Requirements
  ✓ Subsystem Requirements
  
Applied Modules:
  ✓ Electrical
  ✓ Safety
  ⚬ Software (unchecked)
  ⚬ Mechanical (unchecked)

Result: This document powers AI features only for electrical and safety requirements at system and subsystem levels.

Monitoring RAG Status​

Real-Time Monitoring​

The document table includes a Monitoring column showing comprehensive processing status:

  • Enabled: RAG processing active and updated regularly
  • Status: Current processing state (Processed, Processing, Failed)
  • Last Update: When the document was last processed
  • Refresh Rate: How often the system checks for updates
  • Processing Time: How long the last processing took
  • Chunk Count: Number of document chunks created

Example:

Monitoring: ✓ Enabled
Every 1m
Last: 568h ago
Status: ✓ Processed
Chunks: 247

This indicates:

  • Monitoring is active and working properly
  • System checks every 1 minute for changes
  • Document was processed 568 hours ago (about 24 days)
  • Successfully processed into 247 semantic chunks
  • Ready for AI feature use

Understanding Processing Status Colors​

Green Badge: "Processed" ✓

  • Document fully indexed and available
  • All 247 chunks created successfully
  • AI features can access this document
  • Chatbot will cite this document
  • Requirements Assist can use content
  • Ready for production use

Yellow/Orange Badge: "Processing" ⏳

  • Document currently being analyzed
  • Chunks being created in background
  • Partial availability for AI features
  • Usually completes within 1-5 minutes
  • No action required - wait for completion

Red Badge: "Failed" ✗

  • Processing encountered an error
  • Document not available for AI features
  • May be corrupted or unsupported format
  • Action required: Check format, re-upload, or convert
  • Contact support if issue persists

Processing Performance​

Small Documents (< 1 MB):

  • Processing time: 1-2 minutes
  • Chunk count: 50-100 chunks
  • Memory usage: Low
  • Ideal for: Quick reference docs, standards excerpts

Medium Documents (1-10 MB):

  • Processing time: 2-5 minutes
  • Chunk count: 100-500 chunks
  • Memory usage: Moderate
  • Ideal for: Full specifications, detailed requirements

Large Documents (> 10 MB):

  • Processing time: 5-15 minutes
  • Chunk count: 500-2000+ chunks
  • Memory usage: High
  • Consider: Breaking into smaller logical sections for better performance

Edit Monitoring Settings​

Click the Edit button next to a document version to:

  • Enable/Disable Monitoring: Turn RAG processing on or off
  • Adjust Frequency: Change how often content is re-indexed
  • View Processing Logs: See detailed processing history
  • Trigger Re-processing: Force document re-processing if needed

When to Use Monitoring​

Enable Monitoring For:

  • Documents that change frequently
  • Living specifications updated regularly
  • Documents critical to AI accuracy
  • Recently uploaded documents in processing

Disable Monitoring For:

  • Finalized, unchanging documents
  • Obsolete versions no longer in use
  • Documents that successfully processed once
  • Large files to reduce processing load

Version Status Management​

Active vs. Obsolete​

Active Status (Green Badge)

  • Current version in use
  • Referenced by AI features
  • Included in searches
  • Used for new requirements

Obsolete Status (Gray Badge)

  • Previous version, archived
  • Not used by AI features
  • Maintained for audit trail
  • Can be restored if needed

Managing Status​

To change version status:

  1. Locate the document version in the table
  2. Click the status badge or Edit button
  3. Select new status:
    • Set as Active: Make this version current
    • Mark as Obsolete: Archive this version
  4. Confirm the change

Note: Only one version of a document can be "Active" at a time. Setting a new version as Active automatically marks the previous active version as Obsolete.

AI Integration​

How RAG Enhances AI Features​

AI Chatbot

  • Answers questions using document content
  • Cites specific sections and page numbers
  • Provides context from specifications
  • Compares requirements to standards

Requirements Assist

  • Autofill: Uses documents to determine appropriate values
  • Rewrite: References specifications for enhanced clarity
  • Level Generation: Derives requirements from source documents
  • Validation: Checks requirements against documented standards

Test Generation

  • Creates test cases based on specification details
  • Includes acceptance criteria from documents
  • References test procedures in uploaded docs
  • Ensures coverage of documented requirements

Viewing Document Context​

See which documents inform AI responses:

  1. Use the AI Chatbot to ask about a requirement
  2. The AI may cite: "According to EV_Battery_Requirements.pdf (v1.1)..."
  3. Click cited documents to view in context
  4. See exactly which sections influenced the AI response

Supported Document Formats​

Fully Supported​

  • PDF (.pdf): Including scanned documents with OCR
  • Microsoft Word (.doc, .docx)
  • Plain Text (.txt)
  • Markdown (.md)
  • Rich Text (.rtf)

Partially Supported​

  • Microsoft Excel (.xls, .xlsx): Extracts text from cells
  • PowerPoint (.ppt, .pptx): Extracts slide text
  • HTML (.html, .htm): Parses readable content

Not Supported​

  • Image files (.jpg, .png) without OCR
  • CAD files (.dwg, .dxf)
  • Proprietary formats without text extraction
  • Encrypted or password-protected documents

Real-World RAG Use Cases​

Use Case 1: Multi-Version Specification Management​

Scenario: Automotive manufacturer with evolving battery specifications

Setup:

EV_Battery_Car_Requirements.pdf
├── Version 1.0 (Obsolete) - 12V specifications
│   Status: Processed ✓
│   Chunks: 156
│   Applied Levels: System, Subsystem
│   
└── Version 1.1 (Active) - 24V specifications  
    Status: Processed ✓
    Chunks: 189
    Applied Levels: System, Subsystem, Component
    Last processed: 2 hours ago

Workflow:

  1. Version 1.0 uploaded → RAG processes → 156 chunks created
  2. AI Chatbot uses v1.0 for all queries about battery requirements
  3. Specifications updated → Version 1.1 uploaded
  4. RAG processes v1.1 → 189 chunks (more detailed)
  5. v1.1 set as "Active" → AI now references new version
  6. v1.0 marked "Obsolete" → Kept for historical reference

Result:

  • AI always uses latest approved specifications
  • Historical versions maintained for audit trail
  • Requirement changes tracked with document version references
  • Seamless transition with zero AI confusion

Use Case 2: Regulatory Compliance with ISO Standards​

Scenario: Medical device company ensuring ISO 13485 compliance

Setup:

ISO_13485_Quality_Management.pdf
├── Status: Processed ✓
├── Chunks: 847 (large standard document)
├── Processing time: 12 minutes
├── Applied Levels: All levels
├── Applied Modules: Quality, Safety, Documentation
└── Monitoring: Enabled (checks annually for standard updates)

AI Integration:

User in Chatbot: "Does requirement MED-REQ-00015 meet ISO 13485 section 7.3?"

AI Response:
"Analyzing requirement MED-REQ-00015 against ISO 13485:2016 section 7.3 
(Design and Development):

✓ Section 7.3.2 (Design Inputs): COMPLIANT
  Your requirement specifies user needs and regulatory requirements
  
⚠ Section 7.3.3 (Design Outputs): PARTIAL
  Missing: Design output documentation format requirements
  Recommendation: Add clause about design output documentation per 7.3.3.b
  
✗ Section 7.3.4 (Design Review): NON-COMPLIANT
  Not found: Design review checkpoints and criteria
  Required: Add design review stages per 7.3.4.a-e
  
Source: ISO_13485_Quality_Management.pdf, pages 23-26"

Benefits:

  • Real-time compliance checking during requirement authoring
  • Specific citations to standard clauses
  • Gap identification before audits
  • Reduced compliance risk

Use Case 3: Cross-Document Analysis​

Scenario: Aerospace project with multiple related specifications

Setup:

Project Documents (All Processed):
1. DO-178C_Software_Standard.pdf (Active)
   Chunks: 1,247 | Applied: Software modules
   
2. ARP4754A_System_Development.pdf (Active)
   Chunks: 892 | Applied: System level
   
3. Internal_Coding_Standards_v3.pdf (Active)
   Chunks: 234 | Applied: Software component level
   
4. Customer_Requirements_Spec.pdf (Active)
   Chunks: 456 | Applied: Customer requirements level

AI Chatbot Query:

User: "Does our software requirement SW-REQ-0042 satisfy both DO-178C 
      and our customer requirements?"

AI Response:
"Cross-referencing SW-REQ-0042 across multiple documents:

DO-178C Compliance (DO-178C_Software_Standard.pdf):
✓ Section 5.1.1: High-level requirements - COMPLIANT
✓ Section 5.1.3: Derived requirements - COMPLIANT
⚠ Section 6.3.1: Missing test coverage criteria

Customer Requirements (Customer_Requirements_Spec.pdf):
✓ Requirement CUS-001 (page 15): Fully addresses
✓ Requirement CUS-003 (page 18): Partially addresses
✗ Requirement CUS-007 (page 22): Not addressed

Internal Standards (Internal_Coding_Standards_v3.pdf):
✓ Naming conventions: Compliant
✓ Documentation format: Compliant

Recommendation: Add test coverage criteria per DO-178C 6.3.1 and 
link to CUS-007 for complete coverage."

Benefits:

  • Single query analyzes multiple standards simultaneously
  • Identifies gaps across all relevant documents
  • Provides specific page references
  • Saves hours of manual cross-referencing

Use Case 4: Document Version Comparison​

Scenario: Tracking changes between specification versions

Setup:

Battery_Specification.pdf
├── Version 2.0 (Obsolete)
│   Chunks: 178
│   Key content: 12V battery requirements
│   
└── Version 3.0 (Active)
    Chunks: 203
    Key content: 24V battery requirements (updated)

AI Analysis:

User: "What changed between version 2.0 and 3.0 of the battery spec?"

AI Response:
"Comparing Battery_Specification.pdf versions:

Version 2.0 (Obsolete):
- Nominal voltage: 12V (section 3.1, page 8)
- Operating range: 11.5V - 12.5V
- Safety standard: Basic (< 20V category)

Version 3.0 (Active):
- Nominal voltage: 24V (section 3.1, page 8)
- Operating range: 23V - 25V  
- Safety standard: Enhanced (> 20V category, IEC 60950-1)
- NEW: Arc flash protection requirements (section 5.3, page 15)
- NEW: Insulation rating requirements (section 5.4, page 16)

Impact on Requirements:
- 15 existing requirements reference 12V → Need updates
- 3 new safety requirements needed for 24V compliance
- Test coverage gaps: 8 additional tests recommended"

Benefits:

  • Automated change detection between versions
  • Impact analysis on existing requirements
  • Identifies new requirements needed
  • Streamlines version transition process

Use Case 5: Intelligent Test Generation from Specs​

Scenario: Generating test cases directly from specification documents

Setup:

EV_Battery_Test_Procedures.pdf (Active)
├── Status: Processed ✓
├── Chunks: 312
├── Applied Levels: System, Subsystem
└── Content: Detailed test procedures for battery validation

Workflow:

  1. User selects requirement: "Battery shall meet voltage specifications"
  2. Clicks: "Generate Tests with AI"
  3. AI analyzes:
    • Requirement text
    • Linked specification: EV_Battery_Test_Procedures.pdf
    • Industry standards (if uploaded)

AI-Generated Tests:

Test Case TC-BAT-001: Nominal Voltage Measurement
Source: EV_Battery_Test_Procedures.pdf, Section 4.2.1, page 18

Description: "Measure battery voltage at rest (no load) after 
2-hour stabilization period per section 4.2.1"

Steps (from specification):
1. Disconnect battery from all loads
2. Allow 2-hour stabilization at 20°C ± 2°C
3. Connect calibrated multimeter (±0.1V accuracy)
4. Record voltage reading
5. Verify: 24.0V ± 0.6V (23.4V - 24.6V)

Expected Result: Voltage within specified range
Link: SYSTEM-REQ-00003 (Battery Verification)

Benefits:

  • Tests generated from authoritative source documents
  • Includes specific procedures from specifications
  • Automatically linked to requirements
  • Maintains compliance with documented test methods

Best Practices​

Document Naming​

Use clear, consistent naming:

  • Good: EV_Battery_Requirements_v2.1.pdf
  • Good: ISO_26262_Compliance_Standards.pdf
  • Avoid: doc1.pdf, final_FINAL_v2_FINAL.pdf

Version Control​

Maintain clean version history:

  • Upload new versions only when content changes significantly
  • Mark obsolete versions promptly
  • Keep only necessary versions (archive or delete old versions)
  • Document changes in the description field

Scope Configuration​

Set accurate Applied Levels and Modules:

  • Be specific about where documents apply
  • Don't over-apply documents to all levels/modules
  • Update scopes when project structure changes
  • Review scopes quarterly

Monitoring Strategy​

Balance accuracy with performance:

  • Enable monitoring for active documents
  • Disable monitoring for stable, finalized documents
  • Use longer intervals (5m+) for large documents
  • Re-process after document updates

Document Quality​

Ensure AI can effectively process documents:

  • Use text-based PDFs, not just scanned images
  • Ensure good scan quality if using OCR
  • Structure documents with clear headings
  • Break very large documents into logical sections
  • Avoid excessive formatting that obscures text

RAG Performance Optimization​

Maximizing AI Accuracy​

Tip 1: Document Quality Matters

  • Use text-based PDFs (not scanned images when possible)
  • Ensure good scan quality if using OCR (300+ DPI)
  • Avoid heavily formatted documents with complex layouts
  • Use clear section headings and structure

Tip 2: Optimal Document Size

  • Split very large documents (>50 MB) into logical sections
  • Example: Instead of "Complete_ISO_Standards.pdf" (100 MB)
    • Use: "ISO_13485_Section_7_Design.pdf" (15 MB)
    • Use: "ISO_13485_Section_8_Production.pdf" (12 MB)
  • Benefits: Faster processing, more accurate retrieval, easier maintenance

Tip 3: Applied Levels Configuration

  • Be specific about where documents apply
  • Don't apply all documents to all levels
  • Example for battery specification: 
    EV_Battery_Requirements.pdf
    ✓ Applied to: System Requirements, Subsystem Requirements
    ✗ Not applied to: Software Component Requirements

    Result: AI won't confuse software developers with battery specs

Tip 4: Active vs Obsolete Management

  • Keep only ONE version as "Active" per document name
  • Mark superseded versions as "Obsolete" immediately
  • Prevents AI from mixing information from different versions
  • Example: 
    Battery_Spec_v1.pdf → Obsolete (12V era)
    Battery_Spec_v2.pdf → Active (24V current)

Tip 5: Monitoring Strategy

  • Enable monitoring for living documents that change frequently
  • Disable monitoring for finalized, stable specifications
  • Adjust frequency based on update cadence:
    • Active development: Every 1 minute
    • Weekly updates: Every 1 hour
    • Stable docs: Every 24 hours or disable

Troubleshooting RAG Issues​

Issue: AI Not Citing My Document

Checklist:

  1. ✓ Document status is "Processed" (green badge)
  2. ✓ Document version is "Active" (not Obsolete)
  3. ✓ Applied Levels include the requirement level you're working on
  4. ✓ Applied Modules include the functional area
  5. ✓ Monitoring is "Enabled"
  6. ✓ Ask chatbot specifically: "What does [DocumentName] say about X?"

Issue: AI Gives Inconsistent Answers

Possible causes:

  • Multiple document versions marked as "Active"
  • Conflicting information in different documents
  • Document not fully processed (yellow status)

Solution:

  1. Check document versions - mark only one as Active
  2. Wait for processing to complete (green status)
  3. Review Applied Levels to ensure appropriate scope
  4. Regenerate RAG index if needed (Edit → Re-process)

Issue: Processing Stuck on "Processing" Status

If document shows processing for > 30 minutes:

  1. Check document size (may timeout if > 100 MB)
  2. Verify file format is supported
  3. Try re-uploading the document
  4. Split large documents into smaller files
  5. Contact support with document details

Troubleshooting​

Document Not Processing​

If a document shows "Processing" for extended periods:

  1. Check document size (> 50 MB may time out)
  2. Verify file format is supported
  3. Try re-uploading the document
  4. Break large documents into smaller parts
  5. Contact support if issue persists

AI Not Referencing Document​

If AI features don't use your document:

  1. Verify status is "Active" and "Processed"
  2. Check Applied Levels and Modules match your needs
  3. Ensure monitoring is enabled
  4. Wait for processing to complete (check status)
  5. Try asking the chatbot specifically about the document

Slow Processing​

To improve processing speed:

  1. Reduce monitoring frequency (e.g., every 5m instead of 1m)
  2. Disable monitoring for documents that don't change
  3. Upload documents during off-peak hours
  4. Optimize document size before uploading
  5. Use text-based PDFs instead of scanned images

Version Confusion​

If unsure which version is current:

  1. Look for the green "Active" badge
  2. Check the "Uploaded" date for latest version
  3. Review the version badge (e.g., "v1.1")
  4. Read document descriptions for clarification

Advanced Features​

Bulk Version Management​

Manage multiple document versions efficiently:

  1. Select multiple documents using checkboxes
  2. Use bulk actions to:
    • Mark as Active/Obsolete
    • Enable/Disable monitoring
    • Delete unused versions
    • Update Applied Levels/Modules

Document Analytics​

Track document usage and effectiveness:

  • View Processing History: See all processing events
  • AI Reference Count: How often AI cites this document
  • Coverage Metrics: Which requirements link to this document
  • Version Comparison: Compare content between versions

API Integration​

Automate document management:

  • Upload documents via API
  • Trigger RAG processing programmatically
  • Query document status
  • Retrieve version history
  • Update metadata and settings

See Integration & API for details.