hypogenic

•

24.4k23993 months ago

Automated hypothesis generation and testing using large language models. Use this skill when generating scientific hypotheses from datasets, combining literature insights with empirical data, testing hypotheses against observational data, or conducting systematic hypothesis exploration for research discovery in domains like deception detection, AI content detection, mental health analysis, or other empirical research tasks.

Quick Install

npxskills add davila7/claude-code-templates--skill hypogenic

Instructions

Hypogenic

Overview

Hypogenic provides automated hypothesis generation and testing using large language models to accelerate scientific discovery. The framework supports three approaches: HypoGeniC (data-driven hypothesis generation), HypoRefine (synergistic literature and data integration), and Union methods (mechanistic combination of literature and data-driven hypotheses).

Quick Start

Get started with Hypogenic in minutes:

# Install the package
uv pip install hypogenic

# Clone example datasets
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# Run basic hypothesis generation
hypogenic_generation --config ./data/your_task/config.yaml --method hypogenic --num_hypotheses 20

# Run inference on generated hypotheses
hypogenic_inference --config ./data/your_task/config.yaml --hypotheses output/hypotheses.json

Or use Python API:

from hypogenic import BaseTask

# Create task with your configuration
task = BaseTask(config_path="./data/your_task/config.yaml")

# Generate hypotheses
task.generate_hypotheses(method="hypogenic", num_hypotheses=20)

# Run inference
results = task.inference(hypothesis_bank="./output/hypotheses.json")

When to Use This Skill

Use this skill when working on:

Generating scientific hypotheses from observational datasets
Testing multiple competing hypotheses systematically
Combining literature insights with empirical patterns
Accelerating research discovery through automated hypothesis ideation
Domains requiring hypothesis-driven analysis: deception detection, AI-generated content identification, mental health indicators, predictive modeling, or other empirical research

Key Features

Automated Hypothesis Generation

Generate 10-20+ testable hypotheses from data in minutes
Iterative refinement based on validation performance
Support for both API-based (OpenAI, Anthropic) and local LLMs

Literature Integration

Extract insights from research papers via PDF processing
Combine theoretical foundations with empirical patterns
Systematic literature-to-hypothesis pipeline with GROBID

Performance Optimization

Redis caching reduces API costs for repeated experiments
Parallel processing for large-scale hypothesis testing
Adaptive refinement focuses on challenging examples

Flexible Configuration

Template-based prompt engineering with variable injection
Custom label extraction for domain-specific tasks
Modular architecture for easy extension

Proven Results

8.97% improvement over few-shot baselines
15.75% improvement over literature-only approaches
80-84% hypothesis diversity (non-redundant insights)
Human evaluators report significant decision-making improvements

Core Capabilities

1. HypoGeniC: Data-Driven Hypothesis Generation

Generate hypotheses solely from observational data through iterative refinement.

Process:

Initialize with a small data subset to generate candidate hypotheses
Iteratively refine hypotheses based on performance
Replace poorly-performing hypotheses with new ones from challenging examples

Best for: Exploratory research without existing literature, pattern discovery in novel datasets

2. HypoRefine: Literature and Data Integration

Synergistically combine existing literature with empirical data through an agentic framework.

Process:

Extract insights from relevant research papers (typically 10 papers)
Generate theory-grounded hypotheses from literature
Generate data-driven hypotheses from observational patterns
Refine both hypothesis banks through iterative improvement

Best for: Research with established theoretical foundations, validating or extending existing theories

3. Union Methods

Mechanistically combine literature-only hypotheses with framework outputs.

Variants:

Literature ∪ HypoGeniC: Combines literature hypotheses with data-driven generation
Literature ∪ HypoRefine: Combines literature hypotheses with integrated approach

Best for: Comprehensive hypothesis coverage, eliminating redundancy while maintaining diverse perspectives

Installation

Install via pip:

uv pip install hypogenic

Optional dependencies:

Redis server (port 6832): Enables caching of LLM responses to significantly reduce API costs during iterative hypothesis generation
s2orc-doc2json: Required for processing literature PDFs in HypoRefine workflows
GROBID: Required for PDF preprocessing (see Literature Processing section)

Clone example datasets:

# For HypoGeniC examples
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# For HypoRefine/Union examples
git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

Dataset Format

Datasets must follow HuggingFace datasets format with specific naming conventions:

Required files:

<TASK>_train.json: Training data
<TASK>_val.json: Validation data
<TASK>_test.json: Test data

Required keys in JSON:

text_features_1 through text_features_n: Lists of strings containing feature values
label: List of strings containing ground truth labels

Example (headline click prediction):

{
  "headline_1": [
    "What Up, Comet? You Just Got *PROBED*",
    "Scientists Made a Breakthrough in Quantum Computing"
  ],
  "headline_2": [
    "Scientists Everywhere Were Holding Their Breath Today. Here's Why.",
    "New Quantum Computer Achieves Milestone"
  ],
  "label": [
    "Headline 2 has more clicks than Headline 1",
    "Headline 1 has more clicks than Headline 2"
  ]
}

Important notes:

All lists must have the same length
Label format must match your extract_label() function output format
Feature keys can be customized to match your domain (e.g., review_text, post_content, etc.)

Configuration

Each task requires a config.yaml file specifying:

Required elements:

Dataset paths (train/val/test)
Prompt templates for:
- Observations generation
- Batched hypothesis generation
- Hypothesis inference
- Relevance checking
- Adaptive methods (for HypoRefine)

Template capabilities:

Dataset placeholders for dynamic variable injection (e.g., ${text_features_1}, ${num_hypotheses})
Custom label extraction functions for domain-specific parsing
Role-based prompt structure (system, user, assistant roles)

Configuration structure:

task_name: your_task_name

train_data_path: ./your_task_train.json
val_data_path: ./your_task_val.json
test_data_path: ./your_task_test.json

prompt_templates:
  # Extra keys for reusable prompt components
  observations: |
    Feature 1: ${text_features_1}
    Feature 2: ${text_features_2}
    Observation: ${label}
  
  # Required templates
  batched_generation:
    system: "Your system prompt here"
    user: "Your user prompt with ${num_hypotheses} placeholder"
  
  inference:
    system: "Your inference system prompt"
    user: "Your inference user prompt"
  
  # Optional templates for advanced features
  few_shot_baseline: {...}
  is_relevant: {...}
  adaptive_inference: {...}
  adaptive_selection: {...}

Refer to references/config_template.yaml for a complete example configuration.

Literature Processing (HypoRefine/Union Methods)

To use literature-based hypothesis generation, you must preprocess PDF papers:

Step 1: Setup GROBID (first time only)

bash ./modules/setup_grobid.sh

Step 2: Add PDF files Place research papers in literature/YOUR_TASK_NAME/raw/

Step 3: Process PDFs

# Start GROBID service
bash ./modules/run_grobid.sh

# Process PDFs for your task
cd examples
python pdf_preprocess.py --task_name YOUR_TASK_NAME

This converts PDFs to structured format for hypothesis extraction. Automated literature search will be supported in future releases.

CLI Usage

Hypothesis Generation

hypogenic_generation --help

Key parameters:

Task configuration file path
Model selection (API-based or local)
Generation method (HypoGeniC, HypoRefine, or Union)
Number of hypotheses to generate
Output directory for hypothesis banks

Hypothesis Inference

hypogenic_inference --help

Key parameters:

Task configuration file path
Hypothesis bank file path
Test dataset path
Inference method (default or multi-hypothesis)
Output file for results

Python API Usage

For programmatic control and custom workflows, use Hypogenic directly in your Python code:

Basic HypoGeniC Generation

from hypogenic import BaseTask

# Clone example datasets first
# git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# Load your task with custom extract_label function
task = BaseTask(
    config_path="./data/your_task/config.yaml",
    extract_label=lambda text: extract_your_label(text)
)

# Generate hypotheses
task.generate_hypotheses(
    method="hypogenic",
    num_hypotheses=20,
    output_path="./output/hypotheses.json"
)

# Run inference
results = task.inference(
    hypothesis_bank="./output/hypotheses.json",
    test_data="./data/your_task/your_task_test.json"
)

HypoRefine/Union Methods

# For literature-integrated approaches
# git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

# Generate with HypoRefine
task.generate_hypotheses(
    method="hyporefine",
    num_hypotheses=15,
    literature_path="./literature/your_task/",
    output_path="./output/"
)
# This generates 3 hypothesis banks:
# - HypoRefine (integrated approach)
# - Literature-only hypotheses
# - Literature∪HypoRefine (union)

Multi-Hypothesis Inference

from examples.multi_hyp_inference import run_multi_hypothesis_inference

# Test multiple hypotheses simultaneously
results = run_multi_hypothesis_inference(
    config_path="./data/your_task/config.yaml",
    hypothesis_bank="./output/hypotheses.json",
    test_data="./data/your_task/your_task_test.json"
)

Custom Label Extraction

The extract_label() function is critical for parsing LLM outputs. Implement it based on your task:

def extract_label(llm_output: str) -> str:
    """Extract predicted label from LLM inference text.
    
    Default behavior: searches for 'final answer:\s+(.*)' pattern.
    Customize for your domain-specific output format.
    """
    import re
    match = re.search(r'final answer:\s+(.*)', llm_output, re.IGNORECASE)
    if match:
        return match.group(1).strip()
    return llm_output.strip()

Important: Extracted labels must match the format of label values in your dataset for correct accuracy calculation.

Workflow Examples

Example 1: Data-Driven Hypothesis Generation (HypoGeniC)

Scenario: Detecting AI-generated content without prior theoretical framework

Steps:

Prepare dataset with text samples and labels (human vs. AI-generated)
Create config.yaml with appropriate prompt templates

Run hypothesis generation:

hypogenic_generation --config config.yaml --method hypogenic --num_hypotheses 20

Run inference on test set:

hypogenic_inference --config config.yaml --hypotheses output/hypotheses.json --test_data data/test.json

Analyze results for patterns like formality, grammatical precision, and tone differences

Example 2: Literature-Informed Hypothesis Testing (HypoRefine)

Scenario: Deception detection in hotel reviews building on existing research

Steps:

Collect 10 relevant papers on linguistic deception cues
Prepare dataset with genuine and fraudulent reviews
Configure config.yaml with literature processing and data generation templates

Run HypoRefine:

hypogenic_generation --config config.yaml --method hyporefine --papers papers/ --num_hypotheses 15

Test hypotheses examining pronoun frequency, detail specificity, and other linguistic patterns
Compare literature-based and data-driven hypothesis performance

Example 3: Comprehensive Hypothesis Coverage (Union Method)

Scenario: Mental stress detection maximizing hypothesis diversity

Steps:

Generate literature hypotheses from mental health research papers
Generate data-driven hypotheses from social media posts

Run Union method to combine and deduplicate:

hypogenic_generation --config config.yaml --method union --literature_hypotheses lit_hyp.json

Inference captures both theoretical constructs (posting behavior changes) and data patterns (emotional language shifts)

Performance Optimization

Caching: Enable Redis caching to reduce API costs and computation time for repeated LLM calls

Parallel Processing: Leverage multiple workers for large-scale hypothesis generation and testing

Adaptive Refinement: Use challenging examples to iteratively improve hypothesis quality

Expected Outcomes

Research using hypogenic has demonstrated:

14.19% accuracy improvement in AI-content detection tasks
7.44% accuracy improvement in deception detection tasks
80-84% of hypothesis pairs offering distinct, non-redundant insights
High helpfulness ratings from human evaluators across multiple research domains

Troubleshooting

Issue: Generated hypotheses are too generic Solution: Refine prompt templates in config.yaml to request more specific, testable hypotheses

Issue: Poor inference performance Solution: Ensure dataset has sufficient training examples, adjust hypothesis generation parameters, or increase number of hypotheses

Issue: Label extraction failures Solution: Implement custom extract_label() function for domain-specific output parsing

Issue: GROBID PDF processing fails Solution: Ensure GROBID service is running (bash ./modules/run_grobid.sh) and PDFs are valid research papers

Creating Custom Tasks

To add a new task or dataset to Hypogenic:

Step 1: Prepare Your Dataset

Create three JSON files following the required format:

your_task_train.json
your_task_val.json
your_task_test.json

Each file must have keys for text features (text_features_1, etc.) and label.

Step 2: Create config.yaml

Define your task configuration with:

Task name and dataset paths
Prompt templates for observations, generation, inference
Any extra keys for reusable prompt components
Placeholder variables (e.g., ${text_features_1}, ${num_hypotheses})

Step 3: Implement extract_label Function

Create a custom label extraction function that parses LLM outputs for your domain:

from hypogenic import BaseTask

def extract_my_label(llm_output: str) -> str:
    """Custom label extraction for your task.
    
    Must return labels in same format as dataset 'label' field.
    """
    # Example: Extract from specific format
    if "Final prediction:" in llm_output:
        return llm_output.split("Final prediction:")[-1].strip()
    
    # Fallback to default pattern
    import re
    match = re.search(r'final answer:\s+(.*)', llm_output, re.IGNORECASE)
    return match.group(1).strip() if match else llm_output.strip()

# Use your custom task
task = BaseTask(
    config_path="./your_task/config.yaml",
    extract_label=extract_my_label
)

Step 4: (Optional) Process Literature

For HypoRefine/Union methods:

Create literature/your_task_name/raw/ directory
Add relevant research paper PDFs
Run GROBID preprocessing
Process with pdf_preprocess.py

Step 5: Generate and Test

Run hypothesis generation and inference using CLI or Python API:

# CLI approach
hypogenic_generation --config your_task/config.yaml --method hypogenic --num_hypotheses 20
hypogenic_inference --config your_task/config.yaml --hypotheses output/hypotheses.json

# Or use Python API (see Python API Usage section)

Repository Structure

Understanding the repository layout:

hypothesis-generation/
├── hypogenic/              # Core package code
├── hypogenic_cmd/          # CLI entry points
├── hypothesis_agent/       # HypoRefine agent framework
├── literature/            # Literature processing utilities
├── modules/               # GROBID and preprocessing modules
├── examples/              # Example scripts
│   ├── generation.py      # Basic HypoGeniC generation
│   ├── union_generation.py # HypoRefine/Union generation
│   ├── inference.py       # Single hypothesis inference
│   ├── multi_hyp_inference.py # Multiple hypothesis inference
│   └── pdf_preprocess.py  # Literature PDF processing
├── data/                  # Example datasets (clone separately)
├── tests/                 # Unit tests
└── IO_prompting/          # Prompt templates and experiments

Key directories:

hypogenic/: Main package with BaseTask and generation logic
examples/: Reference implementations for common workflows
literature/: Tools for PDF processing and literature extraction
modules/: External tool integrations (GROBID, etc.)

Related Publications

HypoBench (2025)

Liu, H., Huang, S., Hu, J., Zhou, Y., & Tan, C. (2025). HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation. arXiv preprint arXiv:2504.11524.

Paper: https://arxiv.org/abs/2504.11524
Description: Benchmarking framework for systematic evaluation of hypothesis generation methods

BibTeX:

@misc{liu2025hypobenchsystematicprincipledbenchmarking,
      title={HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation}, 
      author={Haokun Liu and Sicong Huang and Jingyu Hu and Yangqiaoyu Zhou and Chenhao Tan},
      year={2025},
      eprint={2504.11524},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.11524}, 
}

Literature Meets Data (2024)

Liu, H., Zhou, Y., Li, M., Yuan, C., & Tan, C. (2024). Literature Meets Data: A Synergistic Approach to Hypothesis Generation. arXiv preprint arXiv:2410.17309.

Paper: https://arxiv.org/abs/2410.17309
Code: https://github.com/ChicagoHAI/hypothesis-generation
Description: Introduces HypoRefine and demonstrates synergistic combination of literature-based and data-driven hypothesis generation

BibTeX:

@misc{liu2024literaturemeetsdatasynergistic,
      title={Literature Meets Data: A Synergistic Approach to Hypothesis Generation}, 
      author={Haokun Liu and Yangqiaoyu Zhou and Mingxuan Li and Chenfei Yuan and Chenhao Tan},
      year={2024},
      eprint={2410.17309},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.17309}, 
}

Hypothesis Generation with Large Language Models (2024)

Zhou, Y., Liu, H., Srivastava, T., Mei, H., & Tan, C. (2024). Hypothesis Generation with Large Language Models. In Proceedings of EMNLP Workshop of NLP for Science.

Paper: https://aclanthology.org/2024.nlp4science-1.10/
Description: Original HypoGeniC framework for data-driven hypothesis generation

BibTeX:

@inproceedings{zhou2024hypothesisgenerationlargelanguage,
      title={Hypothesis Generation with Large Language Models}, 
      author={Yangqiaoyu Zhou and Haokun Liu and Tejes Srivastava and Hongyuan Mei and Chenhao Tan},
      booktitle = {Proceedings of EMNLP Workshop of NLP for Science},
      year={2024},
      url={https://aclanthology.org/2024.nlp4science-1.10/},
}

Additional Resources

Official Links

GitHub Repository: https://github.com/ChicagoHAI/hypothesis-generation
PyPI Package: https://pypi.org/project/hypogenic/
License: MIT License
Issues & Support: https://github.com/ChicagoHAI/hypothesis-generation/issues

Example Datasets

Clone these repositories for ready-to-use examples:

# HypoGeniC examples (data-driven only)
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# HypoRefine/Union examples (literature + data)
git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

Community & Contributions

Contributors: 7+ active contributors
Stars: 89+ on GitHub
Topics: research-tool, interpretability, hypothesis-generation, scientific-discovery, llm-application

For contributions or questions, visit the GitHub repository and check the issues page.

Local Resources

references/

config_template.yaml - Complete example configuration file with all required prompt templates and parameters. This includes:

Full YAML structure for task configuration
Example prompt templates for all methods
Placeholder variable documentation
Role-based prompt examples

scripts/

Scripts directory is available for:

Custom data preparation utilities
Format conversion tools
Analysis and evaluation scripts
Integration with external tools

assets/

Assets directory is available for:

Example datasets and templates
Sample hypothesis banks
Visualization outputs
Documentation supplements

Loading…

Related Skills

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,360

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,309

1password

Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.

365,915

himalaya

CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).

365,885

Tags & Topics

anthropicanthropic-claudeclaudeclaude-code

Hypogenic

Overview

Quick Start

Get started with Hypogenic in minutes:

# Install the package
uv pip install hypogenic

# Clone example datasets
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# Run basic hypothesis generation
hypogenic_generation --config ./data/your_task/config.yaml --method hypogenic --num_hypotheses 20

# Run inference on generated hypotheses
hypogenic_inference --config ./data/your_task/config.yaml --hypotheses output/hypotheses.json

Or use Python API:

from hypogenic import BaseTask

# Create task with your configuration
task = BaseTask(config_path="./data/your_task/config.yaml")

# Generate hypotheses
task.generate_hypotheses(method="hypogenic", num_hypotheses=20)

# Run inference
results = task.inference(hypothesis_bank="./output/hypotheses.json")

When to Use This Skill

Use this skill when working on:

Generating scientific hypotheses from observational datasets
Testing multiple competing hypotheses systematically
Combining literature insights with empirical patterns
Accelerating research discovery through automated hypothesis ideation
Domains requiring hypothesis-driven analysis: deception detection, AI-generated content identification, mental health indicators, predictive modeling, or other empirical research

Key Features

Automated Hypothesis Generation

Generate 10-20+ testable hypotheses from data in minutes
Iterative refinement based on validation performance
Support for both API-based (OpenAI, Anthropic) and local LLMs

Literature Integration

Extract insights from research papers via PDF processing
Combine theoretical foundations with empirical patterns
Systematic literature-to-hypothesis pipeline with GROBID

Performance Optimization

Redis caching reduces API costs for repeated experiments
Parallel processing for large-scale hypothesis testing
Adaptive refinement focuses on challenging examples

Flexible Configuration

Template-based prompt engineering with variable injection
Custom label extraction for domain-specific tasks
Modular architecture for easy extension

Proven Results

8.97% improvement over few-shot baselines
15.75% improvement over literature-only approaches
80-84% hypothesis diversity (non-redundant insights)
Human evaluators report significant decision-making improvements

Core Capabilities

1. HypoGeniC: Data-Driven Hypothesis Generation

Generate hypotheses solely from observational data through iterative refinement.

Process:

Initialize with a small data subset to generate candidate hypotheses
Iteratively refine hypotheses based on performance
Replace poorly-performing hypotheses with new ones from challenging examples

Best for: Exploratory research without existing literature, pattern discovery in novel datasets

2. HypoRefine: Literature and Data Integration

Synergistically combine existing literature with empirical data through an agentic framework.

Process:

Extract insights from relevant research papers (typically 10 papers)
Generate theory-grounded hypotheses from literature
Generate data-driven hypotheses from observational patterns
Refine both hypothesis banks through iterative improvement

Best for: Research with established theoretical foundations, validating or extending existing theories

3. Union Methods

Mechanistically combine literature-only hypotheses with framework outputs.

Variants:

Literature ∪ HypoGeniC: Combines literature hypotheses with data-driven generation
Literature ∪ HypoRefine: Combines literature hypotheses with integrated approach

Best for: Comprehensive hypothesis coverage, eliminating redundancy while maintaining diverse perspectives

Installation

Install via pip:

uv pip install hypogenic

Optional dependencies:

Redis server (port 6832): Enables caching of LLM responses to significantly reduce API costs during iterative hypothesis generation
s2orc-doc2json: Required for processing literature PDFs in HypoRefine workflows
GROBID: Required for PDF preprocessing (see Literature Processing section)

Clone example datasets:

# For HypoGeniC examples
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# For HypoRefine/Union examples
git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

Dataset Format

Datasets must follow HuggingFace datasets format with specific naming conventions:

Required files:

<TASK>_train.json: Training data
<TASK>_val.json: Validation data
<TASK>_test.json: Test data

Required keys in JSON:

text_features_1 through text_features_n: Lists of strings containing feature values
label: List of strings containing ground truth labels

Example (headline click prediction):

{
  "headline_1": [
    "What Up, Comet? You Just Got *PROBED*",
    "Scientists Made a Breakthrough in Quantum Computing"
  ],
  "headline_2": [
    "Scientists Everywhere Were Holding Their Breath Today. Here's Why.",
    "New Quantum Computer Achieves Milestone"
  ],
  "label": [
    "Headline 2 has more clicks than Headline 1",
    "Headline 1 has more clicks than Headline 2"
  ]
}

Important notes:

All lists must have the same length
Label format must match your extract_label() function output format
Feature keys can be customized to match your domain (e.g., review_text, post_content, etc.)

Configuration

Each task requires a config.yaml file specifying:

Required elements:

Dataset paths (train/val/test)
Prompt templates for:
- Observations generation
- Batched hypothesis generation
- Hypothesis inference
- Relevance checking
- Adaptive methods (for HypoRefine)

Template capabilities:

Dataset placeholders for dynamic variable injection (e.g., ${text_features_1}, ${num_hypotheses})
Custom label extraction functions for domain-specific parsing
Role-based prompt structure (system, user, assistant roles)

Configuration structure:

task_name: your_task_name

train_data_path: ./your_task_train.json
val_data_path: ./your_task_val.json
test_data_path: ./your_task_test.json

prompt_templates:
  # Extra keys for reusable prompt components
  observations: |
    Feature 1: ${text_features_1}
    Feature 2: ${text_features_2}
    Observation: ${label}
  
  # Required templates
  batched_generation:
    system: "Your system prompt here"
    user: "Your user prompt with ${num_hypotheses} placeholder"
  
  inference:
    system: "Your inference system prompt"
    user: "Your inference user prompt"
  
  # Optional templates for advanced features
  few_shot_baseline: {...}
  is_relevant: {...}
  adaptive_inference: {...}
  adaptive_selection: {...}

Refer to references/config_template.yaml for a complete example configuration.

Literature Processing (HypoRefine/Union Methods)

To use literature-based hypothesis generation, you must preprocess PDF papers:

Step 1: Setup GROBID (first time only)

bash ./modules/setup_grobid.sh

Step 2: Add PDF files Place research papers in literature/YOUR_TASK_NAME/raw/

Step 3: Process PDFs

# Start GROBID service
bash ./modules/run_grobid.sh

# Process PDFs for your task
cd examples
python pdf_preprocess.py --task_name YOUR_TASK_NAME

This converts PDFs to structured format for hypothesis extraction. Automated literature search will be supported in future releases.

CLI Usage

Hypothesis Generation

hypogenic_generation --help

Key parameters:

Task configuration file path
Model selection (API-based or local)
Generation method (HypoGeniC, HypoRefine, or Union)
Number of hypotheses to generate
Output directory for hypothesis banks

Hypothesis Inference

hypogenic_inference --help

Key parameters:

Task configuration file path
Hypothesis bank file path
Test dataset path
Inference method (default or multi-hypothesis)
Output file for results

Python API Usage

For programmatic control and custom workflows, use Hypogenic directly in your Python code:

Basic HypoGeniC Generation

from hypogenic import BaseTask

# Clone example datasets first
# git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# Load your task with custom extract_label function
task = BaseTask(
    config_path="./data/your_task/config.yaml",
    extract_label=lambda text: extract_your_label(text)
)

# Generate hypotheses
task.generate_hypotheses(
    method="hypogenic",
    num_hypotheses=20,
    output_path="./output/hypotheses.json"
)

# Run inference
results = task.inference(
    hypothesis_bank="./output/hypotheses.json",
    test_data="./data/your_task/your_task_test.json"
)

HypoRefine/Union Methods

# For literature-integrated approaches
# git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

# Generate with HypoRefine
task.generate_hypotheses(
    method="hyporefine",
    num_hypotheses=15,
    literature_path="./literature/your_task/",
    output_path="./output/"
)
# This generates 3 hypothesis banks:
# - HypoRefine (integrated approach)
# - Literature-only hypotheses
# - Literature∪HypoRefine (union)

Multi-Hypothesis Inference

from examples.multi_hyp_inference import run_multi_hypothesis_inference

# Test multiple hypotheses simultaneously
results = run_multi_hypothesis_inference(
    config_path="./data/your_task/config.yaml",
    hypothesis_bank="./output/hypotheses.json",
    test_data="./data/your_task/your_task_test.json"
)

Custom Label Extraction

The extract_label() function is critical for parsing LLM outputs. Implement it based on your task:

def extract_label(llm_output: str) -> str:
    """Extract predicted label from LLM inference text.
    
    Default behavior: searches for 'final answer:\s+(.*)' pattern.
    Customize for your domain-specific output format.
    """
    import re
    match = re.search(r'final answer:\s+(.*)', llm_output, re.IGNORECASE)
    if match:
        return match.group(1).strip()
    return llm_output.strip()

Important: Extracted labels must match the format of label values in your dataset for correct accuracy calculation.

Workflow Examples

Example 1: Data-Driven Hypothesis Generation (HypoGeniC)

Scenario: Detecting AI-generated content without prior theoretical framework

Steps:

Prepare dataset with text samples and labels (human vs. AI-generated)
Create config.yaml with appropriate prompt templates

Run hypothesis generation:

hypogenic_generation --config config.yaml --method hypogenic --num_hypotheses 20

Run inference on test set:

hypogenic_inference --config config.yaml --hypotheses output/hypotheses.json --test_data data/test.json

Analyze results for patterns like formality, grammatical precision, and tone differences

Example 2: Literature-Informed Hypothesis Testing (HypoRefine)

Scenario: Deception detection in hotel reviews building on existing research

Steps:

Collect 10 relevant papers on linguistic deception cues
Prepare dataset with genuine and fraudulent reviews
Configure config.yaml with literature processing and data generation templates

Run HypoRefine:

hypogenic_generation --config config.yaml --method hyporefine --papers papers/ --num_hypotheses 15

Test hypotheses examining pronoun frequency, detail specificity, and other linguistic patterns
Compare literature-based and data-driven hypothesis performance

Example 3: Comprehensive Hypothesis Coverage (Union Method)

Scenario: Mental stress detection maximizing hypothesis diversity

Steps:

Generate literature hypotheses from mental health research papers
Generate data-driven hypotheses from social media posts

Run Union method to combine and deduplicate:

hypogenic_generation --config config.yaml --method union --literature_hypotheses lit_hyp.json

Inference captures both theoretical constructs (posting behavior changes) and data patterns (emotional language shifts)

Performance Optimization

Caching: Enable Redis caching to reduce API costs and computation time for repeated LLM calls

Parallel Processing: Leverage multiple workers for large-scale hypothesis generation and testing

Adaptive Refinement: Use challenging examples to iteratively improve hypothesis quality

Expected Outcomes

Research using hypogenic has demonstrated:

14.19% accuracy improvement in AI-content detection tasks
7.44% accuracy improvement in deception detection tasks
80-84% of hypothesis pairs offering distinct, non-redundant insights
High helpfulness ratings from human evaluators across multiple research domains

Troubleshooting

Issue: Generated hypotheses are too generic Solution: Refine prompt templates in config.yaml to request more specific, testable hypotheses

Issue: Poor inference performance Solution: Ensure dataset has sufficient training examples, adjust hypothesis generation parameters, or increase number of hypotheses

Issue: Label extraction failures Solution: Implement custom extract_label() function for domain-specific output parsing

Issue: GROBID PDF processing fails Solution: Ensure GROBID service is running (bash ./modules/run_grobid.sh) and PDFs are valid research papers

Creating Custom Tasks

To add a new task or dataset to Hypogenic:

Step 1: Prepare Your Dataset

Create three JSON files following the required format:

your_task_train.json
your_task_val.json
your_task_test.json

Each file must have keys for text features (text_features_1, etc.) and label.

Step 2: Create config.yaml

Define your task configuration with:

Task name and dataset paths
Prompt templates for observations, generation, inference
Any extra keys for reusable prompt components
Placeholder variables (e.g., ${text_features_1}, ${num_hypotheses})

Step 3: Implement extract_label Function

Create a custom label extraction function that parses LLM outputs for your domain:

from hypogenic import BaseTask

def extract_my_label(llm_output: str) -> str:
    """Custom label extraction for your task.
    
    Must return labels in same format as dataset 'label' field.
    """
    # Example: Extract from specific format
    if "Final prediction:" in llm_output:
        return llm_output.split("Final prediction:")[-1].strip()
    
    # Fallback to default pattern
    import re
    match = re.search(r'final answer:\s+(.*)', llm_output, re.IGNORECASE)
    return match.group(1).strip() if match else llm_output.strip()

# Use your custom task
task = BaseTask(
    config_path="./your_task/config.yaml",
    extract_label=extract_my_label
)

Step 4: (Optional) Process Literature

For HypoRefine/Union methods:

Create literature/your_task_name/raw/ directory
Add relevant research paper PDFs
Run GROBID preprocessing
Process with pdf_preprocess.py

Step 5: Generate and Test

Run hypothesis generation and inference using CLI or Python API:

# CLI approach
hypogenic_generation --config your_task/config.yaml --method hypogenic --num_hypotheses 20
hypogenic_inference --config your_task/config.yaml --hypotheses output/hypotheses.json

# Or use Python API (see Python API Usage section)

Repository Structure

Understanding the repository layout:

hypothesis-generation/
├── hypogenic/              # Core package code
├── hypogenic_cmd/          # CLI entry points
├── hypothesis_agent/       # HypoRefine agent framework
├── literature/            # Literature processing utilities
├── modules/               # GROBID and preprocessing modules
├── examples/              # Example scripts
│   ├── generation.py      # Basic HypoGeniC generation
│   ├── union_generation.py # HypoRefine/Union generation
│   ├── inference.py       # Single hypothesis inference
│   ├── multi_hyp_inference.py # Multiple hypothesis inference
│   └── pdf_preprocess.py  # Literature PDF processing
├── data/                  # Example datasets (clone separately)
├── tests/                 # Unit tests
└── IO_prompting/          # Prompt templates and experiments

Key directories:

hypogenic/: Main package with BaseTask and generation logic
examples/: Reference implementations for common workflows
literature/: Tools for PDF processing and literature extraction
modules/: External tool integrations (GROBID, etc.)

Related Publications

HypoBench (2025)

Liu, H., Huang, S., Hu, J., Zhou, Y., & Tan, C. (2025). HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation. arXiv preprint arXiv:2504.11524.

Paper: https://arxiv.org/abs/2504.11524
Description: Benchmarking framework for systematic evaluation of hypothesis generation methods

BibTeX:

@misc{liu2025hypobenchsystematicprincipledbenchmarking,
      title={HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation}, 
      author={Haokun Liu and Sicong Huang and Jingyu Hu and Yangqiaoyu Zhou and Chenhao Tan},
      year={2025},
      eprint={2504.11524},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.11524}, 
}

Literature Meets Data (2024)

Liu, H., Zhou, Y., Li, M., Yuan, C., & Tan, C. (2024). Literature Meets Data: A Synergistic Approach to Hypothesis Generation. arXiv preprint arXiv:2410.17309.

Paper: https://arxiv.org/abs/2410.17309
Code: https://github.com/ChicagoHAI/hypothesis-generation
Description: Introduces HypoRefine and demonstrates synergistic combination of literature-based and data-driven hypothesis generation

BibTeX:

@misc{liu2024literaturemeetsdatasynergistic,
      title={Literature Meets Data: A Synergistic Approach to Hypothesis Generation}, 
      author={Haokun Liu and Yangqiaoyu Zhou and Mingxuan Li and Chenfei Yuan and Chenhao Tan},
      year={2024},
      eprint={2410.17309},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.17309}, 
}

Hypothesis Generation with Large Language Models (2024)

Zhou, Y., Liu, H., Srivastava, T., Mei, H., & Tan, C. (2024). Hypothesis Generation with Large Language Models. In Proceedings of EMNLP Workshop of NLP for Science.

Paper: https://aclanthology.org/2024.nlp4science-1.10/
Description: Original HypoGeniC framework for data-driven hypothesis generation

BibTeX:

@inproceedings{zhou2024hypothesisgenerationlargelanguage,
      title={Hypothesis Generation with Large Language Models}, 
      author={Yangqiaoyu Zhou and Haokun Liu and Tejes Srivastava and Hongyuan Mei and Chenhao Tan},
      booktitle = {Proceedings of EMNLP Workshop of NLP for Science},
      year={2024},
      url={https://aclanthology.org/2024.nlp4science-1.10/},
}

Additional Resources

Official Links

GitHub Repository: https://github.com/ChicagoHAI/hypothesis-generation
PyPI Package: https://pypi.org/project/hypogenic/
License: MIT License
Issues & Support: https://github.com/ChicagoHAI/hypothesis-generation/issues

Example Datasets

Clone these repositories for ready-to-use examples:

# HypoGeniC examples (data-driven only)
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data

# HypoRefine/Union examples (literature + data)
git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data

Community & Contributions

Contributors: 7+ active contributors
Stars: 89+ on GitHub
Topics: research-tool, interpretability, hypothesis-generation, scientific-discovery, llm-application

For contributions or questions, visit the GitHub repository and check the issues page.

Local Resources

references/

config_template.yaml - Complete example configuration file with all required prompt templates and parameters. This includes:

Full YAML structure for task configuration
Example prompt templates for all methods
Placeholder variable documentation
Role-based prompt examples

scripts/

Scripts directory is available for:

Custom data preparation utilities
Format conversion tools
Analysis and evaluation scripts
Integration with external tools

assets/

Assets directory is available for:

Example datasets and templates
Sample hypothesis banks
Visualization outputs
Documentation supplements