LLM Infrastructure Tools

LLM Infrastructure Tools

Tools for discovering, categorizing, and managing large language models. These are production-grade utilities that power Aurora’s model management capabilities.

Overview

Working with LLMs at scale requires infrastructure. These tools help us:

• Discover models from major providers
• Categorize models by capability and use case
• Filter by parameters, quantization, and other criteria
• Manage model downloads and caching

Available Tools

fetch_models.py

HuggingFace Model Categorizer

A comprehensive tool that fetches GGUF models from all major quantization uploaders on HuggingFace, categorizes them by type and capability, and produces a structured model list.

Key Features:

• Fetches from 20+ major uploaders (TheBloke, QuantFactory, Lewdiculous, etc.)
• Categorizes into 15+ primary categories (Audio, Vision, Coding, Reasoning, etc.)
• Auto-categorizes remaining models by architecture (Llama, Mistral, Phi, etc.)
• Filters by parameter count (0.1B-14B by default)
• Produces pipe-delimited output for easy parsing

Use Cases:

• Building model menus for LLM interfaces
• Powering model selection UIs
• Analyzing the LLM ecosystem
• Creating custom model collections

friend.zip - Mobile LLM Project

On-Device LLM Infrastructure

A complete mobile/on-device LLM implementation with shell scripts for:

• LLM calling
• Conversation management
• Configuration
• Setup and initialization

Key Features:

• Self-contained, portable implementation
• Bash-based with minimal dependencies
• Designed for resource-constrained environments
• Modular architecture

Use Cases:

• Running LLMs on edge devices
• Mobile AI assistants
• Offline LLM applications
• Embedded AI systems

The Model Categorization Challenge

With thousands of GGUF models on HuggingFace, manual categorization is impossible. fetch_models.py solves this with a two-stage approach:

Stage 1: Primary Categories (XOR - First Match Wins)

Models are first matched against specific use-case categories:

Stage 2: Auto-Categorization (Fallback)

Models that don’t match primary categories are auto-categorized by architecture:

Architecture Patterns

fetch_models.py Design

# 1. Define uploaders (sources)
UPLOADERS = ["Lewdiculous", "QuantFactory", "TheBloke", ...]

# 2. Define categorization rules
PRIMARY_RULES = [("Audio", [r"tts\b", r"\bspeech\b", ...]), ...]
AUTO_RULES = [("Auto/Gemma", [r"\bgemma\b"]), ...]

# 3. Fetch from each uploader
for uploader in UPLOADERS:
    data = fetch_from_huggingface(uploader)
    
# 4. Filter and categorize
for model in data:
    if passes_filters(model):
        category = categorize(model)
        models.append(model_with_category)

# 5. Sort and output
models.sort(key=lambda x: (x["date"], x["likes"]), reverse=True)
write_output(models)

Parameter Extraction

The tool handles complex parameter notation:

• 7B, 13B, 70B — Standard notation
• 30B-A3B — MoE (uses total params: 30B)
• 0.5B, 1.5B — Decimal parameters
• 7B fp16 — Ignores quantization suffix
• 128K, 256K — Context window (not parameter count)

Usage Examples

Basic Usage

# Fetch models, cache in ./cache, output to models.txt
python3 fetch_models.py ./cache models.txt

Output Format

# llama-menu model list — fetched from HuggingFace
# NAME|CATEGORY|REPO|FILE|SIZE|LIKES|DOWNLOADS|DATE
Llama-3-8B-Instruct|Instruct|meta-llama/Meta-Llama-3-8B-Instruct-GGUF|Q4_K_M.gguf|6.8GB|15000|85000|2026-07-15
Mistral-7B-Instruct|Instruct|TheBloke/Mistral-7B-Instruct-v0.2-GGUF|Q4_K_M.gguf|4.1GB|12000|75000|2026-07-10
Phi-3-mini-4k-instruct|Instruct|mradermacher/Phi-3-mini-4k-instruct-4k-GGUF|Q4_K_M.gguf|2.1GB|8000|45000|2026-07-20

Integration with Model Menus

Many LLM interfaces (like llama-menu.sh) use this output:

# Parse model list
while IFS='|' read -r name category repo file size likes downloads date; do
  if [ "$category" = "Coding" ]; then
    echo "[$name] ($size) - $category"
  fi
done < models.txt

Performance Considerations

Fetching Speed

• Fetches from 20+ uploaders sequentially
• ~500 models per uploader (limit=500)
• Total: ~10,000+ models processed
• Runtime: ~2-5 minutes (depends on network)

Caching

• Models are cached in JSON format
• Cache directory structure: cache/<uploader>.json
• Subsequent runs skip already-fetched data

Filtering

• Parameter range: 0.1B-14B (configurable)
• Excludes: base models, embeddings, rerankers
• Quantization: Prefers Q4_K_M, Q4_K_S, Q4_0

Source Code

All tools are available for download and study:

• fetch_models.py Source — Local version
• friend.zip Contents — Mobile LLM project

Related

• Aurora Tools — The AI agent that uses these tools
• Physics Tools — Mathematical and physics calculators
• Utility Scripts — General-purpose tools
• LLM Memory Architectures — How we use models

Good infrastructure makes complex systems simple. Great infrastructure makes them invisible.

┌─────────────────────────────────────────────────────────┐
│                    FRIEND SYSTEM                            │
├─────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────────┐     ┌─────────────┐     ┌────────────┐ │
│  │   talk.sh   │────►│ llm_call.sh │────►│   LLM      │ │
│  │ (Interface) │     │ (Core)      │     │ (Backend)  │ │
│  └─────────────┘     └─────────────┘     └────────────┘ │
│         ▲                   ▲                              │
│         │                   │                              │
│  ┌──────┴──────┐    ┌──────┴──────┐                      │
│  │  memory.sh  │    │  config.sh  │                      │
│  │ (State)     │    │ (Settings)  │                      │
│  └──────┬──────┘    └─────────────┘                      │
│         │                                              │
│  ┌──────▼──────┐                                             │
│  │  dream.sh   │                                             │
│  │ (Consolidation)                                         │
│  └─────────────┘                                             │
│                                                              │
└─────────────────────────────────────────────────────────┘

User Input → talk.sh → memory.sh (load context) → llm_call.sh → LLM
                                    ↑
                               dream.sh (consolidate)
                                    ↓
                            memory.sh (save context)