Current section
Files
Jump to
Current section
Files
guides/continual-learning.md
# Continual Learning: LC at Inference Time
This guide explores how the Liquid Conglomerate architecture can be useful beyond training - enabling **continual adaptation during deployment**.
## Overview
Traditional neuroevolution follows a simple pattern:
```
[Training] ──→ [Deploy Best Network] ──→ [Static Inference]
```
But what if the environment changes after deployment? What if the task distribution shifts? The deployed network becomes stale.
**Continual Learning** extends the LC to monitor and adapt networks during inference:
```
[Training] ──→ [Deploy with LC Monitoring] ──→ [Adaptive Inference]
↓
[Detect Drift] ──→ [Adapt/Retrain]
```
## Why Continual Learning?
### The Deployment Problem
Evolved networks face challenges in production:
| Challenge | Example | Impact |
|-----------|---------|--------|
| **Concept drift** | User behavior changes over time | Model accuracy degrades |
| **Distribution shift** | New input patterns appear | Model makes wrong predictions |
| **Non-stationarity** | Environment dynamics change | Control policy fails |
| **Adversarial adaptation** | Opponents learn your strategy | Game AI becomes exploitable |
### Traditional Solutions and Their Limits
1. **Periodic retraining**: Expensive, reactive (damage already done)
2. **Online learning**: Risk of catastrophic forgetting
3. **Ensemble methods**: High inference cost, doesn't solve drift detection
4. **Monitoring dashboards**: Manual intervention required
The LC offers **automated, continuous adaptation** with minimal overhead.
## LC Inference Modes
The LC can operate in several modes during inference:
### 1. Monitoring Only (Lowest Overhead)
Only the Resource Silo runs, tracking performance metrics:
```erlang
%% Start LC in monitoring mode
{ok, Pid} = lc_supervisor:start_link(#{
mode => inference,
silos => [resource], %% Only Resource Silo
thresholds => #{
latency_p99 => 50, %% ms
error_rate => 0.01, %% 1%
throughput_min => 1000 %% requests/sec
}
}).
```
**What it monitors:**
- Inference latency (p50, p95, p99)
- Error rates (if feedback available)
- Throughput metrics
- Resource utilization (memory, CPU)
**Actions:** Alerts only, no automatic adaptation.
### 2. Portfolio Switching (Low Overhead)
Task Silo selects from a portfolio of evolved networks:
```erlang
{ok, Pid} = lc_supervisor:start_link(#{
mode => inference,
silos => [resource, task],
portfolio => #{
networks => [Network1, Network2, Network3],
selection_strategy => performance_based,
switch_threshold => 0.1 %% 10% performance drop triggers switch
}
}).
```
**How it works:**
1. Monitor current network's performance
2. Detect degradation (error rate increase, reward drop)
3. Switch to alternative network from portfolio
4. Track which network works best for current conditions
**Use cases:**
- Multi-task domains with periodic task changes
- Environments with distinct operating modes
- Fallback to simpler network under resource pressure
### 3. Micro-Evolution (Medium Overhead)
Task Silo performs small weight perturbations during inference:
```erlang
{ok, Pid} = lc_supervisor:start_link(#{
mode => inference,
silos => [resource, task],
micro_evolution => #{
enabled => true,
mutation_strength => 0.01, %% Very small (1% of training)
evaluation_window => 100, %% Evaluate over 100 inferences
acceptance_threshold => 0.0 %% Accept if not worse
}
}).
```
**How it works:**
1. Periodically perturb network weights (very small mutations)
2. Evaluate performance over a window of inferences
3. If better (or not worse), keep the mutation
4. If worse, revert to previous weights
This is [Evolution Strategies](https://arxiv.org/abs/1703.03864) at inference time.
**Trade-off:** Slight performance variance during exploration vs. gradual adaptation.
### 4. Full Continual Learning (Higher Overhead)
Multiple silos work together for comprehensive adaptation:
```erlang
{ok, Pid} = lc_supervisor:start_link(#{
mode => inference,
silos => [resource, task, temporal, ecological],
continual_learning => #{
concept_drift_detector => true,
retraining_trigger => #{
drift_threshold => 0.2,
performance_drop => 0.15,
time_since_training => 86400 %% 24 hours
}
}
}).
```
**Components:**
- **Resource Silo**: Performance monitoring
- **Task Silo**: Network switching and micro-evolution
- **Temporal Silo**: Tracks performance over time, detects trends
- **Ecological Silo**: Monitors input distribution for concept drift
## Key Components
### 1. Performance Monitor
Extends the Resource Silo for inference-time metrics:
```erlang
-record(inference_metrics, {
latency_histogram, %% Latency distribution
error_count, %% Errors in current window
total_count, %% Total inferences in window
feedback_scores, %% Reward/accuracy if available
window_start, %% Timestamp
trend %% improving | stable | degrading
}).
```
**Key signals:**
- `inference_latency_exceeded` - P99 latency above threshold
- `error_rate_elevated` - Error rate above threshold
- `throughput_degraded` - Throughput below threshold
- `performance_trending_down` - Gradual degradation detected
### 2. Concept Drift Detector
Monitors input distribution for changes:
```erlang
-record(drift_detector, {
reference_distribution, %% Input stats from training
current_window, %% Recent input stats
drift_score, %% How much has distribution changed
drift_type %% gradual | sudden | recurring
}).
```
**Detection methods:**
- **Statistical tests**: Kolmogorov-Smirnov, Chi-squared on input features
- **Page-Hinkley**: Detects mean shift in performance metrics
- **ADWIN**: Adaptive windowing for gradual drift
- **Feature monitoring**: Track individual input feature distributions
### 3. Network Portfolio Manager
Manages multiple evolved networks:
```erlang
-record(portfolio, {
networks, %% List of {NetworkId, NetworkData}
performance_history, %% #{NetworkId => [scores]}
active_network, %% Currently deployed network
switch_count, %% How often we've switched
context_mapping %% #{Context => PreferredNetwork}
}).
```
**Portfolio strategies:**
- **Best performer**: Always use highest-scoring network
- **Context-aware**: Different networks for different input contexts
- **Exploration**: Occasionally try other networks to update scores
- **Aging**: Retire networks that haven't been good recently
### 4. Micro-Evolution Module
Applies ES-style updates during inference:
```erlang
micro_evolve(Network, Performance, Config) ->
%% Generate small perturbation
Delta = generate_perturbation(Network, Config#config.mutation_strength),
PerturbedNetwork = apply_perturbation(Network, Delta),
%% Evaluate over window
NewPerformance = evaluate_window(PerturbedNetwork, Config#config.window_size),
%% Accept or reject
case NewPerformance >= Performance - Config#config.acceptance_threshold of
true -> {ok, PerturbedNetwork, NewPerformance};
false -> {reject, Network, Performance}
end.
```
## Implementation Phases
We recommend implementing continual learning in phases:
| Phase | Feature | Complexity | Value |
|-------|---------|------------|-------|
| 1 | Performance monitoring (latency, error rate) | Low | High |
| 2 | Network switching (manual portfolio) | Low | Medium |
| 3 | Automatic concept drift detection | Medium | High |
| 4 | Micro-evolution during inference | Medium | Medium |
| 5 | Automatic network selection | High | High |
| 6 | Full retraining integration | High | Medium |
### Phase 1: Performance Monitoring
Start with basic metrics collection:
```erlang
%% In your inference path
Result = network:evaluate(Network, Input),
lc_resource_silo:record_inference(#{
latency => timer:now_diff(End, Start) / 1000,
success => is_valid(Result),
input_features => extract_features(Input)
}).
%% Resource Silo publishes alerts
handle_info({alert, inference_latency_exceeded, #{p99 := Latency}}, State) ->
%% Log, notify, or take action
...
```
### Phase 2: Network Switching
Add portfolio management:
```erlang
%% Initialize portfolio during deployment
Portfolio = portfolio_manager:init([
{champion, ChampionNetwork}, %% Best from training
{runner_up, RunnerUpNetwork}, %% Second best
{generalist, GeneralistNetwork} %% Robust to variation
]),
%% Switch on performance drop
handle_info({performance_drop, NetworkId, Drop}, State) when Drop > 0.1 ->
{ok, NewActive} = portfolio_manager:select_alternative(Portfolio, NetworkId),
{noreply, State#state{active_network = NewActive}}.
```
### Phase 3: Drift Detection
Implement statistical monitoring:
```erlang
%% During training, capture reference distribution
ReferenceStats = drift_detector:capture_training_distribution(TrainingData),
%% During inference, monitor for drift
handle_inference(Input, State) ->
drift_detector:update(State#state.detector, Input),
DriftScore = drift_detector:get_drift_score(State#state.detector),
case DriftScore > 0.2 of
true ->
%% Significant drift detected
trigger_adaptation(State);
false ->
State
end.
```
## Research Questions
Continual learning opens several research directions:
### Stability-Plasticity Tradeoff
How much should the network adapt without forgetting what it learned during training?
- **Too much plasticity**: Catastrophic forgetting, overfit to recent data
- **Too little plasticity**: Cannot adapt to genuine changes
**Approaches:**
- Elastic Weight Consolidation (EWC): Penalize changes to important weights
- Progressive networks: Add new capacity rather than modifying old
- Rehearsal: Occasionally replay training examples
### When to Switch vs Adapt
Is it better to switch to a different network or adapt the current one?
- **Switch**: Fast, no risk to current network, limited expressiveness
- **Adapt**: Slower, risk of degradation, potentially better fit
**Factors to consider:**
- Speed of change (sudden vs gradual)
- Reversibility (will conditions return?)
- Performance gap between portfolio members
### Feedback Signal Design
What defines "good" inference performance when we don't have explicit rewards?
**Proxy signals:**
- User engagement (clicks, time on task)
- Downstream system behavior
- Anomaly detection (unusual outputs)
- Consistency with recent inferences
### Catastrophic Forgetting Prevention
How to prevent unlearning useful knowledge during adaptation?
**Strategies:**
- Keep a frozen copy of the original network for comparison
- Use very small mutation strengths (1% of training)
- Implement change budgets (max total change allowed)
- Require improvements on validation set, not just recent data
## Use Cases
### 1. Game AI
**Scenario:** Opponents learn and adapt their strategies.
```erlang
%% Competitive Silo tracks opponent strategies
%% Network portfolio includes counter-strategies
%% Micro-evolution allows fine-tuning against current opponent
```
**Approach:**
- Track opponent behavior patterns (Competitive Silo)
- Switch to network trained against similar patterns
- Micro-evolve to specialize against this specific opponent
### 2. Robotics Control
**Scenario:** Physical wear changes robot dynamics over time.
```erlang
%% Resource Silo monitors control errors
%% Temporal Silo detects gradual degradation trends
%% Micro-evolution adjusts motor commands
```
**Approach:**
- Monitor position/velocity errors
- Detect when errors trend upward
- Small weight adjustments to compensate for wear
### 3. Anomaly Detection
**Scenario:** Normal behavior patterns shift seasonally.
```erlang
%% Ecological Silo monitors input distribution
%% Concept drift detection triggers adaptation
%% Portfolio includes networks trained on different seasons
```
**Approach:**
- Track input feature distributions
- Detect seasonal shifts in patterns
- Switch to appropriate seasonal model
### 4. Trading Systems
**Scenario:** Market regime changes (volatility, trends).
```erlang
%% Ecological Silo detects regime changes
%% Portfolio includes regime-specific strategies
%% Risk management via Resource Silo
```
**Approach:**
- Monitor market indicators for regime detection
- Switch to appropriate strategy (momentum vs mean-reversion)
- Conservative mode under uncertainty
## Summary
| Mode | Overhead | Adaptation Speed | Risk | Best For |
|------|----------|------------------|------|----------|
| **Monitoring only** | Minimal | None (manual) | None | Alerting, debugging |
| **Portfolio switching** | Low | Fast | Low | Multi-regime environments |
| **Micro-evolution** | Medium | Gradual | Medium | Gradual drift |
| **Full continual** | Higher | Comprehensive | Medium | Complex, dynamic environments |
**Key insight:** Start simple with monitoring and portfolio switching. Add micro-evolution only when you've validated that gradual adaptation helps. Full continual learning is powerful but requires careful design to avoid catastrophic forgetting.
## References
### Continual Learning
- Parisi, G. I., et al. (2019). "Continual lifelong learning with neural networks: A review." *Neural Networks*, 113, 54-71.
- Kirkpatrick, J., et al. (2017). "Overcoming catastrophic forgetting in neural networks." *PNAS*, 114(13), 3521-3526.
### Concept Drift
- Gama, J., et al. (2014). "A survey on concept drift adaptation." *ACM computing surveys*, 46(4), 1-37.
- Lu, J., et al. (2018). "Learning under concept drift: A review." *IEEE Transactions on Knowledge and Data Engineering*, 31(12), 2346-2363.
### Evolution Strategies
- Salimans, T., et al. (2017). "Evolution strategies as a scalable alternative to reinforcement learning." *arXiv:1703.03864*.
### Multi-Armed Bandits (for Network Selection)
- Auer, P., et al. (2002). "Finite-time analysis of the multiarmed bandit problem." *Machine learning*, 47(2), 235-256.
## Related Guides
- [Liquid Conglomerate](liquid-conglomerate.md) - Full LC architecture
- [Training Strategies](training-strategies.md) - Training phase strategies
- [Inference Scenarios](inference-scenarios.md) - Deployment patterns
- [Custom Evaluator](custom-evaluator.md) - Domain-specific metrics