Refactor UI to Operator Control Plane and add operator documentation
Co-authored-by: Junie <junie@jetbrains.com>
This commit is contained in:
parent
7c7024d66c
commit
3cff4db8f3
9
docs/operator/incident-review.md
Normal file
9
docs/operator/incident-review.md
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Incident Review Flow
|
||||
|
||||
When an incident occurs:
|
||||
1. Identify the incident in the Dashboard or Events view.
|
||||
2. Note the severity and the affected node/service.
|
||||
3. Switch to the Events view to see the correlation chain.
|
||||
4. Look for related events preceding the incident (e.g., latency spikes before a disconnect).
|
||||
5. Check the Recommendations view for suggested fixes.
|
||||
6. Once resolved, the incident will clear from the Active Incidents list.
|
||||
12
docs/operator/reconcile-review.md
Normal file
12
docs/operator/reconcile-review.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# Reconcile Review Flow
|
||||
|
||||
The system continuously monitors for drift between desired and actual state.
|
||||
|
||||
1. If a service is in RECONCILING state, check the Services view.
|
||||
2. Review the Recommendations view for automated or guarded actions.
|
||||
3. For 'safe' actions with high confidence, the system may act autonomously if enabled.
|
||||
4. For 'guarded' or 'dangerous' actions, an operator must manually approve the action.
|
||||
5. Risk Levels:
|
||||
- **Safe**: Minimal impact, high success rate.
|
||||
- **Guarded**: Potential brief service interruption.
|
||||
- **Dangerous**: Significant impact, potential data loss, or hardware interaction required.
|
||||
13
docs/operator/workflow-examples.md
Normal file
13
docs/operator/workflow-examples.md
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# Operator Workflow Examples
|
||||
|
||||
## Daily Check
|
||||
1. Open the Operator Control Plane.
|
||||
2. Check the System Status in the sidebar (should be NOMINAL).
|
||||
3. Review the Dashboard for any active incidents.
|
||||
4. Switch to Nodes view to ensure all nodes are connected and healthy.
|
||||
|
||||
## New Service Deployment
|
||||
1. Trigger deployment via CLI/API.
|
||||
2. Monitor progress in the Deployments view.
|
||||
3. If a stage fails, check the Diagnostics field in the deployment card.
|
||||
4. Once stable, verify the service health in the Services view.
|
||||
70
docs/runtime-supervisor.md
Normal file
70
docs/runtime-supervisor.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# Runtime Supervisor
|
||||
|
||||
The Runtime Supervisor is a reconciliation-oriented component responsible for ensuring the homelab infrastructure aligns with its desired state. It operates in a recommendation-only mode, detecting drifts and proposing actions without directly mutating the runtime.
|
||||
|
||||
## Reconciliation Loop
|
||||
|
||||
The supervisor performs a periodic (or on-demand) reconciliation loop:
|
||||
|
||||
1. **Load Desired State**: Reads configuration from the inventory model (`hosts/*/services.yaml`).
|
||||
2. **Load Actual State**: Reads the current world state from the filesystem-first observer output (`/opt/homelab/world/`).
|
||||
3. **Detect Drift**: Compares the two states to identify discrepancies.
|
||||
4. **Recommend Actions**: Uses a recommendation engine to propose remediations.
|
||||
5. **Emit Events**: Publishes reconciliation events to the platform's event system.
|
||||
|
||||
## Desired vs Actual State
|
||||
|
||||
- **Desired State**: Defined by the user in the inventory. It specifies which services should run on which nodes.
|
||||
- **Actual State (World State)**: Produced by the observer runtime. It represents the ground truth of what is currently happening in the physical and virtual infrastructure.
|
||||
|
||||
The gap between these two states is known as **Drift**.
|
||||
|
||||
## Drift Conditions
|
||||
|
||||
The supervisor detects the following drift conditions:
|
||||
|
||||
- **Missing Service**: A service is defined in the inventory but not found in the world state.
|
||||
- **Unhealthy Service**: A service is present but reporting a non-ok status.
|
||||
- **Failed Deployment**: Recent deployment attempts for a service have failed.
|
||||
- **Offline Node**: A node defined in the inventory is unreachable or inactive.
|
||||
- **Unresolved Incidents**: Active incidents in the world state that require attention.
|
||||
|
||||
## Recommendation Engine
|
||||
|
||||
Based on the detected drift, the supervisor emits one of three reconciliation event types:
|
||||
|
||||
- `reconcile_required`: Immediate action recommended to restore service (e.g., redeploy unhealthy service).
|
||||
- `reconcile_recommended`: Action suggested to improve stability or resolve minor issues (e.g., review unresolved incidents).
|
||||
- `reconcile_blocked`: Action required but blocked by external factors or repeated failures (e.g., repeated deployment failures requiring manual diagnostics).
|
||||
|
||||
### Examples:
|
||||
- **Unhealthy service** -> Recommend `redeploy`.
|
||||
- **Repeated deployment failures** -> Recommend `diagnostics`.
|
||||
- **Node offline** -> Recommend `failover review`.
|
||||
- **Dependency unavailable** -> Recommend `delayed deployment`.
|
||||
|
||||
## Summary States
|
||||
|
||||
The supervisor extends the platform's runtime summary states:
|
||||
|
||||
- `nominal`: Desired and actual states are in sync.
|
||||
- `degraded`: Non-critical drift detected; reconciliation required.
|
||||
- `unstable`: Critical drift or blocked reconciliation detected.
|
||||
- `reconciling`: (Future) Remediations are actively being applied.
|
||||
|
||||
## Future Autonomous Remediation Architecture
|
||||
|
||||
While the current stage is recommendation-only, the architecture is designed for future autonomy:
|
||||
|
||||
1. **Policy Engine**: A future component will ingest reconciliation recommendations and apply policies to decide whether to auto-remediate.
|
||||
2. **Executor**: A mutation-capable runtime that will execute the proposed actions (restarts, redeploys, failovers).
|
||||
3. **Closed-Loop Feedback**: The observer will immediately reflect the results of remediations, allowing the supervisor to verify success or escalate if the drift persists.
|
||||
4. **Guardrails**: Implementation of rate-limiting, maintenance windows, and manual overrides to ensure autonomous actions remain safe.
|
||||
|
||||
## Filesystem-First Design
|
||||
|
||||
The supervisor adheres to the platform's filesystem-first philosophy:
|
||||
- **Input**: Files in `/opt/homelab/world/` and `hosts/`.
|
||||
- **Output**: Append-only logs in `/tmp/agent-events.log`.
|
||||
- **Persistence**: Checkpoints stored in `/tmp/supervisor-checkpoint.json`.
|
||||
- **Idempotency**: Every run is independent and produces the same recommendations for the same input state.
|
||||
889
webui/index.html
889
webui/index.html
|
|
@ -3,558 +3,495 @@
|
|||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Agent System</title>
|
||||
<title>Operator Control Plane</title>
|
||||
<style>
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
background: #101418;
|
||||
color: #e7edf3;
|
||||
:root {
|
||||
--bg-color: #0a0c0e;
|
||||
--sidebar-color: #14171a;
|
||||
--card-color: #1c2024;
|
||||
--border-color: #2a3540;
|
||||
--text-color: #e7edf3;
|
||||
--text-muted: #94a3b8;
|
||||
--accent-color: #3eaf7c;
|
||||
--nominal: #3eaf7c;
|
||||
--degraded: #e7c000;
|
||||
--unstable: #e67e22;
|
||||
--reconciling: #3498db;
|
||||
--error: #c0392b;
|
||||
--safe: #3eaf7c;
|
||||
--guarded: #e67e22;
|
||||
--dangerous: #c0392b;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 1100px;
|
||||
margin: 0 auto;
|
||||
body {
|
||||
margin: 0;
|
||||
font-family: 'Inter', system-ui, -apple-system, sans-serif;
|
||||
background: var(--bg-color);
|
||||
color: var(--text-color);
|
||||
display: flex;
|
||||
height: 100vh;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* Sidebar */
|
||||
.sidebar {
|
||||
width: 240px;
|
||||
background: var(--sidebar-color);
|
||||
border-right: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.sidebar-header {
|
||||
padding: 24px;
|
||||
font-weight: 800;
|
||||
font-size: 14px;
|
||||
letter-spacing: 0.1em;
|
||||
color: var(--accent-color);
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.nav-list {
|
||||
list-style: none;
|
||||
padding: 12px 0;
|
||||
margin: 0;
|
||||
flex-grow: 1;
|
||||
}
|
||||
|
||||
.nav-item {
|
||||
padding: 12px 24px;
|
||||
cursor: pointer;
|
||||
font-size: 14px;
|
||||
color: var(--text-muted);
|
||||
transition: all 0.2s;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.nav-item:hover {
|
||||
background: rgba(255, 255, 255, 0.05);
|
||||
color: var(--text-color);
|
||||
}
|
||||
|
||||
.nav-item.active {
|
||||
background: rgba(62, 175, 124, 0.1);
|
||||
color: var(--accent-color);
|
||||
border-left: 3px solid var(--accent-color);
|
||||
}
|
||||
|
||||
.sidebar-footer {
|
||||
padding: 16px;
|
||||
border-top: 1px solid var(--border-color);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
/* Content Area */
|
||||
.main-content {
|
||||
flex-grow: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
header {
|
||||
height: 64px;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 0 24px;
|
||||
justify-content: space-between;
|
||||
background: var(--bg-color);
|
||||
}
|
||||
|
||||
.view-title {
|
||||
font-size: 18px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.content-scroll {
|
||||
flex-grow: 1;
|
||||
overflow-y: auto;
|
||||
padding: 24px;
|
||||
}
|
||||
|
||||
h1 {
|
||||
margin: 0 0 16px;
|
||||
font-size: 24px;
|
||||
font-weight: 650;
|
||||
/* Cards & Grids */
|
||||
.grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(350px, 1fr));
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
#logs {
|
||||
height: 70vh;
|
||||
overflow-y: auto;
|
||||
border: 1px solid #2a3540;
|
||||
background: #0b0f13;
|
||||
padding: 12px;
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
font-size: 13px;
|
||||
line-height: 1.45;
|
||||
white-space: pre-wrap;
|
||||
.card {
|
||||
background: var(--card-color);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 20px;
|
||||
border-radius: 4px;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
#services {
|
||||
border: 1px solid #2a3540;
|
||||
background: #0b0f13;
|
||||
padding: 12px;
|
||||
margin-bottom: 12px;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.panel-title {
|
||||
font-weight: 700;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
.service {
|
||||
border-top: 1px solid #202a33;
|
||||
padding: 10px 0;
|
||||
}
|
||||
|
||||
.service:first-of-type {
|
||||
border-top: 0;
|
||||
padding-top: 0;
|
||||
}
|
||||
|
||||
.service-header {
|
||||
.card-header {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: baseline;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 6px;
|
||||
}
|
||||
|
||||
.service-name {
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.service-history {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
margin-top: 6px;
|
||||
}
|
||||
|
||||
.history-entry {
|
||||
border: 1px solid #2a3540;
|
||||
padding: 3px 6px;
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
}
|
||||
|
||||
.run {
|
||||
border: 1px solid #333;
|
||||
margin: 10px;
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.run.stopped {
|
||||
opacity: 0.45;
|
||||
}
|
||||
|
||||
.header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
color: #ffffff;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.card-title {
|
||||
font-weight: 700;
|
||||
margin-bottom: 8px;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.run-title {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
/* Status Badges */
|
||||
.badge {
|
||||
padding: 4px 8px;
|
||||
border-radius: 4px;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.proposal {
|
||||
border: 1px solid #4c5b2b;
|
||||
background: #171d12;
|
||||
color: #edf5d4;
|
||||
margin: 8px 0;
|
||||
padding: 10px;
|
||||
}
|
||||
.status-nominal { background: rgba(62, 175, 124, 0.1); color: var(--nominal); }
|
||||
.status-degraded { background: rgba(231, 192, 0, 0.1); color: var(--degraded); }
|
||||
.status-unstable { background: rgba(230, 126, 34, 0.1); color: var(--unstable); }
|
||||
.status-reconciling { background: rgba(52, 152, 219, 0.1); color: var(--reconciling); }
|
||||
.status-error { background: rgba(192, 57, 43, 0.1); color: var(--error); }
|
||||
|
||||
.proposal-actions {
|
||||
/* Timeline */
|
||||
.timeline {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
flex-direction: column;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.proposal-divider {
|
||||
border-top: 1px solid #3c4930;
|
||||
margin: 10px 0;
|
||||
}
|
||||
|
||||
.row {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr auto auto;
|
||||
gap: 8px;
|
||||
margin-top: 12px;
|
||||
}
|
||||
|
||||
.config-panel {
|
||||
border: 1px solid #2a3540;
|
||||
background: #111820;
|
||||
margin-top: 12px;
|
||||
.event {
|
||||
padding: 12px;
|
||||
border-left: 2px solid var(--border-color);
|
||||
background: rgba(255, 255, 255, 0.02);
|
||||
font-family: ui-monospace, monospace;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.config-title {
|
||||
font-weight: 700;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
.event.high { border-left-color: var(--error); }
|
||||
.event.medium { border-left-color: var(--unstable); }
|
||||
.event.low { border-left-color: var(--nominal); }
|
||||
|
||||
.config-grid {
|
||||
display: grid;
|
||||
grid-template-columns: auto minmax(96px, 140px) auto;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.toggle {
|
||||
.event-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
border: 1px solid #354453;
|
||||
background: #151c23;
|
||||
color: #e7edf3;
|
||||
padding: 10px 12px;
|
||||
white-space: nowrap;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 4px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.toggle input {
|
||||
padding: 0;
|
||||
/* Forms & Inputs */
|
||||
.controls {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
input, button {
|
||||
font: inherit;
|
||||
border: 1px solid #354453;
|
||||
background: #151c23;
|
||||
color: #e7edf3;
|
||||
padding: 10px 12px;
|
||||
background: var(--card-color);
|
||||
border: 1px solid var(--border-color);
|
||||
color: var(--text-color);
|
||||
padding: 8px 16px;
|
||||
font-size: 14px;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
button {
|
||||
cursor: pointer;
|
||||
min-width: 88px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
button:hover {
|
||||
background: #1d2730;
|
||||
}
|
||||
button:hover { background: var(--border-color); }
|
||||
|
||||
.btn-primary { background: var(--accent-color); color: white; border: none; }
|
||||
.btn-primary:hover { background: #359b6d; }
|
||||
|
||||
/* Utility */
|
||||
.hidden { display: none !important; }
|
||||
.mono { font-family: ui-monospace, monospace; }
|
||||
.label { color: var(--text-muted); font-size: 12px; margin-bottom: 4px; }
|
||||
.value { font-weight: 500; margin-bottom: 12px; }
|
||||
|
||||
.risk-safe { color: var(--safe); }
|
||||
.risk-guarded { color: var(--guarded); }
|
||||
.risk-dangerous { color: var(--dangerous); }
|
||||
|
||||
.error { color: #ff6b6b; }
|
||||
.warning { color: #f4a261; }
|
||||
.info { color: #5dade2; }
|
||||
.action { color: #58d68d; }
|
||||
.auto_action { color: #a7f3d0; }
|
||||
.learning { color: #c4b5fd; }
|
||||
.log, .result { color: #ccd6df; }
|
||||
.status.running { color: yellow; }
|
||||
.status.done { color: green; }
|
||||
.status.error { color: red; }
|
||||
.status.stopped { color: gray; }
|
||||
.health-ok { color: #58d68d; }
|
||||
.health-warning { color: #f4d03f; }
|
||||
.health-error { color: #ff6b6b; }
|
||||
.health-unknown { color: #95a5a6; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Agent System</h1>
|
||||
<section id="services">
|
||||
<div class="panel-title">Services</div>
|
||||
<div id="services-list"></div>
|
||||
</section>
|
||||
<div id="logs"></div>
|
||||
<div class="row">
|
||||
<input id="cmd" autocomplete="off" placeholder="Command">
|
||||
<button onclick="send()">Send</button>
|
||||
<button id="stop">STOP</button>
|
||||
<aside class="sidebar">
|
||||
<div class="sidebar-header">HOMELAB OPERATOR</div>
|
||||
<ul class="nav-list">
|
||||
<li class="nav-item active" onclick="showView('dashboard', this)">
|
||||
<span>Dashboard</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('nodes', this)">
|
||||
<span>Nodes</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('services', this)">
|
||||
<span>Services</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('deployments', this)">
|
||||
<span>Deployments</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('events', this)">
|
||||
<span>Events</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('recommendations', this)">
|
||||
<span>Recommendations</span>
|
||||
</li>
|
||||
<li class="nav-item" onclick="showView('settings', this)">
|
||||
<span>Settings</span>
|
||||
</li>
|
||||
</ul>
|
||||
<div class="sidebar-footer">
|
||||
<div id="summary-status">System Status: Loading...</div>
|
||||
</div>
|
||||
<section class="config-panel">
|
||||
<div class="config-title">Auto Mode Config</div>
|
||||
<div class="config-grid">
|
||||
<label class="toggle">
|
||||
<input id="config-auto-mode" type="checkbox">
|
||||
auto_mode
|
||||
</label>
|
||||
<span></span>
|
||||
<button id="save-config">Save</button>
|
||||
</aside>
|
||||
|
||||
<label for="restart-threshold">restart_ha threshold</label>
|
||||
<input id="restart-threshold" type="number" min="0" max="1" step="0.01">
|
||||
<span></span>
|
||||
|
||||
<label for="network-threshold">check_network threshold</label>
|
||||
<input id="network-threshold" type="number" min="0" max="1" step="0.01">
|
||||
<span></span>
|
||||
<main class="main-content">
|
||||
<header>
|
||||
<div class="view-title" id="current-view-title">Dashboard</div>
|
||||
<div class="header-actions">
|
||||
<button onclick="refreshData()">Refresh</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="content-scroll">
|
||||
<!-- Dashboard View -->
|
||||
<div id="view-dashboard" class="view">
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<div class="card-title">System Overview</div>
|
||||
<div id="dashboard-summary" style="margin-top:20px"></div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="card-title">Active Incidents</div>
|
||||
<div id="dashboard-incidents" style="margin-top:20px"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Nodes View -->
|
||||
<div id="view-nodes" class="view hidden">
|
||||
<div class="grid" id="nodes-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Services View -->
|
||||
<div id="view-services" class="view hidden">
|
||||
<div class="grid" id="services-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Deployments View -->
|
||||
<div id="view-deployments" class="view hidden">
|
||||
<div class="grid" id="deployments-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Events View -->
|
||||
<div id="view-events" class="view hidden">
|
||||
<div class="timeline" id="events-timeline"></div>
|
||||
</div>
|
||||
|
||||
<!-- Recommendations View -->
|
||||
<div id="view-recommendations" class="view hidden">
|
||||
<div class="grid" id="recommendations-list"></div>
|
||||
</div>
|
||||
|
||||
<!-- Settings View -->
|
||||
<div id="view-settings" class="view hidden">
|
||||
<div class="card">
|
||||
<div class="card-title">Configuration</div>
|
||||
<div id="settings-content" style="margin-top:20px"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script>
|
||||
const logs = document.getElementById("logs");
|
||||
const servicesList = document.getElementById("services-list");
|
||||
let polling = true;
|
||||
let currentConfig = {
|
||||
auto_mode: true,
|
||||
action_thresholds: {
|
||||
restart_ha: 0.8,
|
||||
check_network: 0.9
|
||||
},
|
||||
default_threshold: 0.9,
|
||||
allowed_auto_actions: ["restart_ha"]
|
||||
};
|
||||
let currentView = 'dashboard';
|
||||
const pollInterval = 5000;
|
||||
|
||||
function eventClass(event) {
|
||||
const message = event.message || "";
|
||||
if (event.type === "action") return "action";
|
||||
if (message.startsWith("[error]")) return "error";
|
||||
if (message.startsWith("[warning]")) return "warning";
|
||||
if (message.startsWith("[info]")) return "info";
|
||||
return event.type || "log";
|
||||
function showView(viewId, el) {
|
||||
document.querySelectorAll('.view').forEach(v => v.classList.add('hidden'));
|
||||
document.getElementById('view-' + viewId).classList.remove('hidden');
|
||||
document.querySelectorAll('.nav-item').forEach(i => i.classList.remove('active'));
|
||||
if (el) el.classList.add('active');
|
||||
currentView = viewId;
|
||||
document.getElementById('current-view-title').textContent = viewId.charAt(0).toUpperCase() + viewId.slice(1);
|
||||
refreshData();
|
||||
}
|
||||
|
||||
function confidenceLabel(confidence) {
|
||||
if (typeof confidence !== "number") return "";
|
||||
return `${Math.round(confidence * 100)}%`;
|
||||
}
|
||||
|
||||
function healthClass(status) {
|
||||
const normalized = String(status || "unknown").toLowerCase();
|
||||
if (["ok", "healthy", "up"].includes(normalized)) return "health-ok";
|
||||
if (["warning", "degraded"].includes(normalized)) return "health-warning";
|
||||
if (["error", "down", "failed"].includes(normalized)) return "health-error";
|
||||
return "health-unknown";
|
||||
}
|
||||
|
||||
function formatTime(timestamp) {
|
||||
if (!timestamp) return "unknown";
|
||||
return new Date(timestamp * 1000).toLocaleString();
|
||||
}
|
||||
|
||||
async function fetchServices() {
|
||||
const response = await fetch("/services", {cache: "no-store"});
|
||||
const services = await response.json();
|
||||
servicesList.innerHTML = "";
|
||||
|
||||
if (!Array.isArray(services) || services.length === 0) {
|
||||
servicesList.textContent = "No service health data yet.";
|
||||
return;
|
||||
}
|
||||
|
||||
services.forEach((service) => {
|
||||
const item = document.createElement("div");
|
||||
item.className = "service";
|
||||
|
||||
const header = document.createElement("div");
|
||||
header.className = "service-header";
|
||||
|
||||
const name = document.createElement("span");
|
||||
name.className = "service-name";
|
||||
name.textContent = service.name || "unknown";
|
||||
header.appendChild(name);
|
||||
|
||||
const status = document.createElement("span");
|
||||
status.className = healthClass(service.status);
|
||||
status.textContent = String(service.status || "unknown").toUpperCase();
|
||||
header.appendChild(status);
|
||||
item.appendChild(header);
|
||||
|
||||
const lastCheck = document.createElement("div");
|
||||
lastCheck.textContent = `Last check: ${formatTime(service.last_check)}`;
|
||||
item.appendChild(lastCheck);
|
||||
|
||||
const history = document.createElement("div");
|
||||
history.className = "service-history";
|
||||
(service.history || []).slice(-5).forEach((entry) => {
|
||||
const node = document.createElement("span");
|
||||
node.className = `history-entry ${healthClass(entry.status)}`;
|
||||
node.textContent = `${entry.status || "unknown"} ${formatTime(entry.timestamp)}`;
|
||||
history.appendChild(node);
|
||||
});
|
||||
item.appendChild(history);
|
||||
|
||||
servicesList.appendChild(item);
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchLogs() {
|
||||
if (!polling) return;
|
||||
const response = await fetch("/logs", {cache: "no-store"});
|
||||
const text = await response.text();
|
||||
logs.innerHTML = "";
|
||||
|
||||
const runs = {};
|
||||
text.trim().split("\n").filter(Boolean).forEach((line) => {
|
||||
let event;
|
||||
async function fetchData(endpoint) {
|
||||
try {
|
||||
event = JSON.parse(line);
|
||||
} catch {
|
||||
event = {
|
||||
type: "log",
|
||||
message: line,
|
||||
run_id: "legacy",
|
||||
timestamp: Date.now() / 1000
|
||||
};
|
||||
const res = await fetch(endpoint, {cache: 'no-store'});
|
||||
return await res.json();
|
||||
} catch (e) {
|
||||
console.error('Fetch error:', endpoint, e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
const runId = event.run_id || "no-run";
|
||||
if (!runs[runId]) {
|
||||
runs[runId] = {
|
||||
events: [],
|
||||
status: "running",
|
||||
received: 0,
|
||||
expected: 0
|
||||
};
|
||||
function formatTime(ts) {
|
||||
if (!ts) return 'N/A';
|
||||
return new Date(ts * 1000).toLocaleString();
|
||||
}
|
||||
|
||||
const run = runs[runId];
|
||||
run.events.push(event);
|
||||
|
||||
if (event.type === "run_status") {
|
||||
run.status = event.status || run.status;
|
||||
} else if (event.type === "run_progress") {
|
||||
run.received = Number(event.received) || 0;
|
||||
run.expected = Number(event.expected) || 0;
|
||||
}
|
||||
});
|
||||
|
||||
Object.entries(runs).forEach(([runId, runState]) => {
|
||||
const run = document.createElement("div");
|
||||
run.className = "run";
|
||||
const status = runState.status;
|
||||
if (status === "stopped") {
|
||||
run.classList.add("stopped");
|
||||
function getStatusClass(status) {
|
||||
status = (status || '').toLowerCase();
|
||||
if (['nominal', 'healthy', 'ok', 'up'].includes(status)) return 'status-nominal';
|
||||
if (['degraded', 'warning'].includes(status)) return 'status-degraded';
|
||||
if (['unstable'].includes(status)) return 'status-unstable';
|
||||
if (['reconciling'].includes(status)) return 'status-reconciling';
|
||||
if (['error', 'down', 'failed'].includes(status)) return 'status-error';
|
||||
return '';
|
||||
}
|
||||
|
||||
const header = document.createElement("div");
|
||||
header.className = "header";
|
||||
async function refreshData() {
|
||||
// Refresh summary always
|
||||
const summary = await fetchData('/summary');
|
||||
if (summary) {
|
||||
const statusEl = document.getElementById('summary-status');
|
||||
statusEl.textContent = `System Status: ${summary.status.toUpperCase()}`;
|
||||
statusEl.className = 'sidebar-footer ' + getStatusClass(summary.status);
|
||||
|
||||
const title = document.createElement("div");
|
||||
title.className = "run-title";
|
||||
title.appendChild(document.createTextNode(`RUN ${runId} [`));
|
||||
|
||||
const statusNode = document.createElement("span");
|
||||
statusNode.className = `status ${status}`;
|
||||
statusNode.textContent = status.toUpperCase();
|
||||
title.appendChild(statusNode);
|
||||
title.appendChild(
|
||||
document.createTextNode(`] (${runState.received}/${runState.expected})`)
|
||||
);
|
||||
header.appendChild(title);
|
||||
|
||||
if (runId !== "no-run" && status === "running") {
|
||||
const stopButton = document.createElement("button");
|
||||
stopButton.textContent = "STOP";
|
||||
stopButton.setAttribute("onclick", `stop(${JSON.stringify(runId)})`);
|
||||
header.appendChild(stopButton);
|
||||
if (currentView === 'dashboard') {
|
||||
const dashSummary = document.getElementById('dashboard-summary');
|
||||
dashSummary.innerHTML = `
|
||||
<div class="label">Nodes</div><div class="value">${summary.node_count}</div>
|
||||
<div class="label">Services</div><div class="value">${summary.service_count}</div>
|
||||
<div class="label">Last Update</div><div class="value">${formatTime(summary.last_update)}</div>
|
||||
`;
|
||||
}
|
||||
}
|
||||
|
||||
run.appendChild(header);
|
||||
|
||||
const eventList = document.createElement("div");
|
||||
eventList.className = "events";
|
||||
|
||||
runState.events.forEach((event) => {
|
||||
if (event.type === "proposal") {
|
||||
const proposal = document.createElement("div");
|
||||
proposal.className = "proposal";
|
||||
|
||||
const message = document.createElement("div");
|
||||
message.textContent = event.message || "";
|
||||
proposal.appendChild(message);
|
||||
|
||||
const confidence = document.createElement("div");
|
||||
confidence.textContent = `Confidence: ${confidenceLabel(event.confidence)}`;
|
||||
proposal.appendChild(confidence);
|
||||
|
||||
const actions = document.createElement("div");
|
||||
actions.className = "proposal-actions";
|
||||
const allOptions = event.options || event.actions || [];
|
||||
const options = allOptions.filter((option) => !option.is_ignore).sort(
|
||||
(a, b) => (b.confidence || 0) - (a.confidence || 0)
|
||||
);
|
||||
const ignoreOptions = allOptions.filter((option) => option.is_ignore);
|
||||
options.forEach((option) => {
|
||||
const button = document.createElement("button");
|
||||
const optionConfidence = confidenceLabel(option.confidence);
|
||||
button.textContent = optionConfidence
|
||||
? `${option.label || option.command} (${optionConfidence})`
|
||||
: option.label || option.command;
|
||||
button.setAttribute(
|
||||
"onclick",
|
||||
`apply(${JSON.stringify(runId)}, ${JSON.stringify(option.command)})`
|
||||
);
|
||||
actions.appendChild(button);
|
||||
});
|
||||
proposal.appendChild(actions);
|
||||
|
||||
if (ignoreOptions.length) {
|
||||
const divider = document.createElement("div");
|
||||
divider.className = "proposal-divider";
|
||||
proposal.appendChild(divider);
|
||||
|
||||
const ignoreActions = document.createElement("div");
|
||||
ignoreActions.className = "proposal-actions";
|
||||
ignoreOptions.forEach((option) => {
|
||||
const button = document.createElement("button");
|
||||
button.textContent = option.label || option.command;
|
||||
button.setAttribute(
|
||||
"onclick",
|
||||
`apply(${JSON.stringify(runId)}, ${JSON.stringify(option.command)})`
|
||||
);
|
||||
ignoreActions.appendChild(button);
|
||||
});
|
||||
proposal.appendChild(ignoreActions);
|
||||
if (currentView === 'dashboard' || currentView === 'events') {
|
||||
const incidents = await fetchData('/incidents');
|
||||
if (currentView === 'dashboard') {
|
||||
const dashIncidents = document.getElementById('dashboard-incidents');
|
||||
if (!incidents || incidents.length === 0) {
|
||||
dashIncidents.textContent = 'No active incidents.';
|
||||
} else {
|
||||
dashIncidents.innerHTML = incidents.map(inc => `
|
||||
<div class="event ${inc.severity}">
|
||||
<strong>${inc.severity.toUpperCase()}:</strong> ${inc.message}<br>
|
||||
<small>${formatTime(inc.timestamp)} | Node: ${inc.node}</small>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
eventList.appendChild(proposal);
|
||||
return;
|
||||
if (currentView === 'nodes') {
|
||||
const nodes = await fetchData('/nodes');
|
||||
const list = document.getElementById('nodes-list');
|
||||
list.innerHTML = nodes.map(node => `
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<div class="card-title">${node.hostname}</div>
|
||||
<span class="badge ${getStatusClass(node.health)}">${node.health}</span>
|
||||
</div>
|
||||
<div class="label">ID</div><div class="value mono">${node.id}</div>
|
||||
<div class="label">Capabilities</div><div class="value">${node.capabilities.join(', ')}</div>
|
||||
<div class="label">Connectivity</div><div class="value">${node.connectivity}</div>
|
||||
<div class="label">Incidents (24h)</div><div class="value">${node.incidents}</div>
|
||||
<div class="label">Last Seen</div><div class="value">${formatTime(node.last_seen)}</div>
|
||||
<div class="label">Runtime Status</div><div class="value">${node.status}</div>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
const div = document.createElement("div");
|
||||
div.className = eventClass(event);
|
||||
div.textContent = `${new Date(event.timestamp * 1000).toLocaleTimeString()} ${event.message}`;
|
||||
eventList.appendChild(div);
|
||||
});
|
||||
|
||||
run.appendChild(eventList);
|
||||
logs.appendChild(run);
|
||||
});
|
||||
|
||||
logs.scrollTop = logs.scrollHeight;
|
||||
if (currentView === 'services') {
|
||||
const services = await fetchData('/services');
|
||||
const list = document.getElementById('services-list');
|
||||
list.innerHTML = services.map(svc => `
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<div class="card-title">${svc.name}</div>
|
||||
<span class="badge ${getStatusClass(svc.health)}">${svc.health}</span>
|
||||
</div>
|
||||
<div class="label">State (Desired/Actual)</div><div class="value">${svc.desired_state} / ${svc.actual_state}</div>
|
||||
<div class="label">Deployment</div><div class="value">${svc.deployment_state}</div>
|
||||
<div class="label">Dependencies</div><div class="value">${svc.dependencies.join(', ') || 'None'}</div>
|
||||
<div class="label">Recommendations</div><div class="value">${svc.recommendations.join(', ') || 'None'}</div>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
async function send() {
|
||||
const cmd = document.getElementById("cmd");
|
||||
const value = cmd.value.trim();
|
||||
if (!value) return;
|
||||
await fetch("/command", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({command: value})
|
||||
});
|
||||
cmd.value = "";
|
||||
if (currentView === 'deployments') {
|
||||
const deps = await fetchData('/deployments');
|
||||
const list = document.getElementById('deployments-list');
|
||||
list.innerHTML = deps.map(dep => `
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<div class="card-title">${dep.service}</div>
|
||||
<span class="badge ${dep.status === 'failed' ? 'status-error' : 'status-reconciling'}">${dep.status}</span>
|
||||
</div>
|
||||
<div class="label">ID</div><div class="value mono">${dep.id}</div>
|
||||
<div class="label">Stage</div><div class="value">${dep.stage}</div>
|
||||
<div class="label">Diagnostics</div><div class="value">${dep.diagnostics || 'No data'}</div>
|
||||
<div class="label">Resumable</div><div class="value">${dep.resumable ? 'Yes' : 'No'}</div>
|
||||
${dep.resumable ? '<button class="btn-primary">Resume</button>' : ''}
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
async function stop(run_id) {
|
||||
await fetch("/stop", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({run_id})
|
||||
});
|
||||
if (currentView === 'events') {
|
||||
const events = await fetchData('/events');
|
||||
const timeline = document.getElementById('events-timeline');
|
||||
timeline.innerHTML = events.map(ev => `
|
||||
<div class="event ${ev.severity}">
|
||||
<div class="event-header">
|
||||
<span>${ev.type.toUpperCase()}</span>
|
||||
<span>${formatTime(ev.timestamp)}</span>
|
||||
</div>
|
||||
<div>${ev.message}</div>
|
||||
<div class="label" style="margin-top:8px">Node: ${ev.node} ${ev.service ? '| Service: ' + ev.service : ''}</div>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
async function apply(run_id, command) {
|
||||
await fetch("/action", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({run_id, command})
|
||||
});
|
||||
if (currentView === 'recommendations') {
|
||||
const recs = await fetchData('/recommendations');
|
||||
const list = document.getElementById('recommendations-list');
|
||||
list.innerHTML = recs.map(rec => `
|
||||
<div class="card">
|
||||
<div class="card-header">
|
||||
<div class="card-title">${rec.title}</div>
|
||||
<span class="badge risk-${rec.risk_level}">${rec.risk_level}</span>
|
||||
</div>
|
||||
<p>${rec.description}</p>
|
||||
<div class="label">Confidence</div><div class="value">${Math.round(rec.confidence * 100)}%</div>
|
||||
<div class="label">Autonomous Eligible</div><div class="value">${rec.autonomous_eligible ? 'Yes' : 'No'}</div>
|
||||
<div class="label">Blocked Actions</div><div class="value">${rec.blocked_actions.join(', ') || 'None'}</div>
|
||||
<div class="controls">
|
||||
<button class="btn-primary" ${rec.risk_level === 'dangerous' ? 'style="background:var(--dangerous)"' : ''}>Approve Action</button>
|
||||
</div>
|
||||
</div>
|
||||
`).join('');
|
||||
}
|
||||
|
||||
function populateConfig(config) {
|
||||
currentConfig = config;
|
||||
document.getElementById("config-auto-mode").checked = Boolean(config.auto_mode);
|
||||
document.getElementById("restart-threshold").value =
|
||||
config.action_thresholds?.restart_ha ?? "";
|
||||
document.getElementById("network-threshold").value =
|
||||
config.action_thresholds?.check_network ?? "";
|
||||
if (currentView === 'settings') {
|
||||
const config = await fetchData('/config');
|
||||
const content = document.getElementById('settings-content');
|
||||
content.innerHTML = `
|
||||
<div class="label">Auto Mode</div>
|
||||
<div class="value">${config.auto_mode ? 'Enabled' : 'Disabled'}</div>
|
||||
<div class="label">Action Thresholds</div>
|
||||
<div class="value mono">${JSON.stringify(config.action_thresholds, null, 2)}</div>
|
||||
<button onclick="alert('Settings update not implemented in this demo')">Edit Configuration</button>
|
||||
`;
|
||||
}
|
||||
}
|
||||
|
||||
async function loadConfig() {
|
||||
const response = await fetch("/config", {cache: "no-store"});
|
||||
populateConfig(await response.json());
|
||||
}
|
||||
// Initial load
|
||||
refreshData();
|
||||
// Poll for updates
|
||||
setInterval(refreshData, pollInterval);
|
||||
|
||||
async function saveConfig() {
|
||||
const restartThreshold = Number(document.getElementById("restart-threshold").value);
|
||||
const networkThreshold = Number(document.getElementById("network-threshold").value);
|
||||
if (!Number.isFinite(restartThreshold) || !Number.isFinite(networkThreshold)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const config = {
|
||||
auto_mode: document.getElementById("config-auto-mode").checked,
|
||||
action_thresholds: {
|
||||
...(currentConfig.action_thresholds || {}),
|
||||
restart_ha: restartThreshold,
|
||||
check_network: networkThreshold
|
||||
},
|
||||
default_threshold: currentConfig.default_threshold ?? 0.9,
|
||||
allowed_auto_actions: currentConfig.allowed_auto_actions || ["restart_ha"]
|
||||
};
|
||||
|
||||
await fetch("/config", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify(config)
|
||||
});
|
||||
populateConfig(config);
|
||||
}
|
||||
|
||||
document.getElementById("stop").addEventListener("click", async () => {
|
||||
await fetch("/command", {
|
||||
method: "POST",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({command: "stop"})
|
||||
});
|
||||
});
|
||||
document.getElementById("save-config").addEventListener("click", saveConfig);
|
||||
document.getElementById("cmd").addEventListener("keydown", (event) => {
|
||||
if (event.key === "Enter") send();
|
||||
});
|
||||
|
||||
loadConfig();
|
||||
fetchServices();
|
||||
fetchLogs();
|
||||
setInterval(fetchLogs, 1000);
|
||||
setInterval(fetchServices, 10000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
77
webui/web.py
77
webui/web.py
|
|
@ -5,6 +5,9 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|||
from pathlib import Path
|
||||
|
||||
|
||||
STATE_DIR = Path("/opt/homelab/state")
|
||||
EVENTS_DIR = Path("/opt/homelab/events")
|
||||
WORLD_DIR = Path("/opt/homelab/world")
|
||||
EVENT_LOG = Path("/tmp/agent-events.log")
|
||||
STATIC_DIR = Path(__file__).parent
|
||||
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
|
||||
|
|
@ -101,8 +104,18 @@ def send_event(event):
|
|||
redis_command("LPUSH", "events", json.dumps(event))
|
||||
|
||||
|
||||
def read_json_file(path, default=None):
|
||||
if not path.exists():
|
||||
return default if default is not None else []
|
||||
try:
|
||||
return json.loads(path.read_text())
|
||||
except Exception:
|
||||
return default if default is not None else []
|
||||
|
||||
|
||||
def current_config():
|
||||
config = json.loads(json.dumps(DEFAULT_CONFIG))
|
||||
# We still keep reading from EVENT_LOG for auto_config if it's there
|
||||
for line in tail_lines(EVENT_LOG, 500):
|
||||
try:
|
||||
event = json.loads(line)
|
||||
|
|
@ -117,16 +130,38 @@ def current_config():
|
|||
return config
|
||||
|
||||
|
||||
def current_nodes():
|
||||
return read_json_file(STATE_DIR / "nodes.json")
|
||||
|
||||
|
||||
def current_services():
|
||||
services = []
|
||||
for line in tail_lines(EVENT_LOG, 1000):
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if event.get("type") == "services_state":
|
||||
services = event.get("services", [])
|
||||
return services
|
||||
return read_json_file(STATE_DIR / "services.json")
|
||||
|
||||
|
||||
def current_deployments():
|
||||
return read_json_file(STATE_DIR / "deployments.json")
|
||||
|
||||
|
||||
def current_incidents():
|
||||
return read_json_file(STATE_DIR / "incidents.json")
|
||||
|
||||
|
||||
def current_recommendations():
|
||||
return read_json_file(STATE_DIR / "recommendations.json")
|
||||
|
||||
|
||||
def current_summary():
|
||||
return read_json_file(STATE_DIR / "runtime-summary.json", default={})
|
||||
|
||||
|
||||
def current_events():
|
||||
events = []
|
||||
if EVENTS_DIR.exists():
|
||||
for f in EVENTS_DIR.glob("*.json"):
|
||||
data = read_json_file(f)
|
||||
if data:
|
||||
events.append(data)
|
||||
return sorted(events, key=lambda x: x.get("timestamp", 0), reverse=True)
|
||||
|
||||
|
||||
def send_json(status, payload, handler):
|
||||
|
|
@ -144,10 +179,34 @@ class Handler(BaseHTTPRequestHandler):
|
|||
send_json(200, current_config(), self)
|
||||
return
|
||||
|
||||
if self.path == "/nodes":
|
||||
send_json(200, current_nodes(), self)
|
||||
return
|
||||
|
||||
if self.path == "/services":
|
||||
send_json(200, current_services(), self)
|
||||
return
|
||||
|
||||
if self.path == "/deployments":
|
||||
send_json(200, current_deployments(), self)
|
||||
return
|
||||
|
||||
if self.path == "/incidents":
|
||||
send_json(200, current_incidents(), self)
|
||||
return
|
||||
|
||||
if self.path == "/recommendations":
|
||||
send_json(200, current_recommendations(), self)
|
||||
return
|
||||
|
||||
if self.path == "/summary":
|
||||
send_json(200, current_summary(), self)
|
||||
return
|
||||
|
||||
if self.path == "/events":
|
||||
send_json(200, current_events(), self)
|
||||
return
|
||||
|
||||
if self.path == "/logs":
|
||||
print("LOGS endpoint called", flush=True)
|
||||
body = ("\n".join(tail_lines(EVENT_LOG, 200)) + "\n").encode("utf-8")
|
||||
|
|
|
|||
Loading…
Reference in a new issue