test(evals): measure Java code intelligence as the user sees it

`pnpm eval` drives the production workbench in Chromium against a release
`openidea-server` and jdtls, and judges each scenario on what the user sees: time to
feedback, to the first result and to the final one, and the results against
`evals/java/golden/`. Scenarios cover navigation, usages, implementations, gutter
markers, code lenses, diagnostics, completion, quick documentation and rename, in
`fixtures/java-app` and a pinned spring-petclinic, warm and cold. The report
(`target/eval/report.json` / `.md`) names each failure's kind, a repro command and the
server commands in the slowest run's window; `--save-baseline` compares later runs.

- The frontend gets `data-*` hooks the scenarios read: usage / chooser rows carry their
  location, the usages popup its loading state and scope, a hint whether it is progress
  and what it answers.
- `smoke/tauriShim.ts`: the in-page Tauri runtime shared by the jsdom smoke tests and
  the evals; `perf/lib/bundle.ts` builds the perf bundle for both.
- `.github/tool-versions.env` pins the language-server versions for Language smoke and
  the evals.
- `.agents/skills/eval`: the measure → fix → prove loop for agents.

.agents/skills/eval/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: eval
+description: Run the code-intelligence evals (`pnpm eval`), pick the worst failing scenario, diagnose it from the attached timeline, fix it, prove the fix with before / after numbers measured on this machine without regressing other scenarios, and open one PR. Use for "跑 eval", "优化 Java 代码智能", "eval 失败了", an issue labelled `eval`, or any change to Java navigation / usages / gutter markers that needs numbers.
+---
+
+# Eval loop: measure → fix → prove
+
+`pnpm eval` drives the production workbench in Chromium against a release `openidea-server`
+with real language servers (Java / jdtls for now) and judges every scenario on what the user
+sees: time to feedback, to the first result, to the final one, and the results themselves vs
+`evals/<language>/golden/`. One machine, before and after: numbers from different machines or
+from the nightly host don't judge a fix.
+
+## 0. Preconditions
+
+- `jdtls` on PATH and `JAVA_HOME` pointing at a JDK of the pinned `JAVA` version (projects
+  compile against it, and the JDK scenarios' results are in its sources; jdtls itself runs on
+  a newer installed JDK when it needs one). Pinned versions: `.github/tool-versions.env`. A
+  mismatch prints a warning and the report records both: say so in the PR when it applies.
+- First run builds `cargo build --release -p ide-host --bin openidea-server` and the perf
+  bundle; later iterations of a frontend-only change can use `--no-build` after a
+  `pnpm exec vite build --mode perf --outDir target/eval/dist`, a backend-only one after the
+  cargo build alone.
+- Run evals one at a time on this machine, with nothing heavy (builds, other evals) beside them.
+
+## 1. Baseline on the base commit
+
+```sh
+git switch --detach origin/main     # or the fix branch's merge base
+pnpm eval --save-baseline           # target/eval/baseline.json, this machine only
+```
+
+Read `target/eval/report.json`. Each verdict has `status`, `failures` (`kind`: `error`,
+`timeout`, `degraded`, `correctness`, `budget`, `golden`), `regressions`, `repro` (the command
+that reruns it) and, for failures, `diagnostics`: every command sent to the server in the
+slowest run's window (`cmd`, `start`, `ms`, `ok`, `bytes`, `items`), background tasks, server
+status changes and the server's log tail.
+
+## 2. Pick one problem
+
+Worst first: wrong or missing results and silent degradation, then timeouts, then the
+feedback budget (100 ms), then the slowest `completeMs`. One problem per PR. Say which
+scenario and metric, with its value vs budget.
+
+## 3. Diagnose from the timeline
+
+- Commands in the window ordered by `start`: what ran, what waited on what, what's serial
+  that could overlap, which request returned many `items`/`bytes`, which failed.
+- `feedbackMs` far above 16 ms with a fast `firstResultMs`: the UI shows nothing while it
+  waits (a hint or popup that only appears after a delay).
+- A task still running across the window: background work competing with the request.
+- For the frontend side rerun with `--trace` and open `target/eval/traces/<id>@<cond>.json`
+  in Chrome's Performance panel.
+- Name the cause at class level (as in `issue-fix`): it usually affects sibling scenarios.
+
+## 4. Fix, then rerun that scenario until it passes
+
+```sh
+pnpm eval --only <scenario> --condition <warm|cold>   # the verdict's `repro`
+```
+
+Correctness is enforced, so a faster answer that drops, truncates or serves stale results
+fails. Don't trade a budget miss for a degraded answer.
+
+## 5. Full suite, then PR
+
+```sh
+pnpm eval                           # vs the baseline from step 1: no new failures or regressions
+```
+
+PR per `issue-fix` §6, plus in **How it was verified** the before / after table from
+`target/eval/report.md` (both on this machine, commits named) and the tool versions line.
+
+## Guard rails
+
+- Goldens (`evals/*/golden/`) and budgets (`evals/budgets.json`) are human-owned. Don't edit
+  them, and don't run `--update-golden`, to make a scenario pass. When a golden looks wrong
+  (a tool bump, a deliberate behaviour change), stop and say so: a PR that touches them gets
+  `needs-decision`.
+- Don't weaken a scenario's checks or timeouts in `evals/<language>/scenarios.ts` to pass.
+  Adding a scenario (for a case the sweep or a user found) is fine; record its golden with
+  `--update-golden --only <id>` and ask a person to review the file.
+- A flaky timing (passes on rerun) is not a regression: rerun once before acting on one.

.claude/skills/eval

@@ -0,0 +1 @@
+../../.agents/skills/eval

.github/tool-versions.env

@@ -0,0 +1,14 @@
+# Language server and tool versions for Language smoke (.github/workflows/language-smoke.yml)
+# and the code-intelligence evals (pnpm eval). Pinned so a release can't fail an unrelated
+# PR or move a golden result; bump them deliberately (and re-record goldens: pnpm eval --update-golden).
+VTSLS=0.3.0
+TYPESCRIPT=6.0.3
+BASEDPYRIGHT=1.40.2
+RUFF=0.17.3
+GOPLS=v0.24.0
+JDTLS=1.63.0-202610191322
+KOTLIN_LSP=263.7084.0
+GRADLE=8.14.3
+INTELEPHENSE=1.18.5
+# The JDK projects compile against (JAVA_HOME): library navigation opens its sources.
+JAVA=21

.github/workflows/language-smoke.yml

@@ -24,6 +24,7 @@ on:
       - "package.json"
       - "pnpm-lock.yaml"
       - ".github/workflows/language-smoke.yml"
+      - ".github/tool-versions.env"
   pull_request:
     branches: [main]
     types: [opened, synchronize, reopened]
@@ -38,16 +39,6 @@ concurrency:
 env:
   CARGO_TERM_COLOR: always
   CARGO_INCREMENTAL: 0
-  # Pinned so a server release can't fail an unrelated PR; bump them deliberately.
-  VTSLS: 0.3.0
-  TYPESCRIPT: 6.0.3
-  BASEDPYRIGHT: 1.40.2
-  RUFF: 0.17.3
-  GOPLS: v0.24.0
-  JDTLS: 1.63.0-202610191322
-  KOTLIN_LSP: 263.7084.0
-  GRADLE: 8.14.3
-  INTELEPHENSE: 1.18.5
 
 jobs:
   smoke:
@@ -61,6 +52,8 @@ jobs:
     timeout-minutes: 40
     steps:
       - uses: actions/checkout@v4
+      # Tool versions, shared with the evals (pnpm eval).
+      - run: grep -E '^[A-Z0-9_]+=' .github/tool-versions.env >> "$GITHUB_ENV"
       - uses: dtolnay/rust-toolchain@stable
         with: { components: "rust-analyzer, rustfmt" }
       # One build cache for every language; main saves it.
@@ -96,7 +89,7 @@ jobs:
 
       - if: matrix.language == 'java' || matrix.language == 'kotlin'
         uses: actions/setup-java@v4
-        with: { distribution: temurin, java-version: "21" }
+        with: { distribution: temurin, java-version: "${{ env.JAVA }}" }
       - if: matrix.language == 'java'
         run: |
           mkdir -p "$RUNNER_TEMP/jdtls"

.gitignore

@@ -40,6 +40,7 @@ CLAUDE.local.md
 .claude/skills/*
 !.claude/skills/verify
 !.claude/skills/issue-fix
+!.claude/skills/eval
 
 # Local reference screenshots
 pycharm-screenshots/

.oxfmtrc.json

@@ -9,6 +9,7 @@
     "dist",
     "target",
     "src/bindings.ts",
+    "evals/*/golden",
     "src/editor/core/cm/angular/parser*.ts",
     "src/editor/core/cm/jsp/parser*.ts",
     "src/editor/core/cm/astro/parser*.ts",

AGENTS.md

@@ -24,8 +24,8 @@ Visual style: dark, low-contrast greys, one cool accent
   Plugin docs: `plugins/README.md`, `plugins/themes.md`.
 - `java/` — JDT LS extension bundles (`java-usages`, `kotlin-bridge`) and the patched
   Lombok agent, embedded into the app by `ide-sdk` / `ide-tools`.
-- `e2e/` — Playwright UI tests (`pnpm e2e`). `fixtures/` — sample projects per language
-  for manual and automated testing.
+- `e2e/` — Playwright UI tests (`pnpm e2e`). `evals/` — code-intelligence evals (`pnpm eval`).
+  `fixtures/` — sample projects per language for manual and automated testing.
 - `docs/` — the Mintlify documentation site (`docs.json` navigation, `en/` and `zh/` pages);
   preview with `npx mint dev` in `docs/`.
 
@@ -59,9 +59,12 @@ Unclear product decisions → `needs-decision` and stop.
 
 ## Skills
 
+Do not use Impeccable or its related skills in this repository, including globally installed copies.
+
 Shared agent skills live in `.agents/skills/` (`.claude/skills/` links to them):
 `verify` drives the running app through Tauri MCP and captures screenshot evidence;
-`issue-fix` takes an issue from reproduction to one PR.
+`issue-fix` takes an issue from reproduction to one PR;
+`eval` takes a failing code-intelligence eval to a fix with before / after numbers.
 
 ## Commands
 
@@ -81,6 +84,11 @@ Shared agent skills live in `.agents/skills/` (`.claude/skills/` links to them):
   `pnpm gen:bindings`.
 - `pnpm perf` (UI interaction latency / frames / leaks), `pnpm perf:rust` (criterion
   benches vs budgets). Budgets: `perf/budgets.json`.
+- `pnpm eval [--only <id>] [--runs n] [--save-baseline]` — Java code intelligence (navigation,
+  usages, gutter markers) in the production workbench against a release `openidea-server` and
+  jdtls: time to feedback / first result / complete, results vs `evals/<language>/golden/`.
+  Report: `target/eval/report.json`. Goldens and `evals/budgets.json` are changed by people only.
+  Tool versions: `.github/tool-versions.env` (shared with Language smoke).
 - `pnpm release:prepare <version>` — the release commit's changes: app version everywhere,
   `Cargo.lock`, and the plugin API snapshot when `wit/` moved to a new version.
 

Cargo.lock

@@ -4303,6 +4303,7 @@ dependencies = [
  "ide-core",
  "ide-http",
  "ide-ssh",
+ "notify",
  "oracledb",
  "parking_lot",
  "reqwest",

crates/ide-agent/src/tests.rs

@@ -492,7 +492,7 @@ for line in sys.stdin:
         }
         let concurrent = async {
             if outcome == "session_report" {
-                tokio::time::timeout(Duration::from_secs(2), async {
+                tokio::time::timeout(Duration::from_secs(10), async {
                     while !root.join("initialized").exists() {
                         tokio::time::sleep(Duration::from_millis(10)).await;
                     }
@@ -564,7 +564,7 @@ for line in sys.stdin:
             };
             assert_eq!(subtypes, expected, "{outcome}");
             let pid = std::fs::read_to_string(root.join("pid")).unwrap();
-            tokio::time::timeout(Duration::from_secs(2), async {
+            tokio::time::timeout(Duration::from_secs(10), async {
                 while std::process::Command::new("kill").args(["-0", pid.trim()]).output().unwrap().status.success() {
                     tokio::time::sleep(Duration::from_millis(10)).await;
                 }

crates/ide-core/src/settings.rs

@@ -1203,6 +1203,8 @@ pub struct MavenSettings {
     pub threads: String,
     /// Active Maven profiles (`-P`), comma-separated; empty leaves profiles to the POM/settings.
     pub active_profiles: String,
+    /// Delegate IDE build/run actions to Maven: launches build with Maven (`javac`) instead of jdtls (ECJ).
+    pub delegate_build: bool,
 }
 
 impl MavenSettings {
@@ -1707,10 +1709,12 @@ mod maven_linkage_tests {
     fn maven_runner_jre_defaults_and_round_trips() {
         let old: MavenSettings = serde_json::from_str(r#"{"home":"/opt/maven"}"#).unwrap();
         assert!(old.runner_jre.is_empty());
+        assert!(!old.delegate_build);
         assert!(MavenSettings::default().runner_jre.is_empty());
-        let settings = MavenSettings { runner_jre: "/opt/jdk-21".into(), ..old };
+        let settings = MavenSettings { runner_jre: "/opt/jdk-21".into(), delegate_build: true, ..old };
         let encoded = serde_json::to_string(&settings).unwrap();
         assert!(encoded.contains(r#""runnerJre":"/opt/jdk-21""#));
+        assert!(encoded.contains(r#""delegateBuild":true"#));
         assert_eq!(serde_json::from_str::<MavenSettings>(&encoded).unwrap(), settings);
     }
 

crates/ide-db/Cargo.toml

@@ -17,6 +17,7 @@ blake3.workspace = true
 tokio.workspace = true
 tracing.workspace = true
 parking_lot.workspace = true
+notify.workspace = true
 futures.workspace = true
 sqlx.workspace = true
 tiberius.workspace = true

crates/ide-db/src/lib.rs

@@ -12,6 +12,8 @@ pub mod secrets;
 pub mod source;
 pub mod split;
 pub mod sql;
+mod watch;
+pub use watch::SourcesWatch;
 
 pub use conn::{Held, Pool, QueryResult, ResultColumn};
 pub use history::HistoryEntry;
@@ -235,6 +237,9 @@ pub struct DbManager {
     /// Manual-commit data sources, each with the connection of its open
     /// transaction (taken by the first statement after a commit / rollback).
     sessions: Mutex<HashMap<String, Session>>,
+    /// The data sources as last read from disk, for [`DbManager::reconcile_sources`]
+    /// (`None` until [`DbManager::watch_sources`]).
+    on_disk: Mutex<Option<Vec<DataSource>>>,
 }
 
 type Session = Arc<tokio::sync::Mutex<Option<Held>>>;
@@ -362,6 +367,7 @@ impl DbManager {
             tunnel_open: tokio::sync::Mutex::new(()),
             ssh,
             session_passwords: Mutex::default(),
+            on_disk: Mutex::default(),
             sessions: Mutex::default(),
         }
     }
@@ -474,9 +480,45 @@ impl DbManager {
                 }
             }
         }
+        // Our own write is not reported as an edit on disk.
+        if let Some(known) = self.on_disk.lock().as_mut() {
+            *known = self.sources().unwrap_or(sources);
+        }
         Ok(())
     }
 
+    /// Watches the data source files (the project's and the global one) for edits made
+    /// outside the IDE; `on_change` runs on a background thread once they settle, and
+    /// should call [`DbManager::reconcile_sources`]. Stops when the result is dropped.
+    pub fn watch_sources(&self, on_change: Box<dyn Fn() + Send>) -> IdeResult<SourcesWatch> {
+        *self.on_disk.lock() = Some(self.sources().unwrap_or_default());
+        SourcesWatch::start(&self.root, self.global.as_ref().map(|(dir, _)| dir.as_path()), on_change)
+    }
+
+    /// Re-reads the data sources after their files changed on disk: connections of
+    /// removed or changed sources close, as on [`DbManager::save_sources`] (saved
+    /// passwords stay, the source may come back). Returns whether the list the tree
+    /// shows changed, or the files no longer parse (the tree reports the error).
+    pub async fn reconcile_sources(&self) -> bool {
+        let now = match self.sources() {
+            Ok(now) => now,
+            Err(e) => {
+                tracing::warn!("data sources: {}", e.message);
+                return true;
+            }
+        };
+        let before = self.on_disk.lock().replace(now.clone()).unwrap_or_default();
+        if before == now {
+            return false;
+        }
+        for o in &before {
+            if now.iter().find(|s| s.id == o.id) != Some(o) {
+                self.disconnect_source(&o.id).await;
+            }
+        }
+        true
+    }
+
     /// Remembers `password` for the session, and in the saved passwords when
     /// the data source saves passwords (`None` forgets it).
     pub async fn set_password(&self, id: &str, password: Option<String>, save: bool) -> IdeResult<()> {

crates/ide-db/src/source.rs

@@ -8,7 +8,7 @@ use serde::{Deserialize, Serialize};
 use specta::Type;
 use std::path::Path;
 
-const FILE: &str = "data-sources.toml";
+pub(crate) const FILE: &str = "data-sources.toml";
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize, Type)]
 #[serde(rename_all = "camelCase")]

crates/ide-db/src/watch.rs

@@ -0,0 +1,151 @@
+//! Data source files edited outside the Database tool window (by hand, by a coding
+//! agent, by `git checkout`) are picked up while the project is open, as IDEA does
+//! for `dataSources.xml`. The project file tree's watcher is no help here: it never
+//! reports a modification inside a gitignored `.openidea/`.
+
+use crate::source::FILE;
+use ide_core::{IdeError, IdeResult, project_dir};
+use notify::{RecommendedWatcher, RecursiveMode, Watcher};
+use parking_lot::Mutex;
+use std::path::{Path, PathBuf};
+use std::sync::{Arc, Weak, mpsc};
+use std::time::Duration;
+
+/// How long a burst of writes (an editor's save, an agent's rewrite) settles before
+/// the files are read.
+const SETTLE: Duration = Duration::from_millis(200);
+
+/// Watches `<root>/.openidea/data-sources.toml` and `<global>/data-sources.toml`;
+/// stops when dropped.
+pub struct SourcesWatch {
+    _watcher: Arc<Mutex<RecommendedWatcher>>,
+}
+
+impl SourcesWatch {
+    /// Calls `on_change` once a burst of changes to either file settled, on a
+    /// background thread.
+    pub fn start(root: &Path, global: Option<&Path>, on_change: Box<dyn Fn() + Send>) -> IdeResult<Self> {
+        let config: PathBuf = root.join(project_dir::DIR);
+        let (tx, rx) = mpsc::channel::<()>();
+        let relevant = {
+            let config = config.clone();
+            move |p: &Path| p.file_name().is_some_and(|n| n == FILE) || p == config
+        };
+        let watcher = notify::recommended_watcher(move |res: notify::Result<notify::Event>| {
+            if let Ok(e) = res
+                && e.paths.iter().any(|p| relevant(p))
+            {
+                let _ = tx.send(());
+            }
+        })
+        .map_err(|e| IdeError::internal(format!("data sources watcher: {e}")))?;
+        let watcher = Arc::new(Mutex::new(watcher));
+        {
+            let mut w = watcher.lock();
+            // The root (not recursively) sees `.openidea/` itself appear or go.
+            w.watch(root, RecursiveMode::NonRecursive)
+                .map_err(|e| IdeError::internal(format!("data sources watcher: {e}")))?;
+            if let Some(dir) = global.filter(|d| d.is_dir())
+                && let Err(e) = w.watch(dir, RecursiveMode::NonRecursive)
+            {
+                tracing::warn!("global data sources watcher: {e}");
+            }
+        }
+        let mut watching = watch_config(&watcher, &config, false);
+        let weak: Weak<Mutex<RecommendedWatcher>> = Arc::downgrade(&watcher);
+        std::thread::Builder::new().name("openidea-db-sources-watch".into()).spawn(move || {
+            while rx.recv().is_ok() {
+                // Wait until the writes stop.
+                loop {
+                    match rx.recv_timeout(SETTLE) {
+                        Ok(()) => {}
+                        Err(mpsc::RecvTimeoutError::Timeout) => break,
+                        Err(mpsc::RecvTimeoutError::Disconnected) => return,
+                    }
+                }
+                let Some(watcher) = weak.upgrade() else { return };
+                watching = watch_config(&watcher, &config, watching);
+                drop(watcher);
+                on_change();
+            }
+        })?;
+        Ok(Self { _watcher: watcher })
+    }
+}
+
+/// Watches `.openidea/` once it exists (again: a removed folder loses its watch);
+/// returns whether it is watched.
+fn watch_config(watcher: &Mutex<RecommendedWatcher>, config: &Path, watching: bool) -> bool {
+    if !config.is_dir() {
+        return false;
+    }
+    if watching {
+        return true;
+    }
+    match watcher.lock().watch(config, RecursiveMode::NonRecursive) {
+        Ok(()) => true,
+        Err(e) => {
+            tracing::warn!("data sources watcher {}: {e}", config.display());
+            false
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use std::time::Instant;
+
+    fn wait_for(count: &AtomicUsize, at_least: usize) -> bool {
+        let deadline = Instant::now() + Duration::from_secs(10);
+        while Instant::now() < deadline {
+            if count.load(Ordering::SeqCst) >= at_least {
+                return true;
+            }
+            std::thread::sleep(Duration::from_millis(20));
+        }
+        false
+    }
+
+    fn counting() -> (Arc<AtomicUsize>, Box<dyn Fn() + Send>) {
+        let count = Arc::new(AtomicUsize::new(0));
+        let c = count.clone();
+        (count, Box::new(move || _ = c.fetch_add(1, Ordering::SeqCst)))
+    }
+
+    #[test]
+    fn reports_project_and_global_files_and_a_config_folder_created_later() {
+        let tmp = tempfile::tempdir().unwrap();
+        let root = dunce::canonicalize(tmp.path()).unwrap();
+        let (project, global) = (root.join("p"), root.join("app"));
+        std::fs::create_dir_all(&project).unwrap();
+        std::fs::create_dir_all(&global).unwrap();
+        let (count, on_change) = counting();
+        let _watch = SourcesWatch::start(&project, Some(&global), on_change).unwrap();
+
+        // No `.openidea/` yet: it is created, then the file written into it.
+        std::fs::create_dir_all(project.join(".openidea")).unwrap();
+        assert!(wait_for(&count, 1));
+        let seen = count.load(Ordering::SeqCst);
+        std::fs::write(project.join(".openidea/data-sources.toml"), "").unwrap();
+        assert!(wait_for(&count, seen + 1), "project file");
+
+        let seen = count.load(Ordering::SeqCst);
+        std::fs::write(global.join("data-sources.toml"), "").unwrap();
+        assert!(wait_for(&count, seen + 1), "global file");
+    }
+
+    #[test]
+    fn ignores_other_files() {
+        let tmp = tempfile::tempdir().unwrap();
+        let root = dunce::canonicalize(tmp.path()).unwrap();
+        std::fs::create_dir_all(root.join(".openidea")).unwrap();
+        let (count, on_change) = counting();
+        let _watch = SourcesWatch::start(&root, None, on_change).unwrap();
+        std::fs::write(root.join(".openidea/settings.toml"), "").unwrap();
+        std::fs::write(root.join("main.py"), "").unwrap();
+        std::thread::sleep(SETTLE * 3);
+        assert_eq!(count.load(Ordering::SeqCst), 0);
+    }
+}

crates/ide-db/tests/sqlite.rs

@@ -342,3 +342,31 @@ async fn applies_driver_properties_and_the_startup_script_on_connect() -> IdeRes
     assert_eq!(db.schemas("s1").await?, vec!["main"]);
     Ok(())
 }
+
+#[tokio::test]
+async fn data_sources_edited_on_disk_close_their_connections() -> IdeResult<()> {
+    let (tmp, db) = setup().await;
+    std::fs::File::create(tmp.path().join("other.db")).unwrap();
+    let (tx, rx) = std::sync::mpsc::channel();
+    let _watch = db.watch_sources(Box::new(move || _ = tx.send(())))?;
+    db.pool("s1").await?;
+    assert_eq!(db.connected(), vec!["s1"]);
+
+    // Our own save is no edit on disk.
+    let mut sources = db.sources()?;
+    sources[0].name = "renamed".into();
+    db.save_sources(sources.clone()).await?;
+    assert!(!db.reconcile_sources().await);
+
+    // Edited outside the IDE (by hand, by an agent): the watcher reports it, and the
+    // connection made with the old settings closes.
+    db.pool("s1").await?;
+    sources[0].file = Some("other.db".into());
+    ide_db::source::save(tmp.path(), &sources)?;
+    rx.recv_timeout(std::time::Duration::from_secs(10)).expect("the watcher reports the edit");
+    assert!(db.reconcile_sources().await);
+    assert!(db.connected().is_empty());
+    assert_eq!(db.sources()?[0].file.as_ref().map(|f| f.0.clone()), Some("other.db".into()));
+    assert!(!db.reconcile_sources().await);
+    Ok(())
+}

crates/ide-host/src/notebook.rs

@@ -84,6 +84,14 @@ pub struct NotebookService {
     registry: Arc<Mutex<Registry>>,
     kernels: Arc<KernelManager>,
     sink: SharedSink,
+    observer: parking_lot::RwLock<Option<Arc<dyn NotebookObserver>>>,
+}
+
+/// Learns each open notebook's contents after every change of them, and when it closes (the
+/// language servers' copy of its code cells).
+pub trait NotebookObserver: Send + Sync {
+    fn changed(&self, path: &Path, snapshot: &NotebookSnapshot);
+    fn closed(&self, path: &Path);
 }
 
 impl NotebookService {
@@ -121,7 +129,23 @@ impl NotebookService {
                 }
             }),
         );
-        Arc::new(Self { registry, kernels: Arc::new(kernels), sink })
+        Arc::new(Self { registry, kernels: Arc::new(kernels), sink, observer: Default::default() })
+    }
+
+    pub fn set_observer(&self, observer: Arc<dyn NotebookObserver>) {
+        *self.observer.write() = Some(observer);
+    }
+
+    /// Called with the document's state locked, so observers see its changes in order.
+    fn changed(&self, path: &Path, snapshot: &NotebookSnapshot) {
+        if let Some(observer) = self.observer.read().clone() {
+            observer.changed(path, snapshot);
+        }
+    }
+
+    /// The file of an open notebook.
+    pub fn path_of(&self, id: &str) -> IdeResult<PathBuf> {
+        Ok(self.resolve(id)?.path.clone())
     }
 
     pub fn open(&self, path: &Path) -> IdeResult<NotebookSnapshot> {
@@ -136,6 +160,7 @@ impl NotebookService {
         }
         let document = NotebookDocument::load(&path)?;
         let initial = document.snapshot();
+        self.changed(&path, &initial);
         let id = initial.id.clone();
         registry.paths.insert(path.clone(), id.clone());
         registry.documents.insert(
@@ -173,7 +198,9 @@ impl NotebookService {
         let document = self.resolve(id)?;
         let mut state = document.state.lock();
         mutable(&state)?;
-        state.document.edit(version, edit)
+        let snapshot = state.document.edit(version, edit)?;
+        self.changed(&document.path, &snapshot);
+        Ok(snapshot)
     }
 
     pub fn save(&self, id: &str) -> IdeResult<NotebookSnapshot> {
@@ -190,7 +217,9 @@ impl NotebookService {
         if state.active.is_some() {
             return Err(IdeError::unavailable("stop notebook execution before undo or redo"));
         }
-        if redo { state.document.redo() } else { state.document.undo() }
+        let snapshot = if redo { state.document.redo() } else { state.document.undo() }?;
+        self.changed(&document.path, &snapshot);
+        Ok(snapshot)
     }
 
     pub async fn interrupt(&self, id: &str) -> IdeResult<()> {
@@ -238,7 +267,10 @@ impl NotebookService {
                 snapshot(&document)
             } else {
                 self.kernels.shutdown_kernel(&document.path).await?;
-                document.state.lock().document.reload()
+                let mut state = document.state.lock();
+                let snapshot = state.document.reload()?;
+                self.changed(&document.path, &snapshot);
+                Ok(snapshot)
             }
         }
         .await;
@@ -284,6 +316,10 @@ impl NotebookService {
         let mut registry = self.registry.lock();
         registry.documents.remove(id);
         registry.paths.remove(&document.path);
+        drop(registry);
+        if let Some(observer) = self.observer.read().clone() {
+            observer.closed(&document.path);
+        }
         Ok(())
     }
 
@@ -572,6 +608,49 @@ mod tests {
         KernelOutput { kind: kind.into(), content, msg_id: "actual-parent-request".into() }
     }
 
+    #[tokio::test]
+    async fn the_observer_follows_every_change_of_an_open_notebook_and_its_close() {
+        #[derive(Default)]
+        struct Seen(Mutex<Vec<String>>);
+        impl NotebookObserver for Seen {
+            fn changed(&self, path: &Path, snapshot: &NotebookSnapshot) {
+                let sources: Vec<&str> = snapshot.cells.iter().map(|c| c.source.as_str()).collect();
+                let name = path.file_name().unwrap().to_string_lossy();
+                self.0.lock().push(format!("{name}: {}", sources.join(" | ")));
+            }
+            fn closed(&self, path: &Path) {
+                self.0.lock().push(format!("closed {}", path.file_name().unwrap().to_string_lossy()));
+            }
+        }
+        let (dir, service, initial) = fixture();
+        let seen = Arc::new(Seen::default());
+        service.set_observer(seen.clone());
+        service.close(&initial.id).await.unwrap();
+        let opened = service.open(&dir.path().join("session.ipynb")).unwrap();
+        assert_eq!(
+            service.path_of(&opened.id).unwrap(),
+            dunce::canonicalize(dir.path().join("session.ipynb")).unwrap()
+        );
+        let edit = NotebookEdit::SetSource { cell: "second".into(), source: "print(x + 1)".into() };
+        let edited = service.edit(&opened.id, opened.version, edit).unwrap();
+        service.undo(&opened.id, false).unwrap();
+        // A second view of the open notebook changes nothing.
+        service.open(&dir.path().join("session.ipynb")).unwrap();
+        assert!(edited.version > opened.version);
+        service.close(&opened.id).await.unwrap();
+        assert_eq!(
+            *seen.0.lock(),
+            [
+                "closed session.ipynb",
+                "session.ipynb: x = 1 | print(x) | notes",
+                "session.ipynb: x = 1 | print(x + 1) | notes",
+                "session.ipynb: x = 1 | print(x) | notes",
+                "closed session.ipynb",
+            ]
+        );
+        service.shutdown().await;
+    }
+
     #[tokio::test]
     async fn canonical_split_opens_share_incarnation_but_reopen_and_projects_do_not() {
         let (dir, service, initial) = fixture();

crates/ide-host/src/ops/java.rs

@@ -48,7 +48,7 @@ async fn resolve_launch(
     let build = policy != BuildPolicy::NoBuild;
     let full = policy == BuildPolicy::Rebuild;
     let build_kind = if full { compile::BuildKind::Rebuild } else { compile::BuildKind::Build };
-    let (model, external, delegated) = {
+    let (model, external, delegated, prefer_maven) = {
         let project = project.clone();
         let settings = settings.clone();
         let module = module.clone();
@@ -58,15 +58,17 @@ async fn resolve_launch(
             let non_java = build && compile::launch_has_non_java_sources(&project.root, &model, &module, test);
             let is_maven =
                 model.maven.as_ref().is_some_and(|m| m.projects.iter().any(|p| project.root.join(&p.dir) == module));
+            let prefer_maven = settings.languages.java.maven.delegate_build;
             let delegated = build
                 && delegates_to_external_build(
                     is_maven,
                     policy,
                     non_java,
                     cache::has_record(&project.root, &project.build_cache_dir(), &module),
+                    prefer_maven,
                 );
             let external = non_java || delegated;
-            (model, external, delegated)
+            (model, external, delegated, prefer_maven)
         })
         .await
         .map_err(|e| IdeError::internal(e.to_string()))?
@@ -79,6 +81,7 @@ async fn resolve_launch(
     tokio::task::spawn_blocking(move || {
         let _external_build = external_build;
         let built = if delegated {
+            let policy = delegated_policy(policy, prefer_maven, info.error_count);
             let outputs = project.java_launch_snapshots.lock().build(
                 &project.root,
                 &model,
@@ -169,6 +172,17 @@ async fn resolve_launch(
     .map_err(|e| IdeError::internal(e.to_string()))?
 }
 
+/// With Settings › Maven › Delegate IDE build/run actions to Maven, javac's verdict replaces
+/// JDT's. Classes JDT wrote for sources with errors look up to date to Maven's incremental
+/// compiler, so a launch JDT reports errors for recompiles everything.
+fn delegated_policy(policy: BuildPolicy, prefer_maven: bool, jdt_errors: u32) -> BuildPolicy {
+    if prefer_maven && jdt_errors > 0 && matches!(policy, BuildPolicy::Automatic | BuildPolicy::Make) {
+        BuildPolicy::Rebuild
+    } else {
+        policy
+    }
+}
+
 fn apply_external_build_result(info: &mut JavaLaunchInfo, failure: Option<String>, clean: bool) {
     info.build = Some(if let Some(failure) = failure {
         info.hints.push(failure);
@@ -241,8 +255,15 @@ fn apply_library_classpath(
 /// Whether a launch should be built by the project's build system instead of jdtls.
 /// A Rebuild has to clean the output; jdtls' full build only recompiles sources, so
 /// Maven modules always delegate for it, even when every source is Java.
-fn delegates_to_external_build(is_maven: bool, policy: BuildPolicy, non_java: bool, has_record: bool) -> bool {
-    is_maven && (policy == BuildPolicy::Rebuild || non_java || has_record)
+/// `prefer_maven` is Settings › Maven › Delegate IDE build/run actions to Maven.
+fn delegates_to_external_build(
+    is_maven: bool,
+    policy: BuildPolicy,
+    non_java: bool,
+    has_record: bool,
+    prefer_maven: bool,
+) -> bool {
+    is_maven && (prefer_maven || policy == BuildPolicy::Rebuild || non_java || has_record)
 }
 
 fn same_entry(a: &str, b: &str, windows: bool) -> bool {
@@ -321,14 +342,29 @@ pub(crate) fn register(d: &mut Dispatcher) {
 mod tests {
     use super::*;
 
+    #[test]
+    fn delegated_launches_with_jdt_errors_recompile_with_maven() {
+        assert_eq!(delegated_policy(BuildPolicy::Make, true, 1), BuildPolicy::Rebuild);
+        assert_eq!(delegated_policy(BuildPolicy::Automatic, true, 2), BuildPolicy::Rebuild);
+        assert_eq!(delegated_policy(BuildPolicy::Make, true, 0), BuildPolicy::Make);
+        assert_eq!(delegated_policy(BuildPolicy::Make, false, 1), BuildPolicy::Make);
+        assert_eq!(
+            delegated_policy(BuildPolicy::MakeAndContinueOnErrors, true, 1),
+            BuildPolicy::MakeAndContinueOnErrors
+        );
+    }
+
     #[test]
     fn rebuild_delegates_maven_builds_to_the_build_system() {
-        assert!(delegates_to_external_build(true, BuildPolicy::Rebuild, false, false));
-        assert!(delegates_to_external_build(true, BuildPolicy::Automatic, true, false));
-        assert!(delegates_to_external_build(true, BuildPolicy::Make, false, true));
-        assert!(!delegates_to_external_build(true, BuildPolicy::Automatic, false, false));
-        assert!(!delegates_to_external_build(true, BuildPolicy::Make, false, false));
-        assert!(!delegates_to_external_build(false, BuildPolicy::Rebuild, false, false));
+        assert!(delegates_to_external_build(true, BuildPolicy::Rebuild, false, false, false));
+        assert!(delegates_to_external_build(true, BuildPolicy::Automatic, true, false, false));
+        assert!(delegates_to_external_build(true, BuildPolicy::Make, false, true, false));
+        assert!(!delegates_to_external_build(true, BuildPolicy::Automatic, false, false, false));
+        assert!(!delegates_to_external_build(true, BuildPolicy::Make, false, false, false));
+        assert!(!delegates_to_external_build(false, BuildPolicy::Rebuild, false, false, false));
+        assert!(delegates_to_external_build(true, BuildPolicy::Automatic, false, false, true));
+        assert!(delegates_to_external_build(true, BuildPolicy::Make, false, false, true));
+        assert!(!delegates_to_external_build(false, BuildPolicy::Automatic, false, false, true));
     }
 
     #[test]

crates/ide-host/src/ops/lsp.rs

@@ -314,10 +314,12 @@ pub async fn lsp_goto(host: &Host, window: &str, doc: DocId, pos: Pos, kind: Got
         && let Some(plugins) = crate::plugins::editor_plugin_host()
     {
         let project = host.project(window)?;
-        let (path, version) = project.documents.identity(doc)?;
-        if let Some(found) = plugins.navigation_at(&project.root, &path, version, pos, Some(window), |p| {
-            project.documents.by_path(p).and_then(|id| project.documents.version(id).ok())
-        }) {
+        // A notebook cell is a language-server document only: no plugin navigates in it.
+        if let Ok((path, version)) = project.documents.identity(doc)
+            && let Some(found) = plugins.navigation_at(&project.root, &path, version, pos, Some(window), |p| {
+                project.documents.by_path(p).and_then(|id| project.documents.version(id).ok())
+            })
+        {
             return Ok(found.targets);
         }
     }

crates/ide-host/src/ops/notebook.rs

@@ -19,6 +19,13 @@ pub fn get(host: &Host, window: &str, id: String) -> IdeResult<NotebookSnapshot>
     host.project(window)?.notebooks.get(&id)
 }
 
+/// The language-server documents of the notebook's code cells (empty when no server syncs notebooks).
+pub fn cell_docs(host: &Host, window: &str, id: String) -> IdeResult<Vec<ide_lsp::NotebookCellDoc>> {
+    let project = host.project(window)?;
+    let path = project.notebooks.path_of(&id)?;
+    Ok(project.lsp.notebook_cells(&path))
+}
+
 pub async fn edit(
     host: &Host,
     window: &str,
@@ -132,6 +139,7 @@ impl Drop for RequestGuard<'_> {
 pub(crate) fn register(d: &mut Dispatcher) {
     d.add("notebook_open", |c| async move { open(&c.host, &c.window, c.arg("path")?).await });
     d.add("notebook_get", |c| async move { get(&c.host, &c.window, c.arg("id")?) });
+    d.add("notebook_cell_docs", |c| async move { cell_docs(&c.host, &c.window, c.arg("id")?) });
     d.add("notebook_edit", |c| async move {
         edit(&c.host, &c.window, c.arg("id")?, c.arg("version")?, c.arg("edit")?).await
     });

crates/ide-host/src/project.rs

@@ -69,6 +69,8 @@ pub struct Project {
     /// watchers for (a virtualenv elsewhere, a path dependency).
     lsp_watchers: Mutex<Vec<(PathBuf, Watcher)>>,
     pub(crate) _scratch_watcher: Option<Watcher>,
+    /// Picks up data sources edited on disk outside the Database tool window.
+    _db_sources_watch: Option<ide_db::SourcesWatch>,
     /// A LightEdit window's loose files ([`Host::open_light`]): global settings only,
     /// nothing indexed, no language servers, repositories or project services.
     pub light: bool,
@@ -536,10 +538,12 @@ impl Host {
         tracing::info!("opened {} in window {window} in {} ms {laps:?}", root.display(), started.elapsed().as_millis());
 
         let (db, http_history, agents) = self.project_services(&root, &project_key, &settings);
+        let db_sources_watch = db_sources_watch(&db, &sink, &self.runtime);
         let notebooks = crate::notebook::NotebookService::new(
             self.local_data_dir.join("notebook-runtime").join(uuid::Uuid::new_v4().to_string()),
             sink.clone(),
         );
+        notebooks.set_observer(Arc::new(NotebookLspObserver(lsp.clone())));
         ide_agent::registry::maintain(crate::ops::agent::registry_paths(self));
         let plugin_diagnostics =
             crate::plugins::diagnostics::Diagnostics::new(sink.clone(), self.runtime.clone(), &documents, &root);
@@ -576,6 +580,7 @@ impl Host {
             content_watchers: Mutex::new(content_watchers),
             lsp_watchers: Mutex::new(Vec::new()),
             _scratch_watcher: scratch_watcher,
+            _db_sources_watch: db_sources_watch,
             files_events_active,
             light: false,
         });
@@ -738,6 +743,7 @@ impl Host {
             content_watchers: Mutex::new(Vec::new()),
             lsp_watchers: Mutex::new(Vec::new()),
             _scratch_watcher: None,
+            _db_sources_watch: None,
             files_events_active: Arc::new(AtomicBool::new(false)),
             light: true,
             root: dir,
@@ -790,6 +796,28 @@ fn documents(sink: &SharedSink, languages: &Arc<LanguageRegistry>, root: Option<
     )
 }
 
+/// Data sources edited on disk outside the Database tool window (by hand, by a coding
+/// agent): connections made with the old settings close and the tree re-reads them,
+/// without reopening the project.
+fn db_sources_watch(
+    db: &Arc<ide_db::DbManager>,
+    sink: &SharedSink,
+    runtime: &tokio::runtime::Handle,
+) -> Option<ide_db::SourcesWatch> {
+    let (weak, sink, runtime) = (Arc::downgrade(db), sink.clone(), runtime.clone());
+    db.watch_sources(Box::new(move || {
+        let Some(db) = weak.upgrade() else { return };
+        let sink = sink.clone();
+        runtime.spawn(async move {
+            if db.reconcile_sources().await {
+                sink.emit(IdeEvent::DbSourcesChanged);
+            }
+        });
+    }))
+    .inspect_err(|e| tracing::warn!("{}", e.message))
+    .ok()
+}
+
 fn light_watcher(
     dir: &Path,
     sink: &SharedSink,
@@ -1578,6 +1606,25 @@ impl DocumentObserver for LspObserver {
     }
 }
 
+/// Feeds the code cells of open notebooks to the Python language servers that sync notebooks.
+struct NotebookLspObserver(Arc<LspManager>);
+
+impl crate::notebook::NotebookObserver for NotebookLspObserver {
+    fn changed(&self, path: &Path, snapshot: &ide_notebook::NotebookSnapshot) {
+        let cells: Vec<ide_lsp::NotebookCell> = snapshot
+            .cells
+            .iter()
+            .filter(|c| c.cell_type == "code")
+            .map(|c| ide_lsp::NotebookCell { id: c.id.clone(), source: c.source.clone() })
+            .collect();
+        // Notebooks run Python kernels only.
+        self.0.sync_notebook(path, &LanguageId::new("python"), &cells);
+    }
+    fn closed(&self, path: &Path) {
+        self.0.close_notebook(path);
+    }
+}
+
 /// Feeds document edits into the per-project syntax tree cache.
 struct SyntaxObserver(Arc<SyntaxCache>);
 

crates/ide-lsp/src/caps.rs

@@ -163,7 +163,9 @@ pub fn client_capabilities() -> Value {
             "typeHierarchy": { "dynamicRegistration": false }
         },
         // rust-analyzer: report when the project has loaded (`quiescent`).
-        "experimental": { "serverStatusNotification": true }
+        "experimental": { "serverStatusNotification": true },
+        // Jupyter notebooks: code cells are synced with `notebookDocument/*` (see `notebook.rs`).
+        "notebookDocument": { "synchronization": { "dynamicRegistration": false, "executionSummarySupport": false } }
     })
 }
 
@@ -426,6 +428,34 @@ impl ServerCaps {
             .unwrap_or_default()
     }
 
+    /// Whether `notebookDocumentSync` takes Jupyter notebooks (`file:`) with cells of `language`:
+    /// a selector that names no notebook, or `jupyter-notebook` / `*`, and no cell languages or
+    /// `language` among them.
+    pub fn syncs_notebooks(&self, language: &str) -> bool {
+        let Some(selectors) = self.raw.pointer("/notebookDocumentSync/notebookSelector").and_then(Value::as_array)
+        else {
+            return false;
+        };
+        selectors.iter().any(|s| {
+            let notebook = match s.get("notebook") {
+                None | Some(Value::Null) => true,
+                Some(Value::String(kind)) => kind == "jupyter-notebook" || kind == "*",
+                Some(filter) => {
+                    filter
+                        .get("notebookType")
+                        .and_then(Value::as_str)
+                        .is_none_or(|t| t == "jupyter-notebook" || t == "*")
+                        && filter.get("scheme").and_then(Value::as_str).is_none_or(|s| s == "file")
+                }
+            };
+            let cells = match s.get("cells").and_then(Value::as_array) {
+                None => true,
+                Some(cells) => cells.iter().any(|c| c.get("language").and_then(Value::as_str) == Some(language)),
+            };
+            notebook && cells && (s.get("notebook").is_some() || s.get("cells").is_some())
+        })
+    }
+
     /// `workspace.fileOperations.<op>` registration options (`op`: `willRename`, `didRename`).
     pub fn file_operation(&self, op: &str) -> Option<&Value> {
         self.raw.pointer(&format!("/workspace/fileOperations/{op}")).filter(|v| v.is_object())
@@ -668,6 +698,26 @@ mod tests {
         assert_eq!(c.code_action_kinds(), Some(vec!["refactor.inline".into()]));
     }
 
+    #[test]
+    fn notebook_sync_selectors() {
+        let syncs = |selector: Value| {
+            ServerCaps::parse(json!({ "notebookDocumentSync": { "notebookSelector": [selector] } }))
+                .syncs_notebooks("python")
+        };
+        // basedpyright
+        assert!(syncs(json!({
+            "notebook": { "scheme": "file", "notebookType": "jupyter-notebook" }, "cells": [{ "language": "python" }]
+        })));
+        // Ruff: any notebook with Python cells.
+        assert!(syncs(json!({ "cells": [{ "language": "python" }] })));
+        assert!(syncs(json!({ "notebook": "*" })));
+        assert!(!syncs(json!({ "notebook": "interactive" })));
+        assert!(!syncs(json!({ "notebook": { "notebookType": "jupyter-notebook", "scheme": "untitled" } })));
+        assert!(!syncs(json!({ "cells": [{ "language": "r" }] })));
+        assert!(!syncs(json!({})));
+        assert!(!ServerCaps::parse(json!({ "textDocumentSync": 2 })).syncs_notebooks("python"));
+    }
+
     #[test]
     fn language_definitions_use_known_feature_names() {
         for lang in ide_language::LanguageRegistry::load(None).unwrap().all() {

crates/ide-lsp/src/handler.rs

@@ -10,6 +10,7 @@ use ide_core::{IdeEvent, TaskId};
 use ide_jsonrpc::{Handler, codes};
 use lsp_types as lsp;
 use serde_json::{Value, json};
+use std::path::PathBuf;
 use std::sync::{Arc, Weak};
 use std::time::Duration;
 
@@ -229,7 +230,11 @@ impl ClientHandler {
                 return;
             }
         };
-        let Some(path) = uri_to_path(params.uri.as_str()) else { return };
+        let Some(path) = uri_to_path(params.uri.as_str())
+            .or_else(|| crate::notebook::is_cell_uri(params.uri.as_str()).then(|| PathBuf::from(params.uri.as_str())))
+        else {
+            return;
+        };
         if self.server.name == "jdtls"
             && let Some(failure) = crate::jdtls::build_plan_failure(&path, &self.server.root, &params.diagnostics)
         {

crates/ide-lsp/src/lib.rs

@@ -25,6 +25,7 @@ mod kotlin_lsp;
 mod launch;
 mod library_source;
 mod loading;
+mod notebook;
 mod pull;
 mod scope;
 mod server;
@@ -42,6 +43,7 @@ pub use inspect::{FileInspection, InspectHost, InspectOptions};
 pub use jdtls::jvm_inspect::JdtJvmInspector;
 pub use kotlin_lsp::{KotlinDebugServer, KotlinDebugUnavailable};
 pub use launch::{LaunchError, Launcher, ProcessLauncher};
+pub use notebook::{NotebookCell, NotebookCellDoc, cell_uri};
 pub use scope::{CandidateFiles, CandidateFn, CandidateQuery, MAX_CANDIDATES};
 pub use uri::{is_virtual_uri, path_to_uri_string, uri_target, uri_to_path};
 
@@ -115,6 +117,8 @@ struct DocState {
     servers: Vec<Attached>,
     /// Open YAML document whose filename does not match a Compose glob and whose content is Compose.
     compose_detected: bool,
+    /// The notebook a code cell belongs to: synced through `notebookDocument/*`, not as a file.
+    notebook: Option<PathBuf>,
 }
 
 impl DocState {
@@ -229,6 +233,8 @@ pub struct LspManager {
     /// Contents fetched for virtual URIs.
     virtual_docs: Mutex<HashMap<String, VirtualDocument>>,
     injections: injections::State,
+    /// Open notebooks by path. Locked before `docs` when both are.
+    notebooks: Mutex<HashMap<PathBuf, notebook::NotebookState>>,
 }
 
 /// Fetched virtual documents kept before the cache is reset.
@@ -476,6 +482,7 @@ impl LspManager {
             virtual_origins: Mutex::new(HashMap::new()),
             virtual_docs: Mutex::new(HashMap::new()),
             injections: injections::State::default(),
+            notebooks: Mutex::new(HashMap::new()),
         })
     }
 
@@ -614,8 +621,13 @@ impl LspManager {
     /// enabled in settings.
     fn attach_fallbacks(&self) {
         let _gate = self.injections.gate.lock();
-        let open: Vec<(DocId, LanguageId, PathBuf)> =
-            self.docs.lock().iter().map(|(id, d)| (*id, d.language.clone(), d.path.clone())).collect();
+        let open: Vec<(DocId, LanguageId, PathBuf)> = self
+            .docs
+            .lock()
+            .iter()
+            .filter(|(_, d)| d.notebook.is_none())
+            .map(|(id, d)| (*id, d.language.clone(), d.path.clone()))
+            .collect();
         let wanted: Vec<(DocId, Vec<Arc<Server>>)> =
             open.into_iter().map(|(id, language, path)| (id, self.servers_for(&language, &path))).collect();
         let mut start = Vec::new();
@@ -648,6 +660,11 @@ impl LspManager {
         for (doc, server) in opened {
             self.schedule_pull(doc, Some(&server), Duration::ZERO);
         }
+        for server in self.attach_notebook_servers() {
+            if !start.iter().any(|s| s.id == server.id) {
+                start.push(server);
+            }
+        }
         for server in &start {
             self.ensure_started(server);
         }
@@ -867,7 +884,7 @@ impl LspManager {
                 false
             } else {
                 let open_close = run.caps.read().open_close;
-                for (id, d) in docs.iter_mut() {
+                for (id, d) in docs.iter_mut().filter(|(_, d)| d.notebook.is_none()) {
                     let params = d.did_open_params();
                     for a in
                         d.servers.iter_mut().filter(|a| a.server.id == server.id && a.opened_gen != Some(generation))
@@ -889,6 +906,7 @@ impl LspManager {
         for doc in reopened {
             self.schedule_pull(doc, Some(&server), Duration::ZERO);
         }
+        self.reopen_notebooks(&server, &run);
         self.check_clang_tidy(&server).await;
         self.check_vue_navigation(&server).await;
         server.refresh_busy_status(&self.sink);
@@ -1083,6 +1101,7 @@ impl LspManager {
             text: Arc::new(DocText::new(text)),
             servers: servers.iter().map(|s| Attached { server: s.clone(), opened_gen: None }).collect(),
             compose_detected,
+            notebook: None,
         };
         let compose_added = {
             let mut docs = self.docs.lock();
@@ -1214,7 +1233,7 @@ impl LspManager {
             return;
         }
         let docs = self.docs.lock();
-        for d in docs.values() {
+        for d in docs.values().filter(|d| d.notebook.is_none()) {
             if !d.servers.iter().any(|a| a.server.id == server.id && a.opened_gen == Some(generation)) {
                 continue;
             }
@@ -1288,13 +1307,23 @@ impl LspManager {
     fn snapshot(&self, doc: DocId) -> IdeResult<DocSnap> {
         let docs = self.docs.lock();
         let d = docs.get(&doc).ok_or_else(|| IdeError::not_found(format!("{doc} is not open")))?;
+        // A cell is asked only of the servers its notebook is open on (those that sync notebooks).
+        let servers = d
+            .servers
+            .iter()
+            .filter(|a| {
+                d.notebook.is_none()
+                    || a.opened_gen.is_some_and(|g| a.server.running().is_some_and(|run| run.generation == g))
+            })
+            .map(|a| a.server.clone())
+            .collect();
         Ok(DocSnap {
             version: d.version,
             path: d.path.clone(),
             uri: d.uri.clone(),
             language: d.language.clone(),
             text: d.text.clone(),
-            servers: d.servers.iter().map(|a| a.server.clone()).collect(),
+            servers,
         })
     }
 

crates/ide-lsp/src/notebook.rs

@@ -0,0 +1,382 @@
+//! Jupyter notebooks on servers that sync them with LSP 3.17 `notebookDocument/*` (basedpyright,
+//! Ruff). Each code cell is a document of its own, `vscode-notebook-cell:<notebook path>#<cell id>`,
+//! kept in `docs` under a [`DocId`]: completion, hover and the other requests address a cell like
+//! any open file. Only opening, changing and closing go through the notebook methods; a change in
+//! the cells' order closes and reopens the notebook.
+
+use crate::server::{Running, Server};
+use crate::text::DocText;
+use crate::{Attached, DocState, LspManager, uri};
+use ide_core::{DocId, LanguageId, ServerId};
+use lsp_types as lsp;
+use percent_encoding::{AsciiSet, NON_ALPHANUMERIC, utf8_percent_encode};
+use serde::Serialize;
+use serde_json::{Value, json};
+use specta::Type;
+use std::path::{Path, PathBuf};
+use std::str::FromStr;
+use std::sync::Arc;
+
+const CELL_SCHEME: &str = "vscode-notebook-cell";
+const NOTEBOOK_TYPE: &str = "jupyter-notebook";
+/// `NotebookCellKind.Code`.
+const CODE_CELL: u32 = 2;
+/// Escaped in a cell id: nbformat ids are `[a-zA-Z0-9-_]`, older notebooks may have others.
+const FRAGMENT: &AsciiSet = &NON_ALPHANUMERIC.remove(b'-').remove(b'_');
+
+/// A code cell as the notebook holds it.
+pub struct NotebookCell {
+    pub id: String,
+    pub source: String,
+}
+
+/// The document the language servers know a code cell by.
+#[derive(Clone, Debug, PartialEq, Serialize, Type)]
+#[serde(rename_all = "camelCase")]
+pub struct NotebookCellDoc {
+    pub cell: String,
+    pub doc: DocId,
+    /// Also the `path` of the cell's diagnostics and of locations in it.
+    pub uri: String,
+}
+
+pub(crate) struct NotebookState {
+    uri: lsp::Uri,
+    language: LanguageId,
+    version: i32,
+    /// Code cells in notebook order.
+    cells: Vec<(String, DocId)>,
+    servers: Vec<Arc<Server>>,
+    /// Servers it is open on, by instance generation.
+    opened: Vec<(ServerId, u64)>,
+}
+
+/// The URI of cell `cell` of the notebook at `path`: the notebook's `file:` URI with the cell
+/// scheme and the cell id as fragment (as VS Code names cells; basedpyright expects the scheme).
+pub fn cell_uri(path: &Path, cell: &str) -> String {
+    let file = uri::path_to_uri_string(path);
+    let rest = file.strip_prefix("file://").unwrap_or(&file);
+    // `file:///C:/a` → `…:/C:/a`; a share (`file://server/s/a`) keeps its authority.
+    let authority = if rest.starts_with('/') { "" } else { "//" };
+    format!("{CELL_SCHEME}:{authority}{rest}#{}", utf8_percent_encode(cell, FRAGMENT))
+}
+
+pub(crate) fn is_cell_uri(uri: &str) -> bool {
+    uri.strip_prefix(CELL_SCHEME).is_some_and(|rest| rest.starts_with(':'))
+}
+
+fn parse_uri(s: &str) -> Option<lsp::Uri> {
+    lsp::Uri::from_str(s).inspect_err(|e| tracing::debug!("not a valid URI {s}: {e}")).ok()
+}
+
+impl NotebookState {
+    fn opened_on(&self, server: ServerId, generation: u64) -> bool {
+        self.opened.contains(&(server, generation))
+    }
+
+    /// The servers it is open on whose instance is still the one it was opened on.
+    fn open_targets(&self) -> Vec<(Arc<Server>, Running)> {
+        self.servers
+            .iter()
+            .filter_map(|s| {
+                let run = s.running()?;
+                self.opened_on(s.id, run.generation).then(|| (s.clone(), run))
+            })
+            .collect()
+    }
+}
+
+impl LspManager {
+    /// Opens the notebook at `path` on its language's servers that sync notebooks, or brings
+    /// them up to date with `cells` (its code cells, in order).
+    pub fn sync_notebook(&self, path: &Path, language: &LanguageId, cells: &[NotebookCell]) {
+        let mut notebooks = self.notebooks.lock();
+        let mut changed = Vec::new();
+        let start = match notebooks.get_mut(path) {
+            None => {
+                let Some(notebook_uri) = parse_uri(&uri::path_to_uri_string(path)) else { return };
+                let servers = self.servers_for(language, path);
+                let mut state = NotebookState {
+                    uri: notebook_uri,
+                    language: language.clone(),
+                    version: 1,
+                    cells: Vec::new(),
+                    servers: servers.clone(),
+                    opened: Vec::new(),
+                };
+                self.set_cells(path, &mut state, cells);
+                self.open_notebook(&mut state, None);
+                notebooks.insert(path.to_owned(), state);
+                servers
+            }
+            Some(state) => {
+                let same_cells =
+                    state.cells.len() == cells.len() && state.cells.iter().zip(cells).all(|((id, _), c)| *id == c.id);
+                if same_cells {
+                    changed = self.change_cells(state, cells);
+                } else {
+                    self.close_notebook_on_servers(state);
+                    state.version += 1;
+                    self.set_cells(path, state, cells);
+                    self.open_notebook(state, None);
+                }
+                Vec::new()
+            }
+        };
+        drop(notebooks);
+        for server in &start {
+            self.ensure_started(server);
+        }
+        for doc in changed {
+            self.schedule_pull(doc, None, crate::pull::AFTER_EDIT);
+        }
+    }
+
+    /// Closes the notebook at `path` on its servers and forgets its cells.
+    pub fn close_notebook(&self, path: &Path) {
+        let Some(mut state) = self.notebooks.lock().remove(path) else { return };
+        self.close_notebook_on_servers(&mut state);
+        let removed: Vec<DocId> = state.cells.iter().map(|(_, doc)| *doc).collect();
+        self.remove_cells(&state.servers, &removed);
+    }
+
+    /// The documents of the notebook's code cells, in notebook order; empty when it isn't open.
+    pub fn notebook_cells(&self, path: &Path) -> Vec<NotebookCellDoc> {
+        let notebooks = self.notebooks.lock();
+        let Some(state) = notebooks.get(path) else { return Vec::new() };
+        let docs = self.docs.lock();
+        state
+            .cells
+            .iter()
+            .filter_map(|(cell, doc)| {
+                Some(NotebookCellDoc { cell: cell.clone(), doc: *doc, uri: docs.get(doc)?.uri.as_str().to_owned() })
+            })
+            .collect()
+    }
+
+    /// Opens the notebooks of `server`'s language on its instance that just became ready.
+    pub(crate) fn reopen_notebooks(&self, server: &Arc<Server>, run: &Running) {
+        let mut pull = Vec::new();
+        {
+            let mut notebooks = self.notebooks.lock();
+            for state in notebooks.values_mut() {
+                if state.servers.iter().any(|s| s.id == server.id) && !state.opened_on(server.id, run.generation) {
+                    self.open_notebook(state, Some(server));
+                    pull.extend(state.cells.iter().map(|(_, doc)| *doc));
+                }
+            }
+        }
+        for doc in pull {
+            self.schedule_pull(doc, Some(server), std::time::Duration::ZERO);
+        }
+    }
+
+    /// Adds servers of the notebooks' languages that were not wanted when they opened (a
+    /// fallback for a missing server, one enabled in settings) and returns them to be started.
+    pub(crate) fn attach_notebook_servers(&self) -> Vec<Arc<Server>> {
+        let mut start = Vec::new();
+        let mut notebooks = self.notebooks.lock();
+        for (path, state) in notebooks.iter_mut() {
+            let added: Vec<Arc<Server>> = self
+                .servers_for(&state.language, path)
+                .into_iter()
+                .filter(|s| !state.servers.iter().any(|known| known.id == s.id))
+                .collect();
+            if added.is_empty() {
+                continue;
+            }
+            {
+                let mut docs = self.docs.lock();
+                for (_, doc) in &state.cells {
+                    if let Some(d) = docs.get_mut(doc) {
+                        d.servers.extend(added.iter().map(|s| Attached { server: s.clone(), opened_gen: None }));
+                    }
+                }
+            }
+            state.servers.extend(added.iter().cloned());
+            for server in &added {
+                self.open_notebook(state, Some(server));
+            }
+            start.extend(added);
+        }
+        start
+    }
+
+    /// Replaces the cell documents with `cells`: kept cells keep their [`DocId`].
+    fn set_cells(&self, path: &Path, state: &mut NotebookState, cells: &[NotebookCell]) {
+        let mut docs = self.docs.lock();
+        let previous = std::mem::take(&mut state.cells);
+        for cell in cells {
+            let kept = previous.iter().find(|(id, _)| *id == cell.id).map(|(_, doc)| *doc);
+            if let Some(doc) = kept
+                && let Some(d) = docs.get_mut(&doc)
+            {
+                if d.text.as_str() != cell.source {
+                    Arc::make_mut(&mut d.text).set(&cell.source);
+                }
+                d.version += 1;
+                state.cells.push((cell.id.clone(), doc));
+                continue;
+            }
+            let uri_string = cell_uri(path, &cell.id);
+            let Some(cell_uri) = parse_uri(&uri_string) else { continue };
+            let doc = DocId::next();
+            let key = PathBuf::from(&uri_string);
+            docs.insert(
+                doc,
+                DocState {
+                    canonical: key.clone(),
+                    path: key,
+                    uri: cell_uri,
+                    language: state.language.clone(),
+                    version: 1,
+                    text: Arc::new(DocText::new(&cell.source)),
+                    servers: state.servers.iter().map(|s| Attached { server: s.clone(), opened_gen: None }).collect(),
+                    compose_detected: false,
+                    notebook: Some(path.to_owned()),
+                },
+            );
+            state.cells.push((cell.id.clone(), doc));
+        }
+        drop(docs);
+        let removed: Vec<DocId> = previous
+            .into_iter()
+            .filter(|(_, doc)| !state.cells.iter().any(|(_, d)| d == doc))
+            .map(|(_, d)| d)
+            .collect();
+        self.remove_cells(&state.servers, &removed);
+    }
+
+    /// Sends the cells whose text differs from `cells` as one change; returns their documents.
+    fn change_cells(&self, state: &mut NotebookState, cells: &[NotebookCell]) -> Vec<DocId> {
+        let mut docs = self.docs.lock();
+        let mut changed = Vec::new();
+        let mut content = Vec::new();
+        for ((_, doc), cell) in state.cells.iter().zip(cells) {
+            let Some(d) = docs.get_mut(doc) else { continue };
+            if d.text.as_str() == cell.source {
+                continue;
+            }
+            Arc::make_mut(&mut d.text).set(&cell.source);
+            d.version += 1;
+            content.push(json!({
+                "document": { "uri": d.uri, "version": d.version },
+                "changes": [{ "text": cell.source }],
+            }));
+            changed.push(*doc);
+        }
+        drop(docs);
+        if content.is_empty() {
+            return changed;
+        }
+        state.version += 1;
+        let params = json!({
+            "notebookDocument": { "uri": state.uri, "version": state.version },
+            "change": { "cells": { "textContent": content } },
+        });
+        for (server, run) in state.open_targets() {
+            if let Err(e) = run.peer.notify("notebookDocument/didChange", params.clone()) {
+                tracing::debug!("{}: notebookDocument/didChange: {e}", server.name);
+            }
+        }
+        changed
+    }
+
+    /// Opens the notebook on its running servers that sync notebooks (`only`: on that one).
+    fn open_notebook(&self, state: &mut NotebookState, only: Option<&Arc<Server>>) {
+        let mut docs = self.docs.lock();
+        let mut cells = Vec::new();
+        let mut texts = Vec::new();
+        for (_, doc) in &state.cells {
+            let Some(d) = docs.get(doc) else { continue };
+            cells.push(json!({ "kind": CODE_CELL, "document": d.uri }));
+            texts.push(json!({
+                "uri": d.uri,
+                "languageId": d.language.as_str(),
+                "version": d.version,
+                "text": d.text.as_str(),
+            }));
+        }
+        let params = json!({
+            "notebookDocument": { "uri": state.uri, "notebookType": NOTEBOOK_TYPE, "version": state.version, "cells": cells },
+            "cellTextDocuments": texts,
+        });
+        let servers: Vec<Arc<Server>> =
+            state.servers.iter().filter(|s| only.is_none_or(|o| o.id == s.id)).cloned().collect();
+        for server in servers {
+            let Some(run) = server.running() else { continue };
+            if state.opened_on(server.id, run.generation) || !run.caps.read().syncs_notebooks(state.language.as_str()) {
+                continue;
+            }
+            if let Err(e) = run.peer.notify("notebookDocument/didOpen", params.clone()) {
+                tracing::debug!("{}: notebookDocument/didOpen: {e}", server.name);
+                continue;
+            }
+            state.opened.retain(|(id, _)| *id != server.id);
+            state.opened.push((server.id, run.generation));
+            for (_, doc) in &state.cells {
+                if let Some(a) = docs.get_mut(doc).and_then(|d| d.servers.iter_mut().find(|a| a.server.id == server.id))
+                {
+                    a.opened_gen = Some(run.generation);
+                }
+            }
+        }
+    }
+
+    fn close_notebook_on_servers(&self, state: &mut NotebookState) {
+        let cells: Vec<Value> = {
+            let docs = self.docs.lock();
+            state.cells.iter().filter_map(|(_, doc)| docs.get(doc)).map(|d| json!({ "uri": d.uri })).collect()
+        };
+        let params = json!({ "notebookDocument": { "uri": state.uri }, "cellTextDocuments": cells });
+        for (server, run) in state.open_targets() {
+            if let Err(e) = run.peer.notify("notebookDocument/didClose", params.clone()) {
+                tracing::debug!("{}: notebookDocument/didClose: {e}", server.name);
+            }
+        }
+        state.opened.clear();
+        let mut docs = self.docs.lock();
+        for (_, doc) in &state.cells {
+            if let Some(d) = docs.get_mut(doc) {
+                d.servers.iter_mut().for_each(|a| a.opened_gen = None);
+            }
+        }
+    }
+
+    /// Forgets cell documents and clears what their servers reported about them: a server need
+    /// not clear the diagnostics of a cell that left the notebook.
+    fn remove_cells(&self, servers: &[Arc<Server>], removed: &[DocId]) {
+        let gone: Vec<DocState> = {
+            let mut docs = self.docs.lock();
+            removed.iter().filter_map(|doc| docs.remove(doc)).collect()
+        };
+        for d in gone {
+            for server in servers {
+                let run = server.running();
+                self.forget_pulled(server, &d.path, run.as_ref());
+                let enc = run.as_ref().map(|r| r.caps.read().encoding).unwrap_or_default();
+                let shown = server.set_diagnostics(d.path.clone(), Vec::new());
+                self.emit_diagnostics(server, d.path.clone(), shown, enc);
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn cell_uris_name_the_notebook_file_and_the_cell() {
+        let root = if cfg!(windows) { PathBuf::from(r"C:\work\my nb") } else { PathBuf::from("/work/my nb") };
+        let expected = if cfg!(windows) {
+            "vscode-notebook-cell:/C:/work/my%20nb/a.ipynb#f3-x_1%2E"
+        } else {
+            "vscode-notebook-cell:/work/my%20nb/a.ipynb#f3-x_1%2E"
+        };
+        let uri = cell_uri(&root.join("a.ipynb"), "f3-x_1.");
+        assert_eq!(uri, expected);
+        assert!(is_cell_uri(&uri) && parse_uri(&uri).is_some());
+        assert!(!is_cell_uri("file:///a.ipynb") && !is_cell_uri("vscode-notebook-cellx:/a"));
+    }
+}

crates/ide-lsp/tests/mock_server.rs

@@ -6781,3 +6781,142 @@ async fn gomod_files_are_served_by_gopls() {
     assert_eq!(items[0].message, "missing requirement");
     mgr.shutdown_all().await;
 }
+
+// ------------------------------------------------------------------ notebooks
+
+fn notebook_caps() -> Value {
+    let mut caps = incremental_caps();
+    caps["notebookDocumentSync"] = json!({
+        "notebookSelector": [{
+            "notebook": { "scheme": "file", "notebookType": "jupyter-notebook" },
+            "cells": [{ "language": "python" }]
+        }]
+    });
+    caps
+}
+
+fn cells(cells: &[(&str, &str)]) -> Vec<ide_lsp::NotebookCell> {
+    cells.iter().map(|(id, source)| ide_lsp::NotebookCell { id: (*id).into(), source: (*source).into() }).collect()
+}
+
+fn cell_texts(open: &Value) -> Vec<(String, String)> {
+    open["cellTextDocuments"]
+        .as_array()
+        .unwrap()
+        .iter()
+        .map(|c| (c["uri"].as_str().unwrap().to_owned(), c["text"].as_str().unwrap().to_owned()))
+        .collect()
+}
+
+#[tokio::test(flavor = "multi_thread")]
+async fn notebook_cells_are_documents_of_the_servers_that_sync_notebooks() {
+    let h = harness_for("python", "py", &[("nb", notebook_caps()), ("plain", incremental_caps())], &[]);
+    let path = h.root.join("analysis.ipynb");
+    let python = LanguageId::new("python");
+    h.mgr.sync_notebook(&path, &python, &cells(&[("a", "import os\n"), ("b", "os.pa")]));
+    h.wait_status("nb", ServerStatus::Ready).await;
+    h.wait_status("plain", ServerStatus::Ready).await;
+    let (nb, plain) = (h.launcher.state("nb"), h.launcher.state("plain"));
+    eventually(|| !nb.notified("notebookDocument/didOpen").is_empty()).await;
+
+    let docs = h.mgr.notebook_cells(&path);
+    assert_eq!(docs.iter().map(|d| d.cell.as_str()).collect::<Vec<_>>(), ["a", "b"]);
+    assert_eq!(docs[0].uri, ide_lsp::cell_uri(&path, "a"));
+    let open = &nb.notified("notebookDocument/didOpen")[0];
+    assert_eq!(open["notebookDocument"]["uri"], uri_of(&path));
+    assert_eq!(open["notebookDocument"]["notebookType"], "jupyter-notebook");
+    assert_eq!(open["notebookDocument"]["cells"][1], json!({ "kind": 2, "document": docs[1].uri }));
+    assert_eq!(
+        cell_texts(open),
+        [(docs[0].uri.clone(), "import os\n".to_owned()), (docs[1].uri.clone(), "os.pa".to_owned())]
+    );
+    assert!(plain.notified("notebookDocument/didOpen").is_empty() && plain.notified("textDocument/didOpen").is_empty());
+
+    // Requests address a cell like a file, and go only to the server it is open on.
+    nb.respond("textDocument/completion", json!({ "isIncomplete": false, "items": [{ "label": "path" }] }));
+    plain.respond("textDocument/completion", json!({ "isIncomplete": false, "items": [{ "label": "wrong" }] }));
+    let list = h.mgr.completion(docs[1].doc, Pos::new(0, 5), None, Default::default()).await.unwrap();
+    assert_eq!(list.items.iter().map(|i| i.label.as_str()).collect::<Vec<_>>(), ["path"]);
+    assert_eq!(nb.requested("textDocument/completion")[0]["textDocument"]["uri"], docs[1].uri);
+    assert!(plain.requested("textDocument/completion").is_empty());
+
+    // Its diagnostics are reported under the cell's URI.
+    nb.peer()
+        .notify(
+            "textDocument/publishDiagnostics",
+            json!({ "uri": docs[1].uri, "diagnostics": [{
+                "range": {"start":{"line":0,"character":3},"end":{"line":0,"character":5}}, "message": "no attribute pa"
+            }] }),
+        )
+        .unwrap();
+    let cell_path = PathBuf::from(&docs[1].uri);
+    eventually(|| {
+        h.events
+            .lock()
+            .iter()
+            .any(|e| matches!(e, IdeEvent::Diagnostics { path, items, .. } if path.0 == cell_path && items.len() == 1))
+    })
+    .await;
+
+    h.mgr.close_notebook(&path);
+    eventually(|| !nb.notified("notebookDocument/didClose").is_empty()).await;
+    assert_eq!(nb.notified("notebookDocument/didClose")[0]["cellTextDocuments"].as_array().unwrap().len(), 2);
+    assert!(h.mgr.notebook_cells(&path).is_empty());
+    assert!(h.mgr.completion(docs[1].doc, Pos::new(0, 5), None, Default::default()).await.is_err());
+    let last = h
+        .events
+        .lock()
+        .iter()
+        .filter_map(|e| match e {
+            IdeEvent::Diagnostics { path, items, source } if path.0 == cell_path && source == "nb" => Some(items.len()),
+            _ => None,
+        })
+        .next_back();
+    assert_eq!(last, Some(0), "a closed notebook's diagnostics are cleared");
+    h.mgr.shutdown_all().await;
+}
+
+#[tokio::test(flavor = "multi_thread")]
+async fn notebook_edits_change_cell_text_and_reorders_reopen_it() {
+    let h = harness_for("python", "py", &[("nb", notebook_caps())], &[]);
+    let path = h.root.join("n.ipynb");
+    let python = LanguageId::new("python");
+    h.mgr.sync_notebook(&path, &python, &cells(&[("a", "x = 1\n"), ("b", "print(x)\n")]));
+    h.wait_status("nb", ServerStatus::Ready).await;
+    let nb = h.launcher.state("nb");
+    eventually(|| nb.notified("notebookDocument/didOpen").len() == 1).await;
+    let before = h.mgr.notebook_cells(&path);
+
+    // Typing in one cell sends only that cell's text.
+    h.mgr.sync_notebook(&path, &python, &cells(&[("a", "x = 1\n"), ("b", "print(x + 1)\n")]));
+    h.mgr.sync_notebook(&path, &python, &cells(&[("a", "x = 1\n"), ("b", "print(x + 1)\n")]));
+    eventually(|| !nb.notified("notebookDocument/didChange").is_empty()).await;
+    let changes = nb.notified("notebookDocument/didChange");
+    assert_eq!(changes.len(), 1, "an unchanged notebook sends nothing");
+    let content = &changes[0]["change"]["cells"]["textContent"];
+    assert_eq!(content.as_array().unwrap().len(), 1);
+    assert_eq!(content[0]["document"]["uri"], before[1].uri);
+    assert_eq!(content[0]["changes"], json!([{ "text": "print(x + 1)\n" }]));
+    assert_eq!(changes[0]["notebookDocument"]["version"], 2);
+
+    // Inserting and moving cells reopens the notebook; kept cells keep their documents.
+    h.mgr.sync_notebook(&path, &python, &cells(&[("c", "import math\n"), ("b", "print(x + 1)\n"), ("a", "x = 1\n")]));
+    eventually(|| nb.notified("notebookDocument/didOpen").len() == 2).await;
+    assert_eq!(nb.notified("notebookDocument/didClose").len(), 1);
+    let after = h.mgr.notebook_cells(&path);
+    assert_eq!(after.iter().map(|d| d.cell.as_str()).collect::<Vec<_>>(), ["c", "b", "a"]);
+    assert_eq!((after[1].doc, after[2].doc), (before[1].doc, before[0].doc));
+    let reopened = &nb.notified("notebookDocument/didOpen")[1];
+    assert_eq!(
+        cell_texts(reopened).into_iter().map(|(_, t)| t).collect::<Vec<_>>(),
+        ["import math\n", "print(x + 1)\n", "x = 1\n"]
+    );
+
+    // A restarted server gets the notebook again.
+    let launches = nb.launches.load(Ordering::SeqCst);
+    let server = h.mgr.servers()[0].id;
+    h.mgr.restart(server).await.unwrap();
+    eventually(|| nb.launches.load(Ordering::SeqCst) > launches && nb.notified("notebookDocument/didOpen").len() == 3)
+        .await;
+    h.mgr.shutdown_all().await;
+}

crates/ide-lsp/tests/real_servers.rs

@@ -920,3 +920,99 @@ async fn angular_language_server_end_to_end() {
     }
     tokio::time::timeout(Duration::from_secs(10), mgr.shutdown_all()).await.unwrap();
 }
+
+/// A notebook's code cells on basedpyright and Ruff (LSP 3.17 notebook sync): names flow from
+/// earlier cells, IPython magics parse, and locations and diagnostics name the cells.
+#[tokio::test(flavor = "multi_thread")]
+#[ignore = "requires ruff and basedpyright on PATH"]
+async fn basedpyright_and_ruff_serve_notebook_cells() {
+    use ide_lsp::{NotebookCell, cell_uri};
+    let dir = tempfile::tempdir().unwrap();
+    let root = dir.path().canonicalize().unwrap();
+    let root = dunce::simplified(&root).to_path_buf();
+    std::fs::write(root.join("pyproject.toml"), "[tool.ruff.lint]\nselect = [\"F\"]\n").unwrap();
+    let path = root.join("analysis.ipynb");
+    std::fs::write(&path, "{}").unwrap();
+    let mut python = LanguageRegistry::load(None).unwrap().get(&LanguageId::new("python")).unwrap();
+    python.language_servers.retain(|server| matches!(server.name.as_str(), "basedpyright" | "ruff"));
+    let (mgr, events) = manager(&root, LanguageRegistry::from_configs(vec![python]));
+    let cell = |id: &str, source: &str| NotebookCell { id: id.into(), source: source.into() };
+    let first = "import os\nimport sys\n\ndef area(r: float) -> float:\n    return 3.14 * r * r\n";
+    let second = "%matplotlib inline\nsize: int = area(2)\nos.pa";
+    mgr.sync_notebook(&path, &LanguageId::new("python"), &[cell("first", first), cell("second", second)]);
+    for name in ["basedpyright", "ruff"] {
+        retry(name, Duration::from_secs(60), || {
+            let status = last_status(&events, name);
+            if let Some((ServerStatus::Crashed | ServerStatus::Missing | ServerStatus::Misconfigured, msg)) = &status {
+                panic!("{name} failed: {msg:?}");
+            }
+            async move { matches!(status, Some((ServerStatus::Ready | ServerStatus::Busy, _))).then_some(()) }
+        })
+        .await;
+    }
+    let docs = mgr.notebook_cells(&path);
+    let b = docs[1].doc;
+    let (a_uri, b_uri) = (cell_uri(&path, "first"), cell_uri(&path, "second"));
+    assert_eq!((docs[0].uri.as_str(), docs[1].uri.as_str()), (a_uri.as_str(), b_uri.as_str()));
+
+    // `os`, imported in the first cell, completes in the second.
+    let labels = retry("completion in a cell", Duration::from_secs(30), || {
+        let mgr = mgr.clone();
+        async move {
+            let list = mgr.completion(b, Pos::new(2, 5), Some(".".into()), Default::default()).await.ok()?;
+            let labels: Vec<String> = list.items.into_iter().map(|i| i.label).collect();
+            labels.iter().any(|l| l == "path").then_some(labels)
+        }
+    })
+    .await;
+    assert!(labels.iter().any(|l| l == "path"), "{labels:?}");
+
+    let hover = mgr.hover(b, Pos::new(1, 13), Default::default()).await.unwrap().expect("hover on `area`");
+    assert!(
+        hover.contents.value.contains("area") && hover.contents.value.contains("float"),
+        "{}",
+        hover.contents.value
+    );
+
+    let defs = mgr.goto(b, Pos::new(1, 13), GotoKind::Definition).await.unwrap();
+    assert_eq!(defs.len(), 1, "{defs:?}");
+    assert_eq!((defs[0].path.to_path_buf(), defs[0].range.start.line), (std::path::PathBuf::from(&a_uri), 3));
+
+    let diagnostics = |source: &'static str, uri: String| {
+        let events = events.clone();
+        move || {
+            let items = events.lock().iter().rev().find_map(|e| match e {
+                IdeEvent::Diagnostics { path, source: s, items } if s == source && path.0 == Path::new(&uri) => {
+                    Some(items.clone())
+                }
+                _ => None,
+            });
+            async move { items.filter(|items| !items.is_empty()) }
+        }
+    };
+    // `float` is not assignable to `int`; the magic on line 0 is no error.
+    let typed =
+        retry("basedpyright cell diagnostics", Duration::from_secs(30), diagnostics("basedpyright", b_uri)).await;
+    assert!(typed.iter().any(|d| d.range.start.line == 1), "{typed:?}");
+    assert!(typed.iter().all(|d| d.range.start.line != 0), "{typed:?}");
+    // F401: `sys` is unused.
+    let linted = retry("ruff cell diagnostics", Duration::from_secs(30), diagnostics("ruff", a_uri)).await;
+    assert!(linted.iter().any(|d| d.code.as_deref() == Some("F401") && d.range.start.line == 1), "{linted:?}");
+
+    // An edit reaches both servers: the error is fixed.
+    let fixed = "%matplotlib inline\nsize: float = area(2)\nos.path";
+    mgr.sync_notebook(&path, &LanguageId::new("python"), &[cell("first", first), cell("second", fixed)]);
+    retry("diagnostics after the fix", Duration::from_secs(30), || {
+        let uri = cell_uri(&path, "second");
+        let clear = events.lock().iter().rev().find_map(|e| match e {
+            IdeEvent::Diagnostics { path, source, items } if source == "basedpyright" && path.0 == Path::new(&uri) => {
+                Some(items.is_empty())
+            }
+            _ => None,
+        });
+        async move { (clear == Some(true)).then_some(()) }
+    })
+    .await;
+    mgr.close_notebook(&path);
+    tokio::time::timeout(Duration::from_secs(10), mgr.shutdown_all()).await.unwrap();
+}

docs/en/guide/build-tools.mdx

@@ -61,16 +61,17 @@ When you edit a POM in the editor, the tool window shows **Build files changed s
 
 **Settings › Build, Execution, Deployment › Maven**:
 
-| Setting                     | What it does                                                                                                              |
-| --------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
-| **JRE**                     | JDK for Maven sync, builds, goals and tests                                                                               |
-| **Maven home path**         | A Maven installation to use instead of the project's `mvnw`                                                               |
-| **User settings file**      | Passed as `-s`                                                                                                            |
-| **Toolchains file**         | Passed as `-t`                                                                                                            |
-| **Local repository**        | Passed as `-Dmaven.repo.local`                                                                                            |
-| **Active profiles**         | Comma-separated profiles (`-P`); `!id` excludes one                                                                       |
-| **Thread count**            | Passed as `-T` (for example `4` or `2C`)                                                                                  |
-| **Use Maven Daemon (mvnd)** | Runs builds, sync and run configurations through `mvnd`; falls back to `mvn` if it is missing or the JDK is older than 17 |
+| Setting                                     | What it does                                                                                                              |
+| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
+| **Delegate IDE build/run actions to Maven** | Run and Debug build Maven modules with Maven (`javac`) instead of jdtls                                                   |
+| **JRE**                                     | JDK for Maven sync, builds, goals and tests                                                                               |
+| **Maven home path**                         | A Maven installation to use instead of the project's `mvnw`                                                               |
+| **User settings file**                      | Passed as `-s`                                                                                                            |
+| **Toolchains file**                         | Passed as `-t`                                                                                                            |
+| **Local repository**                        | Passed as `-Dmaven.repo.local`                                                                                            |
+| **Active profiles**                         | Comma-separated profiles (`-P`); `!id` excludes one                                                                       |
+| **Thread count**                            | Passed as `-T` (for example `4` or `2C`)                                                                                  |
+| **Use Maven Daemon (mvnd)**                 | Runs builds, sync and run configurations through `mvnd`; falls back to `mvn` if it is missing or the JDK is older than 17 |
 
 ## Gradle
 

docs/en/guide/database.mdx

@@ -66,6 +66,7 @@ The data source dialog has a name, an optional **Color:** (tints the tree and co
 
 - Project data sources: `.openidea/data-sources.toml` in the project.
 - Global data sources: `data-sources.toml` in the IDE's data directory (see [Data directories](/en/reference/data-directories)).
+- You can edit these files by hand or have a coding agent edit them while the project is open: the Database tool window picks up the change, and connections of data sources whose settings changed or that were removed are closed.
 - Passwords are never written to these files. Saved passwords go to the OS credential store (Keychain, Windows Credential Manager or Secret Service). Unsaved ones are kept in memory for the session; OpenIDEA asks for them when it connects.
 
 ## Browse the database

docs/en/guide/notebook.mdx

@@ -54,6 +54,14 @@ Use **Add Code Cell** and **Add Markdown Cell** at the end of the notebook to ap
 
 Code cells show the execution count (`[3]`) next to the cell toolbar. Raw cells and cell types OpenIDEA doesn't know are shown read-only; their content is kept as it is.
 
+## Code intelligence
+
+Code cells get completion, quick documentation on hover, parameter info and error highlighting from the project's Python language server, as in a `.py` file. The server reads the code cells in order, so names defined in one cell are known in the cells below it, and IPython syntax such as `%matplotlib inline` or `!pip list` isn't reported as an error.
+
+`⌘`-click (macOS) or `Ctrl`-click (Windows/Linux) a name to go to its declaration, in another cell or in a library.
+
+This needs a language server that supports notebooks: **basedpyright** (the default) and **Ruff** do. With other Python servers, code cells only get syntax highlighting.
+
 ## Keyboard
 
 | Action                     | Keys                                                                        |

docs/en/guide/run.mdx

@@ -144,6 +144,8 @@ Add tasks under **Modify options › Add before launch task**. They run in order
 
 For Java, Maven and Gradle configurations, the **Build** option picks how the project is built first: **Automatic (incremental)**, **Build (Make)**, **Rebuild**, **No build** or **Build, no error check**.
 
+Maven modules are built by jdtls (ECJ) for **Automatic** and **Build (Make)**, and by Maven for **Rebuild**. Turn on **Settings › Build, Execution, Deployment › Maven › Delegate IDE build/run actions to Maven** to build them with Maven (the project's `javac`) every time; editor **Problems** still come from jdtls.
+
 ### Compound configurations
 
 A **Compound** configuration starts several configurations at once. Run it to start all of them, or Debug it to debug each of them. Stop from the run widget stops them all.

docs/en/languages/java.mdx

@@ -86,7 +86,7 @@ When the project uses Lombok (in its build file, an inherited parent POM or Grad
 
 jdtls imports Maven and Gradle projects (including the Gradle wrapper) automatically. A folder without a build file is opened as a plain Java project, and jars under `lib/` are added to its classpath. OpenIDEA also shows **Maven** and **Gradle** tool windows for running goals and tasks, reloading, profiles and dependency analysis. Their settings are in **Settings › Build, Execution, Deployment › Maven** and **Gradle**:
 
-- Maven: **JRE**, **Maven home path** (default: the project's `mvnw`, else `mvn` on `PATH`), **User settings file**, **Toolchains file**, **Local repository**, **Active profiles**, **Thread count** and **Use Maven Daemon (mvnd)**. The profiles you choose in the Maven tool window are passed to jdtls too, so the editor sees the same profile-specific dependencies.
+- Maven: **Delegate IDE build/run actions to Maven** (Run and Debug build with Maven's `javac` instead of jdtls), **JRE**, **Maven home path** (default: the project's `mvnw`, else `mvn` on `PATH`), **User settings file**, **Toolchains file**, **Local repository**, **Active profiles**, **Thread count** and **Use Maven Daemon (mvnd)**. The profiles you choose in the Maven tool window are passed to jdtls too, so the editor sees the same profile-specific dependencies.
 - Gradle: **Build and run using** (Gradle or OpenIDEA), **Gradle JVM**, **Distribution** (wrapper or local installation), **Gradle home** and **Gradle user home**.
 
 The IDE passes its proxy settings to jdtls, so dependency resolution works behind a proxy. See [Build tools](/en/guide/build-tools) for the tool windows.
@@ -141,5 +141,6 @@ Navigation between MyBatis mapper interfaces and their XML statements is availab
 - One jdtls runs per project window. With several large projects open at once, memory adds up. Lower `-Xmx` in **JVM options** or close windows you are not using. **Share JDT LS across builds** helps only within one window.
 - Without jdtls (for example with `kmp-lsp` only), Application configurations, classpath-based features and Java debugging are not available.
 - Java tests run through Maven or Gradle. A project without either build tool has no test runner.
+- The jdtls compiler (ECJ) is stricter than javac in some places. Known case: a generic method call used directly as an `if`, `while`, `for` or `do` condition or a ternary condition — for example `if (getCacheObject(key))` with a `<T> T getCacheObject(String key)` method — compiles with javac but ECJ reports `Type mismatch: cannot convert from Object to boolean`. Assigning the call to a `boolean` variable first compiles on both (see [#2624](https://github.com/l0ng-ai/openidea/issues/2624)).
 - The Kotlin bridge is Maven only and exposes compiled Kotlin classes to Java. It does not give Java-to-Kotlin source navigation or refactoring.
 - Groovy needs a `groovy-language-server` you build yourself, and JSP has syntax highlighting only.

docs/zh/guide/build-tools.mdx

@@ -61,16 +61,17 @@ Maven 和 Gradle 使用设置里配置的 wrapper 或本地安装（见下文）
 
 **设置 › 构建、执行、部署 › Maven**：
 
-| 设置                         | 作用                                                                            |
-| ---------------------------- | ------------------------------------------------------------------------------- |
-| **JRE**                      | Maven 同步、构建、目标和测试使用的 JDK                                          |
-| **Maven 主路径**             | 使用指定的 Maven 安装，而不是项目里的 `mvnw`                                    |
-| **用户设置文件**             | 作为 `-s` 传入                                                                  |
-| **Toolchains 文件**          | 作为 `-t` 传入                                                                  |
-| **本地仓库**                 | 作为 `-Dmaven.repo.local` 传入                                                  |
-| **活动 profiles**            | 逗号分隔的 profile（`-P`），`!id` 表示排除                                      |
-| **线程数**                   | 作为 `-T` 传入（如 `4` 或 `2C`）                                                |
-| **使用 Maven Daemon (mvnd)** | 构建、同步和运行配置都通过 `mvnd` 执行；找不到 mvnd 或 JDK 低于 17 时退回 `mvn` |
+| 设置                                 | 作用                                                                            |
+| ------------------------------------ | ------------------------------------------------------------------------------- |
+| **将 IDE 构建/运行操作委托给 Maven** | 运行和调试时用 Maven（`javac`）构建 Maven 模块，而不是 jdtls                    |
+| **JRE**                              | Maven 同步、构建、目标和测试使用的 JDK                                          |
+| **Maven 主路径**                     | 使用指定的 Maven 安装，而不是项目里的 `mvnw`                                    |
+| **用户设置文件**                     | 作为 `-s` 传入                                                                  |
+| **Toolchains 文件**                  | 作为 `-t` 传入                                                                  |
+| **本地仓库**                         | 作为 `-Dmaven.repo.local` 传入                                                  |
+| **活动 profiles**                    | 逗号分隔的 profile（`-P`），`!id` 表示排除                                      |
+| **线程数**                           | 作为 `-T` 传入（如 `4` 或 `2C`）                                                |
+| **使用 Maven Daemon (mvnd)**         | 构建、同步和运行配置都通过 `mvnd` 执行；找不到 mvnd 或 JDK 低于 17 时退回 `mvn` |
 
 ## Gradle
 

docs/zh/guide/database.mdx

@@ -66,6 +66,7 @@ description: "连接 SQL 数据库、浏览架构、在查询控制台中执行
 
 - 项目数据源：项目中的 `.openidea/data-sources.toml`。
 - 全局数据源：IDE 数据目录下的 `data-sources.toml`（见 [数据目录](/zh/reference/data-directories)）。
+- 项目打开时，可以手动或让编码代理编辑这些文件：数据库工具窗口会自动读取改动，设置被修改或被删除的数据源的连接会被关闭。
 - 密码从不写入这些文件。保存的密码放在操作系统凭据存储中（钥匙串、Windows 凭据管理器或 Secret Service）；未保存的密码只在本次会话的内存中，连接时会询问。
 
 ## 浏览数据库

docs/zh/guide/notebook.mdx

@@ -54,6 +54,14 @@ OpenIDEA 用笔记本编辑器打开 `.ipynb` 文件。你可以编辑代码单
 
 代码单元格的工具栏旁显示执行序号（如 `[3]`）。原始单元格和 OpenIDEA 不认识的单元格类型以只读方式显示，内容原样保留。
 
+## 代码智能
+
+和 `.py` 文件一样，代码单元格可以使用项目 Python 语言服务器提供的补全、悬停快速文档、参数信息和错误高亮。语言服务器按顺序读取代码单元格，所以前面单元格定义的名称在后面的单元格里可以识别；`%matplotlib inline`、`!pip list` 这类 IPython 语法也不会被报成错误。
+
+按住 `⌘`（macOS）或 `Ctrl`（Windows/Linux）点击名称，可以跳转到它的声明，无论声明在其他单元格还是库中。
+
+这需要支持笔记本的语言服务器：**basedpyright**（默认）和 **Ruff** 都支持。使用其他 Python 语言服务器时，代码单元格只有语法高亮。
+
 ## 快捷键
 
 | 操作                 | 按键                                                                          |

docs/zh/guide/run.mdx

@@ -140,6 +140,8 @@ description: "创建、编辑和启动运行配置，从装订区运行代码，
 
 Java、Maven、Gradle 配置还可以通过 **构建** 选项决定启动前怎么构建：**自动（增量）**、**构建（Make）**、**重新构建**、**不构建** 或 **构建，不检查错误**。
 
+Maven 模块在 **自动** 和 **构建（Make）** 时由 jdtls（ECJ）构建，**重新构建** 时由 Maven 构建。打开 **设置 › 构建、执行、部署 › Maven › 将 IDE 构建/运行操作委托给 Maven** 后每次都用 Maven（项目的 `javac`）构建；编辑器的 **问题** 仍来自 jdtls。
+
 ### 复合配置
 
 **复合** 配置一次启动多个配置：运行它会启动全部成员，调试它会逐个调试每个成员。在运行微件上点停止会把它们一起停掉。

docs/zh/languages/java.mdx

@@ -86,7 +86,7 @@ jdtls 的工作区保存在 IDE 的数据目录（`lsp/jdtls/`）中，不在项
 
 jdtls 会自动导入 Maven 和 Gradle 项目（包括 Gradle wrapper）。没有构建文件的目录会作为普通 Java 项目打开，`lib/` 下的 jar 会加入类路径。OpenIDEA 还提供 **Maven** 和 **Gradle** 工具窗口，用来执行目标和任务、重新加载、切换 profile 以及分析依赖。相关设置在 **设置 › 构建、执行、部署 › Maven** 和 **Gradle**：
 
-- Maven：**JRE**、**Maven 主路径**（默认用项目的 `mvnw`，没有则用 `PATH` 上的 `mvn`）、**用户设置文件**、**Toolchains 文件**、**本地仓库**、**活动 profiles**、**线程数** 和 **使用 Maven Daemon (mvnd)**。在 Maven 工具窗口里选的 profile 也会传给 jdtls，编辑器看到的依赖与该 profile 一致。
+- Maven：**将 IDE 构建/运行操作委托给 Maven**（运行和调试时用 Maven 的 `javac` 而不是 jdtls 构建）、**JRE**、**Maven 主路径**（默认用项目的 `mvnw`，没有则用 `PATH` 上的 `mvn`）、**用户设置文件**、**Toolchains 文件**、**本地仓库**、**活动 profiles**、**线程数** 和 **使用 Maven Daemon (mvnd)**。在 Maven 工具窗口里选的 profile 也会传给 jdtls，编辑器看到的依赖与该 profile 一致。
 - Gradle：**构建和运行使用**（Gradle 或 OpenIDEA）、**Gradle 使用的 JVM**、**分发**（Gradle 包装器或本地安装）、**Gradle 主路径** 和 **Gradle 用户主目录**。
 
 IDE 会把代理设置传给 jdtls，所以在代理环境下也能解析依赖。工具窗口的用法见[构建工具](/zh/guide/build-tools)。
@@ -141,5 +141,6 @@ MyBatis mapper 接口与 XML 语句之间的跳转由 MyBatis Navigator 插件
 - 每个项目窗口运行一个 jdtls，同时打开多个大项目时内存会叠加。可以在 **JVM 选项** 中调低 `-Xmx`，或关闭暂时不用的窗口。**在构建间共享 JDT LS** 只在单个窗口内有效。
 - 没有 jdtls 时（例如只用 `kmp-lsp`），“应用程序”配置、依赖类路径的功能和 Java 调试都不可用。
 - Java 测试通过 Maven 或 Gradle 运行，两者都没有的项目没有测试运行器。
+- jdtls 自带的 ECJ 编译器有些地方比 javac 严格。已知例子：把泛型方法调用直接用作 `if`、`while`、`for`、`do` 条件或三元运算符条件——例如 `<T> T getCacheObject(String key)` 方法配合 `if (getCacheObject(key))`——javac 可以编译，ECJ 则报 `Type mismatch: cannot convert from Object to boolean`。先把调用赋值给 `boolean` 变量即可两边都编译通过（见 [#2624](https://github.com/l0ng-ai/openidea/issues/2624)）。
 - Kotlin 桥接只支持 Maven，只把编译后的 Kotlin 类暴露给 Java，不提供从 Java 到 Kotlin 源码的跳转或重构。
 - Groovy 需要自行构建 `groovy-language-server`，JSP 只有语法高亮。

evals/budgets.json

@@ -0,0 +1,11 @@
+{
+  "$comment": "Human-owned: agents don't change this file or evals/*/golden/ (a PR that does is labelled needs-decision). `*Ms` are upper bounds on medians (the user's view: input → first painted frame), `recall` / `precision` lower bounds vs the golden. Keys: `<scenario>` or `<scenario>@<condition>`. Regressions vs target/eval/baseline.json need both the relative tolerance and the absolute floor.",
+  "regressionTolerance": 0.2,
+  "regressionFloorMs": 200,
+  "defaults": {
+    "feedbackMs": 100,
+    "recall": 1,
+    "precision": 1
+  },
+  "scenarios": {}
+}

evals/java/golden/java.app.completion.jdkMember.json

@@ -0,0 +1,70 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "completions": [
+      "add",
+      "addAll",
+      "addFirst",
+      "addLast",
+      "cast",
+      "clear",
+      "contains",
+      "containsAll",
+      "else",
+      "equals",
+      "for",
+      "forEach",
+      "fori",
+      "get",
+      "getClass",
+      "getFirst",
+      "getLast",
+      "hashCode",
+      "if",
+      "indexOf",
+      "isEmpty",
+      "iterator",
+      "lastIndexOf",
+      "listIterator",
+      "nn",
+      "not",
+      "notify",
+      "notifyAll",
+      "null",
+      "orders.nnull",
+      "orders.opt",
+      "orders.syserr",
+      "orders.sysouf",
+      "orders.sysout",
+      "orders.sysoutv",
+      "par",
+      "parallelStream",
+      "remove",
+      "removeAll",
+      "removeFirst",
+      "removeIf",
+      "removeLast",
+      "replaceAll",
+      "retainAll",
+      "return",
+      "reversed",
+      "set",
+      "size",
+      "sort",
+      "sout",
+      "spliterator",
+      "stream",
+      "subList",
+      "throw",
+      "toArray",
+      "toString",
+      "try",
+      "var",
+      "wait",
+      "while"
+    ]
+  }
+}

evals/java/golden/java.app.completion.projectMember.json

@@ -0,0 +1,43 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "completions": [
+      "book.nnull",
+      "book.opt",
+      "book.syserr",
+      "book.sysouf",
+      "book.sysout",
+      "book.sysoutv",
+      "byCustomer",
+      "cast",
+      "else",
+      "equals",
+      "for",
+      "fori",
+      "getClass",
+      "hashCode",
+      "if",
+      "largest",
+      "nn",
+      "not",
+      "notify",
+      "notifyAll",
+      "null",
+      "par",
+      "place",
+      "return",
+      "revenueCents",
+      "size",
+      "sout",
+      "throw",
+      "toString",
+      "try",
+      "var",
+      "wait",
+      "while"
+    ]
+  }
+}

evals/java/golden/java.app.declaration.dependency.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/junit-jupiter-api-5.11.3.jar/org.junit.jupiter.api/Assertions.java#assertEquals"
+    ]
+  }
+}

evals/java/golden/java.app.declaration.inJdkSource.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/java.base/java.util/AbstractList.java#AbstractList"
+    ]
+  }
+}

evals/java/golden/java.app.declaration.jdk.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/java.base/java.util/ArrayList.java#ArrayList"
+    ]
+  }
+}

evals/java/golden/java.app.declaration.jdkMethod.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/java.base/java.util/Comparator.java#comparingLong"
+    ]
+  }
+}

evals/java/golden/java.app.declaration.project.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "src/main/java/com/example/orders/OrderBook.java:28"
+    ]
+  }
+}

evals/java/golden/java.app.diagnostics.afterError.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "problems": [
+      "L33:error:Type mismatch: cannot convert from String to int"
+    ]
+  }
+}

evals/java/golden/java.app.diagnostics.afterFix.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "problems": [
+      "L34:warning:The value of the local variable used is not used"
+    ]
+  }
+}

evals/java/golden/java.app.doc.jdk.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "documentation": [
+      "java.util.ArrayList.ArrayList<Order>() Constructs an empty list with an initial capacity of ten. Source: Java 21.0.13.0.2 (module: java.base)"
+    ]
+  }
+}

evals/java/golden/java.app.doc.project.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "documentation": [
+      "Order com.example.orders.OrderBook.place(String customer, String product, int quantity, long unitPriceCents) Places an order and returns it. Parameters: customer product quantity unitPriceCents Source: openidea-fixture-orders"
+    ]
+  }
+}

evals/java/golden/java.app.edit.declaration.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "src/main/java/com/example/orders/OrderBook.java:32"
+    ]
+  }
+}

evals/java/golden/java.app.edit.usagesOtherFile.json

@@ -0,0 +1,14 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/com/example/orders/ListPrice.java:7",
+      "src/main/java/com/example/orders/OrderBook.java:11",
+      "src/main/java/com/example/orders/OrderBook.java:28",
+      "src/main/java/com/example/orders/OrderBook.java:33"
+    ]
+  }
+}

evals/java/golden/java.app.edit.usagesSameFile.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/com/example/orders/App.java:12",
+      "src/main/java/com/example/orders/OrderBook.java:11",
+      "src/test/java/com/example/orders/OrderBookTest.java:15"
+    ]
+  }
+}

evals/java/golden/java.app.gutter.interface.json

@@ -0,0 +1,12 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "markers": [
+      "L4:implemented",
+      "L5:implemented"
+    ]
+  }
+}

evals/java/golden/java.app.gutter.jdkClass.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "markers": [
+      "L109:overridden"
+    ]
+  }
+}

evals/java/golden/java.app.gutter.subclass.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "markers": [
+      "L14:overriding"
+    ]
+  }
+}

evals/java/golden/java.app.implementation.interfaceMethod.json

@@ -0,0 +1,12 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "implementations": [
+      "src/main/java/com/example/orders/BulkDiscount.java:14",
+      "src/main/java/com/example/orders/ListPrice.java:6"
+    ]
+  }
+}

evals/java/golden/java.app.implementation.jdkInterface.json

@@ -0,0 +1,81 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "implementations": [
+      "jdt://contents/java.base/java.lang.invoke/AbstractConstantGroup.java:175",
+      "jdt://contents/java.base/java.util.concurrent/CopyOnWriteArrayList.java:1269",
+      "jdt://contents/java.base/java.util.concurrent/CopyOnWriteArrayList.java:1742",
+      "jdt://contents/java.base/java.util.concurrent/CopyOnWriteArrayList.java:99",
+      "jdt://contents/java.base/java.util/AbstractList.java:75",
+      "jdt://contents/java.base/java.util/AbstractList.java:750",
+      "jdt://contents/java.base/java.util/AbstractList.java:919",
+      "jdt://contents/java.base/java.util/AbstractSequentialList.java:71",
+      "jdt://contents/java.base/java.util/ArrayList.java:109",
+      "jdt://contents/java.base/java.util/ArrayList.java:1194",
+      "jdt://contents/java.base/java.util/Arrays.java:4229",
+      "jdt://contents/java.base/java.util/Collections.java:1486",
+      "jdt://contents/java.base/java.util/Collections.java:1585",
+      "jdt://contents/java.base/java.util/Collections.java:2691",
+      "jdt://contents/java.base/java.util/Collections.java:2789",
+      "jdt://contents/java.base/java.util/Collections.java:3748",
+      "jdt://contents/java.base/java.util/Collections.java:3836",
+      "jdt://contents/java.base/java.util/Collections.java:4779",
+      "jdt://contents/java.base/java.util/Collections.java:5158",
+      "jdt://contents/java.base/java.util/Collections.java:5368",
+      "jdt://contents/java.base/java.util/ImmutableCollections.java:254",
+      "jdt://contents/java.base/java.util/ImmutableCollections.java:434",
+      "jdt://contents/java.base/java.util/ImmutableCollections.java:553",
+      "jdt://contents/java.base/java.util/ImmutableCollections.java:660",
+      "jdt://contents/java.base/java.util/LinkedList.java:1292",
+      "jdt://contents/java.base/java.util/LinkedList.java:90",
+      "jdt://contents/java.base/java.util/ReverseOrderListView.java:40",
+      "jdt://contents/java.base/java.util/ReverseOrderListView.java:55",
+      "jdt://contents/java.base/java.util/Stack.java:50",
+      "jdt://contents/java.base/java.util/Vector.java:92",
+      "jdt://contents/java.base/jdk.internal.classfile.components/ClassPrinter.java:151",
+      "jdt://contents/java.base/jdk.internal.classfile.impl/ClassPrinterImpl.java:78",
+      "jdt://contents/java.base/jdk.internal.classfile.impl/Util.java:161",
+      "jdt://contents/java.base/jdk.internal.org.objectweb.asm.tree/MethodNode.java:281",
+      "jdt://contents/java.base/sun.net.www.http/KeepAliveStreamCleaner.java:49",
+      "jdt://contents/java.base/sun.security.jca/ProviderList.java:153",
+      "jdt://contents/java.base/sun.security.jca/ProviderList.java:431",
+      "jdt://contents/java.desktop/javax.swing/JEditorPane.java:2010",
+      "jdt://contents/java.desktop/sun.awt.util/IdentityArrayList.java:95",
+      "jdt://contents/java.desktop/sun.swing/BakedArrayList.java:47",
+      "jdt://contents/java.management/com.sun.jmx.remote.internal/ArrayQueue.java:31",
+      "jdt://contents/java.management/javax.management.relation/RoleList.java:56",
+      "jdt://contents/java.management/javax.management.relation/RoleUnresolvedList.java:55",
+      "jdt://contents/java.management/javax.management/AttributeList.java:65",
+      "jdt://contents/java.xml/com.sun.org.apache.xalan.internal.xsltc.compiler.util/StringStack.java:30",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.util/ByteListImpl.java:36",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/Base64BinaryDV.java:60",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/HexBinaryDV.java:57",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/ListDV.java:58",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/XSSimpleTypeDecl.java:2263",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/XSSimpleTypeDecl.java:2298",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.dv.xs/XSSimpleTypeDecl.java:3447",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs.util/LSInputListImpl.java:38",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs.util/ObjectListImpl.java:35",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs.util/ShortListImpl.java:36",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs.util/StringListImpl.java:38",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs.util/XSObjectListImpl.java:41",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs/PSVIErrorList.java:35",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.impl.xs/XSModelImpl.java:63",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs.datatypes/ByteList.java:34",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs.datatypes/ObjectList.java:33",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs/LSInputList.java:33",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs/ShortList.java:31",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs/StringList.java:31",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs/XSNamespaceItemList.java:32",
+      "jdt://contents/java.xml/com.sun.org.apache.xerces.internal.xs/XSObjectList.java:32",
+      "jdt://contents/jdk.compiler/com.sun.tools.javac.model/FilteredMemberList.java:46",
+      "jdt://contents/jdk.compiler/com.sun.tools.javac.util/List.java:53",
+      "jdt://contents/jdk.compiler/com.sun.tools.javac.util/List.java:79",
+      "jdt://contents/jdk.jdi/com.sun.tools.jdi/EventSetImpl.java:86",
+      "jdt://contents/jdk.jshell/jdk.jshell/DiagList.java:38"
+    ]
+  }
+}

evals/java/golden/java.app.implementation.projectInterface.json

@@ -0,0 +1,12 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "implementations": [
+      "src/main/java/com/example/orders/BulkDiscount.java:4",
+      "src/main/java/com/example/orders/ListPrice.java:4"
+    ]
+  }
+}

evals/java/golden/java.app.lens.jdkClass.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "lenses": [
+      "L33:2 usages",
+      "L73:no usages",
+      "L92:no usages"
+    ]
+  }
+}

evals/java/golden/java.app.lens.project.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "lenses": [
+      "L10:3 usages",
+      "L3:12 usages",
+      "L6:2 usages"
+    ]
+  }
+}

evals/java/golden/java.app.rename.acrossFiles.json

@@ -0,0 +1,14 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "changes": [
+      "src/main/java/com/example/orders/ListPrice.java:7",
+      "src/main/java/com/example/orders/Order.java:11",
+      "src/main/java/com/example/orders/OrderBook.java:24",
+      "src/main/java/com/example/orders/OrderBook.java:29"
+    ]
+  }
+}

evals/java/golden/java.app.rename.local.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "changes": [
+      "src/main/java/com/example/orders/OrderBook.java:14",
+      "src/main/java/com/example/orders/OrderBook.java:15",
+      "src/main/java/com/example/orders/OrderBook.java:16"
+    ]
+  }
+}

evals/java/golden/java.app.usages.inJdkSource.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/com/example/orders/OrderBook.java:20",
+      "src/main/java/com/example/orders/OrderBook.java:24",
+      "src/main/java/com/example/orders/OrderBook.java:29"
+    ]
+  }
+}

evals/java/golden/java.app.usages.jdkMethodLibraries.json

@@ -0,0 +1,16 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "jdt://contents/java.base/java.util/Comparator.java:302",
+      "jdt://contents/java.base/java.util/Comparator.java:307",
+      "jdt://contents/jdk.jfr/jdk.jfr.internal.dcmd/AbstractDCmd.java:174",
+      "jdt://contents/jdk.jfr/jdk.jfr.internal.periodic/BatchManager.java:84",
+      "jdt://contents/jdk.jfr/jdk.jfr.internal/TypeLibrary.java:109",
+      "src/main/java/com/example/orders/OrderBook.java:24"
+    ]
+  }
+}

evals/java/golden/java.app.usages.method.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/com/example/orders/ListPrice.java:7",
+      "src/main/java/com/example/orders/OrderBook.java:24",
+      "src/main/java/com/example/orders/OrderBook.java:29"
+    ]
+  }
+}

evals/java/golden/java.app.usages.record.json

@@ -0,0 +1,21 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/com/example/orders/BulkDiscount.java:14",
+      "src/main/java/com/example/orders/ListPrice.java:6",
+      "src/main/java/com/example/orders/OrderBook.java:10",
+      "src/main/java/com/example/orders/OrderBook.java:13",
+      "src/main/java/com/example/orders/OrderBook.java:14",
+      "src/main/java/com/example/orders/OrderBook.java:19",
+      "src/main/java/com/example/orders/OrderBook.java:23",
+      "src/main/java/com/example/orders/OrderBook.java:24",
+      "src/main/java/com/example/orders/OrderBook.java:29",
+      "src/main/java/com/example/orders/PricingRule.java:5",
+      "src/test/java/com/example/orders/OrderBookTest.java:21"
+    ]
+  }
+}

evals/java/golden/java.petclinic.completion.libraryClass.json

@@ -0,0 +1,18 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "completions": [
+      "Pageable",
+      "PageableArgumentResolver",
+      "PageableDefault",
+      "PageableExecutionUtils",
+      "PageableHandlerMethodArgumentResolver",
+      "PageableHandlerMethodArgumentResolverCustomizer",
+      "PageableHandlerMethodArgumentResolverSupport",
+      "PageableUtils"
+    ]
+  }
+}

evals/java/golden/java.petclinic.declaration.inheritedLibrary.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/spring-data-commons-4.1.0.jar/org.springframework.data.repository/CrudRepository.java#save"
+    ]
+  }
+}

evals/java/golden/java.petclinic.declaration.libraryClass.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "jdt://contents/spring-data-commons-4.1.0.jar/org.springframework.data.domain/PageRequest.java#PageRequest"
+    ]
+  }
+}

evals/java/golden/java.petclinic.declaration.project.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "declaration": [
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:45"
+    ]
+  }
+}

evals/java/golden/java.petclinic.diagnostics.cleanFile.json

@@ -0,0 +1,9 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "problems": []
+  }
+}

evals/java/golden/java.petclinic.doc.library.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "documentation": [
+      "org.springframework.data.domain.PageRequest Basic Java Bean implementation of Pageable. Author: Oliver Gierke Thomas Darimont Anastasiia Smirnova Mark Paluch Thach Le Source: spring-data-commons-4.1.0.jar"
+    ]
+  }
+}

evals/java/golden/java.petclinic.gutter.baseClass.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "markers": [
+      "L33:overridden"
+    ]
+  }
+}

evals/java/golden/java.petclinic.gutter.midClass.json

@@ -0,0 +1,11 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "markers": [
+      "L29:overridden"
+    ]
+  }
+}

evals/java/golden/java.petclinic.implementation.baseClass.json

@@ -0,0 +1,20 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "implementations": [
+      "src/main/java/org/springframework/samples/petclinic/model/NamedEntity.java:31",
+      "src/main/java/org/springframework/samples/petclinic/model/Person.java:29",
+      "src/main/java/org/springframework/samples/petclinic/owner/Owner.java:50",
+      "src/main/java/org/springframework/samples/petclinic/owner/Pet.java:46",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetType.java:28",
+      "src/main/java/org/springframework/samples/petclinic/owner/Visit.java:37",
+      "src/main/java/org/springframework/samples/petclinic/vet/Specialty.java:30",
+      "src/main/java/org/springframework/samples/petclinic/vet/Vet.java:45",
+      "src/test/java/org/springframework/samples/petclinic/owner/PetTypeFormatterTests.java:84",
+      "src/test/java/org/springframework/samples/petclinic/owner/PetTypeFormatterTests.java:89"
+    ]
+  }
+}

evals/java/golden/java.petclinic.lens.entity.json

@@ -0,0 +1,16 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "lenses": [
+      "L38:82 usages",
+      "L72:no usages",
+      "L76:3 usages",
+      "L80:no usages",
+      "L84:3 usages",
+      "L88:no usages"
+    ]
+  }
+}

evals/java/golden/java.petclinic.rename.acrossFiles.json

@@ -0,0 +1,13 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "changes": [
+      "src/main/java/org/springframework/samples/petclinic/owner/Owner.java:176",
+      "src/main/java/org/springframework/samples/petclinic/owner/VisitController.java:108",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:229"
+    ]
+  }
+}

evals/java/golden/java.petclinic.usages.entity.json

@@ -0,0 +1,83 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:107",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:132",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:133",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:141",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:154",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:181",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:182",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:65",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:66",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:78",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:95",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:36",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:39",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:42",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:45",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:48",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:50",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:51",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:55",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerRepository.java:60",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:101",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:108",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:145",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:186",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:68",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:69",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:70",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:83",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:84",
+      "src/main/java/org/springframework/samples/petclinic/owner/VisitController.java:66",
+      "src/main/java/org/springframework/samples/petclinic/owner/VisitController.java:67",
+      "src/main/java/org/springframework/samples/petclinic/owner/VisitController.java:98",
+      "src/test/java/org/springframework/samples/petclinic/PetClinicConcurrencyTests.java:114",
+      "src/test/java/org/springframework/samples/petclinic/PetClinicConcurrencyTests.java:22",
+      "src/test/java/org/springframework/samples/petclinic/PetClinicConcurrencyTests.java:43",
+      "src/test/java/org/springframework/samples/petclinic/PetClinicConcurrencyTests.java:45",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:176",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:192",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:201",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:215",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:227",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:302",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:73",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:74",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerControllerTests.java:95",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerTests.java:27",
+      "src/test/java/org/springframework/samples/petclinic/owner/OwnerTests.java:40",
+      "src/test/java/org/springframework/samples/petclinic/owner/PetControllerTests.java:167",
+      "src/test/java/org/springframework/samples/petclinic/owner/PetControllerTests.java:258",
+      "src/test/java/org/springframework/samples/petclinic/owner/PetControllerTests.java:77",
+      "src/test/java/org/springframework/samples/petclinic/owner/VisitControllerTests.java:61",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:100",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:102",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:112",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:115",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:131",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:133",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:160",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:162",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:188",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:190",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:220",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:222",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:239",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:241",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:256",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:258",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:285",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:287",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:296",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:298",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:34",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:91"
+    ]
+  }
+}

evals/java/golden/java.petclinic.usages.inheritedMethod.json

@@ -0,0 +1,24 @@
+{
+  "tools": {
+    "jdtls": "1.63.0-202610191322",
+    "java": "21.0.13.0.2 (Homebrew)"
+  },
+  "groups": {
+    "usages": [
+      "src/main/java/org/springframework/samples/petclinic/owner/Owner.java:108",
+      "src/main/java/org/springframework/samples/petclinic/owner/Owner.java:132",
+      "src/main/java/org/springframework/samples/petclinic/owner/Owner.java:161",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:125",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:161",
+      "src/main/java/org/springframework/samples/petclinic/owner/OwnerController.java:86",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:153",
+      "src/main/java/org/springframework/samples/petclinic/owner/PetController.java:187",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:122",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:182",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:229",
+      "src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java:234",
+      "src/test/java/org/springframework/samples/petclinic/service/EntityUtils.java:46",
+      "src/test/java/org/springframework/samples/petclinic/vet/VetTests.java:38"
+    ]
+  }
+}

evals/java/scenarios.ts

@@ -0,0 +1,841 @@
+/**
+ * Java code intelligence with jdtls: the steps users take most, timed as they see them and
+ * checked against `golden/<id>.json`. On `fixtures/java-app` (small, every case visible) and
+ * on Spring PetClinic (`evals/projects.json`: a real Spring Boot / Maven project).
+ */
+import { join } from "node:path";
+import { track } from "../lib/measure.ts";
+import { normalizePath, type EvalScenario, type Measured } from "../lib/scenario.ts";
+import type { Session } from "../lib/session.ts";
+
+const SRC = "src/main/java/com/example/orders";
+const TEST = "src/test/java/com/example/orders";
+const PET = "src/main/java/org/springframework/samples/petclinic";
+const TIMEOUT = 30_000;
+/** Results a whole file's worth of server requests make up. */
+const LONG_TIMEOUT = 60_000;
+
+interface ProbeWindow {
+  __openideaPerf: {
+    editorState(): { path: string; head: number; line: number; word: string } | null;
+    editorText(): string | null;
+    setCursor(offset: number): void;
+    insertAfter(after: string, text: string): number;
+    completions(): string[];
+    completionPending(): boolean;
+    diagnostics(): Array<{ line: number; severity: string; message: string }>;
+    lineAtY(y: number): number;
+  };
+}
+
+/** The visible editor hint, unless it only says a request is pending. */
+async function finalHint(s: Session): Promise<string | null> {
+  return s.page.evaluate(() => {
+    const el = document.querySelector(".cm-lsp-hint:not([data-pending])") as HTMLElement | null;
+    return el && el.getClientRects().length ? el.textContent : null;
+  });
+}
+
+/** Where the caret is: `path:line` in the project, `uri#word` in a library class (its line depends on the JDK build). */
+async function caretLocation(s: Session): Promise<string | null> {
+  const st = await s.page.evaluate(() => (window as unknown as ProbeWindow).__openideaPerf.editorState());
+  if (!st) return null;
+  const path = normalizePath(s.root, st.path);
+  return path.includes("://") ? `${path}#${st.word}` : `${path}:${st.line + 1}`;
+}
+
+/**
+ * Where a step starts: inside `word` on the first line containing `line` of `file`, or of the
+ * library class reached from `file` by going to the declaration of each `via` in turn. `edit`
+ * changes `file` first; `typed` changes the file reached right before the step, with no time
+ * for the server to catch up.
+ */
+interface Place {
+  file: string;
+  edit?: Insertion;
+  via?: Array<[line: string, word: string]>;
+  typed?: Insertion;
+  line: string;
+  word: string;
+}
+
+interface Insertion {
+  after: string;
+  text: string;
+}
+
+async function reach(s: Session, at: Place) {
+  await s.openFile(at.file);
+  if (at.edit) await s.edit(at.edit.after, at.edit.text);
+  for (const [line, word] of at.via ?? []) await s.declarationOf(line, word);
+  if (!at.typed) await s.caretAt(at.line, at.word);
+}
+
+/** What a step does first when its place is `typed`. */
+async function type(s: Session, at: Place) {
+  if (!at.typed) return;
+  await s.edit(at.typed.after, at.typed.text);
+  await s.caretAt(at.line, at.word);
+}
+
+/** Text that marks an answer from syntax instead of the language server. */
+const SYNTAX = /based on syntax|language server not ready/i;
+
+function declaration(
+  project: string,
+  id: string,
+  title: string,
+  at: Place,
+  conditions: EvalScenario["conditions"] = ["warm"],
+): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions,
+    setup: (s) => reach(s, at),
+    async step(s): Promise<Measured> {
+      await type(s, at);
+      // Moved to the declaration, or a hint saying why not (not one saying it is still looking).
+      const answered = (_: null, before: { path: string; head: number } | null) => {
+        const st = (window as unknown as ProbeWindow).__openideaPerf.editorState();
+        if (st && before && (st.path !== before.path || st.head !== before.head)) return true;
+        return !!document.querySelector(".cm-lsp-hint:not([data-pending])");
+      };
+      const timing = await track(s.page, {
+        action: "navigate.declaration",
+        arg: null,
+        first: answered,
+        done: answered,
+        timeoutMs: TIMEOUT,
+      });
+      // A note that the answer came from syntax follows the navigation.
+      await s.page.waitForTimeout(300);
+      const hint = await finalHint(s);
+      const landed = await caretLocation(s);
+      return {
+        timing,
+        results: { declaration: landed ? [landed] : [] },
+        degraded: hint && SYNTAX.test(hint) ? hint : null,
+      };
+    },
+  };
+}
+
+/** Every usage row of the popup, scrolling its virtual list through. */
+async function usageRows(s: Session): Promise<string[]> {
+  const rows = await s.page.evaluate(async () => {
+    const popup = document.querySelector('[data-testid="usages-popup"]');
+    if (!popup) return [];
+    const found = new Set<string>();
+    const collect = () => {
+      for (const r of popup.querySelectorAll<HTMLElement>('[data-testid="usage-row"]'))
+        found.add(`${r.dataset.path}\u0000${Number(r.dataset.line) + 1}`);
+    };
+    const scroller = [...popup.querySelectorAll<HTMLElement>("*")].find((el) => el.scrollHeight > el.clientHeight + 1);
+    collect();
+    if (scroller) {
+      for (let top = 0; top < scroller.scrollHeight; top += scroller.clientHeight / 2) {
+        scroller.scrollTop = top;
+        await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)));
+        collect();
+      }
+    }
+    return [...found];
+  });
+  return rows.map((r) => {
+    const [path, line] = r.split("\u0000") as [string, string];
+    return `${normalizePath(s.root, path)}:${line}`;
+  });
+}
+
+/**
+ * Show Usages at `at`. `widen`: in Project and Libraries, as invoking it again while the popup
+ * or its hint offers to widen a Project Files search does (the step is that second invocation).
+ */
+function showUsages(project: string, id: string, title: string, at: Place, widen = false): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup(s) {
+      await reach(s, at);
+      if (!widen) return;
+      await s.runAction("find.showUsages");
+      await s.page.waitForFunction(
+        () =>
+          document.querySelector('[data-testid="usages-popup"][data-loading="false"][data-scope="project"]') ||
+          document.querySelector(".cm-lsp-hint[data-usages]"),
+        undefined,
+        { timeout: TIMEOUT },
+      );
+    },
+    async step(s): Promise<Measured> {
+      await type(s, at);
+      const timing = await track(s.page, {
+        action: "find.showUsages",
+        arg: widen,
+        // A single usage is jumped to; none says so in a hint.
+        first: (widen, before) => {
+          const popup = document.querySelector<HTMLElement>('[data-testid="usages-popup"]');
+          if (popup)
+            return (
+              (!widen || popup.dataset.scope === "libraries") && !!popup.querySelector('[data-testid="usage-row"]')
+            );
+          const st = (window as unknown as ProbeWindow).__openideaPerf.editorState();
+          if (st && before && (st.path !== before.path || st.head !== before.head)) return true;
+          return !!document.querySelector(widen ? '.cm-lsp-hint[data-scope="libraries"]' : ".cm-lsp-hint[data-usages]");
+        },
+        done: (widen, before) => {
+          const popup = document.querySelector<HTMLElement>('[data-testid="usages-popup"]');
+          if (popup) return (!widen || popup.dataset.scope === "libraries") && popup.dataset.loading === "false";
+          const st = (window as unknown as ProbeWindow).__openideaPerf.editorState();
+          if (st && before && (st.path !== before.path || st.head !== before.head)) return true;
+          return !!document.querySelector(widen ? '.cm-lsp-hint[data-scope="libraries"]' : ".cm-lsp-hint[data-usages]");
+        },
+        timeoutMs: TIMEOUT,
+      });
+      const popup = await s.page.$('[data-testid="usages-popup"]');
+      const rows = popup ? await usageRows(s) : [];
+      // Without a popup the hint gives the count: the only usage was jumped to (or the caret is on it).
+      const count = await s.page.evaluate(
+        () => (document.querySelector(".cm-lsp-hint[data-usages]") as HTMLElement | null)?.dataset.usages ?? null,
+      );
+      const single = !popup && count === "1" ? await caretLocation(s) : null;
+      const note = await s.page.evaluate(
+        () => document.querySelector('[data-testid="usages-note"]')?.textContent ?? null,
+      );
+      const hint = await finalHint(s);
+      return {
+        timing,
+        results: { usages: single ? [single] : rows },
+        degraded: note ?? (hint && SYNTAX.test(hint) ? hint : null),
+      };
+    },
+  };
+}
+
+/** Every row of the targets chooser (`path:line`), scrolling its virtual list through. */
+async function chooserRows(s: Session): Promise<string[]> {
+  const rows = await s.page.evaluate(async () => {
+    const popup = document.querySelector('[data-testid="list-popup"]');
+    if (!popup) return [];
+    const found = new Set<string>();
+    const collect = () => {
+      for (const r of popup.querySelectorAll<HTMLElement>('[data-testid="list-popup-item"]'))
+        if (r.dataset.path) found.add(`${r.dataset.path}\u0000${Number(r.dataset.line) + 1}`);
+    };
+    const scroller = [...popup.querySelectorAll<HTMLElement>("*")].find((el) => el.scrollHeight > el.clientHeight + 1);
+    collect();
+    if (scroller) {
+      for (let top = 0; top < scroller.scrollHeight; top += scroller.clientHeight / 2) {
+        scroller.scrollTop = top;
+        await new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)));
+        collect();
+      }
+    }
+    return [...found];
+  });
+  return rows.map((r) => {
+    const [path, line] = r.split("\u0000") as [string, string];
+    return `${normalizePath(s.root, path)}:${line}`;
+  });
+}
+
+/** Go to Implementation: the chooser's rows once it stops searching, or the only target. */
+function implementation(project: string, id: string, title: string, at: Place): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    setup: (s) => reach(s, at),
+    async step(s): Promise<Measured> {
+      const before = await caretLocation(s);
+      // The chooser (busy while the server looks on), the only target, or a hint.
+      const answered = (searching: boolean, before: { path: string; head: number } | null) => {
+        const popup = document.querySelector<HTMLElement>('[data-testid="list-popup"]');
+        if (popup) return searching || popup.getAttribute("aria-busy") !== "true";
+        const st = (window as unknown as ProbeWindow).__openideaPerf.editorState();
+        if (st && before && (st.path !== before.path || st.head !== before.head)) return true;
+        return !!document.querySelector(".cm-lsp-hint:not([data-pending])");
+      };
+      // Checks run in the page from their source, so the shared one goes in as the argument's source.
+      const timing = await track(s.page, {
+        action: "navigate.implementation",
+        arg: answered.toString(),
+        first: (src, before) => (new Function(`return ${src}`)() as typeof answered)(true, before),
+        done: (src, before) => (new Function(`return ${src}`)() as typeof answered)(false, before),
+        timeoutMs: TIMEOUT,
+      });
+      const popup = await s.page.$('[data-testid="list-popup"]');
+      const rows = popup ? await chooserRows(s) : [];
+      const at = await caretLocation(s);
+      const hint = await finalHint(s);
+      return {
+        timing,
+        results: { implementations: popup ? rows : at && at !== before ? [at] : [] },
+        degraded: hint && SYNTAX.test(hint) ? hint : null,
+      };
+    },
+  };
+}
+
+/**
+ * What opening `file`, or the library class reached from it through `via` (see `Place`), brings
+ * up beside its text: every `selector` with its line, as `L<line>:<its kind or text>`.
+ */
+function onOpen(
+  project: string,
+  id: string,
+  title: string,
+  file: string,
+  via: NonNullable<Place["via"]>,
+  group: string,
+  selector: string,
+): EvalScenario {
+  const last = via[via.length - 1];
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup(s) {
+      if (!last) return;
+      await s.openFile(file);
+      for (const [line, word] of via.slice(0, -1)) await s.declarationOf(line, word);
+      await s.caretAt(last[0], last[1]);
+    },
+    async step(s): Promise<Measured> {
+      const items = (selector: string) => {
+        const probe = (window as unknown as ProbeWindow).__openideaPerf;
+        const out: string[] = [];
+        for (const m of document.querySelectorAll<HTMLElement>(selector)) {
+          // A lens is a row of its own above the line it is about.
+          const r = (m.closest(".cm-openidea-lens")?.nextElementSibling ?? m).getBoundingClientRect();
+          out.push(`L${probe.lineAtY(r.top + r.height / 2) + 1}:${m.dataset.kind ?? m.textContent}`);
+        }
+        return out.sort();
+      };
+      const timing = await track(s.page, {
+        // A library class has no path to open: it is reached the way users do, which also shows its declaration.
+        ...(last ? { action: "navigate.declaration" } : { openFile: join(s.root, file) }),
+        arg: selector,
+        first: (selector, before) =>
+          (window as unknown as ProbeWindow).__openideaPerf.editorState()?.path !== before?.path &&
+          !!document.querySelector(selector),
+        done: () => true,
+        signature: items,
+        settleMs: 2000,
+        timeoutMs: LONG_TIMEOUT,
+      });
+      return { timing, results: { [group]: await s.page.evaluate(items, selector) } };
+    },
+  };
+}
+
+const GUTTER = '.cm-editor [class*="cm-openidea-lineMarker-"]';
+
+function gutterOnOpen(project: string, id: string, title: string, file: string, via: Place["via"] = []): EvalScenario {
+  return onOpen(project, id, title, file, via, "markers", GUTTER);
+}
+
+function lensOnOpen(project: string, id: string, title: string, file: string, via: Place["via"] = []): EvalScenario {
+  return onOpen(project, id, title, file, via, "lenses", ".cm-editor .cm-openidea-lensAction");
+}
+
+/** Errors and warnings in the active editor: hints and notes come and go with the server's settings. */
+const problems = () =>
+  (window as unknown as ProbeWindow).__openideaPerf
+    .diagnostics()
+    .filter((d) => d.severity === "error" || d.severity === "warning")
+    .map((d) => `L${d.line + 1}:${d.severity}:${d.message}`)
+    .sort();
+
+/** Waits for what the previous run left in the active editor to go. */
+async function noErrors(s: Session) {
+  await s.page.waitForFunction(() => !document.querySelector(".cm-lintRange-error"), undefined, { timeout: TIMEOUT });
+}
+
+/** Opening a correct `file`: nothing is reported in it, once the server has had its say. */
+function diagnosticsOnOpen(project: string, id: string, title: string, file: string): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup() {},
+    async step(s): Promise<Measured> {
+      const timing = await track(s.page, {
+        openFile: join(s.root, file),
+        arg: null,
+        first: () => !!document.querySelector(".cm-content .cm-line"),
+        done: () => true,
+        signature: problems,
+        settleMs: 3000,
+        timeoutMs: TIMEOUT,
+      });
+      return { timing, results: { problems: await s.page.evaluate(problems) } };
+    },
+  };
+}
+
+/** Typing `edit` into a correct file: the error shows under it. */
+function diagnosticsAfterEdit(project: string, id: string, title: string, at: Place, edit: Insertion): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup(s) {
+      await reach(s, at);
+      await noErrors(s);
+      await s.keepText();
+    },
+    async step(s): Promise<Measured> {
+      const timing = await track(s.page, {
+        run: (edit) => (window as unknown as ProbeWindow).__openideaPerf.insertAfter(edit.after, edit.text),
+        arg: edit,
+        // The typed text answers the keystroke; the error is the result.
+        feedback: () => true,
+        first: () => !!document.querySelector(".cm-lintRange-error"),
+        done: () => true,
+        signature: problems,
+        settleMs: 2000,
+        timeoutMs: TIMEOUT,
+      });
+      return { timing, results: { problems: await s.page.evaluate(problems) } };
+    },
+  };
+}
+
+/** Typing `fix` elsewhere repairs the mistake `edit` made: only the server can clear its error. */
+function diagnosticsAfterFix(
+  project: string,
+  id: string,
+  title: string,
+  at: Place,
+  edit: Insertion,
+  fix: Insertion,
+): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup(s) {
+      await reach(s, at);
+      await noErrors(s);
+      await s.edit(edit.after, edit.text);
+      await s.page.waitForSelector(".cm-lintRange-error", { timeout: TIMEOUT });
+    },
+    async step(s): Promise<Measured> {
+      const timing = await track(s.page, {
+        run: (fix) => (window as unknown as ProbeWindow).__openideaPerf.insertAfter(fix.after, fix.text),
+        arg: fix,
+        feedback: () => true,
+        first: () => !document.querySelector(".cm-lintRange-error"),
+        done: () => true,
+        signature: problems,
+        settleMs: 2000,
+        timeoutMs: TIMEOUT,
+      });
+      return { timing, results: { problems: await s.page.evaluate(problems) } };
+    },
+  };
+}
+
+/** Code Completion right after typing `edit`: the lookup's labels. */
+function completion(project: string, id: string, title: string, file: string, edit: Insertion): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    setup: (s) => s.openFile(file),
+    async step(s): Promise<Measured> {
+      const end = await s.edit(edit.after, edit.text);
+      await s.page.evaluate((o) => (window as unknown as ProbeWindow).__openideaPerf.setCursor(o), end);
+      const timing = await track(s.page, {
+        action: "code.completion",
+        arg: null,
+        first: () => (window as unknown as ProbeWindow).__openideaPerf.completions().length > 0,
+        // Quicker sources show first: every source has answered once it stops being pending.
+        done: () => !(window as unknown as ProbeWindow).__openideaPerf.completionPending(),
+        timeoutMs: TIMEOUT,
+      });
+      const labels = await s.page.evaluate(() => (window as unknown as ProbeWindow).__openideaPerf.completions());
+      return { timing, results: { completions: labels } };
+    },
+  };
+}
+
+/** Quick Documentation: the start of what the popup says. */
+function quickDoc(project: string, id: string, title: string, at: Place): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    setup: (s) => reach(s, at),
+    async step(s): Promise<Measured> {
+      const shown = () => !!document.querySelector('[data-testid="doc-popup"], .cm-lsp-hint:not([data-pending])');
+      const text = () =>
+        document.querySelector('[data-testid="doc-popup"]')?.textContent?.replace(/\s+/g, " ").trim().slice(0, 300) ??
+        "";
+      const timing = await track(s.page, {
+        action: "code.quickDocumentation",
+        arg: null,
+        feedback: () => !!document.querySelector('[data-testid="doc-popup"]'),
+        first: shown,
+        done: shown,
+        signature: text,
+        settleMs: 500,
+        timeoutMs: TIMEOUT,
+      });
+      const doc = await s.page.evaluate(text);
+      const hint = await finalHint(s);
+      return {
+        timing,
+        results: { documentation: doc ? [doc] : [] },
+        degraded: hint && SYNTAX.test(hint) ? hint : null,
+      };
+    },
+  };
+}
+
+interface RenameAnswer {
+  edit: { files: Array<{ path?: string; edits?: Array<{ range: { start: { line: number } }; new_text: string }> }> };
+}
+
+/** Rename in place to `newName`, then ↵: every edit it makes (a preview is left unapplied). */
+function rename(project: string, id: string, title: string, at: Place, newName: string): EvalScenario {
+  return {
+    id,
+    title,
+    fixture: project,
+    server: "jdtls",
+    conditions: ["warm"],
+    async setup(s) {
+      await reach(s, at);
+      await s.keepText();
+      await s.runAction("refactor.rename");
+      await s.page.waitForSelector(".cm-lsp-inplace-primary", { timeout: TIMEOUT });
+      await s.page.keyboard.type(newName);
+    },
+    async step(s): Promise<Measured> {
+      const answered = (newName: string) => {
+        if (document.querySelector('[data-testid="refactor-preview"]')) return true;
+        if (document.querySelector(".cm-lsp-hint:not([data-pending]):not(.cm-lsp-inplace-hint)")) return true;
+        return (
+          !document.querySelector(".cm-lsp-inplace") &&
+          !!(window as unknown as ProbeWindow).__openideaPerf.editorText()?.includes(newName)
+        );
+      };
+      const timing = await track(s.page, {
+        run: () =>
+          document.querySelector(".cm-content")?.dispatchEvent(
+            new KeyboardEvent("keydown", {
+              key: "Enter",
+              code: "Enter",
+              keyCode: 13,
+              bubbles: true,
+              cancelable: true,
+            }),
+          ),
+        arg: newName,
+        first: answered,
+        done: answered,
+        timeoutMs: TIMEOUT,
+      });
+      // The preview only counts per file: where the name lands is in the answer it shows. A server
+      // may merge nearby occurrences into one edit, so every line of its new text counts.
+      const answer = s.lastAnswer.get("lsp_rename") as RenameAnswer | undefined;
+      const changes = (answer?.edit.files ?? []).flatMap((f) =>
+        (f.edits ?? []).flatMap((e) =>
+          e.new_text
+            .split("\n")
+            .flatMap((l, i) =>
+              l.includes(newName) ? [`${normalizePath(s.root, f.path!)}:${e.range.start.line + i + 1}`] : [],
+            ),
+        ),
+      );
+      return { timing, results: { changes } };
+    },
+  };
+}
+
+const APP = "java-app";
+const PETCLINIC = "spring-petclinic";
+
+/** From `OrderBook` into the JDK's `ArrayList` / `Collection` sources. */
+const BOOK = `${SRC}/OrderBook.java`;
+const TO_ARRAYLIST: [string, string] = ["private final List<Order> orders = new ArrayList<>();", "ArrayList"];
+/** A new method above the rest of `OrderBook`, calling `revenueCents`. */
+const TWICE = "\n    public long twice() {\n        return revenueCents();\n    }\n";
+const TYPE_ERROR: Insertion = { after: "public int size() {", text: '\n        int broken = "text";' };
+const TO_STREAM: [string, string] = ["return orders.stream().mapToLong(Order::totalCents).sum();", "stream"];
+
+export const JAVA_SCENARIOS: EvalScenario[] = [
+  declaration(
+    APP,
+    "java.app.declaration.project",
+    "Go to declaration: project method",
+    { file: `${SRC}/App.java`, line: "long revenue = book.revenueCents();", word: "revenueCents" },
+    ["warm", "cold"],
+  ),
+  declaration(APP, "java.app.declaration.jdk", "Go to declaration: JDK class", {
+    file: BOOK,
+    line: TO_ARRAYLIST[0],
+    word: TO_ARRAYLIST[1],
+  }),
+  declaration(APP, "java.app.declaration.dependency", "Go to declaration: dependency method (JUnit)", {
+    file: `${TEST}/OrderBookTest.java`,
+    line: "assertEquals(2500, book.revenueCents());",
+    word: "assertEquals",
+  }),
+  showUsages(APP, "java.app.usages.record", "Show Usages: record used across files", {
+    file: `${SRC}/Order.java`,
+    line: "public record Order(",
+    word: "Order",
+  }),
+  showUsages(APP, "java.app.usages.method", "Show Usages: method used through calls and method references", {
+    file: `${SRC}/Order.java`,
+    line: "public long totalCents() {",
+    word: "totalCents",
+  }),
+  gutterOnOpen(APP, "java.app.gutter.interface", "Gutter markers: open an interface", `${SRC}/PricingRule.java`),
+  gutterOnOpen(
+    APP,
+    "java.app.gutter.subclass",
+    "Gutter markers: open an overriding subclass",
+    `${SRC}/BulkDiscount.java`,
+  ),
+
+  // The JDK's sources: their lines depend on its build (`java` in .github/tool-versions.env).
+  declaration(APP, "java.app.declaration.jdkMethod", "Go to declaration: JDK static method", {
+    file: BOOK,
+    line: "return orders.stream().max(Comparator.comparingLong(Order::totalCents));",
+    word: "comparingLong",
+  }),
+  declaration(APP, "java.app.declaration.inJdkSource", "Go to declaration from a JDK class: its superclass", {
+    file: BOOK,
+    via: [TO_ARRAYLIST],
+    line: "extends AbstractList<E>",
+    word: "AbstractList",
+  }),
+  showUsages(
+    APP,
+    "java.app.usages.jdkMethodLibraries",
+    "Show Usages in Project and Libraries: JDK method called once in the project",
+    {
+      file: BOOK,
+      line: "return orders.stream().max(Comparator.comparingLong(Order::totalCents));",
+      word: "comparingLong",
+    },
+    true,
+  ),
+  showUsages(APP, "java.app.usages.inJdkSource", "Show Usages from a JDK class: a default method the project calls", {
+    file: BOOK,
+    via: [TO_STREAM],
+    line: "default Stream<E> stream() {",
+    word: "stream",
+  }),
+  implementation(APP, "java.app.implementation.jdkInterface", "Go to Implementation: JDK interface", {
+    file: BOOK,
+    line: TO_ARRAYLIST[0],
+    word: "List",
+  }),
+  // From the import: `new ArrayList<>()` lands on a constructor, with no markers in view.
+  gutterOnOpen(APP, "java.app.gutter.jdkClass", "Gutter markers: open a JDK class", BOOK, [
+    ["import java.util.ArrayList;", "ArrayList"],
+  ]),
+  lensOnOpen(APP, "java.app.lens.project", "Usage lenses: open a project class", `${SRC}/Order.java`),
+  lensOnOpen(APP, "java.app.lens.jdkClass", "Usage lenses: open a JDK class", BOOK, [
+    ["public Optional<Order> largest() {", "Optional"],
+  ]),
+
+  implementation(APP, "java.app.implementation.projectInterface", "Go to Implementation: project interface", {
+    file: `${SRC}/PricingRule.java`,
+    line: "public interface PricingRule {",
+    word: "PricingRule",
+  }),
+  implementation(APP, "java.app.implementation.interfaceMethod", "Go to Implementation: interface method", {
+    file: `${SRC}/PricingRule.java`,
+    line: "long price(Order order);",
+    word: "price",
+  }),
+
+  // Right after an edit: results follow the new text, unsaved and with its lines moved.
+  declaration(APP, "java.app.edit.declaration", "Go to declaration right after typing above the target", {
+    file: BOOK,
+    typed: { after: "public class OrderBook {", text: TWICE },
+    line: "return revenueCents();",
+    word: "revenueCents",
+  }),
+  showUsages(APP, "java.app.edit.usagesSameFile", "Show Usages right after typing a call in the same file", {
+    file: BOOK,
+    typed: { after: "public class OrderBook {", text: TWICE },
+    line: "public long revenueCents() {",
+    word: "revenueCents",
+  }),
+  showUsages(APP, "java.app.edit.usagesOtherFile", "Show Usages after typing a call in another file", {
+    file: BOOK,
+    edit: {
+      after: "public class OrderBook {",
+      text: "\n    public long firstTotal() {\n        return orders.get(0).totalCents();\n    }\n",
+    },
+    via: [["return orders.get(0).totalCents();", "totalCents"]],
+    line: "public long totalCents() {",
+    word: "totalCents",
+  }),
+
+  diagnosticsAfterEdit(
+    APP,
+    "java.app.diagnostics.afterError",
+    "Diagnostics: type a type error",
+    { file: BOOK, line: "public class OrderBook {", word: "OrderBook" },
+    TYPE_ERROR,
+  ),
+  diagnosticsAfterFix(
+    APP,
+    "java.app.diagnostics.afterFix",
+    "Diagnostics: declare the missing variable",
+    { file: BOOK, line: "public class OrderBook {", word: "OrderBook" },
+    { after: "public int size() {", text: "\n        int used = missing;" },
+    { after: "public class OrderBook {", text: "\n    private final int missing = 0;" },
+  ),
+
+  completion(APP, "java.app.completion.jdkMember", "Code Completion: members of a JDK type", BOOK, {
+    after: "public int size() {",
+    text: "\n        orders.",
+  }),
+  completion(
+    APP,
+    "java.app.completion.projectMember",
+    "Code Completion: members of a project class",
+    `${SRC}/App.java`,
+    {
+      after: "long revenue = book.revenueCents();",
+      text: "\n        book.",
+    },
+  ),
+
+  quickDoc(APP, "java.app.doc.project", "Quick Documentation: project method", {
+    file: BOOK,
+    line: "public Order place(String customer, String product, int quantity, long unitPriceCents) {",
+    word: "place",
+  }),
+  quickDoc(APP, "java.app.doc.jdk", "Quick Documentation: JDK class", {
+    file: BOOK,
+    line: TO_ARRAYLIST[0],
+    word: TO_ARRAYLIST[1],
+  }),
+
+  rename(
+    APP,
+    "java.app.rename.acrossFiles",
+    "Rename: method used across files",
+    { file: `${SRC}/Order.java`, line: "public long totalCents() {", word: "totalCents" },
+    "orderTotalCents",
+  ),
+  rename(
+    APP,
+    "java.app.rename.local",
+    "Rename: local variable",
+    { file: BOOK, line: "Order order = new Order(customer, product, quantity, unitPriceCents);", word: "order" },
+    "placedOrder",
+  ),
+
+  declaration(
+    PETCLINIC,
+    "java.petclinic.declaration.project",
+    "Go to declaration: Spring Data query method in the project",
+    {
+      file: `${PET}/owner/OwnerController.java`,
+      line: "return owners.findByLastNameStartingWith(lastname, pageable);",
+      word: "findByLastNameStartingWith",
+    },
+    ["warm", "cold"],
+  ),
+  declaration(
+    PETCLINIC,
+    "java.petclinic.declaration.inheritedLibrary",
+    "Go to declaration: method inherited from a Spring Data interface",
+    { file: `${PET}/owner/OwnerController.java`, line: "this.owners.save(owner);", word: "save" },
+  ),
+  declaration(PETCLINIC, "java.petclinic.declaration.libraryClass", "Go to declaration: Spring Data class", {
+    file: `${PET}/owner/OwnerController.java`,
+    line: "Pageable pageable = PageRequest.of(validatedPage - 1, pageSize);",
+    word: "PageRequest",
+  }),
+  showUsages(
+    PETCLINIC,
+    "java.petclinic.usages.entity",
+    "Show Usages: entity class used across controllers, repositories and tests",
+    { file: `${PET}/owner/Owner.java`, line: "public class Owner extends Person {", word: "Owner" },
+  ),
+  showUsages(
+    PETCLINIC,
+    "java.petclinic.usages.inheritedMethod",
+    "Show Usages: base-class getter called through subclasses",
+    {
+      file: `${PET}/model/BaseEntity.java`,
+      line: "public Integer getId() {",
+      word: "getId",
+    },
+  ),
+  gutterOnOpen(
+    PETCLINIC,
+    "java.petclinic.gutter.baseClass",
+    "Gutter markers: open a base entity",
+    `${PET}/model/BaseEntity.java`,
+  ),
+  gutterOnOpen(
+    PETCLINIC,
+    "java.petclinic.gutter.midClass",
+    "Gutter markers: open a mid-hierarchy entity",
+    `${PET}/model/Person.java`,
+  ),
+  lensOnOpen(PETCLINIC, "java.petclinic.lens.entity", "Usage lenses: open an entity", `${PET}/owner/Owner.java`),
+  implementation(PETCLINIC, "java.petclinic.implementation.baseClass", "Go to Implementation: base entity", {
+    file: `${PET}/model/BaseEntity.java`,
+    line: "public class BaseEntity implements Serializable {",
+    word: "BaseEntity",
+  }),
+  diagnosticsOnOpen(
+    PETCLINIC,
+    "java.petclinic.diagnostics.cleanFile",
+    "Diagnostics: open a controller that compiles",
+    `${PET}/owner/OwnerController.java`,
+  ),
+  completion(
+    PETCLINIC,
+    "java.petclinic.completion.libraryClass",
+    "Code Completion: class from a dependency, not imported",
+    `${PET}/owner/PetValidator.java`,
+    { after: "public class PetValidator implements Validator {", text: "\n    private Pageab" },
+  ),
+  quickDoc(PETCLINIC, "java.petclinic.doc.library", "Quick Documentation: Spring Data class", {
+    file: `${PET}/owner/OwnerController.java`,
+    line: "Pageable pageable = PageRequest.of(validatedPage - 1, pageSize);",
+    word: "PageRequest",
+  }),
+  rename(
+    PETCLINIC,
+    "java.petclinic.rename.acrossFiles",
+    "Rename: entity method used across controllers and tests",
+    { file: `${PET}/owner/Owner.java`, line: "public void addVisit(Integer petId, Visit visit) {", word: "addVisit" },
+    "recordVisit",
+  ),
+];

evals/lib/golden.test.ts

@@ -0,0 +1,40 @@
+import { describe, expect, test } from "vitest";
+import { compare, toolDrift } from "./golden.ts";
+
+describe("compare", () => {
+  test("a perfect answer", () => {
+    expect(compare({ usages: ["a:1", "b:2"] }, { usages: ["b:2", "a:1"] })).toEqual({
+      recall: 1,
+      precision: 1,
+      missing: [],
+      extra: [],
+    });
+  });
+
+  test("dropped and extra results lower recall and precision", () => {
+    const c = compare({ usages: ["a:1", "c:3"] }, { usages: ["a:1", "b:2"] });
+    expect(c.recall).toBe(0.5);
+    expect(c.precision).toBe(0.5);
+    expect(c.missing).toEqual(["usages: b:2"]);
+    expect(c.extra).toEqual(["usages: c:3"]);
+  });
+
+  test("groups are part of the identity", () => {
+    expect(compare({ read: ["a:1"] }, { write: ["a:1"] }).recall).toBe(0);
+  });
+
+  test("nothing expected", () => {
+    expect(compare({}, {})).toMatchObject({ recall: 1, precision: 1 });
+    expect(compare({ usages: ["a:1"] }, {})).toMatchObject({ recall: 1, precision: 0 });
+  });
+
+  test("nothing found", () => {
+    expect(compare({}, { usages: ["a:1"] })).toMatchObject({ recall: 0, precision: 0 });
+  });
+});
+
+test("toolDrift lists tools that changed since the recording", () => {
+  expect(toolDrift({ tools: { jdtls: "1.63.0", java: "21" }, groups: {} }, { jdtls: "1.64.0", java: "21" })).toEqual([
+    "jdtls: 1.63.0 → 1.64.0",
+  ]);
+});

evals/lib/golden.ts

@@ -0,0 +1,62 @@
+/**
+ * Golden results: what a scenario found when a person last confirmed it, keyed to the
+ * language server and JDK that found it. Compared as sets (recall / precision), not by order.
+ */
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { dirname } from "node:path";
+
+/** Result group → items (e.g. `src/…/OrderBook.java:14`); repeated items count once. */
+export type ResultSet = Record<string, string[]>;
+
+export interface Golden {
+  /** Tool versions the results were recorded with (`jdtls`, `java`). */
+  tools: Record<string, string>;
+  groups: ResultSet;
+}
+
+export interface Comparison {
+  recall: number;
+  precision: number;
+  /** In the golden, not found (`group: item`). */
+  missing: string[];
+  /** Found, not in the golden. */
+  extra: string[];
+}
+
+const keys = (set: ResultSet) => new Set(Object.entries(set).flatMap(([g, items]) => items.map((i) => `${g}: ${i}`)));
+
+export function compare(found: ResultSet, golden: ResultSet): Comparison {
+  const got = keys(found);
+  const want = keys(golden);
+  const missing = [...want].filter((k) => !got.has(k));
+  const extra = [...got].filter((k) => !want.has(k));
+  const hit = want.size - missing.length;
+  return {
+    // Nothing expected and nothing found is a perfect answer.
+    recall: want.size ? hit / want.size : 1,
+    precision: got.size ? hit / got.size : want.size ? 0 : 1,
+    missing: missing.sort(),
+    extra: extra.sort(),
+  };
+}
+
+export function readGolden(file: string): Golden | null {
+  return existsSync(file) ? (JSON.parse(readFileSync(file, "utf8")) as Golden) : null;
+}
+
+export function writeGolden(file: string, golden: Golden) {
+  mkdirSync(dirname(file), { recursive: true });
+  const sorted = Object.fromEntries(
+    Object.entries(golden.groups)
+      .sort(([a], [b]) => a.localeCompare(b))
+      .map(([g, items]) => [g, [...new Set(items)].sort()]),
+  );
+  writeFileSync(file, JSON.stringify({ tools: golden.tools, groups: sorted }, null, 2) + "\n");
+}
+
+/** Tools that differ between the golden's recording and this run (`name: was → now`). */
+export function toolDrift(golden: Golden, tools: Record<string, string>): string[] {
+  return Object.entries(golden.tools)
+    .filter(([k, v]) => tools[k] !== undefined && tools[k] !== v)
+    .map(([k, v]) => `${k}: ${v} → ${tools[k]}`);
+}

evals/lib/machine.ts

@@ -0,0 +1,86 @@
+/** The machine and tools a report was measured with: a baseline only holds for the same ones. */
+import { execFileSync, spawnSync } from "node:child_process";
+import { existsSync, readdirSync, readFileSync, realpathSync } from "node:fs";
+import { arch, cpus, platform, release, totalmem } from "node:os";
+import { dirname, join } from "node:path";
+
+export interface Fingerprint {
+  cpu: string;
+  cores: number;
+  memoryGb: number;
+  os: string;
+}
+
+export function fingerprint(): Fingerprint {
+  return {
+    cpu: cpus()[0]?.model.trim() ?? "unknown",
+    cores: cpus().length,
+    memoryGb: Math.round(totalmem() / 2 ** 30),
+    os: `${platform()} ${release()} ${arch()}`,
+  };
+}
+
+export function sameMachine(a: Fingerprint | undefined, b: Fingerprint): boolean {
+  return !!a && a.cpu === b.cpu && a.cores === b.cores && a.memoryGb === b.memoryGb && a.os === b.os;
+}
+
+/** `KEY=value` lines (`#` comments), as GitHub Actions' `$GITHUB_ENV` takes them; keys lower-cased. */
+export function pinnedTools(file: string): Record<string, string> {
+  if (!existsSync(file)) return {};
+  const out: Record<string, string> = {};
+  for (const line of readFileSync(file, "utf8").split("\n")) {
+    const m = /^\s*([A-Z0-9_]+)\s*=\s*(\S+)/.exec(line);
+    if (m) out[m[1]!.toLowerCase()] = m[2]!;
+  }
+  return out;
+}
+
+function which(cmd: string): string | null {
+  try {
+    const found = execFileSync(process.platform === "win32" ? "where" : "which", [cmd], { encoding: "utf8" })
+      .split("\n")[0]!
+      .trim();
+    return found ? realpathSync(found) : null;
+  } catch {
+    return null;
+  }
+}
+
+/** jdtls's version from its core bundle (`org.eclipse.jdt.ls.core_1.63.0.202610191322.jar` → `1.63.0-202610191322`). */
+function jdtlsVersion(): string | null {
+  const bin = which("jdtls");
+  if (!bin) return null;
+  for (let dir = dirname(bin), i = 0; i < 4; dir = dirname(dir), i++) {
+    for (const plugins of [join(dir, "plugins"), join(dir, "libexec", "plugins")]) {
+      if (!existsSync(plugins)) continue;
+      const core = readdirSync(plugins).find((f) => /^org\.eclipse\.jdt\.ls\.core_.*\.jar$/.test(f));
+      const m = core && /_(\d+\.\d+\.\d+)\.(\d+)\.jar$/.exec(core);
+      if (m) return `${m[1]}-${m[2]}`;
+    }
+  }
+  return null;
+}
+
+/**
+ * The JDK projects compile against, whose sources library navigation opens: `JAVA_HOME`'s (it
+ * wins for its Java version; jdtls itself runs on a newer one when it needs to), else the `java`
+ * on PATH. Runtime version and vendor (`-XshowSettings` prints to stderr).
+ */
+function javaVersion(): string | null {
+  const home = process.env.JAVA_HOME;
+  const java = home ? join(home, "bin", process.platform === "win32" ? "java.exe" : "java") : "java";
+  const r = spawnSync(java, ["-XshowSettings:properties", "-version"], { encoding: "utf8" });
+  const props = `${r.stdout ?? ""}${r.stderr ?? ""}`;
+  const version = /java\.runtime\.version = (\S+)/.exec(props)?.[1];
+  const vendor = /java\.vendor = (.+)/.exec(props)?.[1]?.trim();
+  return version ? `${version}${vendor ? ` (${vendor})` : ""}` : null;
+}
+
+export function toolVersions(): Record<string, string> {
+  const tools: Record<string, string> = {};
+  const jdtls = jdtlsVersion();
+  if (jdtls) tools.jdtls = jdtls;
+  const java = javaVersion();
+  if (java) tools.java = java;
+  return tools;
+}

evals/lib/measure.ts

@@ -0,0 +1,154 @@
+/**
+ * Timing one user action in the page: the action is run, then every animation frame checks
+ * what the user would see. A condition true at the start of a frame is painted by it.
+ */
+import type { Page } from "@playwright/test";
+
+/**
+ * A check of the page, run in it every frame: self-contained (it is re-created from its
+ * source), given the arguments passed to `track` and the editor state before the action.
+ */
+export type Check<A> = (arg: A, before: EditorSnapshot | null) => boolean;
+
+export interface EditorSnapshot {
+  path: string;
+  head: number;
+  line: number;
+  word: string;
+}
+
+export interface Tracked {
+  /** Something visible changed in answer to the action (popup, hint, spinner, caret / editor moved). */
+  feedbackMs: number;
+  /** The first result is shown. */
+  firstResultMs: number;
+  /** The final result is shown and nothing is loading. */
+  completeMs: number;
+}
+
+export interface TrackSpec<A> {
+  /** The action run (`actionArg` its argument), the file opened (`openFile`), or a check-like step in the page (`run`). */
+  action?: string;
+  actionArg?: unknown;
+  openFile?: string;
+  run?: (arg: A) => unknown;
+  arg: A;
+  first: Check<A>;
+  done: Check<A>;
+  /** Extra feedback beyond the default (popup, hint, status spinner, editor or caret changed). */
+  feedback?: Check<A>;
+  /**
+   * For results with no loading indicator (gutter markers): done once `done` holds and
+   * `signature` has not changed for `settleMs`; complete is when it last changed.
+   */
+  signature?: (arg: A) => unknown;
+  settleMs?: number;
+  timeoutMs: number;
+}
+
+export class TimeoutError extends Error {
+  readonly partial: Partial<Tracked>;
+  constructor(message: string, partial: Partial<Tracked>) {
+    super(message);
+    this.partial = partial;
+  }
+}
+
+export async function track<A>(page: Page, spec: TrackSpec<A>): Promise<Tracked> {
+  const r = await page.evaluate(
+    ({ action, actionArg, openFile, run, arg, first, done, feedback, signature, settleMs, timeoutMs }) => {
+      const w = window as unknown as {
+        __openideaPerf: {
+          runAction(id: string, arg?: unknown): Promise<boolean>;
+          openFile(path: string): Promise<void>;
+          editorState(): EditorSnapshot | null;
+        };
+      };
+      const make = (src: string | null) =>
+        src
+          ? (new Function("arg", "before", `return (${src})(arg, before)`) as (a: unknown, b: unknown) => boolean)
+          : null;
+      const checks = { first: make(first)!, done: make(done)!, feedback: make(feedback) };
+      const sign = signature
+        ? (new Function("arg", `return JSON.stringify((${signature})(arg))`) as (a: unknown) => string)
+        : null;
+      let lastSig: string | null = null;
+      let lastChange = 0;
+      const before = w.__openideaPerf.editorState();
+      const visible = (sel: string) =>
+        [...document.querySelectorAll(sel)].some((el) => (el as HTMLElement).getClientRects().length > 0);
+      const anyFeedback = () => {
+        if (visible('.cm-lsp-hint, [data-testid="usages-popup"], [role="dialog"], [role="listbox"]')) return true;
+        const now = w.__openideaPerf.editorState();
+        if ((now?.path ?? null) !== (before?.path ?? null) || (now && before && now.head !== before.head)) return true;
+        return checks.feedback?.(arg, before) ?? false;
+      };
+      const times: { feedbackMs?: number; firstResultMs?: number; completeMs?: number } = {};
+      const t0 = performance.now();
+      let failed: unknown = null;
+      const started = run
+        ? new Promise((resolve) => resolve(make(run)!(arg, before)))
+        : openFile
+          ? w.__openideaPerf.openFile(openFile)
+          : w.__openideaPerf.runAction(action!, actionArg);
+      void started.catch((e: unknown) => (failed = e));
+      return new Promise<{ times: typeof times; timedOut: boolean; error: string | null }>((resolve) => {
+        const tick = () => {
+          const t = performance.now() - t0;
+          try {
+            if (times.feedbackMs === undefined && anyFeedback()) times.feedbackMs = t;
+            if (times.firstResultMs === undefined && checks.first(arg, before)) times.firstResultMs = t;
+            if (sign) {
+              const sig = sign(arg);
+              if (sig !== lastSig) {
+                lastSig = sig;
+                lastChange = t;
+              }
+            }
+            const settled = !sign || t - lastChange >= (settleMs ?? 0);
+            if (times.firstResultMs !== undefined && settled && checks.done(arg, before)) {
+              // Session.reset waits out the time in which invoking it again widens it.
+              if (action === "find.showUsages")
+                (window as unknown as { __openideaEvalShowUsagesAt?: number }).__openideaEvalShowUsagesAt =
+                  performance.now();
+              times.completeMs = sign ? Math.max(lastChange, times.firstResultMs) : t;
+              times.feedbackMs ??= t;
+              return resolve({ times, timedOut: false, error: null });
+            }
+          } catch (e) {
+            return resolve({ times, timedOut: false, error: String(e) });
+          }
+          if (failed) return resolve({ times, timedOut: false, error: String((failed as Error).message ?? failed) });
+          if (t > timeoutMs) return resolve({ times, timedOut: true, error: null });
+          requestAnimationFrame(tick);
+        };
+        requestAnimationFrame(tick);
+      });
+    },
+    {
+      action: spec.action,
+      actionArg: spec.actionArg,
+      openFile: spec.openFile,
+      run: spec.run?.toString() ?? null,
+      arg: spec.arg,
+      first: spec.first.toString(),
+      done: spec.done.toString(),
+      feedback: spec.feedback?.toString() ?? null,
+      signature: spec.signature?.toString() ?? null,
+      settleMs: spec.settleMs,
+      timeoutMs: spec.timeoutMs,
+    },
+  );
+  const what = spec.action ?? (spec.openFile ? `open ${spec.openFile}` : "step");
+  if (r.error) throw new Error(`${what}: ${r.error}`);
+  if (r.timedOut) {
+    const seen =
+      r.times.firstResultMs !== undefined
+        ? "a first result but no final one"
+        : r.times.feedbackMs !== undefined
+          ? "feedback but no result"
+          : "nothing";
+    throw new TimeoutError(`${what} timed out after ${spec.timeoutMs} ms (${seen})`, r.times);
+  }
+  return r.times as Tracked;
+}

evals/lib/projects.ts

@@ -0,0 +1,70 @@
+/** Where a scenario's project comes from: `fixtures/<name>`, or an open-source project pinned in `evals/projects.json`. */
+import { execFileSync, spawnSync } from "node:child_process";
+import { cpSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+
+interface Pinned {
+  repo: string;
+  commit: string;
+  remove?: string[];
+  prepare?: string[];
+}
+
+function pinned(repoRoot: string, name: string): Pinned | null {
+  const all = JSON.parse(readFileSync(join(repoRoot, "evals", "projects.json"), "utf8")) as Record<string, Pinned>;
+  return all[name] ?? null;
+}
+
+const quiet = { stdio: "ignore" as const };
+
+/** The pinned checkout, cloned and prepared once; returns its directory. */
+function checkout(repoRoot: string, name: string, p: Pinned): string {
+  const dir = join(repoRoot, "target", "eval", "sources", name);
+  if (!existsSync(join(dir, ".git"))) {
+    console.log(`Cloning ${p.repo}…`);
+    mkdirSync(join(dir, ".."), { recursive: true });
+    execFileSync("git", ["clone", "-q", "--filter=blob:none", p.repo, dir], quiet);
+  }
+  const head = execFileSync("git", ["rev-parse", "HEAD"], { cwd: dir, encoding: "utf8" }).trim();
+  if (head !== p.commit) {
+    const has = spawnSync("git", ["cat-file", "-e", `${p.commit}^{commit}`], { cwd: dir }).status === 0;
+    if (!has) execFileSync("git", ["fetch", "-q", "origin", p.commit], { cwd: dir, ...quiet });
+    execFileSync("git", ["checkout", "-q", "--detach", p.commit], { cwd: dir, ...quiet });
+  }
+  const marker = join(repoRoot, "target", "eval", "sources", `.${name}.prepared`);
+  if (p.prepare && (!existsSync(marker) || readFileSync(marker, "utf8") !== p.commit)) {
+    console.log(`Preparing ${name}: ${p.prepare.join(" ")}…`);
+    const r = spawnSync(p.prepare[0]!, p.prepare.slice(1), { cwd: dir, stdio: ["ignore", "ignore", "inherit"] });
+    if (r.status !== 0) throw new Error(`preparing ${name} failed: ${p.prepare.join(" ")}`);
+    writeFileSync(marker, p.commit);
+  }
+  return dir;
+}
+
+/**
+ * A fresh copy of `name` at `dest`, as its own git repository (edits and IDE state of an
+ * earlier session don't carry over; it isn't part of the checkout `target/` sits in).
+ */
+export function copyProject(repoRoot: string, name: string, dest: string) {
+  const fixture = join(repoRoot, "fixtures", name);
+  const p = existsSync(fixture) ? null : pinned(repoRoot, name);
+  if (!p && !existsSync(fixture)) throw new Error(`no fixtures/${name} and no ${name} in evals/projects.json`);
+  const src = p ? checkout(repoRoot, name, p) : fixture;
+  rmSync(dest, { recursive: true, force: true, maxRetries: 5, retryDelay: 500 });
+  mkdirSync(join(dest, ".."), { recursive: true });
+  cpSync(src, dest, { recursive: true, filter: (f) => !/[\\/](\.git|target|build|\.gradle|\.openidea)$/.test(f) });
+  for (const r of p?.remove ?? []) rmSync(join(dest, r), { recursive: true, force: true });
+  const env = {
+    ...process.env,
+    GIT_AUTHOR_NAME: "eval",
+    GIT_AUTHOR_EMAIL: "eval@localhost",
+    GIT_COMMITTER_NAME: "eval",
+    GIT_COMMITTER_EMAIL: "eval@localhost",
+  };
+  for (const args of [
+    ["init", "-q"],
+    ["add", "-A"],
+    ["commit", "-q", "--no-verify", "-m", "eval project"],
+  ])
+    execFileSync("git", args, { cwd: dest, env, ...quiet });
+}

evals/lib/scenario.ts

@@ -0,0 +1,35 @@
+/** What an eval scenario is: a project, a step the user takes, and what they should see. */
+import type { ResultSet } from "./golden.ts";
+import type { Tracked } from "./measure.ts";
+import type { Session } from "./session.ts";
+import type { Condition } from "./verdict.ts";
+
+export interface Measured {
+  timing: Tracked;
+  results?: ResultSet;
+  /** Why the answer is weaker than a semantic one, when the UI says so. */
+  degraded?: string | null;
+}
+
+export interface EvalScenario {
+  /** Stable id: `<language>.<group>.<case>`, used by `--only`, goldens and budgets. */
+  id: string;
+  title: string;
+  /** `fixtures/<fixture>`. */
+  fixture: string;
+  /** The language server whose readiness the warm runs wait for. */
+  server: string;
+  conditions: Condition[];
+  /** Brings the idle workbench to where the step starts (open a file, place the caret). */
+  setup(s: Session): Promise<void>;
+  /** The measured step. */
+  step(s: Session): Promise<Measured>;
+}
+
+/** Project-relative path, or a library class's URI without its machine-specific query. */
+export function normalizePath(root: string, path: string): string {
+  if (path.startsWith(root + "/")) return path.slice(root.length + 1);
+  const uri = /^[a-z]+:\/\//.exec(path);
+  if (uri) return path.replace(/[?#].*$/, "");
+  return path;
+}

evals/lib/session.ts

@@ -0,0 +1,425 @@
+/**
+ * One eval session: a release `openidea-server --stdio` with its own home, and the production
+ * workbench in Chromium connected to it as remote development connects a window (the Tauri
+ * runtime from `smoke/tauriShim.ts`, commands and notifications relayed by Playwright).
+ */
+import { mkdirSync, rmSync } from "node:fs";
+import { homedir, tmpdir } from "node:os";
+import { join, sep } from "node:path";
+import type { Browser, BrowserContext, Page } from "@playwright/test";
+import { installInPage } from "../../perf/lib/inpage.ts";
+import { Server } from "../../smoke/server.ts";
+import { copyProject } from "./projects.ts";
+import { installTauriShim, type ShimWindow } from "../../smoke/tauriShim.ts";
+
+/** How long after its answer Show Usages invoked again widens it (`HINT_MS` in code-insight/usages.ts), plus a margin. */
+const SHOW_USAGES_HINT_MS = 4200;
+
+export interface IpcCall {
+  cmd: string;
+  /** Milliseconds since the session started. */
+  start: number;
+  ms: number;
+  ok: boolean;
+  /** JSON length of the result (or error). */
+  bytes: number;
+  /** Array length of the result, when it is one. */
+  items?: number;
+  /** Still unanswered when the window closed (`ms` is how long it had waited). */
+  pending?: true;
+}
+
+export interface TaskSpan {
+  id: number;
+  title: string;
+  start: number;
+  end: number | null;
+}
+
+/** What the server did, for diagnostics: every command, background task and server status. */
+export class Timeline {
+  readonly t0 = Date.now();
+  readonly calls: IpcCall[] = [];
+  readonly tasks = new Map<number, TaskSpan>();
+  readonly servers: Array<{ at: number; name: string; status: string; message: string | null }> = [];
+  /** Last time anything happened (a call ended, a task or status changed). */
+  lastActivity = 0;
+  inFlight = 0;
+  /** When the scenario's measured step started, for a failure's window. */
+  stepStart: number | null = null;
+  /** Commands sent and not answered yet, by call number. */
+  readonly unanswered = new Map<number, { cmd: string; start: number }>();
+  private nextCall = 0;
+
+  now() {
+    return Date.now() - this.t0;
+  }
+
+  onEvent(event: { type: string } & Record<string, unknown>) {
+    if (event.type === "task") {
+      const id = event.id as number;
+      const span = this.tasks.get(id) ?? { id, title: String(event.title), start: this.now(), end: null };
+      if (event.done) span.end = this.now();
+      this.tasks.set(id, span);
+      this.lastActivity = this.now();
+    } else if (event.type === "languageServer") {
+      this.servers.push({
+        at: this.now(),
+        name: String(event.name),
+        status: String(event.status),
+        message: (event.message as string | null) ?? null,
+      });
+      this.lastActivity = this.now();
+    }
+  }
+
+  /** Records `cmd` as sent; the returned function marks it answered. */
+  sent(cmd: string, start: number): () => void {
+    const id = this.nextCall++;
+    this.unanswered.set(id, { cmd, start });
+    return () => this.unanswered.delete(id);
+  }
+
+  runningTasks(): TaskSpan[] {
+    return [...this.tasks.values()].filter((t) => t.end === null);
+  }
+
+  serverStatus(name: string): string | null {
+    for (let i = this.servers.length - 1; i >= 0; i--)
+      if (this.servers[i]!.name === name) return this.servers[i]!.status;
+    return null;
+  }
+
+  /** The calls and tasks overlapping `[from, to]` (session milliseconds). */
+  window(from: number, to: number) {
+    return {
+      calls: [
+        ...this.calls.filter((c) => c.start <= to && c.start + c.ms >= from),
+        // A command that never answers is what a timeout usually waits on.
+        ...[...this.unanswered.values()]
+          .filter((c) => c.start <= to)
+          .map((c): IpcCall => ({ ...c, ms: this.now() - c.start, ok: false, bytes: 0, pending: true })),
+      ],
+      tasks: [...this.tasks.values()].filter((t) => t.start <= to && (t.end ?? Infinity) >= from),
+    };
+  }
+}
+
+export interface SessionOptions {
+  /** `fixtures/<name>` or a project pinned in `evals/projects.json`. */
+  fixture: string;
+  /** Where the project copy and the server's home go. */
+  workDir: string;
+  /** Keep the server's home (installed tools, language server workspaces) from an earlier session. */
+  keepHome: boolean;
+  /** `OPENIDEA_LOG` for the server. */
+  log?: string;
+}
+
+export class Session {
+  readonly timeline = new Timeline();
+  readonly root: string;
+  readonly home: string;
+  page!: Page;
+  private context!: BrowserContext;
+  private server!: Server;
+  private relayQueue: Promise<unknown> = Promise.resolve();
+  private pendingNotes: Array<[kind: "channel" | "event", a: unknown, b: unknown]> = [];
+  /** The last answer to each command that succeeded, for what the UI only summarises. */
+  readonly lastAnswer = new Map<string, unknown>();
+  /** Editors changed by a scenario: path → the text to put back. */
+  private readonly edited = new Map<string, string>();
+  readonly pageErrors: string[] = [];
+
+  private constructor(opts: SessionOptions) {
+    this.root = join(opts.workDir, "projects", opts.fixture);
+    this.home = join(opts.workDir, "home");
+  }
+
+  static async start(browser: Browser, url: string, repoRoot: string, opts: SessionOptions): Promise<Session> {
+    const s = new Session(opts);
+    copyProject(repoRoot, opts.fixture, s.root);
+    if (!opts.keepHome) rmSync(s.home, { recursive: true, force: true, maxRetries: 5, retryDelay: 500 });
+    mkdirSync(s.home, { recursive: true });
+
+    s.server = await Server.start(s.home, opts.log ? { OPENIDEA_LOG: opts.log } : {});
+    s.server.onEvent = (target, event) => {
+      if (target !== null && target !== "main") return;
+      s.timeline.onEvent(event as { type: string });
+      s.notify("event", target, event);
+    };
+    s.server.onChannel = (id, data) => s.notify("channel", id, data);
+    await s.server.invoke("project_trust_set", { path: s.root, trusted: true, folder: false });
+
+    s.context = await browser.newContext({
+      baseURL: url,
+      viewport: { width: 1400, height: 900 },
+      deviceScaleFactor: 1,
+    });
+    await s.context.exposeFunction("__openideaRelay", async (cmd: string, args: unknown) => {
+      const start = s.timeline.now();
+      s.timeline.inFlight++;
+      const answered = s.timeline.sent(cmd, start);
+      try {
+        const value = await s.server.invoke(cmd, args);
+        s.record(cmd, start, true, value);
+        return { ok: value ?? null };
+      } catch (err) {
+        s.record(cmd, start, false, err);
+        return { err };
+      } finally {
+        answered();
+        s.timeline.inFlight--;
+        s.timeline.lastActivity = s.timeline.now();
+      }
+    });
+    await s.context.addInitScript(installTauriShim, {
+      directories: { 12: tmpdir(), 21: homedir() },
+      sep: sep as "/" | "\\",
+    });
+    await s.context.addInitScript(() => {
+      const w = window as unknown as ShimWindow & {
+        __openideaRelay(cmd: string, args: unknown): Promise<{ ok: unknown } | { err: unknown }>;
+      };
+      w.__openideaForward = async (cmd, args) => {
+        const r = await w.__openideaRelay(cmd, args);
+        if ("err" in r) throw r.err;
+        return r.ok;
+      };
+    });
+    await s.context.addInitScript(installInPage);
+    s.page = await s.context.newPage();
+    s.page.on("pageerror", (e) => s.pageErrors.push(e.message));
+    if (process.env.OPENIDEA_EVAL_DEBUG) {
+      s.page.on("console", (m) => console.log("[page]", m.type(), m.text().slice(0, 300)));
+      s.page.on("pageerror", (e) => console.log("[pageerror]", e.message));
+    }
+
+    await s.page.goto("/");
+    await s.page.waitForFunction(() => "__openideaPerf" in window && "__openideaShim" in window);
+    await s.page.evaluate(
+      (root) =>
+        (window as unknown as { __openideaPerf: { openProject(p: string): Promise<void> } }).__openideaPerf.openProject(
+          root,
+        ),
+      s.root,
+    );
+    await s.page.waitForSelector('[role="tree"][aria-label="Project"] [role="treeitem"]', { timeout: 60_000 });
+    return s;
+  }
+
+  private record(cmd: string, start: number, ok: boolean, value: unknown) {
+    let bytes = 0;
+    try {
+      bytes = JSON.stringify(value ?? null).length;
+    } catch {
+      // not serialisable: size unknown
+    }
+    const items = Array.isArray(value)
+      ? value.length
+      : value && typeof value === "object" && Array.isArray((value as { locations?: unknown }).locations)
+        ? (value as { locations: unknown[] }).locations.length
+        : undefined;
+    this.timeline.calls.push({ cmd, start, ms: this.timeline.now() - start, ok, bytes, items });
+    if (ok) this.lastAnswer.set(cmd, value);
+  }
+
+  /** Delivers server notifications in order, batched per turn of the event loop. */
+  private notify(kind: "channel" | "event", a: unknown, b: unknown) {
+    this.pendingNotes.push([kind, a, b]);
+    if (this.pendingNotes.length > 1) return;
+    this.relayQueue = this.relayQueue
+      .then(() => new Promise((r) => setImmediate(r)))
+      .then(() => {
+        const batch = this.pendingNotes;
+        this.pendingNotes = [];
+        return this.page
+          ?.evaluate((notes) => {
+            const shim = (window as unknown as ShimWindow).__openideaShim;
+            for (const [kind, a, b] of notes) {
+              if (kind === "channel") shim.channel(a as number, b);
+              else shim.event(a as string | null, b);
+            }
+          }, batch)
+          .catch(() => {});
+      });
+  }
+
+  runAction(id: string, arg?: unknown): Promise<boolean> {
+    return this.page.evaluate(
+      async ({ id, arg }) => {
+        const w = window as unknown as {
+          __openideaPerf: { runAction(id: string, arg?: unknown): Promise<boolean> };
+          __openideaEvalShowUsagesAt?: number;
+        };
+        const ran = await w.__openideaPerf.runAction(id, arg);
+        if (id === "find.showUsages") w.__openideaEvalShowUsagesAt = performance.now();
+        return ran;
+      },
+      { id, arg },
+    );
+  }
+
+  /** Opens `rel` (project-relative, or a URI) in an editor and waits until its text is shown. */
+  async openFile(rel: string) {
+    const path = /^[a-z]+:\/\//.test(rel) ? rel : join(this.root, rel);
+    const bare = rel.replace(/[?#].*$/, "");
+    const name = bare.slice(bare.lastIndexOf("/") + 1);
+    await this.page.evaluate(
+      async ({ path, name }) => {
+        const w = window as unknown as {
+          __perf: { waitShown(s: string, t: string | null, ms: number): Promise<number> };
+          __openideaPerf: { openFile(p: string): Promise<void> };
+        };
+        await w.__openideaPerf.openFile(path);
+        await w.__perf.waitShown('[role="tab"][aria-selected="true"]', name, 30_000);
+        await w.__perf.waitShown(".cm-content .cm-line", null, 30_000);
+      },
+      { path, name },
+    );
+  }
+
+  /**
+   * Goes to the declaration of `word` on `line` of the active editor and waits until another
+   * file shows (a library class: its `jdt://` URI, which carries what only the server knows).
+   */
+  async declarationOf(line: string, word: string): Promise<string> {
+    await this.caretAt(line, word);
+    type Probe = { __openideaPerf: { editorState(): { path: string } | null } };
+    const from = await this.page.evaluate(() => (window as unknown as Probe).__openideaPerf.editorState()?.path);
+    await this.runAction("navigate.declaration");
+    const handle = await this.page.waitForFunction(
+      (from) => {
+        const path = (window as unknown as Probe).__openideaPerf.editorState()?.path;
+        return path && path !== from && document.querySelector(".cm-content .cm-line") ? path : null;
+      },
+      from,
+      { timeout: 30_000 },
+    );
+    return (await handle.jsonValue()) as string;
+  }
+
+  /** Puts the caret inside `word` on the active editor's first line containing `line` (both literal). */
+  async caretAt(line: string, word: string) {
+    const text = await this.editorText();
+    const at = text.indexOf(line);
+    if (at < 0) throw new Error(`no line containing ${JSON.stringify(line)} in the active editor`);
+    const inLine = line.indexOf(word);
+    if (inLine < 0) throw new Error(`${JSON.stringify(word)} is not in ${JSON.stringify(line)}`);
+    // Inside the word, as a click on it would put it.
+    const target = at + inLine + Math.floor(word.length / 2);
+    await this.page.evaluate(
+      (o) => (window as unknown as { __openideaPerf: { setCursor(o: number): void } }).__openideaPerf.setCursor(o),
+      target,
+    );
+  }
+
+  /** The active editor's whole text. */
+  editorText(): Promise<string> {
+    return this.page.evaluate(() => {
+      const text = (
+        window as unknown as { __openideaPerf: { editorText(): string | null } }
+      ).__openideaPerf.editorText();
+      if (text === null) throw new Error("no active editor");
+      return text;
+    });
+  }
+
+  /** Keeps the active editor's text, for `reset` to put back whatever the scenario changes. */
+  async keepText() {
+    const { path, text } = await this.page.evaluate(() => {
+      const probe = (
+        window as unknown as {
+          __openideaPerf: { editorState(): { path: string } | null; editorText(): string | null };
+        }
+      ).__openideaPerf;
+      return { path: probe.editorState()?.path, text: probe.editorText() };
+    });
+    if (path === undefined || text === null) throw new Error("no active editor");
+    if (!this.edited.has(path)) this.edited.set(path, text);
+  }
+
+  /** Inserts `text` after the first `after` in the active editor; the offset after it. */
+  async edit(after: string, text: string): Promise<number> {
+    await this.keepText();
+    return this.page.evaluate(
+      ({ after, text }) =>
+        (
+          window as unknown as { __openideaPerf: { insertAfter(a: string, t: string): number } }
+        ).__openideaPerf.insertAfter(after, text),
+      { after, text },
+    );
+  }
+
+  private async restoreEdits() {
+    for (const [path, text] of this.edited) {
+      await this.page.evaluate(
+        async ({ path, text }) => {
+          const probe = (
+            window as unknown as {
+              __openideaPerf: {
+                openFile(p: string): Promise<void>;
+                editorState(): { path: string } | null;
+                editorText(): string | null;
+                setText(t: string): void;
+              };
+            }
+          ).__openideaPerf;
+          await probe.openFile(path);
+          while (probe.editorState()?.path !== path) await new Promise(requestAnimationFrame);
+          if (probe.editorText() !== text) probe.setText(text);
+        },
+        { path, text },
+      );
+      await this.runAction("file.save");
+    }
+    this.edited.clear();
+  }
+
+  /** Waits until `server` is ready and nothing has happened for `quietMs`: no task, no command in flight. */
+  async settle(server: string, quietMs = 2000, timeoutMs = 5 * 60_000) {
+    const deadline = Date.now() + timeoutMs;
+    for (;;) {
+      const t = this.timeline;
+      const quiet =
+        t.serverStatus(server) === "ready" &&
+        t.runningTasks().length === 0 &&
+        t.inFlight === 0 &&
+        t.now() - t.lastActivity >= quietMs;
+      if (quiet) return;
+      if (Date.now() > deadline) {
+        throw new Error(
+          `the IDE did not settle within ${timeoutMs / 1000}s: ${server} ${t.serverStatus(server) ?? "not started"}, ` +
+            `tasks ${JSON.stringify(t.runningTasks().map((x) => x.title))}, ${t.inFlight} command(s) in flight`,
+        );
+      }
+      await new Promise((r) => setTimeout(r, 200));
+    }
+  }
+
+  /** Back to an idle workbench: popups dismissed, edits undone, every editor closed. */
+  async reset() {
+    // Show Usages invoked again within 4 s of its hint widens the previous search: wait it out.
+    const since = await this.page.evaluate(() => {
+      const at = (window as unknown as { __openideaEvalShowUsagesAt?: number }).__openideaEvalShowUsagesAt;
+      return at === undefined ? Infinity : performance.now() - at;
+    });
+    if (since < SHOW_USAGES_HINT_MS) await this.page.waitForTimeout(SHOW_USAGES_HINT_MS - since);
+    await this.page.keyboard.press("Escape");
+    await this.page.keyboard.press("Escape");
+    await this.restoreEdits();
+    await this.runAction("editor.closeAllTabs");
+    await this.page.waitForFunction(() => !document.querySelector('[role="tab"][aria-selected="true"]'), undefined, {
+      timeout: 10_000,
+    });
+  }
+
+  serverLog(): string {
+    return this.server.log();
+  }
+
+  async close() {
+    await this.context?.close().catch(() => {});
+    await this.server?.close();
+  }
+}

evals/lib/verdict.test.ts

@@ -0,0 +1,86 @@
+import { describe, expect, test } from "vitest";
+import { judge, table, type Budgets, type Outcome } from "./verdict.ts";
+
+const budgets: Budgets = {
+  regressionTolerance: 0.2,
+  regressionFloorMs: 200,
+  defaults: { feedbackMs: 100, recall: 1, precision: 1 },
+  scenarios: { usages: { completeMs: 2000 }, "usages@cold": { completeMs: 10_000 } },
+};
+
+const outcome = (o: Partial<Outcome>): Outcome => ({
+  scenario: "usages",
+  title: "Show Usages",
+  condition: "warm",
+  metrics: { feedbackMs: 40, firstResultMs: 300, completeMs: 500, recall: 1, precision: 1 },
+  samples: {},
+  results: { usages: ["a:1"] },
+  comparison: { recall: 1, precision: 1, missing: [], extra: [] },
+  ...o,
+});
+
+const repro = () => "pnpm eval --only usages";
+
+describe("judge", () => {
+  test("within budgets and matching the golden passes", () => {
+    const [v] = judge([outcome({})], budgets, null, repro);
+    expect(v).toMatchObject({ id: "usages@warm", status: "pass", failures: [], repro: "pnpm eval --only usages" });
+  });
+
+  test("slow feedback, dropped results and silent degradation each fail", () => {
+    const [v] = judge(
+      [
+        outcome({
+          metrics: { feedbackMs: 250, firstResultMs: 300, completeMs: 2500, recall: 0.5, precision: 1 },
+          comparison: { recall: 0.5, precision: 1, missing: ["usages: b:2"], extra: [] },
+          degraded: "Based on syntax",
+        }),
+      ],
+      budgets,
+      null,
+      repro,
+    );
+    expect(v!.status).toBe("fail");
+    expect(v!.failures.map((f) => [f.kind, f.metric])).toEqual([
+      ["degraded", undefined],
+      ["budget", "feedbackMs"],
+      ["correctness", "recall"],
+      ["budget", "completeMs"],
+    ]);
+    expect(v!.failures.find((f) => f.metric === "recall")!.detail).toContain("usages: b:2");
+  });
+
+  test("condition budgets override scenario budgets", () => {
+    const [v] = judge(
+      [outcome({ condition: "cold", metrics: { feedbackMs: 40, completeMs: 5000, recall: 1, precision: 1 } })],
+      budgets,
+      null,
+      repro,
+    );
+    expect(v!.status).toBe("pass");
+  });
+
+  test("timeouts and missing goldens fail", () => {
+    const [timedOut, noGolden] = judge(
+      [outcome({ error: "find.showUsages timed out after 30000 ms" }), outcome({ comparison: undefined })],
+      budgets,
+      null,
+      repro,
+    );
+    expect(timedOut!.failures[0]!.kind).toBe("timeout");
+    expect(noGolden!.failures[0]!.kind).toBe("golden");
+  });
+
+  test("regressions need both the relative tolerance and the absolute floor", () => {
+    const baseline = [outcome({ metrics: { feedbackMs: 40, firstResultMs: 300, completeMs: 500 } })];
+    const [v] = judge(
+      [outcome({ metrics: { feedbackMs: 90, firstResultMs: 450, completeMs: 1200, recall: 1, precision: 1 } })],
+      budgets,
+      baseline,
+      repro,
+    );
+    expect(v!.status).toBe("pass");
+    expect(v!.regressions.map((r) => r.metric)).toEqual(["completeMs"]);
+    expect(table([v!], false)).toContain("regressed: completeMs");
+  });
+});

evals/lib/verdict.ts

@@ -0,0 +1,196 @@
+/**
+ * Verdicts: each scenario run under each condition passes or fails on its own numbers
+ * (budgets, correctness, degradation, errors), and is compared with the local baseline.
+ */
+import { readFileSync } from "node:fs";
+import type { Comparison, ResultSet } from "./golden.ts";
+
+export type Condition = "warm" | "cold";
+
+export interface Diagnostics {
+  /** Commands sent to the server during the slowest measured run (session ms). */
+  calls: Array<{ cmd: string; start: number; ms: number; ok: boolean; bytes: number; items?: number; pending?: true }>;
+  tasks: Array<{ title: string; start: number; end: number | null }>;
+  servers: Array<{ at: number; name: string; status: string; message: string | null }>;
+  /** The slowest run's window in session milliseconds. */
+  window: [number, number];
+  serverLog: string;
+  trace?: string;
+}
+
+export interface Outcome {
+  scenario: string;
+  title: string;
+  condition: Condition;
+  /** Medians of the measured runs (correctness: the worst run). */
+  metrics: Record<string, number>;
+  samples: Record<string, number[]>;
+  results?: ResultSet;
+  comparison?: Comparison;
+  /** Notes of the tools whose versions differ from the golden's. */
+  drift?: string[];
+  /** Why the answer is weaker than a semantic one (syntax fallback, incomplete search), if it is. */
+  degraded?: string | null;
+  error?: string;
+  diagnostics?: Diagnostics;
+}
+
+export type FailureKind = "error" | "timeout" | "degraded" | "correctness" | "budget" | "golden";
+
+export interface Failure {
+  kind: FailureKind;
+  metric?: string;
+  value?: number;
+  limit?: number;
+  detail: string;
+}
+
+export interface Regression {
+  metric: string;
+  value: number;
+  baseline: number;
+}
+
+export interface Verdict {
+  id: string;
+  scenario: string;
+  condition: Condition;
+  title: string;
+  status: "pass" | "fail";
+  failures: Failure[];
+  regressions: Regression[];
+  improvements: Regression[];
+  metrics: Record<string, number>;
+  baseline?: Record<string, number>;
+  repro: string;
+}
+
+export interface Budgets {
+  regressionTolerance: number;
+  regressionFloorMs: number;
+  /** Applied to every scenario. `*Ms`: upper bounds; `recall` / `precision`: lower bounds. */
+  defaults: Record<string, number>;
+  /** By `<scenario>` or `<scenario>@<condition>` (the latter wins). */
+  scenarios: Record<string, Record<string, number>>;
+}
+
+export function loadBudgets(file: string): Budgets {
+  const raw = JSON.parse(readFileSync(file, "utf8")) as Partial<Budgets>;
+  return {
+    regressionTolerance: raw.regressionTolerance ?? 0.2,
+    regressionFloorMs: raw.regressionFloorMs ?? 200,
+    defaults: raw.defaults ?? {},
+    scenarios: raw.scenarios ?? {},
+  };
+}
+
+export const idOf = (o: { scenario: string; condition: Condition }) => `${o.scenario}@${o.condition}`;
+
+function budgetsFor(b: Budgets, o: Outcome): Record<string, number> {
+  return { ...b.defaults, ...b.scenarios[o.scenario], ...b.scenarios[idOf(o)] };
+}
+
+const lowerBound = (metric: string) => metric === "recall" || metric === "precision";
+
+export function judge(
+  outcomes: Outcome[],
+  budgets: Budgets,
+  baseline: Outcome[] | null,
+  repro: (o: Outcome) => string,
+): Verdict[] {
+  const base = new Map((baseline ?? []).map((o) => [idOf(o), o.metrics]));
+  return outcomes.map((o) => {
+    const failures: Failure[] = [];
+    if (o.error) failures.push({ kind: /timed out/.test(o.error) ? "timeout" : "error", detail: o.error });
+    if (o.degraded) failures.push({ kind: "degraded", detail: o.degraded });
+    if (o.results && !o.comparison && !o.error)
+      failures.push({ kind: "golden", detail: "no golden recorded: run with --update-golden and review it" });
+    for (const [metric, limit] of Object.entries(budgetsFor(budgets, o))) {
+      const value = o.metrics[metric];
+      if (value === undefined || !Number.isFinite(value)) continue;
+      const over = lowerBound(metric) ? value < limit : value > limit;
+      if (!over) continue;
+      if (lowerBound(metric)) {
+        const c = o.comparison;
+        const list = metric === "recall" ? c?.missing : c?.extra;
+        const shown = (list ?? []).slice(0, 8).join("; ");
+        failures.push({
+          kind: "correctness",
+          metric,
+          value,
+          limit,
+          detail: `${metric} ${value.toFixed(2)} < ${limit} (${metric === "recall" ? "missing" : "unexpected"}: ${shown}${(list?.length ?? 0) > 8 ? "; …" : ""})`,
+        });
+      } else {
+        failures.push({
+          kind: "budget",
+          metric,
+          value,
+          limit,
+          detail: `${metric} ${Math.round(value)} ms > ${limit} ms`,
+        });
+      }
+    }
+    const before = base.get(idOf(o));
+    const regressions: Regression[] = [];
+    const improvements: Regression[] = [];
+    if (before) {
+      for (const [metric, value] of Object.entries(o.metrics)) {
+        const was = before[metric];
+        if (was === undefined || !metric.endsWith("Ms") || !Number.isFinite(value) || !Number.isFinite(was)) continue;
+        const slack = Math.max(was * budgets.regressionTolerance, budgets.regressionFloorMs);
+        if (value > was + slack) regressions.push({ metric, value, baseline: was });
+        else if (value < was - slack) improvements.push({ metric, value, baseline: was });
+      }
+    }
+    return {
+      id: idOf(o),
+      scenario: o.scenario,
+      condition: o.condition,
+      title: o.title,
+      status: failures.length ? "fail" : "pass",
+      failures,
+      regressions,
+      improvements,
+      metrics: o.metrics,
+      baseline: before,
+      repro: repro(o),
+    };
+  });
+}
+
+const fmt = (metric: string, v: number | undefined) =>
+  v === undefined || !Number.isFinite(v) ? "–" : metric.endsWith("Ms") ? String(Math.round(v)) : v.toFixed(2);
+
+const COLUMNS = ["feedbackMs", "firstResultMs", "completeMs", "recall", "precision"];
+
+/** The verdicts as a table (Markdown when `md`), with the baseline in parentheses. */
+export function table(verdicts: Verdict[], md: boolean): string {
+  const head = ["scenario", ...COLUMNS, "verdict"];
+  const rows = verdicts.map((v) => [
+    v.id,
+    ...COLUMNS.map((m) => {
+      const now = fmt(m, v.metrics[m]);
+      const was = v.baseline?.[m];
+      return was === undefined || !m.endsWith("Ms") ? now : `${now} (${fmt(m, was)})`;
+    }),
+    v.status === "fail"
+      ? `FAIL: ${v.failures.map((f) => f.kind + (f.metric ? ` ${f.metric}` : "")).join(", ")}`
+      : v.regressions.length
+        ? `regressed: ${v.regressions.map((r) => r.metric).join(", ")}`
+        : v.improvements.length
+          ? `improved: ${v.improvements.map((r) => r.metric).join(", ")}`
+          : "ok",
+  ]);
+  if (md) {
+    return [
+      `| ${head.join(" | ")} |`,
+      `| ${head.map((_, i) => (i === 0 || i === head.length - 1 ? "---" : "---:")).join(" | ")} |`,
+      ...rows.map((r) => `| ${r.join(" | ")} |`),
+    ].join("\n");
+  }
+  const widths = head.map((h, i) => Math.max(h.length, ...rows.map((r) => r[i]!.length)));
+  const line = (cells: string[]) =>
+    cells.map((c, i) => (i === 0 || i === head.length - 1 ? c.padEnd(widths[i]!) : c.padStart(widths[i]!))).join("  ");
+  return [line(head), ...rows.map(line)].join("\n");
+}

evals/projects.json

@@ -0,0 +1,16 @@
+{
+  "$comment": "Open-source projects the evals run on besides fixtures/: cloned at `commit` into target/eval/sources/, `remove`d paths dropped from the copy, `prepare` run once per checkout (dependencies downloaded before anything is timed; Maven needs to reach Central, a proxy goes in ~/.m2/settings.xml).",
+  "spring-petclinic": {
+    "repo": "https://github.com/spring-projects/spring-petclinic.git",
+    "commit": "500158f732419217507c7656904b8e6aa1bcc0d6",
+    "remove": ["build.gradle", "settings.gradle", "gradle", "gradlew", "gradlew.bat"],
+    "prepare": [
+      "mvn",
+      "-q",
+      "-B",
+      "dependency:go-offline",
+      "dependency:sources",
+      "-DfailOnMissingClassifierArtifact=false"
+    ]
+  }
+}

evals/run.ts

@@ -0,0 +1,365 @@
+/**
+ * Code-intelligence evals (`pnpm eval`): the production workbench in Chromium driving a
+ * release `openidea-server` with real language servers, each scenario timed as the user
+ * sees it (feedback, first result, complete) and checked against its golden results.
+ *
+ *   pnpm eval                              every scenario (warm: 1 warm-up + median of 3; cold: once)
+ *   pnpm eval --only java.usages --runs 5  scenarios whose id starts with java.usages
+ *   pnpm eval --save-baseline              store this run as target/eval/baseline.json
+ *   pnpm eval --update-golden              re-record evals/<language>/golden/ (review the diff)
+ *   pnpm eval --only <id> --trace          Chrome trace of the slowest run → target/eval/traces/
+ *
+ * Writes target/eval/report.json (verdicts, failures with diagnostics, machine fingerprint)
+ * and target/eval/report.md. Timings go over the remote-development route (one stdio hop
+ * more than Tauri IPC): compare them before / after on one machine, not as absolute numbers.
+ */
+import { execFileSync, spawnSync } from "node:child_process";
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { join, resolve } from "node:path";
+import { parseArgs } from "node:util";
+import { chromium, type Browser } from "@playwright/test";
+import { buildBundle, serveBundle } from "../perf/lib/bundle.ts";
+import { median } from "../perf/lib/stats.ts";
+import { JAVA_SCENARIOS } from "./java/scenarios.ts";
+import { compare, readGolden, toolDrift, writeGolden, type ResultSet } from "./lib/golden.ts";
+import { TimeoutError } from "./lib/measure.ts";
+import { fingerprint, pinnedTools, sameMachine, toolVersions, type Fingerprint } from "./lib/machine.ts";
+import type { EvalScenario } from "./lib/scenario.ts";
+import { Session } from "./lib/session.ts";
+import { idOf, judge, loadBudgets, table, type Condition, type Diagnostics, type Outcome } from "./lib/verdict.ts";
+
+const ROOT = resolve(import.meta.dirname, "..");
+const OUT = join(ROOT, "target", "eval");
+const DIST = join(OUT, "dist");
+const SCENARIOS: EvalScenario[] = [...JAVA_SCENARIOS];
+
+const { values: args } = parseArgs({
+  options: {
+    only: { type: "string" },
+    condition: { type: "string" },
+    runs: { type: "string", default: "3" },
+    warmup: { type: "string", default: "1" },
+    channel: { type: "string", default: process.env.OPENIDEA_PERF_CHANNEL ?? "chromium" },
+    headed: { type: "boolean", default: false },
+    "no-build": { type: "boolean", default: false },
+    "save-baseline": { type: "boolean", default: false },
+    "update-golden": { type: "boolean", default: false },
+    "fail-on-regression": { type: "boolean", default: false },
+    trace: { type: "boolean", default: false },
+    list: { type: "boolean", default: false },
+  },
+  allowPositionals: false,
+});
+
+if (args.list) {
+  for (const s of SCENARIOS) console.log(`${s.id.padEnd(44)} ${s.conditions.join(",").padEnd(10)} ${s.title}`);
+  process.exit(0);
+}
+
+const runs = Math.max(1, Number(args.runs));
+const warmup = Math.max(0, Number(args.warmup));
+const only = args.only?.split(",").map((s) => s.trim());
+const conditionFilter = args.condition as Condition | undefined;
+const selected = SCENARIOS.filter((s) => !only || only.some((o) => s.id === o || s.id.startsWith(`${o}.`)));
+if (only && !selected.length) throw new Error(`no scenario matches ${args.only} (see --list)`);
+
+const goldenFile = (id: string) => join(ROOT, "evals", id.slice(0, id.indexOf(".")), "golden", `${id}.json`);
+
+function serverBinary(): string {
+  const exe = process.platform === "win32" ? ".exe" : "";
+  const bin = process.env.OPENIDEA_SERVER ?? join(ROOT, "target", "release", `openidea-server${exe}`);
+  if (!args["no-build"]) {
+    console.log("Building openidea-server (cargo build --release)…");
+    const r = spawnSync("cargo", ["build", "--release", "-p", "ide-host", "--bin", "openidea-server"], {
+      cwd: ROOT,
+      stdio: ["ignore", "ignore", "inherit"],
+    });
+    if (r.status !== 0) throw new Error("cargo build failed");
+  }
+  if (!existsSync(bin)) throw new Error(`no server at ${bin}; run without --no-build`);
+  return bin;
+}
+
+function git(...a: string[]): string {
+  try {
+    return execFileSync("git", a, { cwd: ROOT, encoding: "utf8" }).trim();
+  } catch {
+    return "";
+  }
+}
+
+/** Medians of the timings; correctness from the worst run. */
+function summarize(samples: Array<Record<string, number>>): Record<string, number> {
+  const keys = [...new Set(samples.flatMap((s) => Object.keys(s)))];
+  return Object.fromEntries(
+    keys.map((k) => {
+      const values = samples.map((s) => s[k]!).filter(Number.isFinite);
+      return [k, k === "recall" || k === "precision" ? Math.min(...values) : median(values)];
+    }),
+  );
+}
+
+interface Run {
+  metrics: Record<string, number>;
+  results?: ResultSet;
+  degraded?: string | null;
+  window: [number, number];
+}
+
+async function measureOnce(s: Session, scenario: EvalScenario, trace: string | null, browser: Browser): Promise<Run> {
+  await s.reset();
+  await scenario.setup(s);
+  await s.settle(scenario.server, 1000);
+  const from = (s.timeline.stepStart = s.timeline.now());
+  if (trace) await browser.startTracing(s.page, { screenshots: true });
+  try {
+    const m = await scenario.step(s);
+    return {
+      metrics: { ...m.timing },
+      results: m.results,
+      degraded: m.degraded,
+      window: [from, s.timeline.now()],
+    };
+  } finally {
+    if (trace) writeFileSync(trace, await browser.stopTracing());
+  }
+}
+
+function diagnose(s: Session, window: [number, number], trace?: string): Diagnostics {
+  const w = s.timeline.window(window[0] - 500, window[1]);
+  return {
+    calls: w.calls.map((c) => ({ ...c })),
+    tasks: w.tasks.map((t) => ({ title: t.title, start: t.start, end: t.end })),
+    servers: s.timeline.servers.filter((x) => x.at <= window[1]),
+    window,
+    serverLog: s.serverLog(),
+    trace,
+  };
+}
+
+function outcomeOf(
+  scenario: EvalScenario,
+  condition: Condition,
+  done: Run[],
+  tools: Record<string, string>,
+  extra: Record<string, number> = {},
+): Outcome {
+  const golden = readGolden(goldenFile(scenario.id));
+  const comparisons = golden ? done.map((r) => compare(r.results ?? {}, golden.groups)) : [];
+  const worst = comparisons.reduce<(typeof comparisons)[number] | undefined>(
+    (w, c) => (!w || c.recall + c.precision < w.recall + w.precision ? c : w),
+    undefined,
+  );
+  const samples: Array<Record<string, number>> = done.map((r, i) =>
+    comparisons[i] ? { ...r.metrics, recall: comparisons[i].recall, precision: comparisons[i].precision } : r.metrics,
+  );
+  return {
+    scenario: scenario.id,
+    title: scenario.title,
+    condition,
+    metrics: { ...summarize(samples), ...extra },
+    samples: Object.fromEntries(Object.keys(samples[0] ?? {}).map((k) => [k, samples.map((x) => x[k]!)])),
+    results: done.at(-1)?.results,
+    comparison: worst,
+    drift: golden ? toolDrift(golden, tools) : undefined,
+    degraded: done.find((r) => r.degraded)?.degraded ?? null,
+  };
+}
+
+async function runScenario(
+  s: Session,
+  scenario: EvalScenario,
+  condition: Condition,
+  browser: Browser,
+  tools: Record<string, string>,
+  extra: Record<string, number> = {},
+): Promise<Outcome> {
+  const total = condition === "cold" ? 1 : warmup + runs;
+  const done: Run[] = [];
+  process.stdout.write(`${idOf({ scenario: scenario.id, condition }).padEnd(52)} `);
+  try {
+    for (let i = 0; i < total; i++) {
+      const last = i === total - 1;
+      const trace = args.trace && last ? join(OUT, "traces", `${scenario.id}@${condition}.json`) : null;
+      const r = await measureOnce(s, scenario, trace, browser);
+      if (condition === "cold" || i >= warmup) done.push(r);
+      process.stdout.write(condition === "warm" && i < warmup ? "w" : ".");
+    }
+    console.log();
+    const o = outcomeOf(scenario, condition, done, tools, extra);
+    const slowest = done.reduce((a, b) => ((b.metrics.completeMs ?? 0) > (a.metrics.completeMs ?? 0) ? b : a));
+    o.diagnostics = diagnose(
+      s,
+      slowest.window,
+      args.trace ? join(OUT, "traces", `${scenario.id}@${condition}.json`) : undefined,
+    );
+    return o;
+  } catch (e) {
+    console.log(" failed");
+    const message = e instanceof Error ? e.message : String(e);
+    const partial = e instanceof TimeoutError ? (e.partial as Record<string, number>) : {};
+    const now = s.timeline.now();
+    return {
+      ...outcomeOf(scenario, condition, done, tools, extra),
+      metrics: { ...partial, ...extra },
+      error: message,
+      diagnostics: diagnose(s, [s.timeline.stepStart ?? Math.max(0, now - 35_000), now]),
+    };
+  }
+}
+
+async function main() {
+  mkdirSync(join(OUT, "traces"), { recursive: true });
+  process.env.OPENIDEA_SERVER = serverBinary();
+  if (!args["no-build"]) buildBundle(ROOT, DIST);
+  if (!existsSync(join(DIST, "index.html"))) throw new Error(`no build in ${DIST}; run without --no-build`);
+
+  const tools = toolVersions();
+  const pins = pinnedTools(join(ROOT, ".github", "tool-versions.env"));
+  const machine: Fingerprint = fingerprint();
+  for (const [k, want] of Object.entries(pins)) {
+    if (tools[k] !== undefined && !tools[k]!.startsWith(want))
+      console.warn(`warning: ${k} ${tools[k]} is not the pinned ${want} (.github/tool-versions.env)`);
+  }
+
+  const { server, url } = await serveBundle(DIST);
+  const browser = await chromium.launch({
+    channel: args.channel === "chromium" ? undefined : args.channel,
+    headless: !args.headed,
+    args: [
+      "--disable-renderer-backgrounding",
+      "--disable-background-timer-throttling",
+      "--disable-backgrounding-occluded-windows",
+    ],
+  });
+  const outcomes: Outcome[] = [];
+  try {
+    // Warm: one session per fixture, its project imported once, scenarios one after another.
+    const warm = selected.filter(
+      (s) => s.conditions.includes("warm") && (!conditionFilter || conditionFilter === "warm"),
+    );
+    for (const fixture of [...new Set(warm.map((s) => s.fixture))]) {
+      const scenarios = warm.filter((s) => s.fixture === fixture);
+      const s = await Session.start(browser, url, ROOT, { fixture, workDir: join(OUT, "warm"), keepHome: false });
+      try {
+        await scenarios[0]!.setup(s);
+        await s.settle(scenarios[0]!.server);
+        for (const scenario of scenarios) outcomes.push(await runScenario(s, scenario, "warm", browser, tools));
+      } finally {
+        await s.close();
+      }
+    }
+    // Cold: a fresh server and project import per scenario, measured once after the import settles.
+    const cold = selected.filter(
+      (s) => s.conditions.includes("cold") && (!conditionFilter || conditionFilter === "cold"),
+    );
+    for (const scenario of cold) {
+      const started = Date.now();
+      const s = await Session.start(browser, url, ROOT, {
+        fixture: scenario.fixture,
+        workDir: join(OUT, "cold"),
+        keepHome: false,
+      });
+      try {
+        await scenario.setup(s);
+        await s.settle(scenario.server);
+        // Settling waits out the quiet period at its end.
+        const readyMs = Date.now() - started - 2000;
+        outcomes.push(await runScenario(s, scenario, "cold", browser, tools, { readyMs }));
+      } finally {
+        await s.close();
+      }
+    }
+  } finally {
+    await browser.close();
+    server.close();
+  }
+
+  if (args["update-golden"]) {
+    for (const o of outcomes) {
+      if (o.error || !o.results) continue;
+      writeGolden(goldenFile(o.scenario), { tools, groups: o.results });
+      const c = compare(o.results, o.results);
+      o.comparison = c;
+      o.metrics.recall = c.recall;
+      o.metrics.precision = c.precision;
+      o.drift = [];
+    }
+    console.log("Goldens re-recorded: review `git diff evals/` before committing.");
+  }
+
+  const baselineFile = join(OUT, "baseline.json");
+  let baseline: Outcome[] | null = null;
+  if (existsSync(baselineFile)) {
+    const saved = JSON.parse(readFileSync(baselineFile, "utf8")) as { machine: Fingerprint; outcomes: Outcome[] };
+    if (sameMachine(saved.machine, machine)) baseline = saved.outcomes;
+    else console.warn("warning: target/eval/baseline.json is from another machine or OS: not compared");
+  }
+  const repro = (o: Outcome) => `pnpm eval --only ${o.scenario} --condition ${o.condition}`;
+  const verdicts = judge(outcomes, loadBudgets(join(ROOT, "evals", "budgets.json")), baseline, repro);
+
+  const meta = {
+    date: new Date().toISOString(),
+    commit: git("rev-parse", "--short", "HEAD"),
+    dirty: git("status", "--porcelain", "--untracked-files=no") !== "",
+    runs,
+    warmup,
+    machine,
+    tools,
+    pins,
+    browser: browser.version(),
+  };
+  const report = {
+    meta,
+    pass: verdicts.every((v) => v.status === "pass"),
+    verdicts: verdicts.map((v) => {
+      const o = outcomes.find((x) => idOf(x) === v.id)!;
+      return {
+        ...v,
+        samples: o.samples,
+        results: o.results,
+        comparison: o.comparison,
+        drift: o.drift,
+        degraded: o.degraded,
+        error: o.error,
+        // Diagnostics only where something needs a look.
+        diagnostics: v.status === "fail" || v.regressions.length ? o.diagnostics : undefined,
+      };
+    }),
+  };
+  writeFileSync(join(OUT, "report.json"), JSON.stringify(report, null, 2));
+  const failing = verdicts.filter((v) => v.status === "fail");
+  const md = [
+    "# OpenIDEA eval report",
+    "",
+    `${meta.date} · \`${meta.commit}\`${meta.dirty ? " (dirty)" : ""} · ${machine.cpu} · ${machine.os} · jdtls ${tools.jdtls ?? "?"} · java ${tools.java ?? "?"}`,
+    "",
+    table(verdicts, true),
+    "",
+    ...failing.flatMap((v) => [`- **${v.id}**: ${v.failures.map((f) => f.detail).join("; ")} — \`${v.repro}\``]),
+  ].join("\n");
+  writeFileSync(join(OUT, "report.md"), md + "\n");
+
+  console.log();
+  console.log(table(verdicts, false));
+  for (const v of failing) {
+    console.log(`\n${v.id} FAILED (${v.repro})`);
+    for (const f of v.failures) console.log(`  ${f.kind}: ${f.detail}`);
+    const o = outcomes.find((x) => idOf(x) === v.id)!;
+    if (o.drift?.length) console.log(`  golden recorded with other tools: ${o.drift.join(", ")}`);
+    const slow = [...(o.diagnostics?.calls ?? [])].sort((a, b) => b.ms - a.ms).slice(0, 5);
+    if (slow.length)
+      console.log(
+        `  slowest commands: ${slow.map((c) => `${c.cmd} ${Math.round(c.ms)} ms${c.pending ? " (no answer)" : c.ok ? "" : " (failed)"}`).join(", ")}`,
+      );
+  }
+  console.log(`\nReport: ${join(OUT, "report.json")}`);
+  if (args["save-baseline"]) {
+    writeFileSync(baselineFile, JSON.stringify({ machine, meta, outcomes }, null, 2));
+    console.log(`Baseline saved to ${baselineFile}`);
+  }
+  const regressed = verdicts.some((v) => v.regressions.length);
+  process.exit(failing.length || (args["fail-on-regression"] && regressed) ? 1 : 0);
+}
+
+await main();

evals/tsconfig.json

@@ -0,0 +1,4 @@
+{
+  "extends": "../perf/tsconfig.json",
+  "include": ["./**/*.ts", "../perf/lib/**/*.ts", "../smoke/server.ts", "../smoke/tauriShim.ts"]
+}

fixtures/java-app/src/main/java/com/example/orders/BulkDiscount.java

@@ -0,0 +1,18 @@
+package com.example.orders;
+
+/** Takes a percentage off orders of at least {@code minQuantity} items. */
+public class BulkDiscount extends ListPrice {
+    private final int minQuantity;
+    private final int percent;
+
+    public BulkDiscount(int minQuantity, int percent) {
+        this.minQuantity = minQuantity;
+        this.percent = percent;
+    }
+
+    @Override
+    public long price(Order order) {
+        long list = super.price(order);
+        return order.quantity() >= minQuantity ? list * (100 - percent) / 100 : list;
+    }
+}

fixtures/java-app/src/main/java/com/example/orders/ListPrice.java

@@ -0,0 +1,9 @@
+package com.example.orders;
+
+/** Charges the order's total. */
+public class ListPrice implements PricingRule {
+    @Override
+    public long price(Order order) {
+        return order.totalCents();
+    }
+}

fixtures/java-app/src/main/java/com/example/orders/PricingRule.java

@@ -0,0 +1,6 @@
+package com.example.orders;
+
+/** Turns an order into the price charged for it, in cents. */
+public interface PricingRule {
+    long price(Order order);
+}

knip.ts

@@ -11,6 +11,7 @@ const config: KnipConfig = {
     "!src/editor/core/cm/astro/parser.terms.ts",
     "scripts/**/*.ts",
     "perf/**/*.ts",
+    "evals/**/*.ts",
     "e2e/**/*.ts",
     "smoke/**/*.ts",
   ],

package.json

@@ -17,7 +17,7 @@
     "format": "oxfmt",
     "format:check": "oxfmt --check",
     "knip": "knip",
-    "typecheck": "tsrs --noEmit && tsrs -p tsconfig.node.json --noEmit && tsrs -p perf --noEmit && tsrs -p scripts --noEmit && tsrs -p smoke --noEmit",
+    "typecheck": "tsrs --noEmit && tsrs -p tsconfig.node.json --noEmit && tsrs -p perf --noEmit && tsrs -p evals --noEmit && tsrs -p scripts --noEmit && tsrs -p smoke --noEmit",
     "gen:bindings": "cargo test -p openidea --test export_bindings",
     "gen:windows-icons": "node scripts/gen-windows-icons.ts",
     "gen:angular-grammar": "lezer-generator src/editor/core/cm/angular/angular.grammar --typeScript -o src/editor/core/cm/angular/parser.ts",
@@ -31,6 +31,7 @@
     "e2e": "playwright test",
     "perf": "node perf/run.ts",
     "perf:rust": "node perf/rust.ts",
+    "eval": "node evals/run.ts",
     "release:prepare": "node scripts/release-prepare.ts",
     "nightly:prepare": "node scripts/nightly-prepare.ts"
   },

perf/lib/bundle.ts

@@ -0,0 +1,47 @@
+/** The workbench's `--mode perf` production bundle: building it and serving it to Chromium. */
+import { spawnSync } from "node:child_process";
+import { createReadStream, existsSync, statSync } from "node:fs";
+import { createServer, type Server } from "node:http";
+import { extname, join } from "node:path";
+
+export function buildBundle(root: string, outDir: string, flags: string[] = []) {
+  console.log("Building the workbench (vite build --mode perf)…");
+  const r = spawnSync(
+    "pnpm",
+    ["exec", "vite", "build", "--mode", "perf", ...flags, "--outDir", outDir, "--emptyOutDir"],
+    {
+      cwd: root,
+      stdio: ["ignore", "ignore", "inherit"],
+      shell: process.platform === "win32",
+    },
+  );
+  if (r.status !== 0) throw new Error("vite build failed");
+}
+
+const MIME: Record<string, string> = {
+  ".html": "text/html",
+  ".js": "text/javascript",
+  ".css": "text/css",
+  ".json": "application/json",
+  ".svg": "image/svg+xml",
+  ".png": "image/png",
+  ".woff2": "font/woff2",
+  ".wasm": "application/wasm",
+};
+
+/** Serves `dir` on a free local port (unknown paths: `index.html`). */
+export function serveBundle(dir: string): Promise<{ server: Server; url: string }> {
+  const server = createServer((req, res) => {
+    const path = decodeURIComponent(new URL(req.url ?? "/", "http://x").pathname);
+    let file = join(dir, path);
+    if (!file.startsWith(dir) || !existsSync(file) || statSync(file).isDirectory()) file = join(dir, "index.html");
+    res.writeHead(200, { "content-type": MIME[extname(file)] ?? "application/octet-stream" });
+    createReadStream(file).pipe(res);
+  });
+  return new Promise((ok) =>
+    server.listen(0, "127.0.0.1", () => {
+      const addr = server.address();
+      ok({ server, url: `http://127.0.0.1:${typeof addr === "object" && addr ? addr.port : 0}` });
+    }),
+  );
+}

perf/run.ts

@@ -11,13 +11,12 @@
  *   pnpm perf -- --only typing --trace   Chrome trace of the last run → target/perf/traces/
  *   pnpm perf -- --only typing --profile --unminified   CPU profile + top functions
  */
-import { spawnSync } from "node:child_process";
-import { createReadStream, existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
-import { createServer, type Server } from "node:http";
-import { extname, join, resolve } from "node:path";
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { join, resolve } from "node:path";
 import { parseArgs } from "node:util";
 import { gzipSync } from "node:zlib";
 import { chromium, type Browser } from "@playwright/test";
+import { buildBundle, serveBundle } from "./lib/bundle.ts";
 import { Harness, type Metrics, type Scenario } from "./lib/harness.ts";
 import { topFunctions, type CpuProfile } from "./lib/profile.ts";
 import { installInPage } from "./lib/inpage.ts";
@@ -65,18 +64,7 @@ if (only && scenarios.length !== only.length) {
 }
 
 function build() {
-  console.log("Building the workbench (vite build --mode perf)…");
-  const flags = args.unminified ? ["--minify", "false"] : [];
-  const r = spawnSync(
-    "pnpm",
-    ["exec", "vite", "build", "--mode", "perf", ...flags, "--outDir", DIST, "--emptyOutDir"],
-    {
-      cwd: ROOT,
-      stdio: ["ignore", "ignore", "inherit"],
-      shell: process.platform === "win32",
-    },
-  );
-  if (r.status !== 0) throw new Error("vite build failed");
+  buildBundle(ROOT, DIST, args.unminified ? ["--minify", "false"] : []);
 }
 
 /** Initial-load JavaScript / CSS from the built index.html (gzip sizes). */
@@ -92,33 +80,6 @@ function bundleMetrics(): Metrics {
   };
 }
 
-const MIME: Record<string, string> = {
-  ".html": "text/html",
-  ".js": "text/javascript",
-  ".css": "text/css",
-  ".json": "application/json",
-  ".svg": "image/svg+xml",
-  ".png": "image/png",
-  ".woff2": "font/woff2",
-  ".wasm": "application/wasm",
-};
-
-function serve(): Promise<{ server: Server; url: string }> {
-  const server = createServer((req, res) => {
-    const path = decodeURIComponent(new URL(req.url ?? "/", "http://x").pathname);
-    let file = join(DIST, path);
-    if (!file.startsWith(DIST) || !existsSync(file) || statSync(file).isDirectory()) file = join(DIST, "index.html");
-    res.writeHead(200, { "content-type": MIME[extname(file)] ?? "application/octet-stream" });
-    createReadStream(file).pipe(res);
-  });
-  return new Promise((ok) =>
-    server.listen(0, "127.0.0.1", () => {
-      const addr = server.address();
-      ok({ server, url: `http://127.0.0.1:${typeof addr === "object" && addr ? addr.port : 0}` });
-    }),
-  );
-}
-
 interface Capture {
   trace?: string;
   profile?: string;
@@ -180,7 +141,7 @@ async function main() {
   const results: ScenarioResult[] = args.unminified
     ? []
     : [{ id: "bundle", title: "Initial bundle (gzip)", metrics: bundleMetrics() }];
-  const { server, url } = await serve();
+  const { server, url } = await serveBundle(DIST);
   const browser = await chromium
     .launch({
       channel: args.channel === "chromium" ? undefined : args.channel,

smoke/desktop.ts

@@ -1,76 +1,24 @@
 /**
  * The frontend's view of the desktop app, with a real backend: the window's Tauri runtime
- * (callbacks, `Channel`s, the event plugin) in-process, and every app command sent to an
- * `openidea-server` through the route remote development uses (`src/platform/tauriCore.ts`).
+ * in-process (`tauriShim.ts`), and every app command sent to an `openidea-server`, as
+ * remote development sends them.
  */
 import { homedir, tmpdir } from "node:os";
-import { join } from "node:path";
-import { setInvokeRoute } from "@/platform/tauriCore";
-import { encodeArgs } from "@/platform/remote";
-import { Server } from "./server";
+import { sep } from "node:path";
+import { installTauriShim, type ShimWindow } from "./tauriShim";
+import type { Server } from "./server";
 
-type Callback = (message: unknown) => void;
-
-// One runtime per page, there from the start as in the app: the frontend registers its event
-// listener once (`src/platform/ipc.ts`), possibly before a server is connected, and keeps it
-// across the servers connected one after another.
-const callbacks = new Map<number, Callback>();
-const listeners: { event: string; handler: number }[] = [];
-const channels = new Map<number, { onmessage: (data: unknown) => void }>();
-let nextCallback = 1;
-/** `BaseDirectory` values (`@tauri-apps/api/path`) the frontend resolves. */
-const DIRECTORIES: Record<number, string> = { 12: tmpdir(), 21: homedir() };
-
-{
-  const internals = {
-    metadata: { currentWindow: { label: "main" }, currentWebview: { label: "main", windowLabel: "main" } },
-    transformCallback(cb: Callback) {
-      const id = nextCallback++;
-      callbacks.set(id, cb);
-      return id;
-    },
-    unregisterCallback(id: number) {
-      callbacks.delete(id);
-    },
-    convertFileSrc: (path: string) => path,
-    // Only the window's own plugins reach this: app commands take the route below.
-    async invoke(cmd: string, args: Record<string, unknown> = {}) {
-      if (cmd === "plugin:event|listen") {
-        listeners.push({ event: args.event as string, handler: args.handler as number });
-        return listeners.length;
-      }
-      if (cmd === "plugin:window|get_all_windows") return [];
-      // This machine's directories (`@tauri-apps/api/path`), as on the desktop.
-      if (cmd === "plugin:path|resolve_directory") return DIRECTORIES[args.directory as number] ?? null;
-      if (cmd === "plugin:path|join") return join(...(args.paths as string[]));
-      if (cmd.startsWith("plugin:")) return null;
-      throw { kind: "unavailable", message: `${cmd} has no route`, detail: null };
-    },
-  };
-  Object.assign(window, {
-    __TAURI_INTERNALS__: internals,
-    __TAURI_EVENT_PLUGIN_INTERNALS__: { unregisterListener: () => {} },
-  });
-}
+installTauriShim({ directories: { 12: tmpdir(), 21: homedir() }, sep: sep as "/" | "\\" });
 
 /** Connects this page's frontend to `server`; returns a function that disconnects it. */
 export function connectDesktop(server: Server): () => void {
-  channels.clear();
-  server.onChannel = (id, data) => channels.get(id)?.onmessage(data);
-  server.onEvent = (target, event) => {
-    if (target !== null && target !== "main") return;
-    for (const l of listeners) {
-      if (l.event === "ide://event") callbacks.get(l.handler)?.({ event: l.event, id: 0, payload: event });
-    }
-  };
-  setInvokeRoute((cmd, args) => {
-    if (cmd.startsWith("plugin:")) return null;
-    const { json, channels: used } = encodeArgs(args);
-    for (const [id, channel] of used) channels.set(id, channel as unknown as { onmessage: (data: unknown) => void });
-    return server.invoke(cmd, JSON.parse(json));
-  });
+  const w = window as unknown as ShimWindow;
+  w.__openideaShim.reset();
+  server.onChannel = (id, data) => w.__openideaShim.channel(id, data);
+  server.onEvent = (target, event) => w.__openideaShim.event(target, event);
+  w.__openideaForward = (cmd, args) => server.invoke(cmd, args);
   return () => {
-    setInvokeRoute(null);
+    delete w.__openideaForward;
     server.onChannel = () => {};
     server.onEvent = () => {};
   };

smoke/server.ts

@@ -121,6 +121,7 @@ export class Server {
   onEvent: (window: string | null, event: unknown) => void = () => {};
 
   private closing: Promise<void> | undefined;
+  // A plain field, not a parameter property: the evals load this file with Node's type stripping.
   private readonly child: ChildProcessWithoutNullStreams;
 
   private constructor(child: ChildProcessWithoutNullStreams) {