Порівняти коміти

...
Автор SHA1 Повідомлення Дата
l0ng-ai cccefcc078 feat(plugins): open the plugins page on Available 2026-11-11 20:06:13 +08:00
l0ng-ai 4bb76acf66 feat(plugins): drop the Preview badge from the plugins page 2026-11-11 20:04:43 +08:00
l0ng-ai 187c0324fd fix(plugins): plugin icons fill their tile like the initials 2026-11-11 20:03:46 +08:00
l0ng-ai 2fca3e0607 feat(plugins): a marketplace that helps pick plugins
Rows show each plugin's description, publisher, downloads and an inline
Install / Update button; plugins without an icon get a tinted initial.
While browsing, the list opens with suggestions for the open project and
the maintainers' picks, then all community plugins by downloads; the
official plugins come last as examples. The first plugin is selected so
the details are never empty, and they show release notes, the homepage
and why a plugin is suggested.

The index gains optional homepage, notes, downloads, featured and suggest
fields; openidea-plugin index fills homepage and notes from the manifest.
2026-11-11 19:43:34 +08:00
l0ng-ai 02ee2880fe fix: translate New Project popup and error text; green CI on main (#2715)
Translate the project popup actions and the non-empty target error; translate the JDK field label. Fix clippy in ide-jdwp, formatting, RUSTSEC-2026-0486 (rustix 1.1.6) and agent E2E tests that expected the removed floating Quote button.
2026-11-11 19:25:09 +08:00
l0ng-ai 5a316e81d6 fix: Compile File sees its module's classes; list external agent sessions per project off the main thread (#2707)
Compile File passed javac only the dependencies and upstream modules, so any file using
another class of its own module failed with "cannot find symbol". The module's own
output folders now come first on the classpath, and its source folders (generated ones
included; main ones for a test) are the -sourcepath with -implicit:none, so classes that
were never built resolve from source while only the file itself is written.

Listing sessions started outside OpenIDEA parsed every Claude Code transcript of every
project on a synchronous command, which runs on the main thread and froze the window
(1.2 GB of transcripts here). It now reads only the transcript folders of the open
project, keeps sessions whose working directory is in the project, reads Codex rollouts
line by line, and runs on the blocking pool; importing does the same.
2026-11-11 18:22:28 +08:00
l0ng-ai 7c2ca4d061 fix: build Maven again when outputs were deleted; main menu in a separate toolbar (#2706)
Build Project / Build Module skipped Maven when no input changed, even after the
modules' target/classes were deleted outside the IDE (mvn clean in a terminal).
The up-to-date record now also lists the output folders that existed after the
build, and a build is skipped only while they all still exist.

The "Always show the main menu" toggle duplicated Main menu › Merged with main
toolbar, showed on macOS where it does nothing, and kept the titles in a title
bar too narrow for them. It is replaced by a third Main menu choice, Separate
toolbar: the menu titles get their own row below the title bar and never
collapse into the hamburger button.
2026-11-11 18:09:01 +08:00
ntbang0901 8d09479a65 feat(debugger): heap snapshots of a debugged JVM (#2649)
Capture a heap dump of the debugged JVM over the attach listener (GC.heap_dump, after checking free disk space and the JDK's rules for suspended VMs) or open an .hprof file, and browse it in an editor tab: classes with shallow and retained sizes, biggest objects, dominators, the path from an object to a GC root, and a comparison of two snapshots. A snapshot captured at a stop adds a Retained Size column to that stop's Memory tab.

New crate ide-heap: HotSpot hprof parser (bounds-checked; unfinished or crafted dumps are refused), object graph, dominator tree (semi-NCA), retained sizes and GC-root paths. Snapshots live in private directories removed with their window; an opened dump is analyzed from a private copy so changes to the user's file can't affect the mapped data.

Co-authored-by: ntbang0901 <[email protected]>
2026-11-11 17:43:07 +08:00
Bin 8fafde10b4 fix(editor): keep tab actions at the pane edge (#2682)
The trailing slot of the editor tab strip had flex: none, so the split / more buttons followed the last tab. An automatic left margin keeps them at the right edge of the pane without changing how wrapped tabs share space.

Co-authored-by: ffgenius <[email protected]>
2026-11-11 17:35:16 +08:00
Muhammad N. Fadhil b1cd917a96 fix(gradle): process resources before launches from the language server's classpath (#2678)
The Java language server copies resources unchanged into Buildship's bin/<source set>, so launches with "Build and run using: OpenIDEA" saw unfiltered resources (e.g. `version: @version@`). The Gradle sync now records which source sets Gradle changes resources for (filters, expand, renames, includes/excludes, extra sources); before such a launch the IDE runs only those processResources tasks and copies their output over the plain copies. Builds that copy resources as they are run no extra Gradle step; if Gradle fails, the launch continues with a hint.

Closes #2675

Co-authored-by: MuhammadNFadhil <[email protected]>
2026-11-11 17:33:43 +08:00
l0ng-ai 605d5f4985 feat(new-project): project generators for mainstream languages and frameworks (#2689)
File › New Project offers 38 generators grouped by language (Java, Spring Boot, Quarkus, Micronaut, Kotlin, Ktor, Scala, Python/Django/FastAPI/Flask, Node/Vite/Next.js/Nuxt/Angular/SvelteKit/Astro/NestJS/Express, Go, Rust, C/C++, Zig, .NET, PHP/Laravel/Symfony, Ruby/Rails, Dart/Flutter, Swift, Elixir/Phoenix, Haskell, static website, empty). Each is a declaration in generators/<id>.toml run by the new ide-scaffold crate: built-in templates, the framework's official CLI run non-interactively, or a project service zip (Spring Initializr, code.quarkus.io, Micronaut Launch) whose metadata becomes form fields. Projects are built in a staging directory and moved into place only on success. The dialog renders each form from its schema, flags missing toolchains with install hints and streams progress; created projects open trusted. New Project is also on the welcome screen and the title bar project menu.

Closes #709
2026-11-11 17:22:57 +08:00
l0ng-ai 7d52f058a7 chore: record ide-search's encoding_rs dev-dependency in Cargo.lock (#2695) 2026-11-11 17:17:30 +08:00
l0ng-ai c44558b4aa fix(lsp): report a server missing while a variable it requires is unset (#2671)
With pwsh installed but PSES_BUNDLE_PATH unset, PowerShell Editor Services was launched with a broken path and restarted in a loop with a crash notification each time. Language servers can now declare requires-env; while one of those variables is unset or empty (in the server's env and the IDE's environment) the server counts as missing and shows its install hint instead of starting. powershell.toml declares PSES_BUNDLE_PATH; the field is documented.
2026-11-11 17:16:11 +08:00
l0ng-ai 8455761181 fix(lsp): show only the crashed instance's stderr, without colour codes (#2668)
A language server crash notification quoted stderr verbatim: coloured output showed raw escape sequences, and since the buffer was never cleared between restarts each report repeated earlier instances' output. Stderr lines are now stripped of ANSI escapes when recorded and the buffer is cleared when a new instance starts; strip_ansi moves to ide-core::output_encoding so ide-build and ide-lsp share it.
2026-11-11 17:16:07 +08:00
l0ng-ai dabf9a954e fix(search): find text in GBK and UTF-16 files (#2665)
Closed files that are not UTF-8 were searched as raw bytes: Chinese in GBK files never matched and ASCII matches showed mojibake, UTF-16 files were skipped as binary, and comment/string-limited searches skipped non-UTF-8 files. The engine now checks UTF-8 while streaming and searches non-UTF-8 files again in text decoded like the editor does (BOM / UTF-16 as detected, else the project's fallback encoding). Find in Files, Search Everywhere › Text, MCP and plugin search share this path.
2026-11-11 17:14:42 +08:00
ntbang0901 a3d1e57f3e feat(debugger): memory view for a paused JVM (#2648)
Adds a Memory tab to the Debug tool window for JVM sessions: loaded classes with instance counts and shallow sizes (with deltas between refreshes), read over the HotSpot attach listener (`GC.class_histogram`), falling back to counting reachable objects through the debugger on JDKs that cannot answer while suspended. While paused, a class's instances, an instance's fields and its referrers are listed through a JDWP link (the Kotlin adapter's path, and a jdtls bundle `java/jvm-inspect` for java-debug). Debug launches started by the IDE get `-XX:+StartAttachListener` unless the user's VM options decide otherwise (not on Windows).

New crates: ide-jdwp, ide-jvm-attach, ide-jvm-inspect. The bundled jar rebuilds byte-for-byte from build.py against the pinned JDT LS plugins.

Co-authored-by: ntbang0901 <[email protected]>
2026-11-11 17:10:21 +08:00
l0ng-ai 0ff3f188de feat(agents): add built-in Qoder (#2687)
Qoder's global CLI starts with `qodercli --acp` like Qoder CN; both editions share the icon.
2026-11-11 16:49:13 +08:00
wangfeiтаhapput 1654d22756 feat(agents): add built-in Qoder CN support (#2647)
Qoder CN appears in the built-in agent list and starts with `qoderclicn --acp` through the existing ACP driver, with sign-in guidance and a row in the agent guides. Its icon is a 16px glyph with a dark-theme variant.

Co-authored-by: happut <[email protected]>
2026-11-11 16:46:45 +08:00
Emanyzwww 14d9cdff11 feat(app): show a What's New page after an update (#2664)
After an update the restarted app opens what the new version brought as an editor tab, and Help > What's New reopens it at any time. Its checkbox turns the automatic opening off for later updates; the same switch sits on System Settings > Updates next to the update checks. The update balloon keeps a one-paragraph summary.

The release workflow fills the updater manifest's notes from the release body without its Download section, the install records them, and a version installed outside the app reads them from the manifest. The first start of a different version is recognised from the last run version kept in settings.
2026-11-11 16:42:32 +08:00
Muuтаmuutot f7a507a5db feat(vcs): allow rewriting history of pushed commits (#2644)
Edit Commit Message, Fixup, Squash Into, Drop Commit, Squash Commits and Interactively Rebase from Here are no longer disabled for pushed commits. A rewrite that reaches commits already on the upstream asks for confirmation first (pushing afterwards needs a force push); on a protected branch such a rewrite is refused, while its local commits can still be rewritten. Undo Commit follows the same rule.

Co-authored-by: muutot <[email protected]>
2026-11-11 16:41:45 +08:00
Charlie.Weiтаcharli117 c67e07a4f9 feat(plugins): cancel plugin commands per request (#2620)
Plugin commands could only be cancelled as a whole plugin. Local IPC and remote RPC now take a request id: a cancelled request is dropped before the plugin is activated, a running command gets a short grace period to exit and is interrupted after it, and other calls to the same plugin keep running. The frontend runCommand takes an AbortSignal that settles the caller at once and forwards the cancellation.

Co-authored-by: charli117 <[email protected]>
2026-11-11 16:37:16 +08:00
misswellтаmisswell 4e08893eae fix(run): build Maven modules through Maven on Rebuild (#2617)
A Rebuild launch of a Maven module used jdtls' java/buildWorkspace(true), which only recompiles sources: stale classes stayed in target/classes, and JDT rejected sources javac accepts (a package that doesn't match its directory), so Debug stopped with "Build has compile errors" although mvn clean compile succeeded. Rebuild now always delegates Maven modules to their Maven build (clean compile / clean test-compile), and a successful clean build clears the stale JDT diagnostics. Automatic and Make keep the JDT path.

Co-authored-by: misswell <[email protected]>
2026-11-11 16:33:19 +08:00
l0ng-ai 779070c3d7 feat(vcs): open diff follows the change tree selection; file buttons in the diff toolbar (#2628)
With a diff tab open from the Commit or Local Changes tree, selecting another file in the tree shows that file in the same tab and keeps focus in the tree; with no diff open, selecting opens nothing. The diff toolbar gets Compare Previous File / Compare Next File buttons when the diff has sibling files.

Closes #2492
2026-11-11 16:30:53 +08:00
l0ng-ai 4e86161c3c fix(run): debug Maven tests from the gutter and attach when Surefire starts the fork (#2684)
The run gutter and context menu offered Debug only for configurations with a debug plan or adapter, so Maven / Gradle tests (debugged through a JDWP agent) had none. Debugging a Maven test attached to the pinned Surefire port right away and java-debug's 30-second retry could run out while Maven recompiled; it now attaches when Maven prints the T E S T S banner just before starting the fork. A failed attach to a JVM started for the debugger now stops that run (as for Python), so its suspended JVM doesn't keep it alive.

Fixes #2656
2026-11-11 16:28:14 +08:00
arookieofc d7393b53f3 feat(menu): add always-show main menu option (#1521) (#2661)
* feat(menu): add always-show main menu option (#1521)

* fix(i18n): add zh catalog entries for always-show main menu strings
2026-11-11 16:22:52 +08:00
Deng Liuтаdengliu 276a91973c fix(terminal): use editor search colors, keep the active match above the selection, and seed ⌘F (#2606)
The terminal Find bar painted matches with the old text-search tokens; it now uses --bg-search-match / --bg-search-match-current (blended over the panel background) like the editor. The active match's decoration goes to xterm's top layer so it stays visible above the selection the search places, and ⌘F with the Find bar already open seeds it from a fresh user selection while keeping the query when the selection is the current match.

Co-authored-by: dengliu <[email protected]>
2026-11-11 16:17:36 +08:00
l0ng-ai fc75b15519 perf(editor): find a type's overridden methods with one subtype walk (#2680)
The gutter's "is implemented / overridden" markers sent one implementation request per visible method, and jdtls builds the type hierarchy again for each, so a large interface in a big project kept it busy long enough to hold up navigation. When outline names carry the signature (jdtls), a type's methods are now resolved from one walk of its subtypes and their outlines, as the super-method markers are; used only while it takes fewer requests than it replaces, and a method whose name a subtype reuses with another signature (generic overloads) still gets its own request.

Refs #2314
2026-11-11 16:04:10 +08:00
arookieofc 8798c2a542 fix(remote): sync outside-root classpath entries for SSH Java launches (#2641) (#2669) 2026-11-11 15:55:30 +08:00
Deng Liuтаdengliu 8ea2787490 fix(editor): keep selection within its own row when font glyphs exceed line height (#2674)
With a font whose glyph box is taller than the line height (e.g. font size 18–20 with the default 22px line height), selecting text on one line also painted the rows above and below: growToLineBoxes probed the rows at the glyph box's top and bottom edges, which then fell in the neighbouring rows, and .cm-openidea-selected painted an unclipped inline background at glyph height. Outer rows are now resolved at each marker's vertical midpoint and row seams align to the line boxes (block widgets and zero-width trailing markers handled), and the inline selection background is capped at 1lh in the editor, notebook cells and diff/merge viewers, with a plain colour fallback where lh is unsupported.

Co-authored-by: dengliu <[email protected]>
2026-11-11 15:54:26 +08:00
l0ng-ai dc4e8403de fix(java): keep a language server JDK that ran usable while it is unchanged (#2679)
A JDK whose java -version succeeded was probed again every 10 seconds; on a busy machine one probe could time out and the working JDK was reported as "cannot run on this machine". A successful probe now stays valid while the executable's size, modification time and permissions are unchanged; failed probes are still retried after 10 seconds, and a java that hangs on its first probe is still rejected.

Fixes #2470
2026-11-11 15:50:10 +08:00
l0ng-ai 4a0efbae25 fix(agents): keep a run or turn the reader opened open across remounts
A run's open state lived in its row, so it closed when the row remounted: when a
finished turn folded its work away, or when a run grew from one call to two.
Opened runs and turns are now remembered by their first entry for the
conversation's lifetime.
2026-11-11 15:36:22 +08:00
l0ng-ai cb3c0bafd7 fix(boot): count the startup timeout only while the window shows
A window behind others or on another Space gets no animation frames, and the
splash waits for two of them before startup finishes. The 30-second timeout kept
running meanwhile and reported a healthy but slow start, such as a cold dev
build, as "OpenIDEA could not start". It now pauses while the document is hidden
and resumes with the time left.
2026-11-11 15:34:30 +08:00
l0ng-ai 71072dbe4b fix(agents): pace streaming reasoning, cut only between graphemes, cap at 60 Hz
- Reasoning in view streams at the reply's pace instead of in bursts.
- The painted end of a streaming text falls between graphemes (Intl.Segmenter):
  an emoji with joiners or a skin tone, a flag or a letter with combining marks
  no longer flashes in parts. A chunk ending mid-cluster holds that fragment back
  until the rest arrives.
- The reveal commits at most at 60 Hz, so a 120 Hz display doesn't re-render the
  Markdown twice as often.
2026-11-11 15:33:53 +08:00
l0ng-ai 18b286e7e0 fix(agents): one spinner per conversation, counts on one line
Work in progress shimmers on its label (a run's counts and current call, a
running tool, "Thinking") instead of showing a spinner beside it; the working
row at the conversation's end is the only spinner. A run's counts keep their
line, and what runs now ellipsizes in the space left.
2026-11-11 15:32:40 +08:00
l0ng-ai 9474af4de1 fix(agents): steadier tool-call runs, working row in the stream, selection and scroll fixes
- Reasoning and tool calls fold into one run including running calls, so a run
  keeps its place while a call finishes. Its summary names what runs now; it no
  longer leads with "Reasoning". A folded thought shows its first line. An open
  run is bounded and follows its latest call.
- The working / starting / retrying row with its timer ends the conversation
  instead of sitting above the composer, for sessions and subagents alike.
- Selecting text no longer pops up a Quote button; Quote stays in the context menu.
- A selection clamped to the conversation ends in its last selectable text, not on
  its edge: WebKit painted a selection ending there into the composer, as a
  triple-click on the last paragraph showed.
- Sending a message while scrolled up returns to the end, where it shows.
2026-11-11 15:20:09 +08:00
arookieofc 8044d99ea0 feat(build): compile single Java files and skip unchanged Maven builds (#2476) (#2646)
* feat(build): compile a single Java file with javac (#2476)

* feat(build): add Compile File action for single Java sources (#2476)

* feat(build): skip unchanged Maven menu builds via fingerprint records (#2476)

* fix(build): drop shadowed fixture model in unix-only classpath test (#2476)

* fix(build): pass cancel into shared executeBuild helper (#2476)

* chore: retrigger CI after mass-cancel wave (#2646)

* chore: probe CI queue recovery (#2646)
2026-11-11 15:03:42 +08:00
Charlie.Weiтаcharli117 0abaf6c777 fix(plugins): keep plugin settings that match their defaults (#2619)
Choosing a plugin setting's default value deleted the override, so in a project the global value showed through instead of the user's choice. The setting control now always writes the chosen value.

Co-authored-by: charli117 <[email protected]>
2026-11-11 14:45:40 +08:00
LU XIANGRUIтаwildcatDownstairs 0d9b08ac8c fix(color-scheme): import WebStorm JavaScript and TypeScript colours from .icls (#2634)
WebStorm stores JavaScript and TypeScript colours under JS.* / TS.* keys, which the .icls importer did not map, and DEFAULT_SEMICOLON / _COMMA / _DOT all mapped to Braces and Operators, so a scheme recolouring semicolons painted every brace and paren. JS.* and TS.* now map onto the Language Defaults attributes with TS > JS > PY > defaults precedence, and the separator keys are no longer mapped.

Co-authored-by: wildcatDownstairs <[email protected]>
2026-11-11 14:45:32 +08:00
Maskerтаzed1995 e54580265d fix(agents): show Codex ACP collaboration mode in the composer (#2631)
codex-acp reports its Default/Plan choice as a collaboration_mode config option, which the composer dropped because it only rendered model, mode, permissions, approval and thought_level. collaboration_mode is now one of the composer's ordered settings, so a Codex session can switch to Plan.

Co-authored-by: zed1995 <[email protected]>
2026-11-11 14:45:24 +08:00
LU XIANGRUIтаwildcatDownstairs 4fa700b337 fix(editor): collapse the bookmark gutter fully in files without bookmarks (#2632)
The bookmark gutter's 2px side padding applied to the empty spacer element too, so the column kept 4px in files without bookmarks. The padding now applies only to elements that hold a marker, so the column is 0px until the file has a bookmark.

Co-authored-by: wildcatDownstairs <[email protected]>
2026-11-11 14:45:16 +08:00
DrPepperтаmakise2060 173d5edcdc fix(run): render the Run console with the terminal font (#2615)
The Run console hard-coded the --font-mono stack and --fs-code, so Settings › Tools › Terminal › Font (family, fallback, size, or Use editor font) never reached process output. resolveMonoFont() now resolves the terminal font choice for both xterm and the DOM console, and the console's virtual row height follows the font size (round(size × 1.5), 20px at the default 13px) so rows don't overlap.

Co-authored-by: makise2060 <[email protected]>
2026-11-11 14:43:59 +08:00
452 змінених файлів з 43220 додано та 1313 видалено

No files matched your search

+22 -1
Переглянути файл
@@ -251,7 +251,28 @@ jobs:
core.setFailed(`latest.json would miss ${missing.join(", ")}`);
return;
}
const manifest = { version: tag.replace(/^v/, ""), notes: "", pub_date: new Date().toISOString(), platforms };
// The update's notes: the release body without its Download section. That
// section only matters on the release page, and What's new follows it.
const { data: release } = await github.rest.repos.getRelease({ owner, repo, release_id });
const withoutDownloadSection = (body) => {
const kept = [];
let skip = 0;
for (const line of body.split(/\r?\n/)) {
const heading = /^(#{1,6})\s+(.*)$/.exec(line);
if (heading) {
const level = heading[1].length;
if (!skip && level <= 3 && /^download$/i.test(heading[2].trim())) {
skip = level;
continue;
}
if (skip && level <= skip) skip = 0;
}
if (!skip) kept.push(line);
}
return kept.join("\n").trim();
};
const notes = withoutDownloadSection(String(release.body ?? ""));
const manifest = { version: tag.replace(/^v/, ""), notes, pub_date: new Date().toISOString(), platforms };
const old = assets.find((a) => a.name === "latest.json");
if (old) await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: old.id });
await github.rest.repos.uploadReleaseAsset({
Створено
+79 -2
Переглянути файл
@@ -4271,11 +4271,16 @@ version = "0.1.41"
dependencies = [
"futures",
"ide-core",
"ide-heap",
"ide-jdwp",
"ide-jsonrpc",
"ide-jvm-attach",
"ide-jvm-inspect",
"ide-language",
"libc",
"memchr",
"parking_lot",
"percent-encoding",
"portable-pty",
"serde",
"serde_json",
@@ -4375,6 +4380,24 @@ dependencies = [
"which",
]
[[package]]
name = "ide-heap"
version = "0.1.41"
dependencies = [
"criterion",
"ide-core",
"libc",
"memmap2",
"proptest",
"serde",
"serde_json",
"specta",
"tempfile",
"thiserror 2.0.21",
"tracing",
"uuid",
]
[[package]]
name = "ide-host"
version = "0.1.41"
@@ -4397,9 +4420,11 @@ dependencies = [
"ide-db",
"ide-docker",
"ide-extension",
"ide-heap",
"ide-http",
"ide-index",
"ide-jsonrpc",
"ide-jvm-attach",
"ide-language",
"ide-lsp",
"ide-mcp",
@@ -4407,6 +4432,7 @@ dependencies = [
"ide-plugin",
"ide-remote",
"ide-run",
"ide-scaffold",
"ide-sdk",
"ide-search",
"ide-spell",
@@ -4482,6 +4508,18 @@ dependencies = [
"tracing",
]
[[package]]
name = "ide-jdwp"
version = "0.1.41"
dependencies = [
"ide-jvm-inspect",
"proptest",
"tempfile",
"thiserror 2.0.21",
"tokio",
"tracing",
]
[[package]]
name = "ide-jsonrpc"
version = "0.1.41"
@@ -4499,6 +4537,26 @@ dependencies = [
"tracing",
]
[[package]]
name = "ide-jvm-attach"
version = "0.1.41"
dependencies = [
"libc",
"tempfile",
"thiserror 2.0.21",
]
[[package]]
name = "ide-jvm-inspect"
version = "0.1.41"
dependencies = [
"futures",
"serde",
"serde_json",
"specta",
"thiserror 2.0.21",
]
[[package]]
name = "ide-language"
version = "0.1.41"
@@ -4564,6 +4622,7 @@ dependencies = [
"globset",
"ide-core",
"ide-jsonrpc",
"ide-jvm-inspect",
"ide-language",
"ide-text",
"ide-vfs",
@@ -4702,6 +4761,22 @@ dependencies = [
"zip",
]
[[package]]
name = "ide-scaffold"
version = "0.1.41"
dependencies = [
"ide-core",
"ide-tools",
"ide-vcs",
"insta",
"serde",
"serde_json",
"specta",
"tempfile",
"toml 0.9.12+spec-1.1.0",
"tracing",
]
[[package]]
name = "ide-sdk"
version = "0.1.41"
@@ -4727,6 +4802,7 @@ name = "ide-search"
version = "0.1.41"
dependencies = [
"criterion",
"encoding_rs",
"globset",
"grep-matcher",
"grep-regex",
@@ -6513,6 +6589,7 @@ dependencies = [
"ide-profiler",
"ide-remote",
"ide-run",
"ide-scaffold",
"ide-sdk",
"ide-search",
"ide-spell",
@@ -7719,9 +7796,9 @@ dependencies = [
[[package]]
name = "rustix"
version = "1.1.5"
version = "1.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
dependencies = [
"bitflags 2.13.2",
"errno",
+6
Переглянути файл
@@ -43,10 +43,15 @@ ide-ssh = { path = "crates/ide-ssh" }
ide-remote = { path = "crates/ide-remote" }
ide-host = { path = "crates/ide-host" }
ide-mcp = { path = "crates/ide-mcp" }
ide-scaffold = { path = "crates/ide-scaffold" }
ide-notebook = { path = "crates/ide-notebook" }
ide-extension = { path = "crates/ide-extension" }
ide-plugin = { path = "crates/ide-plugin" }
ide-cli = { path = "crates/ide-cli" }
ide-jvm-attach = { path = "crates/ide-jvm-attach" }
ide-jvm-inspect = { path = "crates/ide-jvm-inspect" }
ide-jdwp = { path = "crates/ide-jdwp" }
ide-heap = { path = "crates/ide-heap" }
# serialization & IPC typing
serde = { version = "1", features = ["derive"] }
@@ -82,6 +87,7 @@ trash = "5"
blake3 = "1"
dunce = "1"
dirs = "6"
memmap2 = "0.9"
zip = { version = "9.2.3", default-features = false, features = ["deflate"] }
# search, index, language
+3 -4
Переглянути файл
@@ -1,11 +1,10 @@
# Third-party licenses
## Flutter and Vite project icons
## Project type icons
`src/ui/brands/flutter.svg` and `vite.svg` use the unmodified logo paths from
`src/ui/brands/*.svg` use the unmodified logo paths from
[Simple Icons](https://github.com/simple-icons/simple-icons), retrieved through
Iconify (`simple-icons:flutter`, `simple-icons:vite`) and filled with the project
icon colors. Licensed under [CC0 1.0 Universal](https://github.com/simple-icons/simple-icons/blob/develop/LICENSE.md).
Iconify (`simple-icons:angular`, `simple-icons:astro`, `simple-icons:cplusplus`, `simple-icons:dart`, `simple-icons:django`, `simple-icons:dotnet`, `simple-icons:elixir`, `simple-icons:express`, `simple-icons:fastapi`, `simple-icons:flask`, `simple-icons:flutter`, `simple-icons:go`, `simple-icons:haskell`, `simple-icons:html5`, `simple-icons:kotlin`, `simple-icons:ktor`, `simple-icons:laravel`, `simple-icons:nestjs`, `simple-icons:nextdotjs`, `simple-icons:nodedotjs`, `simple-icons:nuxt`, `simple-icons:openjdk`, `simple-icons:phoenixframework`, `simple-icons:php`, `simple-icons:python`, `simple-icons:quarkus`, `simple-icons:ruby`, `simple-icons:rubyonrails`, `simple-icons:rust`, `simple-icons:scala`, `simple-icons:springboot`, `simple-icons:svelte`, `simple-icons:swift`, `simple-icons:symfony`, `simple-icons:vite`, `simple-icons:zig`) and filled with the brand colors (the standard icon grey for black logos). Licensed under [CC0 1.0 Universal](https://github.com/simple-icons/simple-icons/blob/develop/LICENSE.md).
## Project Lombok
+70 -15
Переглянути файл
@@ -8,6 +8,7 @@
use super::native::claude::history;
use serde::Serialize;
use specta::Type;
use std::io::BufRead;
use std::path::{Path, PathBuf};
/// A session on this machine that no task holds yet.
@@ -62,11 +63,28 @@ fn message_text(e: &serde_json::Value) -> Option<String> {
}
}
/// Claude Code transcripts: `<config>/projects/*/<id>.jsonl`.
pub fn claude_sessions(config_dir: &Path) -> Vec<ExternalSession> {
/// Whether a session that ran in `cwd` belongs to the project at `root` (in it or below it).
fn in_project(cwd: Option<&str>, root: &Path) -> bool {
cwd.is_some_and(|cwd| Path::new(cwd).starts_with(root))
}
/// Claude Code transcripts of the project at `root`: `<config>/projects/<escaped cwd>/<id>.jsonl`.
/// Only the folders named after `root` or a folder below it are read, so other projects'
/// transcripts (often gigabytes) are never opened.
pub fn claude_sessions(config_dir: &Path, root: &Path) -> Vec<ExternalSession> {
let mut out = Vec::new();
let projects = config_dir.join("projects");
let dirs: Vec<PathBuf> = std::fs::read_dir(&projects).into_iter().flatten().flatten().map(|d| d.path()).collect();
let escaped = history::escape(root);
let dirs: Vec<PathBuf> = std::fs::read_dir(&projects)
.into_iter()
.flatten()
.flatten()
.filter(|d| {
let name = d.file_name().to_string_lossy().into_owned();
name == escaped || name.starts_with(&format!("{escaped}-"))
})
.map(|d| d.path())
.collect();
for dir in dirs {
let files: Vec<PathBuf> = std::fs::read_dir(&dir)
.into_iter()
@@ -93,6 +111,10 @@ pub fn claude_sessions(config_dir: &Path) -> Vec<ExternalSession> {
.filter_map(|e| e.get("cwd").and_then(serde_json::Value::as_str))
.next()
.map(str::to_owned);
// Escaping is lossy (`a/b-c` and `a/b/c` share a folder): the transcript decides.
if !in_project(cwd.as_deref(), root) {
continue;
}
out.push(ExternalSession {
provider: "claude".to_owned(),
session_id: id,
@@ -138,12 +160,13 @@ fn codex_user_text(line: &serde_json::Value) -> Option<String> {
/// One rollout file's session, or `None` when it names no `session_meta`.
fn codex_file(path: &Path) -> Option<ExternalSession> {
let text = std::fs::read_to_string(path).ok()?;
let file = std::fs::File::open(path).ok()?;
let mut id: Option<(String, Option<String>)> = None;
let mut title: Option<String> = None;
// `session_meta` is first, but cap the scan: giant rollouts must not stall listing.
for line in text.lines().take(400) {
let Ok(v) = serde_json::from_str::<serde_json::Value>(line) else { continue };
for line in std::io::BufReader::new(file).lines().take(400) {
let Ok(line) = line else { break };
let Ok(v) = serde_json::from_str::<serde_json::Value>(&line) else { continue };
if id.is_none()
&& let Some(meta) = codex_meta(&v)
{
@@ -183,11 +206,11 @@ fn rollout_files(dir: &Path, depth: usize, out: &mut Vec<PathBuf>) {
}
}
/// Codex sessions: `<home>/.codex/sessions/**`.
pub fn codex_sessions(codex_home: &Path) -> Vec<ExternalSession> {
/// Codex sessions of the project at `root`: `<home>/.codex/sessions/**`.
pub fn codex_sessions(codex_home: &Path, root: &Path) -> Vec<ExternalSession> {
let mut files = Vec::new();
rollout_files(&codex_home.join("sessions"), 4, &mut files);
files.iter().filter_map(|p| codex_file(p)).collect()
files.iter().filter_map(|p| codex_file(p)).filter(|s| in_project(s.cwd.as_deref(), root)).collect()
}
#[cfg(test)]
@@ -222,7 +245,7 @@ mod tests {
],
);
write(&tmp.path().join("projects/-work-app/empty.jsonl"), &[]);
let sessions = claude_sessions(tmp.path());
let sessions = claude_sessions(tmp.path(), Path::new("/work/app"));
assert_eq!(sessions.len(), 1);
let s = &sessions[0];
assert_eq!((s.provider.as_str(), s.session_id.as_str()), ("claude", "abc123"));
@@ -234,8 +257,8 @@ mod tests {
#[test]
fn claude_title_falls_back_to_the_first_prompt() {
let tmp = tempfile::tempdir().unwrap();
write(&tmp.path().join("projects/-x/def456.jsonl"), &[claude_user("u1", "hello there")]);
let sessions = claude_sessions(tmp.path());
write(&tmp.path().join("projects/-work-app/def456.jsonl"), &[claude_user("u1", "hello there")]);
let sessions = claude_sessions(tmp.path(), Path::new("/work/app"));
assert_eq!(sessions.len(), 1);
assert_eq!(sessions[0].title, "hello there");
}
@@ -253,7 +276,7 @@ mod tests {
],
);
write(&tmp.path().join("sessions/notes.jsonl"), &[json!({"nope": true})]);
let sessions = codex_sessions(tmp.path());
let sessions = codex_sessions(tmp.path(), Path::new("/work"));
assert_eq!(sessions.len(), 1);
let s = &sessions[0];
assert_eq!((s.provider.as_str(), s.session_id.as_str()), ("codex", "thread-1"));
@@ -264,7 +287,39 @@ mod tests {
#[test]
fn missing_directories_list_nothing() {
let tmp = tempfile::tempdir().unwrap();
assert!(claude_sessions(&tmp.path().join("nope")).is_empty());
assert!(codex_sessions(&tmp.path().join("nope")).is_empty());
assert!(claude_sessions(&tmp.path().join("nope"), Path::new("/work")).is_empty());
assert!(codex_sessions(&tmp.path().join("nope"), Path::new("/work")).is_empty());
}
#[test]
fn only_sessions_of_the_project_are_listed() {
let tmp = tempfile::tempdir().unwrap();
let user = |cwd: &str| {
json!({"type": "user", "uuid": "u1", "parentUuid": null,
"message": {"role": "user", "content": "hi"}, "cwd": cwd})
};
write(&tmp.path().join("projects/-work-app/in-root.jsonl"), &[user("/work/app")]);
write(&tmp.path().join("projects/-work-app-web/below.jsonl"), &[user("/work/app/web")]);
// `/work/app-old` escapes like a folder below `/work/app`; its transcript says otherwise.
write(&tmp.path().join("projects/-work-app-old/sibling.jsonl"), &[user("/work/app-old")]);
write(&tmp.path().join("projects/-other/elsewhere.jsonl"), &[user("/other")]);
let mut ids: Vec<String> =
claude_sessions(tmp.path(), Path::new("/work/app")).into_iter().map(|s| s.session_id).collect();
ids.sort();
assert_eq!(ids, ["below", "in-root"]);
let rollout = |name: &str, cwd: &str| {
write(
&tmp.path().join(format!("codex/sessions/2026/01/02/{name}.jsonl")),
&[json!({"payload": {"type": "session_meta", "payload": {"id": name, "cwd": cwd}}})],
)
};
rollout("mine", "/work/app/api");
rollout("theirs", "/other");
let codex: Vec<String> = codex_sessions(&tmp.path().join("codex"), Path::new("/work/app"))
.into_iter()
.map(|s| s.session_id)
.collect();
assert_eq!(codex, ["mine"]);
}
}
+5 -3
Переглянути файл
@@ -592,17 +592,19 @@ impl AgentManager {
self.tasks.read().values().filter_map(|t| t.state.lock().session_id.clone()).collect()
}
/// Sessions on this machine that no task holds yet, newest first (#312).
/// Sessions of this project on this machine that no task holds yet, newest first. Reads
/// transcripts from disk: call it off the UI and command threads.
pub fn external_sessions(&self) -> Vec<ExternalSession> {
let mut out = Vec::new();
let root = self.shared.paths.project_root.clone();
if let Some(spec) = self.shared.spec("claude")
&& let Some(dir) = native::claude::cli::config_dir(&spec.env)
{
out.extend(external::claude_sessions(&dir));
out.extend(external::claude_sessions(&dir, &root));
}
let home = self.shared.paths.home_dir.clone().or_else(home_dir);
if let Some(home) = home {
out.extend(external::codex_sessions(&home.join(".codex")));
out.extend(external::codex_sessions(&home.join(".codex"), &root));
}
let known = self.imported_session_ids();
out.retain(|s| !known.contains(&s.session_id));
+1 -1
Переглянути файл
@@ -31,7 +31,7 @@ pub(super) fn transcript(config_dir: &Path, cwd: &Path, id: &str) -> PathBuf {
}
/// The project directory name Claude Code uses for `cwd`.
pub(super) fn escape(cwd: &Path) -> String {
pub(crate) fn escape(cwd: &Path) -> String {
cwd.to_string_lossy().chars().map(|c| if c.is_ascii_alphanumeric() { c } else { '-' }).collect()
}
+41
Переглянути файл
@@ -59,6 +59,8 @@ const BUILTIN: &[(&str, &str, Driver, &[&str], &str)] = &[
("cursor", "Cursor", Driver::Acp, &["cursor-agent", "acp"], "Run `cursor-agent login` in a terminal."),
("opencode", "OpenCode", Driver::Acp, &["opencode", "acp"], "Run `opencode auth login` in a terminal."),
("qwen", "Qwen Code", Driver::Acp, &["qwen", "--experimental-acp"], "Run `qwen` in a terminal and sign in."),
("qoder", "Qoder", Driver::Acp, &["qodercli", "--acp"], "Run `qodercli login` in a terminal."),
("qodercn", "Qoder CN", Driver::Acp, &["qoderclicn", "--acp"], "Run `qoderclicn login` in a terminal."),
("kimi", "Kimi CLI", Driver::Acp, &["kimi", "acp"], "Run `kimi` in a terminal and sign in."),
("goose", "Goose", Driver::Acp, &["goose", "acp"], "Run `goose configure` in a terminal."),
("auggie", "Auggie", Driver::Acp, &["auggie", "--acp"], "Run `auggie login` in a terminal."),
@@ -409,6 +411,45 @@ mod tests {
use super::*;
use serde_json::json;
#[test]
fn qoder_global_and_cn_editions_are_both_built_in() {
let info = |id: &str| describe(&load(None).into_iter().find(|s| s.id == id).unwrap());
let (global, cn) = (info("qoder"), info("qodercn"));
assert_eq!((global.name.as_str(), global.command.as_str()), ("Qoder", "qodercli --acp"));
assert_eq!((cn.name.as_str(), cn.command.as_str()), ("Qoder CN", "qoderclicn --acp"));
assert_eq!(global.login_hint.as_deref(), Some("Run `qodercli login` in a terminal."));
}
#[test]
fn qodercn_is_available_without_a_file_and_preserves_existing_customizations() {
let dir = tempfile::tempdir().unwrap();
let qodercn = load(None).into_iter().find(|s| s.id == "qodercn").unwrap();
assert_eq!(qodercn.driver, Driver::Acp);
assert!(!is_unlisted(&qodercn));
let info = describe(&qodercn);
assert_eq!(info.name, "Qoder CN");
assert_eq!(info.command, "qoderclicn --acp");
assert_eq!(info.source, AgentSource::Builtin);
assert!(info.command_editable);
assert_eq!(info.login_hint.as_deref(), Some("Run `qoderclicn login` in a terminal."));
std::fs::write(
dir.path().join("qodercn.toml"),
"command = \"/custom/qoderclicn\"\nargs = [\"--acp\"]\n[env]\nK = \"v\"\n",
)
.unwrap();
let specs = load(Some(dir.path()));
assert_eq!(specs.iter().filter(|s| s.id == "qodercn").count(), 1);
let custom = specs.iter().find(|s| s.id == "qodercn").unwrap();
assert_eq!(custom.command, "/custom/qoderclicn");
assert_eq!(custom.env, [("K".into(), "v".into())]);
set_builtin_command(dir.path(), "qodercn", None).unwrap();
let reset = load(Some(dir.path())).into_iter().find(|s| s.id == "qodercn").unwrap();
assert_eq!(reset.command, "qoderclicn");
assert_eq!(reset.args, ["--acp"]);
assert_eq!(reset.env, custom.env);
}
#[test]
fn login_hints_cover_all_builtin_commands_and_preserve_separate_login_tools() {
for spec in builtins() {
+110 -22
Переглянути файл
@@ -79,23 +79,26 @@ fn scan(
Ok(())
}
/// `None` means an input cannot be resolved safely: build normally without reuse.
/// All filesystem work belongs on the caller's blocking thread.
pub fn launch_fingerprint(
/// One planned step as fingerprinted: resolved program, args, cwd and env.
type StepFingerprint = (PathBuf, Vec<String>, PathBuf, BTreeMap<String, String>);
/// Environment, planned steps and input paths shared by every Maven build
/// fingerprint: the filtered process environment plus the wrapper, settings
/// and JDK files the planned invocation reads.
struct Collected {
inputs: Vec<PathBuf>,
environment: BTreeMap<String, String>,
steps: Vec<StepFingerprint>,
inherited_hash: u64,
}
fn collect(
root: &Path,
model: &BuildModel,
module: &Path,
test: bool,
projects: &[&maven::MavenProject],
plan: &BuildPlan,
local_repo: Option<PathBuf>,
cancel: &CancellationToken,
) -> IdeResult<Option<LaunchFingerprint>> {
let Some(m) = &model.maven else { return Ok(None) };
let projects = maven_launch_projects(root, m, module, test);
if projects.is_empty() {
return Ok(None);
}
let Some(mut inputs) = maven::launch_inputs(root, &projects, local_repo) else { return Ok(None) };
let plan = launch_plan(root, model, module, test, BuildKind::Build)?;
) -> IdeResult<Option<Collected>> {
let Some(mut inputs) = maven::launch_inputs(root, projects, local_repo) else { return Ok(None) };
let mut environment: BTreeMap<String, String> = std::env::vars()
.map(|(key, value)| (key.to_ascii_uppercase(), value))
.filter(|(key, _)| {
@@ -165,26 +168,111 @@ pub fn launch_fingerprint(
Path::new(home).join(if cfg!(windows) { "bin/java.exe" } else { "bin/java" }),
]);
}
let steps: Vec<_> =
plan.steps.iter().map(|s| (process::program_path(&s.program, &s.env), &s.args, &s.cwd, &s.env)).collect();
let steps = plan
.steps
.iter()
.map(|s| (process::program_path(&s.program, &s.env), s.args.clone(), s.cwd.clone(), s.env.clone()))
.collect();
// Profiles and plugins can read arbitrary environment variables. Keep a signature
// of the inherited environment without persisting unrelated credentials.
let inherited: BTreeMap<_, _> = std::env::vars_os().collect();
let mut inherited_hash = std::hash::DefaultHasher::new();
inherited.hash(&mut inherited_hash);
let parameters =
serde_json::to_string(&(root, module, test, steps, environment, inherited_hash.finish(), projects))
.map_err(|e| IdeError::internal(e.to_string()))?;
Ok(Some(Collected { inputs, environment, steps, inherited_hash: inherited_hash.finish() }))
}
/// Scans every input, storing project inputs relative to the project.
fn scan_all(
root: &Path,
inputs: Vec<PathBuf>,
cancel: &CancellationToken,
) -> IdeResult<BTreeMap<PathBuf, Option<Stamp>>> {
let mut files = BTreeMap::new();
for path in inputs {
scan(root, &path, &mut files, &mut HashSet::new(), cancel)?;
}
// Store project inputs relative to the project, including each file's relative path.
let files = files.into_iter().map(|(p, stamp)| (p.strip_prefix(root).unwrap_or(&p).to_path_buf(), stamp)).collect();
Ok(Some(LaunchFingerprint { version: 2, parameters, files }))
Ok(files.into_iter().map(|(p, stamp)| (p.strip_prefix(root).unwrap_or(&p).to_path_buf(), stamp)).collect())
}
/// `None` means an input cannot be resolved safely: build normally without reuse.
/// All filesystem work belongs on the caller's blocking thread.
pub fn launch_fingerprint(
root: &Path,
model: &BuildModel,
module: &Path,
test: bool,
local_repo: Option<PathBuf>,
cancel: &CancellationToken,
) -> IdeResult<Option<LaunchFingerprint>> {
let Some(m) = &model.maven else { return Ok(None) };
let projects = maven_launch_projects(root, m, module, test);
if projects.is_empty() {
return Ok(None);
}
let plan = launch_plan(root, model, module, test, BuildKind::Build)?;
let Some(collected) = collect(root, &projects, &plan, local_repo)? else { return Ok(None) };
let Collected { inputs, environment, steps, inherited_hash } = collected;
let parameters = serde_json::to_string(&(root, module, test, steps, environment, inherited_hash, projects))
.map_err(|e| IdeError::internal(e.to_string()))?;
Ok(Some(LaunchFingerprint { version: 2, parameters, files: scan_all(root, inputs, cancel)? }))
}
/// Inputs fingerprint of a menu build (`Build Project` / `Build Module`): the
/// built modules' sources, resources and POM chain plus the exact planned
/// steps, so an unchanged project skips Maven entirely (#2476). Only `Build`
/// callers reuse it, never `Rebuild` or `Artifacts`; `None` disables reuse.
/// All filesystem work belongs on the caller's blocking thread.
pub fn menu_fingerprint(
root: &Path,
model: &BuildModel,
file: Option<&Path>,
cancel: &CancellationToken,
) -> IdeResult<Option<LaunchFingerprint>> {
let plan = plan(root, model, BuildKind::Build, file, None)?;
if plan.system != BuildSystem::Maven {
return Ok(None);
}
let Some(m) = &model.maven else { return Ok(None) };
let projects: Vec<&maven::MavenProject> = match file {
Some(file) => {
// The innermost linked project owning the file, as in single-file compilation.
let rel = rel_str(root, file);
let module = m
.projects
.iter()
.filter(|p| rel == p.dir || rel.starts_with(&format!("{}/", p.dir)) || p.dir.is_empty())
.max_by_key(|p| p.dir.len());
match module {
Some(project) if project.packaging != "pom" => {
maven_launch_projects(root, m, &subdir(root, &project.dir), true)
}
_ => return Ok(None),
}
}
// A project build walks every linked project.
None => m.projects.iter().collect(),
};
if projects.is_empty() {
return Ok(None);
}
let Some(collected) = collect(root, &projects, &plan, None)? else { return Ok(None) };
let Collected { inputs, environment, steps, inherited_hash } = collected;
let scope = file.map(|f| rel_str(root, f));
let parameters = serde_json::to_string(&("menu-build", scope, steps, environment, inherited_hash))
.map_err(|e| IdeError::internal(e.to_string()))?;
Ok(Some(LaunchFingerprint { version: 2, parameters, files: scan_all(root, inputs, cancel)? }))
}
impl LaunchFingerprint {
/// Folders (relative to the project) of the Maven projects whose POM is an input.
pub(super) fn project_dirs(&self) -> impl Iterator<Item = &Path> {
self.files
.keys()
.filter(|p| p.is_relative() && p.file_name().is_some_and(|n| n == "pom.xml"))
.map(|pom| pom.parent().unwrap_or(Path::new("")))
}
/// Relative paths whose content changed while `(length, mtime)` stayed the same
/// compared to `previous` (timestamp-preserving external edits, coarse-granularity
/// filesystems). Timestamp-incremental builders (Maven, the Kotlin plugin) would
+253
Переглянути файл
@@ -0,0 +1,253 @@
//! Resources a Gradle build changes while copying them (filters, `expand`, renames, extra
//! sources). The Java language server copies them as they are into Buildship's
//! `bin/<source set>` folders, which launches run from; IntelliJ IDEA runs Gradle's copy.
//! Before a launch, Gradle processes them and its output replaces those plain copies, so
//! each resource is on the classpath once, as the build produces it.
use super::{BuildPlan, BuildStep, BuildSystem, Format, run, subdir};
use crate::BuildModel;
use crate::gradle::{self, GradleModel, ReportedProject};
use ide_core::{CancellationToken, IdeError, IdeResult, Severity};
use std::fs;
use std::path::{Path, PathBuf};
/// A source set whose resources Gradle processes, with the folder the launch reads them from.
#[derive(Debug, PartialEq)]
struct Target {
/// Task path from the root build (`:app:processResources`).
task: String,
/// Where the task writes the resources.
output: PathBuf,
/// The Java language server's output folder of the source set, on the launch classpath.
jdtls: PathBuf,
}
fn targets(reported: &[ReportedProject], class_paths: &[String]) -> Vec<Target> {
// buildSrc is built before the build, never addressable as `:buildSrc:<task>`.
let addressable = |p: &&ReportedProject| !p.build.as_deref().is_some_and(|b| b.ends_with(":buildSrc"));
let sets: Vec<_> = reported
.iter()
.filter(addressable)
.flat_map(|p| p.source_sets.iter().filter_map(move |s| Some((p, s, s.processed.as_ref()?))))
.collect();
if sets.is_empty() {
return Vec::new();
}
// Canonical, as Gradle reports its folders: jdtls may spell them differently.
let entries: Vec<PathBuf> = class_paths.iter().filter_map(|p| dunce::canonicalize(p).ok()).collect();
sets.into_iter()
.filter_map(|(p, set, resources)| {
// Buildship's output folder of the source set.
let jdtls = dunce::canonicalize(Path::new(&p.dir).join("bin").join(&set.name)).ok()?;
let project = if p.path == ":" { "" } else { p.path.as_str() };
entries.contains(&jdtls).then(|| Target {
task: format!("{project}:{}", resources.task),
output: PathBuf::from(&resources.dir),
jdtls,
})
})
.collect()
}
/// Before a JVM launch from the Java language server's output folders on `class_paths`:
/// runs the Gradle tasks that process resources of those folders' source sets (see the
/// module docs) and copies their output over the plain copies. `cache_dir` holds the
/// linked builds' last syncs. Returns the tasks run, none for builds that copy their
/// resources as they are.
pub fn process_launch_resources(
root: &Path,
model: &BuildModel,
cache_dir: &Path,
class_paths: &[String],
cancel: &CancellationToken,
) -> IdeResult<Vec<String>> {
// A composite also reports the builds it includes: the deepest build owning a folder runs its task.
let mut builds: Vec<&GradleModel> = model.gradle.iter().collect();
builds.sort_by_key(|g| std::cmp::Reverse(subdir(root, &g.dir).components().count()));
let mut done: Vec<PathBuf> = Vec::new();
let mut tasks = Vec::new();
for g in builds {
let reported = crate::sync::reported_gradle_projects(&gradle::cache_dir(cache_dir, &g.dir));
let targets: Vec<Target> =
targets(&reported, class_paths).into_iter().filter(|t| !done.contains(&t.jdtls)).collect();
if targets.is_empty() {
continue;
}
let mut step = BuildStep::new(&g.command, &["--console=plain"], &subdir(root, &g.dir), Format::Gradle);
step.args.extend(targets.iter().map(|t| t.task.clone()));
step.env = g.env.clone();
let plan = BuildPlan { system: BuildSystem::Gradle, title: "Process resources".into(), steps: vec![step] };
let result = run(root, &plan, cancel, &|_| {})?;
if !result.success {
let errors: Vec<&str> = result
.messages
.iter()
.filter(|m| m.severity == Severity::Error)
.take(5)
.map(|m| m.text.as_str())
.collect();
return Err(IdeError::internal(errors.join("\n")));
}
for t in targets {
if cancel.is_cancelled() {
return Err(IdeError::cancelled());
}
overlay(&t.output, &t.output, &t.jdtls)?;
done.push(t.jdtls);
tasks.push(t.task);
}
}
Ok(tasks)
}
/// Copies the files under `dir` to the same place under `to` (relative to `base`) where
/// they differ, so unchanged resources are not rewritten on every launch.
fn overlay(base: &Path, dir: &Path, to: &Path) -> IdeResult<()> {
let entries = match fs::read_dir(dir) {
// No resources: the task had nothing to copy.
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()),
entries => entries?,
};
for entry in entries {
let entry = entry?;
let path = entry.path();
let kind = entry.file_type()?;
if kind.is_dir() {
overlay(base, &path, to)?;
} else if kind.is_file() {
let dest = to.join(path.strip_prefix(base).map_err(|e| IdeError::internal(e.to_string()))?);
if !same_content(&path, &dest) {
if let Some(parent) = dest.parent() {
fs::create_dir_all(parent)?;
}
fs::copy(&path, &dest)?;
}
}
}
Ok(())
}
fn same_content(a: &Path, b: &Path) -> bool {
match (fs::metadata(a), fs::metadata(b)) {
(Ok(m), Ok(n)) if m.len() == n.len() => matches!((fs::read(a), fs::read(b)), (Ok(x), Ok(y)) if x == y),
_ => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::gradle::{ProcessedResources, ReportedSourceSet};
fn write(path: &Path, text: &str) {
fs::create_dir_all(path.parent().unwrap()).unwrap();
fs::write(path, text).unwrap();
}
fn set(name: &str, processed: Option<&Path>) -> ReportedSourceSet {
ReportedSourceSet {
name: name.into(),
test: name == "test",
processed: processed.map(|dir| ProcessedResources {
task: if name == "main" { "processResources".into() } else { "processTestResources".into() },
dir: dir.to_string_lossy().into_owned(),
}),
..Default::default()
}
}
fn project(path: &str, dir: &Path, sets: Vec<ReportedSourceSet>) -> ReportedProject {
ReportedProject {
path: path.into(),
dir: dir.to_string_lossy().into_owned(),
source_sets: sets,
..Default::default()
}
}
/// Only processed source sets whose jdtls output is on the classpath: the launched
/// module, upstream modules of the build, tests only for a test classpath; never buildSrc.
#[test]
fn targets_are_processed_sets_on_the_classpath() {
let tmp = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(tmp.path()).unwrap();
let (app, lib, plain, src) = (root.join("app"), root.join("lib"), root.join("plain"), root.join("buildSrc"));
for dir in [&root, &app, &lib, &plain, &src] {
for set in ["main", "test"] {
fs::create_dir_all(dir.join("bin").join(set)).unwrap();
}
}
let out = |dir: &Path, set: &str| dir.join("build/resources").join(set);
let mut build_src = project(":buildSrc", &src, vec![set("main", Some(&out(&src, "main")))]);
build_src.build = Some(":buildSrc".into());
let reported = [
project(":", &root, vec![set("main", Some(&out(&root, "main")))]),
project(":app", &app, vec![set("main", Some(&out(&app, "main"))), set("test", Some(&out(&app, "test")))]),
project(":lib", &lib, vec![set("main", Some(&out(&lib, "main")))]),
project(":plain", &plain, vec![set("main", None)]),
build_src,
];
let entry = |dir: &Path, set: &str| dir.join("bin").join(set).to_string_lossy().into_owned();
let runtime = [
entry(&app, "main"),
// Spelled differently from Gradle's canonical folders.
format!("{}/", entry(&lib, "main")),
entry(&plain, "main"),
entry(&src, "main"),
"/m2/spring-core.jar".into(),
];
let tasks = |cp: &[String]| targets(&reported, cp).into_iter().map(|t| t.task).collect::<Vec<_>>();
assert_eq!(tasks(&runtime), [":app:processResources", ":lib:processResources"]);
let mut test = runtime.to_vec();
test.insert(0, entry(&app, "test"));
assert_eq!(tasks(&test), [":app:processResources", ":app:processTestResources", ":lib:processResources"]);
assert_eq!(tasks(&[entry(&root, "main")]), [":processResources"], "the root project");
let app_targets = targets(&reported, &runtime);
let t = &app_targets[0];
assert_eq!(
(t.output.as_path(), t.jdtls.as_path()),
(out(&app, "main").as_path(), app.join("bin/main").as_path())
);
let unprocessed = [project(":app", &app, vec![set("main", None)])];
assert!(targets(&unprocessed, &runtime).is_empty());
}
/// The processed copy replaces jdtls's plain one (the reported `@version@` case), new
/// files are added, identical ones are left alone and files Gradle doesn't produce stay.
#[test]
fn overlay_replaces_plain_copies_with_gradle_output() {
let tmp = tempfile::tempdir().unwrap();
let (output, jdtls) = (tmp.path().join("build/resources/main"), tmp.path().join("bin/main"));
write(&output.join("application.yaml"), "spring:\n application:\n version: 0.0.1\n");
write(&output.join("static/extra.txt"), "from an extra source");
write(&output.join("migrations/V1__init.sql"), "create table t (id int);");
write(&jdtls.join("application.yaml"), "spring:\n application:\n version: @version@\n");
write(&jdtls.join("migrations/V1__init.sql"), "create table t (id int);");
write(&jdtls.join("com/example/App.class"), "bytecode");
let unchanged = fs::metadata(jdtls.join("migrations/V1__init.sql")).unwrap().modified().unwrap();
std::thread::sleep(std::time::Duration::from_millis(20));
overlay(&output, &output, &jdtls).unwrap();
let read = |rel: &str| fs::read_to_string(jdtls.join(rel)).unwrap();
assert_eq!(read("application.yaml"), "spring:\n application:\n version: 0.0.1\n");
assert_eq!(read("static/extra.txt"), "from an extra source");
assert_eq!(read("com/example/App.class"), "bytecode");
let modified = fs::metadata(jdtls.join("migrations/V1__init.sql")).unwrap().modified().unwrap();
assert_eq!(modified, unchanged, "identical files are not rewritten");
overlay(&tmp.path().join("missing"), &tmp.path().join("missing"), &jdtls).unwrap();
}
#[test]
fn builds_without_processed_resources_run_nothing() {
let tmp = tempfile::tempdir().unwrap();
// Running this command would fail the call.
let missing = tmp.path().join("missing-gradle").to_string_lossy().into_owned();
let model =
BuildModel { gradle: vec![GradleModel { command: missing, ..Default::default() }], ..Default::default() };
let class_paths = [tmp.path().join("bin/main").to_string_lossy().into_owned()];
let tasks =
process_launch_resources(tmp.path(), &model, tmp.path(), &class_paths, &CancellationToken::new()).unwrap();
assert!(tasks.is_empty());
}
}
+9 -1
Переглянути файл
@@ -4,9 +4,17 @@
//! for the Build tool window, the Problems view and the editor.
mod fingerprint;
mod launch_resources;
mod parse;
mod single;
mod up_to_date;
pub use fingerprint::{LaunchFingerprint, launch_fingerprint};
pub use fingerprint::{LaunchFingerprint, launch_fingerprint, menu_fingerprint};
pub use launch_resources::process_launch_resources;
pub use single::{
SingleFileTarget, compile_file_plan, find_javac, javac_step, single_file_classpath, single_file_target,
};
pub use up_to_date::{menu_up_to_date, record_menu_build};
use crate::process::{self, Stream};
use crate::{BuildModel, maven};
+1 -26
Переглянути файл
@@ -7,6 +7,7 @@
//! [`super::resolve`] turns them into IDE coordinates.
use ide_core::Severity;
pub(crate) use ide_core::output_encoding::strip_ansi;
use serde_json::Value;
/// Output dialect of a build step.
@@ -98,32 +99,6 @@ pub(crate) struct OutputParser {
pending: Option<Pending>,
}
/// Removes ANSI escape sequences (colour, cursor movement).
pub(crate) fn strip_ansi(s: &str) -> String {
if !s.contains('\u{1b}') {
return s.to_owned();
}
let mut out = String::with_capacity(s.len());
let mut chars = s.chars().peekable();
while let Some(c) = chars.next() {
if c != '\u{1b}' {
out.push(c);
continue;
}
if chars.peek() == Some(&'[') {
chars.next();
for c in chars.by_ref() {
if c.is_ascii_alphabetic() || c == '~' {
break;
}
}
} else {
chars.next();
}
}
out
}
fn num(s: &str) -> Option<u32> {
s.trim().parse().ok().filter(|n| *n > 0)
}
+331
Переглянути файл
@@ -0,0 +1,331 @@
//! Single-file Java compilation with `javac` for the Compile File action (#2476).
//!
//! A module build (`-pl … -am`) still walks every upstream module, and packaging
//! walks the whole reactor; compiling one file only needs that file's module
//! outputs plus its dependency classpath, which is much faster to assemble.
use super::*;
use std::fs;
use std::process::Command as ProcCommand;
/// Output file of the `dependency:build-classpath` probe, relative to each module.
const CP_FILE: &str = "target/ide-compile-cp.txt";
/// A `.java` source and the module build that compiles it.
pub struct SingleFileTarget {
/// Absolute module directory.
pub module: PathBuf,
/// The source is under `src/test`: compile to test classes with the test classpath.
pub test: bool,
/// Absolute `target/classes` (or `target/test-classes`).
pub out_dir: PathBuf,
/// Reactor root building the module (owns the Maven command and options).
pub top: maven::MavenProject,
/// `-pl` value of the module relative to the reactor root; empty when the
/// module is the root itself.
pub pl: String,
/// `maven.compiler.release` (or equivalent) of the module's project.
pub java_version: Option<u32>,
}
/// Locates the linked Maven module owning a `.java` source file.
pub fn single_file_target(root: &Path, model: &BuildModel, file: &Path) -> IdeResult<SingleFileTarget> {
if file.extension().is_none_or(|ext| ext != "java") {
return Err(IdeError::invalid("only Java sources can be compiled as a single file"));
}
if !file.is_file() {
return Err(IdeError::not_found("no Java source found for single-file compilation"));
}
let Some(m) = &model.maven else {
return Err(IdeError::not_found("no linked Maven project found for module"));
};
// The innermost project whose directory contains the file owns it; the
// aggregator root (`""`) is a prefix of everything, so take the longest match.
let rel = rel_str(root, file);
let project = m
.projects
.iter()
.filter(|p| rel == p.dir || rel.starts_with(&format!("{}/", p.dir)) || p.dir.is_empty())
.max_by_key(|p| p.dir.len())
.ok_or_else(|| IdeError::not_found("no linked Maven project found for module"))?;
if project.packaging == "pom" {
// An aggregator owns no sources itself: a file that only matches it
// belongs to no linked module.
return Err(IdeError::not_found("no linked Maven project found for module"));
}
let module = subdir(root, &project.dir);
let top = maven_reactor(m, &project.dir)
.ok_or_else(|| IdeError::not_found("no linked Maven project found for module"))?;
let top_dir = subdir(root, &top.dir);
let pl = module.strip_prefix(&top_dir).map(|r| r.to_string_lossy().replace('\\', "/")).unwrap_or_default();
// Test sources see the test classpath and compile into test classes.
let parts: Vec<&str> = rel.split('/').collect();
let test = parts.windows(2).any(|w| w == ["src", "test"]);
let out_dir = module.join("target").join(if test { "test-classes" } else { "classes" });
Ok(SingleFileTarget { module, test, out_dir, top: top.clone(), pl, java_version: project.java_version })
}
/// `javac` for the module: its `JAVA_HOME`, else `PATH`.
pub fn find_javac(env: &BTreeMap<String, String>) -> String {
let home = env.get("JAVA_HOME").map(Path::new);
let exe = if cfg!(windows) { "javac.exe" } else { "javac" };
if let Some(javac) = home.map(|h| h.join("bin").join(exe)).filter(|p| p.is_file()) {
return javac.to_string_lossy().into_owned();
}
exe.to_owned()
}
/// Dependency classpath of the module: existing reactor class directories plus
/// `mvn dependency:build-classpath` (which resolves without compiling anything,
/// so `-am` stays cheap). Entries are platform-separated (`;` on Windows).
pub fn single_file_classpath(root: &Path, model: &BuildModel, target: &SingleFileTarget) -> IdeResult<String> {
let mut entries: Vec<String> = Vec::new();
// The module's own classes first: the file almost always uses classes next to it.
let own: &[&str] = if target.test { &["target/test-classes", "target/classes"] } else { &["target/classes"] };
for out in own {
let path = target.module.join(out);
if path.is_dir() {
entries.push(path.to_string_lossy().into_owned());
}
}
if let Some(m) = &model.maven {
// Upstream reactor modules first: their freshly built classes shadow
// installed snapshots of the same artifacts.
for project in maven_launch_projects(root, m, &target.module, target.test) {
let dir = subdir(root, &project.dir);
if dir == target.module {
continue;
}
for classes in ["target/classes", "target/test-classes"] {
let path = dir.join(classes);
if path.is_dir() {
entries.push(path.to_string_lossy().into_owned());
}
}
}
}
let probe = CP_FILE;
let mut args = vec!["-B", "-Dstyle.color=never", maven::LOCAL_REPO_TRACKING_ARG];
args.extend(target.top.options.iter().map(String::as_str));
if !target.pl.is_empty() {
args.extend(["-pl", target.pl.as_str(), "-am"]);
}
let output_arg = format!("-Dmdep.outputFile={probe}");
args.extend(["dependency:build-classpath", output_arg.as_str(), "-Dmdep.includeScope=test", "-q"]);
let top_dir = subdir(root, &target.top.dir);
let status = ProcCommand::new(&target.top.command)
.args(&args)
.envs(&target.top.env)
.current_dir(&top_dir)
.output()
.map_err(|e| IdeError::internal(format!("cannot resolve the compile classpath: {e}")))?;
if !status.status.success() {
let stderr = String::from_utf8_lossy(&status.stderr);
let tail: Vec<&str> = stderr.lines().filter(|l| !l.trim().is_empty()).collect();
let tail = tail.iter().rev().take(5).rev().cloned().collect::<Vec<_>>().join("\n");
return Err(IdeError::internal(format!("cannot resolve the compile classpath:\n{tail}")));
}
// With `-am` every visited module writes its own file; only this module's counts.
let cp_file = target.module.join(probe);
let content = fs::read_to_string(&cp_file).map_err(|e| {
IdeError::internal(format!("cannot resolve the compile classpath: {} ({e})", cp_file.display()))
})?;
let separator = if cfg!(windows) { ';' } else { ':' };
entries.extend(content.split([separator, '\n', '\r']).map(str::trim).filter(|e| !e.is_empty()).map(str::to_owned));
entries.dedup();
Ok(entries.join(&separator.to_string()))
}
/// The `javac` step compiling `file` into `target`'s output directory.
pub fn javac_step(target: &SingleFileTarget, file: &Path, classpath: &str, javac: &str) -> IdeResult<BuildStep> {
fs::create_dir_all(&target.out_dir)
.map_err(|e| IdeError::internal(format!("cannot create {} ({e})", target.out_dir.display())))?;
let mut step = BuildStep::new(javac, &[], &target.module, Format::Gradle);
// Raw `javac` prints `path:line: error: …`, the same form Gradle forwards.
step.args.push("-encoding".into());
step.args.push("UTF-8".into());
step.args.push("-d".into());
step.args.push(target.out_dir.to_string_lossy().into_owned());
step.args.push("-cp".into());
step.args.push(classpath.to_owned());
// Classes of the module that were never built (or are older than their source) are read
// from its sources; only `file` is written, so a test compile never puts main classes
// into test-classes.
let sources = source_roots(target);
if !sources.is_empty() {
let separator = if cfg!(windows) { ";" } else { ":" };
step.args.push("-sourcepath".into());
step.args.push(sources.iter().map(|p| p.to_string_lossy()).collect::<Vec<_>>().join(separator));
step.args.push("-implicit:none".into());
}
if let Some(version) = target.java_version {
step.args.push("--release".into());
step.args.push(version.to_string());
}
step.args.push(file.to_string_lossy().into_owned());
step.env = target.top.env.clone();
Ok(step)
}
/// The module's Java source folders that exist, generated ones included; test sources see
/// the main ones too.
fn source_roots(target: &SingleFileTarget) -> Vec<PathBuf> {
let scopes: &[(&str, &str)] = if target.test {
&[("src/test/java", "target/generated-test-sources"), ("src/main/java", "target/generated-sources")]
} else {
&[("src/main/java", "target/generated-sources")]
};
let mut roots = Vec::new();
for (src, generated) in scopes {
roots.push(target.module.join(src));
let generated = target.module.join(generated);
let mut dirs: Vec<PathBuf> =
fs::read_dir(&generated).into_iter().flatten().flatten().map(|d| d.path()).filter(|p| p.is_dir()).collect();
dirs.sort();
roots.extend(dirs);
}
roots.retain(|p| p.is_dir());
roots
}
/// Resolves the classpath and plans the `javac` invocation for one Java source.
pub fn compile_file_plan(root: &Path, model: &BuildModel, file: &Path) -> IdeResult<BuildPlan> {
let target = single_file_target(root, model, file)?;
let classpath = single_file_classpath(root, model, &target)?;
let javac = find_javac(&target.top.env);
let name = file.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default();
Ok(BuildPlan {
system: BuildSystem::Maven,
title: format!("Compile {name}"),
steps: vec![javac_step(&target, file, &classpath, &javac)?],
})
}
#[cfg(test)]
mod tests {
use super::*;
fn write(root: &Path, rel: &str, value: &str) {
let path = root.join(rel);
fs::create_dir_all(path.parent().unwrap()).unwrap();
fs::write(path, value).unwrap();
}
fn fixture(root: &Path) -> BuildModel {
write(
root,
"pom.xml",
r#"<project><groupId>demo</groupId><artifactId>parent</artifactId><version>1</version>
<packaging>pom</packaging><modules><module>lib</module><module>app</module></modules></project>"#,
);
write(
root,
"lib/pom.xml",
r#"<project><parent><groupId>demo</groupId><artifactId>parent</artifactId><version>1</version></parent>
<artifactId>lib</artifactId></project>"#,
);
write(
root,
"app/pom.xml",
r#"<project><parent><groupId>demo</groupId><artifactId>parent</artifactId><version>1</version></parent>
<artifactId>app</artifactId><dependencies><dependency><groupId>demo</groupId><artifactId>lib</artifactId><version>1</version></dependency></dependencies></project>"#,
);
write(root, "app/src/main/java/demo/Main.java", "class Main {}");
write(root, "app/src/test/java/demo/MainTest.java", "class MainTest {}");
write(root, "lib/target/classes/Lib.class", "built");
crate::load(root, &root.join("cache"), None)
}
#[test]
fn target_finds_the_owning_module_and_the_test_scope() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let main = single_file_target(root, &model, &root.join("app/src/main/java/demo/Main.java")).unwrap();
assert_eq!(main.module, root.join("app"));
assert!(!main.test);
assert_eq!(main.out_dir, root.join("app/target/classes"));
let test = single_file_target(root, &model, &root.join("app/src/test/java/demo/MainTest.java")).unwrap();
assert!(test.test);
assert_eq!(test.out_dir, root.join("app/target/test-classes"));
}
#[test]
fn target_rejects_non_java_missing_and_unlinked_files() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
write(root, "app/pom.xml", "<project/>");
assert!(single_file_target(root, &model, &root.join("app/pom.xml")).is_err());
assert!(single_file_target(root, &model, &root.join("app/src/main/java/demo/Missing.java")).is_err());
write(root, "other/Main.java", "class Main {}");
assert!(single_file_target(root, &model, &root.join("other/Main.java")).is_err());
}
#[test]
fn javac_prefers_the_module_jdk_and_scopes_test_sources() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let file = root.join("app/src/test/java/demo/MainTest.java");
let target = single_file_target(root, &model, &file).unwrap();
let jdk = root.join("jdk");
fs::create_dir_all(jdk.join("bin")).unwrap();
let exe = if cfg!(windows) { "javac.exe" } else { "javac" };
fs::write(jdk.join("bin").join(exe), "fake").unwrap();
let mut target = target;
target.top.env.insert("JAVA_HOME".into(), jdk.to_string_lossy().into_owned());
let step = javac_step(&target, &file, "/repo/a.jar", &find_javac(&target.top.env)).unwrap();
assert!(step.program.ends_with(exe));
assert!(target.out_dir.is_dir());
let args = step.args.join(" ");
assert!(args.contains("-encoding UTF-8"), "{args}");
assert!(args.contains("-d ") && args.contains("test-classes"), "{args}");
assert!(args.contains("-cp /repo/a.jar"), "{args}");
assert!(args.contains("MainTest.java"), "{args}");
// Test sources resolve their own and the main sources, without writing them.
let sourcepath = step.args.iter().position(|a| a == "-sourcepath").map(|i| step.args[i + 1].clone());
let roots = sourcepath.expect("a -sourcepath");
assert!(roots.contains("src/test/java") && roots.contains("src/main/java"), "{roots}");
assert!(step.args.iter().any(|a| a == "-implicit:none"), "{args}");
// No `JAVA_HOME`: plain `javac` from `PATH`.
assert_eq!(find_javac(&BTreeMap::new()), exe);
}
/// The classpath probe runs the project's Maven command with `-pl … -am`
/// `dependency:build-classpath`; the fake `mvnw` below only records its
/// arguments and writes a canned classpath file for the `-pl` module.
#[cfg(unix)]
#[test]
fn classpath_probe_writes_one_file_per_module_and_reads_the_compiled_one() {
use std::os::unix::fs::PermissionsExt;
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
fixture(root);
let script = root.join("mvnw");
fs::write(
&script,
"#!/bin/sh\npl='.'\nwhile [ $# -gt 0 ]; do\nif [ \"$1\" = \"-pl\" ]; then pl=\"$2\"; shift 2; else shift; fi\ndone\nmkdir -p \"$pl/target\"\nprintf '/repo/a.jar:/repo/b.jar' > \"$pl/target/ide-compile-cp.txt\"\n",
)
.unwrap();
fs::set_permissions(&script, fs::Permissions::from_mode(0o755)).unwrap();
let model = crate::load(root, &root.join("cache"), None);
let file = root.join("app/src/main/java/demo/Main.java");
let target = single_file_target(root, &model, &file).unwrap();
assert!(target.top.command.ends_with("mvnw"));
assert_eq!(target.pl, "app");
let cp = single_file_classpath(root, &model, &target).unwrap();
let separator = ':';
// Reactor class directories first, so fresh outputs shadow installed snapshots.
let lib = root.join("lib/target/classes").to_string_lossy().into_owned();
assert!(cp.starts_with(&lib), "{cp}");
// The module's own classes come first once they exist.
fs::create_dir_all(root.join("app/target/classes")).unwrap();
let cp = single_file_classpath(root, &model, &target).unwrap();
let own = root.join("app/target/classes").to_string_lossy().into_owned();
assert!(cp.starts_with(&format!("{own}:{lib}")), "{cp}");
assert!(cp.contains("/repo/a.jar"), "{cp}");
assert!(cp.contains("/repo/b.jar"), "{cp}");
assert!(cp.split(separator).count() >= 3, "{cp}");
}
}
+253
Переглянути файл
@@ -0,0 +1,253 @@
//! Up-to-date checks for menu builds (`Build Project` / `Build Module`, #2476).
//!
//! Menu builds have no fingerprint record on the launch path, so a second
//! click always pays a full offline Maven run. This module stores one JSON
//! record per scope (project, or one module file's build) after a successful
//! build and compares it before the next build; on a match the build is
//! skipped entirely. Like launch reuse, timestamp-preserving external edits
//! are re-stamped first so Maven actually recompiles them on the next run.
use super::*;
use std::fs;
/// Record of the last successful menu build of one scope.
#[derive(serde::Serialize, serde::Deserialize)]
struct Record {
fingerprint: LaunchFingerprint,
/// Output folders of the built projects that existed after the build, relative to the
/// project. Deleting one outside the IDE (`mvn clean` in a terminal) leaves every input
/// unchanged, so a build is skipped only while they are all still there.
#[serde(default)]
outputs: Vec<PathBuf>,
}
/// The build's output folders that exist now, relative to `root`.
fn existing_outputs(root: &Path, fingerprint: &LaunchFingerprint) -> Vec<PathBuf> {
fingerprint
.project_dirs()
.flat_map(|dir| ["target/classes", "target/test-classes"].map(|out| dir.join(out)))
.filter(|rel| root.join(rel).is_dir())
.collect()
}
fn record_path(cache: &Path, root: &Path, file: Option<&Path>) -> PathBuf {
let base = cache.join("menu-build").join("records");
match file {
Some(f) => base.join(f.strip_prefix(root).unwrap_or(f)).with_extension("json"),
None => base.join("project.json"),
}
}
/// Re-stamps files whose content changed without advancing mtime (git
/// checkout, external sync): timestamp-incremental Maven would otherwise skip
/// them as "unchanged" and the rebuild would compile nothing.
fn restamp(root: &Path, edits: &[PathBuf]) {
let now = std::time::SystemTime::now();
for rel in edits {
let path = root.join(rel);
let _ = fs::File::options().write(true).open(&path).and_then(|f| f.set_modified(now)).or_else(|_| {
fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&path)
.and_then(|f| f.set_modified(now))
});
}
}
/// Whether the last successful menu build of this scope is still fresh: its
/// record exists and the fingerprint matches. On a mismatch caused by
/// timestamp-preserving external edits the files are re-stamped first, so the
/// rebuild Maven runs actually recompiles them.
pub fn menu_up_to_date(
cache: &Path,
root: &Path,
model: &BuildModel,
file: Option<&Path>,
cancel: &CancellationToken,
) -> bool {
let current = match menu_fingerprint(root, model, file, cancel) {
Ok(Some(current)) => current,
_ => return false,
};
let bytes = match fs::read(record_path(cache, root, file)) {
Ok(bytes) => bytes,
_ => return false,
};
let record: Record = match serde_json::from_slice(&bytes) {
Ok(record) => record,
_ => return false,
};
if record.fingerprint != current {
restamp(root, &record.fingerprint.mtime_preserved_edits(&current));
return false;
}
record.outputs.iter().all(|out| root.join(out).is_dir())
}
/// Records a successful menu build as the baseline for future up-to-date
/// checks. Failures are swallowed: a missing record only costs one Maven run.
pub fn record_menu_build(
cache: &Path,
root: &Path,
model: &BuildModel,
file: Option<&Path>,
cancel: &CancellationToken,
) {
let current = match menu_fingerprint(root, model, file, cancel) {
Ok(Some(current)) => current,
_ => return,
};
// A torn write fails to parse later and is treated as "no record", so a
// plain write is enough.
let path = record_path(cache, root, file);
if path.parent().is_some_and(|p| fs::create_dir_all(p).is_err()) {
return;
}
let outputs = existing_outputs(root, &current);
let bytes = match serde_json::to_vec(&Record { fingerprint: current, outputs }) {
Ok(bytes) => bytes,
_ => return,
};
let _ = fs::write(&path, bytes);
}
#[cfg(test)]
mod tests {
use super::*;
fn write(root: &Path, rel: &str, value: &str) {
let path = root.join(rel);
fs::create_dir_all(path.parent().unwrap()).unwrap();
fs::write(path, value).unwrap();
}
fn fixture(root: &Path) -> BuildModel {
maven_two_module(root, "int x;");
BuildModel { maven: Some(maven::load(root)), ..Default::default() }
}
fn maven_two_module(root: &Path, extra_app: &str) {
write(
root,
"pom.xml",
r#"<project><modelVersion>4.0.0</modelVersion><groupId>t</groupId><artifactId>root</artifactId><version>1</version><packaging>pom</packaging><modules><module>app</module><module>lib</module></modules></project>"#,
);
write(
root,
"app/pom.xml",
r#"<project><modelVersion>4.0.0</modelVersion><parent><groupId>t</groupId><artifactId>root</artifactId><version>1</version></parent><artifactId>app</artifactId><dependencies><dependency><groupId>t</groupId><artifactId>lib</artifactId><version>1</version></dependency></dependencies></project>"#,
);
write(
root,
"lib/pom.xml",
r#"<project><modelVersion>4.0.0</modelVersion><parent><groupId>t</groupId><artifactId>root</artifactId><version>1</version></parent><artifactId>lib</artifactId></project>"#,
);
write(root, "app/src/main/java/t/A.java", &format!("package t; public class A {{ {extra_app} }}"));
write(root, "lib/src/main/java/t/L.java", "package t; public class L {}");
}
#[test]
fn record_then_skip_without_changes() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let file = root.join("app/src/main/java/t/A.java");
let cancel = CancellationToken::new();
assert!(!menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
record_menu_build(&cache, root, &model, Some(&file), &cancel);
assert!(menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
// The project scope is independent of the module scope.
assert!(!menu_up_to_date(&cache, root, &model, None, &cancel));
record_menu_build(&cache, root, &model, None, &cancel);
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
}
#[test]
fn deleted_outputs_invalidate_the_record() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let cancel = CancellationToken::new();
for out in ["app/target/classes/t", "lib/target/classes/t", "app/target/test-classes"] {
fs::create_dir_all(root.join(out)).unwrap();
}
record_menu_build(&cache, root, &model, None, &cancel);
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
// `mvn clean` outside the IDE: no input changed, but the classes are gone.
fs::remove_dir_all(root.join("lib/target")).unwrap();
assert!(!menu_up_to_date(&cache, root, &model, None, &cancel));
fs::create_dir_all(root.join("lib/target/classes")).unwrap();
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
}
#[test]
fn edit_invalidates_and_rerecord_restores_skip() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let file = root.join("app/src/main/java/t/A.java");
let cancel = CancellationToken::new();
record_menu_build(&cache, root, &model, Some(&file), &cancel);
assert!(menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
write(root, "app/src/main/java/t/A.java", "package t; public class A { int y; }");
assert!(!menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
record_menu_build(&cache, root, &model, Some(&file), &cancel);
assert!(menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
}
#[test]
fn mtime_preserved_edit_forces_rebuild_and_restamps() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let rel = "app/src/main/java/t/A.java";
let file = root.join(rel);
let cancel = CancellationToken::new();
record_menu_build(&cache, root, &model, Some(&file), &cancel);
let before = fs::metadata(&file).unwrap().modified().unwrap();
// Same length, same mtime, different content: an external edit that
// preserved the timestamp.
write(root, rel, "package t; public class A { int z; }");
fs::File::options().write(true).open(&file).and_then(|f| f.set_modified(before)).unwrap();
assert!(!menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
assert!(fs::metadata(&file).unwrap().modified().unwrap() > before);
}
#[test]
fn corrupt_record_never_skips() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let file = root.join("app/src/main/java/t/A.java");
let cancel = CancellationToken::new();
fs::create_dir_all(cache.join("menu-build").join("records").join("app/src/main/java/t")).unwrap();
fs::write(cache.join("menu-build").join("records").join("app/src/main/java/t/A.json"), b"{not json").unwrap();
assert!(!menu_up_to_date(&cache, root, &model, Some(&file), &cancel));
}
#[test]
fn unsupported_builds_never_skip() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
write(root, "other/Main.java", "public class Main {}");
let cache = root.join("cache");
let cancel = CancellationToken::new();
// A file outside any non-aggregator module has no fingerprint.
let stray = root.join("other/Main.java");
record_menu_build(&cache, root, &model, Some(&stray), &cancel);
assert!(!menu_up_to_date(&cache, root, &model, Some(&stray), &cancel));
}
}
+22 -1
Переглянути файл
@@ -8,6 +8,24 @@ def canon = { File f -> f.canonicalPath }
// the identity path (`:lib:core`, `:buildSrc`) tells them apart and addresses their tasks
// from the root build. Internal API, hence the fallback.
def identity = { Project p -> try { p.identityPath.toString() } catch (ignored) { p.path } }
// The task copying a source set's resources and its output folder, when it changes them on the
// way (filters, `expand`, renames, includes / excludes, extra sources): jdtls copies resources as
// they are, so launches take them from Gradle. Internal API, hence the fallback.
def processedResources = { Project p, s ->
try {
def task = p.tasks.findByName(s.processResourcesTaskName)
if (task == null) return null
def specs = 0
def changes = task.mainSpec.sourcePaths.size() > 1
task.rootSpec.buildRootResolver().walk { r ->
specs++
changes |= !r.allCopyActions.isEmpty() || !r.allIncludes.isEmpty() || !r.allExcludes.isEmpty() ||
!r.allIncludeSpecs.isEmpty() || !r.allExcludeSpecs.isEmpty() || r.destPath.segments.length > 0
}
// The root and main specs exist for every task; more come from `from(…) { … }`.
(changes || specs > 2) ? [task: task.name, dir: canon(task.destinationDir)] : null
} catch (ignored) { null }
}
def describe = { Project p ->
def m = [path: identity(p), dir: canon(p.projectDir), buildDir: canon(p.layout.buildDirectory.get().asFile)]
@@ -26,7 +44,10 @@ def describe = { Project p ->
def test = s.name == 'test' || s.name == 'testFixtures' || suites.contains(s.name) ||
s.name.endsWith('Test') || s.name.endsWith('Tests') ||
(!src.isEmpty() && src.every { ideaTests.contains(it) })
[name: s.name, test: test, src: src, res: res]
def set = [name: s.name, test: test, src: src, res: res]
def processed = processedResources(p, s)
if (processed != null) set.processed = processed
set
}
}
try {
+13
Переглянути файл
@@ -335,6 +335,19 @@ pub struct ReportedSourceSet {
/// Absolute resource folders.
#[serde(default)]
pub res: Vec<String>,
/// Set when Gradle changes the resources while copying them (filters, `expand`, renames,
/// includes / excludes, extra sources), which the Java language server's copy does not.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub processed: Option<ProcessedResources>,
}
/// The task that copies a source set's resources, and where it writes them.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProcessedResources {
/// Task name in its project (`processResources`).
pub task: String,
/// Absolute output folder.
pub dir: String,
}
const MODEL_LINE: &str = "@openidea-model\t";
+25 -3
Переглянути файл
@@ -73,6 +73,9 @@ struct GradleCache {
/// The projects as Gradle evaluated them; empty in caches from before they were read.
#[serde(default)]
projects: Vec<ReportedProject>,
/// The projects report how their resources are processed; older caches are stale.
#[serde(default)]
resources: bool,
}
/// Init script printing every project, its source sets and Java version once configured.
@@ -244,11 +247,12 @@ pub fn apply_gradle_cache(root: &Path, cache_dir: &Path, model: &mut GradleModel
}
}
}
// A successful sync from before projects were cached has not seen computed includes.
let without_projects = cache.error.is_none() && cache.projects.is_empty();
// A successful sync from before projects were cached has not seen computed includes,
// nor, before resources were read, which launches need Gradle to process them.
let outdated = cache.error.is_none() && (cache.projects.is_empty() || !cache.resources);
model.synced_at = Some(cache.at_ms);
model.sync_error = cache.error;
model.stale = without_projects || cache.hash != hash_gradle_files(&gradle::build_files(root, model));
model.stale = outdated || cache.hash != hash_gradle_files(&gradle::build_files(root, model));
}
// ---------------------------------------------------------------- processes
@@ -462,6 +466,7 @@ pub fn sync_gradle_model(
error: (!out.success).then(|| error_summary(&out.text)),
tasks: if out.success { gradle::parse_tasks(&out.text) } else { Vec::new() },
projects,
resources: true,
};
write_cache(&cache_dir, GRADLE_CACHE, &cache)
}
@@ -713,6 +718,7 @@ mod tests {
test,
src: src.iter().map(|p| core_src(p)).collect(),
res: res.iter().map(|p| core_src(p)).collect(),
processed: None,
};
// A settings script's own println comes first; task listing lines follow.
let out = format!(
@@ -737,6 +743,15 @@ mod tests {
);
let projects = parse_reported_projects(&out);
assert_eq!(projects.len(), 3, "the malformed line is skipped");
// As the init script prints a set whose resources Gradle filters or renames.
let processed = parse_reported_projects(&format!(
"@openidea-model\t{}",
r#"{"path":":","dir":"/p","sourceSets":[{"name":"main","test":false,"src":[],"res":[],"processed":{"task":"processResources","dir":"/p/build/resources/main"}}]}"#
));
assert_eq!(
processed[0].source_sets[0].processed,
Some(gradle::ProcessedResources { task: "processResources".into(), dir: "/p/build/resources/main".into() })
);
let src = |p: &str| PathBuf::from(core_src(p));
assert_eq!(
gradle::reported_source_roots(&projects),
@@ -802,12 +817,18 @@ mod tests {
..Default::default()
})
.collect(),
resources: true,
};
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
apply_gradle_cache(&root, &cache_dir, &mut model);
assert_eq!(model.projects[1].tasks.len(), 1);
assert!(model.projects[0].tasks.iter().any(|t| t.name == "wrapper"), "root keeps its defaults");
assert!(!model.stale);
write_cache(&cache_dir, GRADLE_CACHE, &GradleCache { resources: false, ..cache.clone() }).unwrap();
let mut m = gradle::load(&root).unwrap();
apply_gradle_cache(&root, &cache_dir, &mut m);
assert!(m.stale, "synced before resource processing was read");
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
let stale_after = |script: &str| {
std::fs::write(root.join("app/build.gradle"), script).unwrap();
@@ -841,6 +862,7 @@ mod tests {
dir: root.to_string_lossy().into_owned(),
..Default::default()
}],
resources: true,
..Default::default()
};
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
+37
Переглянути файл
@@ -273,6 +273,27 @@ pub enum DebugEvent {
change_type: String,
message: String,
},
/// The session's capabilities changed after it started: fetch them again.
Capabilities,
JvmBackend {
state: JvmBackendState,
},
}
/// The backend that reads a debugged JVM's objects (instances, fields, referrers).
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmBackendState {
Ready,
/// Turned off for the rest of the session: the debug adapter lacks what it needs.
Unsupported {
detail: String,
},
/// The Kotlin JDWP link only forwards now; inspecting stops.
Degraded {
detail: String,
},
Closed,
}
/// Where backend crates publish [`IdeEvent`]s. Cheap to clone via `Arc`.
@@ -294,3 +315,19 @@ pub struct NullSink;
impl EventSink for NullSink {
fn emit(&self, _: IdeEvent) {}
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn jvm_backend_events_json_shape() {
let degraded = DebugEvent::JvmBackend { state: JvmBackendState::Degraded { detail: "x".into() } };
assert_eq!(
serde_json::to_value(&degraded).unwrap(),
json!({"kind": "jvmBackend", "state": {"kind": "degraded", "detail": "x"}})
);
assert_eq!(serde_json::to_value(DebugEvent::Capabilities).unwrap(), json!({"kind": "capabilities"}));
}
}
+26
Переглянути файл
@@ -21,6 +21,32 @@
use std::collections::BTreeMap;
/// Removes ANSI escape sequences (colour, cursor movement).
pub fn strip_ansi(s: &str) -> String {
if !s.contains('\u{1b}') {
return s.to_owned();
}
let mut out = String::with_capacity(s.len());
let mut chars = s.chars().peekable();
while let Some(c) = chars.next() {
if c != '\u{1b}' {
out.push(c);
continue;
}
if chars.peek() == Some(&'[') {
chars.next();
for c in chars.by_ref() {
if c.is_ascii_alphabetic() || c == '~' {
break;
}
}
} else {
chars.next();
}
}
out
}
/// Decodes complete lines or collected output from IDE-managed tools, preferring UTF-8.
/// Windows launcher/native diagnostics fall back to the system ANSI code page.
/// Callers must buffer incomplete lines so multibyte characters are never split.
+112
Переглянути файл
@@ -72,6 +72,75 @@ pub fn kill_tree_of(pid: u32) {
let _ = pid;
}
/// Whether a process with this id exists. PID reuse makes this conservative: a reused id keeps
/// whatever was tied to the old process alive, a live process is never reported dead.
pub fn process_alive(pid: u32) -> bool {
// `kill(0, 0)` would signal the caller's own process group.
if pid == 0 {
return false;
}
#[cfg(unix)]
{
// Above `i32::MAX` the id would wrap to a negative value, which `kill` reads as a group.
let Ok(pid) = libc::pid_t::try_from(pid) else { return false };
// SAFETY: signal 0 delivers nothing; it only checks that the process exists and is reachable.
unsafe { libc::kill(pid, 0) == 0 || std::io::Error::last_os_error().raw_os_error() == Some(libc::EPERM) }
}
#[cfg(windows)]
{
use std::os::windows::io::{AsRawHandle, FromRawHandle, OwnedHandle};
use windows_sys::Win32::Foundation::{ERROR_ACCESS_DENIED, GetLastError, STILL_ACTIVE};
use windows_sys::Win32::System::Threading::{
GetExitCodeProcess, OpenProcess, PROCESS_QUERY_LIMITED_INFORMATION,
};
// SAFETY: plain call; a non-null result is a process handle owned by this function.
let handle = unsafe { OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, 0, pid) };
if handle.is_null() {
// A process we may not open still exists.
// SAFETY: reads the thread's last error, set by the call above.
return unsafe { GetLastError() } == ERROR_ACCESS_DENIED;
}
// SAFETY: `handle` is valid and nobody else owns it.
let handle = unsafe { OwnedHandle::from_raw_handle(handle) };
let mut code = 0u32;
// SAFETY: a live process handle opened with query access; one out-parameter.
let ok = unsafe { GetExitCodeProcess(handle.as_raw_handle(), &mut code) };
// A handle that cannot be queried is not evidence that the process is gone.
ok == 0 || code == STILL_ACTIVE as u32
}
#[cfg(not(any(unix, windows)))]
{
true
}
}
/// The program and everything it started: an adapter may report the launcher shell, whose
/// child is the debugged program, so stopping only the reported pid leaves the program alive.
#[cfg(unix)]
pub fn process_tree(pid: i32) -> Vec<i32> {
use std::collections::HashMap;
let Ok(out) = std::process::Command::new("ps").args(["-A", "-o", "pid=,ppid="]).output() else {
return vec![pid];
};
let mut children: HashMap<i32, Vec<i32>> = HashMap::new();
for line in String::from_utf8_lossy(&out.stdout).lines() {
let mut it = line.split_whitespace();
let (Some(p), Some(parent)) = (it.next(), it.next()) else { continue };
let (Ok(p), Ok(parent)) = (p.parse::<i32>(), parent.parse::<i32>()) else { continue };
children.entry(parent).or_default().push(p);
}
let mut tree = vec![pid];
let mut at = 0;
while at < tree.len() {
if let Some(kids) = children.get(&tree[at]) {
tree.extend(kids.iter().copied());
}
at += 1;
}
tree
}
/// [`kill_tree`] that first asks the group to exit: `SIGTERM`, then `grace`, then `SIGKILL`.
/// Git removes its `index.lock` on `SIGTERM`, where `SIGKILL` leaves it behind and the next
/// git command fails. Windows has no such signal to send from outside the child's console,
@@ -730,6 +799,30 @@ fn append_path_entries(path: &OsStr, extra: &OsStr) -> OsString {
mod tests {
use super::*;
#[test]
fn process_alive_tells_live_from_dead_pids() {
assert!(process_alive(std::process::id()));
assert!(!process_alive(0));
let mut cmd = if cfg!(windows) {
let mut cmd = Command::new("cmd");
cmd.args(["/C", "exit 0"]);
cmd
} else {
Command::new("true")
};
let mut child = cmd.spawn().unwrap();
let pid = child.id();
child.wait().unwrap();
drop(child);
assert!(!process_alive(pid));
}
#[cfg(unix)]
#[test]
fn process_alive_rejects_ids_that_are_not_a_single_process() {
assert!(!process_alive(u32::MAX));
}
#[cfg(unix)]
#[test]
fn kill_tree_of_reaches_what_the_child_started() {
@@ -1050,6 +1143,25 @@ mod tests {
}
assert_eq!(find_executable("nope", &search), None);
}
#[cfg(unix)]
#[test]
fn process_tree_lists_the_child_of_a_shell() {
let mut cmd = Command::new("sh");
own_group(&mut cmd).args(["-c", "sleep 5 & wait"]);
let mut child = cmd.spawn().unwrap();
let sh_pid = child.id() as i32;
// The shell forks `sleep` after it starts, so the tree grows a moment later.
let deadline = std::time::Instant::now() + Duration::from_secs(2);
let mut tree = process_tree(sh_pid);
while tree.len() < 2 && std::time::Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(20));
tree = process_tree(sh_pid);
}
kill_tree(&mut child);
assert_eq!(tree[0], sh_pid, "the pid itself comes first");
assert!(tree.len() >= 2 && tree[1..].iter().all(|p| *p != sh_pid), "the shell's child follows: {tree:?}");
}
}
#[cfg(all(test, unix))]
+7 -1
Переглянути файл
@@ -453,6 +453,10 @@ pub struct GeneralSettings {
/// Where updates come from: `auto` (the CNB mirror for the Chinese UI, else GitHub) | `github` | `cnb`.
pub update_source: String,
pub auto_check_updates: bool,
/// System Settings › Updates: open the What's New page after the app starts a new version.
pub show_whats_new_after_update: bool,
/// The version the app last started with, so the first start after an update is recognisable.
pub last_run_version: String,
}
impl Default for GeneralSettings {
@@ -471,6 +475,8 @@ impl Default for GeneralSettings {
project_name: String::new(),
update_source: "auto".into(),
auto_check_updates: true,
show_whats_new_after_update: true,
last_run_version: String::new(),
}
}
}
@@ -574,7 +580,7 @@ pub struct AppearanceSettings {
pub project_colors_in_toolbar: bool,
pub dnd_with_alt_only: bool,
pub menu_icons: bool,
/// `hamburger` | `merged`: main menu presentation in the frameless title bar.
/// `hamburger` | `merged` | `toolbar` (a separate row): main menu presentation in the frameless title bar.
pub main_menu_style: String,
/// A lone Alt press focuses or opens the main menu (IDEA `ide.windows.main.menu.focus.on.alt`).
pub main_menu_alt: bool,
+5
Переглянути файл
@@ -11,12 +11,17 @@ tempfile.workspace = true
ide-core.workspace = true
ide-language.workspace = true
ide-jsonrpc.workspace = true
ide-jvm-attach.workspace = true
ide-jvm-inspect.workspace = true
ide-jdwp.workspace = true
ide-heap.workspace = true
serde.workspace = true
serde_json.workspace = true
specta.workspace = true
tokio = { workspace = true, features = ["net"] }
parking_lot.workspace = true
memchr = "2"
percent-encoding = "2"
portable-pty.workspace = true
tracing.workspace = true
+100 -13
Переглянути файл
@@ -12,6 +12,35 @@ use tokio::net::{TcpListener, TcpStream};
/// How long the relay waits for java-debug to attach once the JVM is connected.
const ATTACH_TIMEOUT: Duration = Duration::from_secs(60);
/// Loopback ports relaying a connected JVM, while their relay runs.
static RELAY_PORTS: std::sync::Mutex<Vec<u16>> = std::sync::Mutex::new(Vec::new());
fn relay_ports() -> std::sync::MutexGuard<'static, Vec<u16>> {
RELAY_PORTS.lock().unwrap_or_else(std::sync::PoisonError::into_inner)
}
/// Whether `port` is a relay of [`listen_for_jvm`]: a session attaching to it debugs a JVM that
/// connected to the IDE, so the JVM has no JDWP listener the memory view could find it by.
pub(crate) fn is_listen_relay(port: u16) -> bool {
relay_ports().contains(&port)
}
/// Registers a relay port for as long as it lives.
struct RelayPort(u16);
impl RelayPort {
fn new(port: u16) -> Self {
relay_ports().push(port);
Self(port)
}
}
impl Drop for RelayPort {
fn drop(&mut self) {
relay_ports().retain(|&p| p != self.0);
}
}
/// The address to listen on for `host`: the interface it names, or every interface when
/// it is not one of this machine's (a DNS name or a NAT address the JVM connects to).
async fn bind(host: &str, port: u16) -> IdeResult<TcpListener> {
@@ -40,11 +69,11 @@ pub async fn listen_for_jvm(host: &str, port: u16, cancel: CancellationToken) ->
let _ = jvm.set_nodelay(true);
let relay = TcpListener::bind(("127.0.0.1", 0)).await?;
let local = relay.local_addr()?.port();
tokio::spawn(relay_one(relay, jvm));
tokio::spawn(relay_one(relay, jvm, RelayPort::new(local)));
Ok(local)
}
async fn relay_one(relay: TcpListener, mut jvm: TcpStream) {
async fn relay_one(relay: TcpListener, mut jvm: TcpStream, _registered: RelayPort) {
let debugger = match tokio::time::timeout(ATTACH_TIMEOUT, relay.accept()).await {
Ok(Ok((s, _))) => s,
_ => return,
@@ -58,22 +87,64 @@ async fn relay_one(relay: TcpListener, mut jvm: TcpStream) {
}
#[cfg(test)]
mod tests {
pub(crate) mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
#[tokio::test]
async fn relays_the_jdwp_handshake_both_ways() {
let port = TcpListener::bind(("127.0.0.1", 0)).await.unwrap().local_addr().unwrap().port();
let waiting = tokio::spawn(listen_for_jvm("127.0.0.1", port, CancellationToken::new()));
// The remote JVM (`server=n`) connects to the IDE.
let mut jvm = loop {
if let Ok(s) = TcpStream::connect(("127.0.0.1", port)).await {
break s;
/// A remote JVM (`server=n`) connected to [`listen_for_jvm`], and the relay port. A port
/// picked by binding to 0 can be taken by a parallel test before `listen_for_jvm` binds it,
/// so a failed listen is retried on a fresh one.
pub(crate) async fn connected_jvm() -> (TcpStream, u16) {
for _ in 0..20 {
let port = TcpListener::bind(("127.0.0.1", 0)).await.unwrap().local_addr().unwrap().port();
if let Some(connected) = jvm_on(port).await {
return connected;
}
}
panic!("listen_for_jvm never accepted a JVM");
}
/// One attempt of [`connected_jvm`]; `None` when the port turned out to be taken.
async fn jvm_on(port: u16) -> Option<(TcpStream, u16)> {
let mut waiting = tokio::spawn(listen_for_jvm("127.0.0.1", port, CancellationToken::new()));
let mut jvm = None;
for _ in 0..100 {
// A listen that failed to bind leaves nothing to connect to.
if waiting.is_finished() {
break;
}
// macOS does not refuse a connect to a port a socket holds without listening: the
// SYN goes unanswered until the connect times out.
let connect = TcpStream::connect(("127.0.0.1", port));
if let Ok(Ok(s)) = tokio::time::timeout(Duration::from_secs(1), connect).await {
jvm = Some(s);
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
};
let local = waiting.await.unwrap().unwrap();
}
// A connection to another test's listener never reaches `listen_for_jvm`.
let relay = tokio::time::timeout(Duration::from_secs(2), &mut waiting).await;
if let (Some(jvm), Ok(Ok(Ok(relay)))) = (jvm, relay) {
return Some((jvm, relay));
}
waiting.abort();
None
}
#[tokio::test]
async fn a_port_another_socket_holds_is_given_up() {
// Bound but not listening, as a parallel test's outgoing connection holds its local port.
let held = tokio::net::TcpSocket::new_v4().unwrap();
held.bind(([127, 0, 0, 1], 0).into()).unwrap();
let port = held.local_addr().unwrap().port();
let attempt = tokio::time::timeout(Duration::from_secs(5), jvm_on(port)).await;
assert!(matches!(attempt, Ok(None)), "an attempt on a taken port gives up instead of waiting forever");
}
#[tokio::test]
async fn relays_the_jdwp_handshake_both_ways() {
// The remote JVM (`server=n`) connects to the IDE.
let (mut jvm, local) = connected_jvm().await;
// java-debug attaches to the relay and starts the handshake.
let mut debugger = TcpStream::connect(("127.0.0.1", local)).await.unwrap();
debugger.write_all(b"JDWP-Handshake").await.unwrap();
@@ -85,6 +156,22 @@ mod tests {
assert_eq!(&buf, b"JDWP-Handshake");
}
#[tokio::test]
async fn relay_port_is_known_while_the_relay_runs() {
let (jvm, local) = connected_jvm().await;
assert!(is_listen_relay(local));
let debugger = TcpStream::connect(("127.0.0.1", local)).await.unwrap();
drop((jvm, debugger));
for _ in 0..200 {
if !is_listen_relay(local) {
return;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
panic!("relay port {local} still registered after the relay ended");
}
#[tokio::test]
async fn waiting_is_cancellable() {
let cancel = CancellationToken::new();
+92
Переглянути файл
@@ -0,0 +1,92 @@
//! The class table counted by the debugger (JDWP `InstanceCounts`) where no histogram can be
//! taken: JDK 8-16 while suspended, Windows, a remote JVM. Only while suspended, and slow on a
//! large heap (seconds on JDK 17+).
use super::inspect::Call;
use super::snapshot::diff_rows;
use super::{ClassCounts, JvmMemoryResult, JvmMemorySnapshot, JvmMemorySource, JvmMemoryState, JvmMemoryUnavailable};
use crate::session::Session;
use ide_core::IdeResult;
use ide_jvm_attach::HistogramRow;
use ide_jvm_inspect::ClassCount;
/// Reasons the debugger may be able to work around; the others (the program exited, the JVM
/// did not answer) would fail for it too.
pub(super) fn falls_back(reason: &JvmMemoryUnavailable) -> bool {
use JvmMemoryUnavailable as U;
matches!(
reason,
U::UnsafeWhileSuspended { .. }
| U::UnsupportedPlatform
| U::AttachedJvmNotFound
| U::ListenerNotRunning
| U::ListenModeNotSupported
| U::NoJvmProcess
)
}
/// `histogram` unless it is unavailable for a reason `session`'s inspector can work around.
pub(super) async fn or_debugger_counts(
state: &mut JvmMemoryState,
histogram: Result<JvmMemorySnapshot, JvmMemoryUnavailable>,
session: Option<&Session>,
) -> IdeResult<JvmMemoryResult> {
let reason = match histogram {
Ok(snapshot) => return Ok(JvmMemoryResult::Ready { snapshot }),
Err(reason) => reason,
};
let unavailable = |can_count_when_paused| Ok(JvmMemoryResult::Unavailable { reason, can_count_when_paused });
let Some(session) = session.filter(|_| falls_back(&reason)) else {
return unavailable(false);
};
let call = match Call::begin(session, None).await {
Ok(call) => call,
Err(super::JvmInspectUnavailable::NotSuspended) => return unavailable(true),
Err(_) => return unavailable(false),
};
let started = std::time::Instant::now();
let result = call.run(call.inspector.class_counts(&call.target)).await;
match call.finish(result) {
Ok(counts) => {
let elapsed_ms = u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX);
let previous = state.baseline(JvmMemorySource::Debugger);
let (snapshot, counts) = debugger_snapshot(counts, previous.as_deref(), elapsed_ms);
state.remember(JvmMemorySource::Debugger, counts);
Ok(JvmMemoryResult::Ready { snapshot })
}
Err(e) => {
tracing::debug!("debugger class counts: {e}");
let running = !session.suspended.get();
unavailable(running && session.jvm_inspector.lock().is_some())
}
}
}
/// Classes without instances are left out; most instances first.
pub(super) fn debugger_snapshot(
counts: Vec<ClassCount>,
previous: Option<&ClassCounts>,
elapsed_ms: u32,
) -> (JvmMemorySnapshot, ClassCounts) {
let rows = counts
.into_iter()
.filter(|c| c.instances > 0)
.map(|c| HistogramRow { name: c.name, module: None, instances: c.instances, bytes: 0 })
.collect();
let (mut rows, counts) = diff_rows(rows, previous);
for row in &mut rows {
row.delta_bytes = None;
}
rows.sort_by(|a, b| b.instances.cmp(&a.instances).then_with(|| a.name.cmp(&b.name)));
let snapshot = JvmMemorySnapshot {
total_instances: rows.iter().map(|r| r.instances).sum(),
total_bytes: 0,
rows,
java_version: String::new(),
taken_while_suspended: true,
has_baseline: previous.is_some(),
elapsed_ms,
source: JvmMemorySource::Debugger,
};
(snapshot, counts)
}
+293
Переглянути файл
@@ -0,0 +1,293 @@
//! A heap dump of a debugged JVM, written over its attach listener into a snapshot directory.
use super::snapshot::{AttachProbe, JdkProbe, attach_failure, gone, pick_jvm_pid};
use super::{
JvmMemoryState, JvmMemoryUnavailable, JvmTarget, histogram_pid, keep_found, lookup, resolve_jvm,
safe_while_suspended, session_target,
};
use crate::session::Session;
use ide_core::{CancellationToken, IdeError, IdeResult};
use ide_jvm_attach::{AttachError, HeapDumpReply};
use serde::{Deserialize, Serialize};
use specta::Type;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::sync::Arc;
const DUMP_FILE: &str = "dump.hprof";
/// For a refusal whose reply holds no reason: empty, or only the line announcing the dump.
const NO_REASON: &str = "the JVM wrote no heap dump and gave no reason";
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmHeapUnavailable {
/// The reason a class table could not be read applies to the dump too.
Memory {
reason: JvmMemoryUnavailable,
},
/// Bytes.
NotEnoughDiskSpace {
needed: u64,
available: u64,
},
DumpFailed {
message: String,
},
DumpTimedOut {
seconds: u32,
},
}
#[derive(Clone, Debug, PartialEq)]
pub struct JvmHeapDump {
pub path: PathBuf,
/// The class histogram taken just before the dump.
pub histogram: Vec<ide_heap::HistogramEntry>,
/// Raw `VM.flags`; empty when the JVM did not answer.
pub vm_flags: String,
/// 0 when unknown.
pub java_major: u32,
/// Empty when unknown.
pub java_version: String,
pub pid: u32,
pub taken_while_suspended: bool,
pub dump_ms: u32,
}
#[derive(Clone, Debug, PartialEq)]
pub enum JvmHeapOutcome {
Ready(JvmHeapDump),
/// The snapshot directory has been deleted.
Unavailable(JvmHeapUnavailable),
/// The snapshot directory has been deleted.
Cancelled,
}
pub(crate) async fn session_heap_dump(
session: Arc<Session>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let launched = histogram_pid(&session);
let capture = {
let (dir, cancel) = (dir.clone(), cancel.clone());
async move {
let target = session_target(&session, launched);
let probe = |suspended| -> Arc<dyn AttachProbe> { Arc::new(JdkProbe { suspended }) };
let stopped = session.clone();
let suspended = move || capture_suspended(&stopped);
heap_dump_with(&session.jvm_memory, suspended, probe, target, dir, cancel).await
}
};
detached(capture, dir, cancel).await
}
/// The JVM can be suspended before the adapter reports the stop, so a session with a JDWP link
/// also counts the link's view.
pub(super) fn capture_suspended(session: &Session) -> bool {
session.suspended.get()
|| session
.jdwp_link
.lock()
.as_ref()
.is_some_and(|link| matches!(link.state(), ide_jdwp::LinkState::Suspended { .. }))
}
/// Runs `capture` in a task of its own, so a caller that stops waiting cannot release the
/// session's lock while the JVM is still writing the dump and let a class-table request run
/// next to it. On `cancel` returns at once and deletes `dir`.
pub(super) async fn detached(
capture: impl Future<Output = IdeResult<JvmHeapOutcome>> + Send + 'static,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let task = tokio::spawn(capture);
tokio::select! {
joined = task => joined.map_err(|e| IdeError::internal(format!("JVM heap dump failed: {e}")))?,
() = cancel.cancelled() => {
discard_later(dir).await;
Ok(JvmHeapOutcome::Cancelled)
}
}
}
/// Holds `state`'s lock throughout, as a class-table request does; `suspended` and then `probe`
/// are read once it is held, and `suspended` again by [`take_heap_dump`]. Anything but `Ready`
/// deletes `dir`.
pub(super) async fn heap_dump_with(
state: &tokio::sync::Mutex<JvmMemoryState>,
suspended: impl Fn() -> bool + Send + Sync + 'static,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
// The capture's histogram never becomes the Memory view's baseline, so the state is unused.
let _turn = state.lock().await;
let outcome = capture(suspended, probe, target, dir.clone(), cancel).await;
if !matches!(outcome, Ok(JvmHeapOutcome::Ready(_))) {
discard_later(dir).await;
}
outcome
}
async fn capture(
suspended_now: impl Fn() -> bool + Send + Sync + 'static,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
let suspended = suspended_now();
let lookup = match lookup(target) {
Ok(lookup) => lookup,
Err(reason) => return Ok(memory(reason)),
};
let probe = probe(suspended);
let (found, outcome) = tokio::task::spawn_blocking(move || match resolve_jvm(lookup, &*probe, suspended) {
Ok((tree, found)) => (found, take_heap_dump(&*probe, &tree, suspended, &suspended_now, &dir, &cancel)),
Err(reason) => (None, Ok(memory(reason))),
})
.await
.map_err(|e| IdeError::internal(format!("JVM heap dump failed: {e}")))?;
keep_found(target, found);
outcome
}
/// `tree` starts with the session's reported pid, followed by its descendants. `suspended` is
/// the stop read when the capture began; `suspended_now` reads it again. Anything but `Ready`
/// deletes `dir`.
pub(crate) fn take_heap_dump(
probe: &dyn AttachProbe,
tree: &[i32],
suspended: bool,
suspended_now: &dyn Fn() -> bool,
dir: &Path,
cancel: &CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let outcome = dump(probe, tree, suspended, suspended_now, dir, cancel);
if !matches!(outcome, Ok(JvmHeapOutcome::Ready(_))) {
discard(dir);
}
outcome
}
fn dump(
probe: &dyn AttachProbe,
tree: &[i32],
suspended: bool,
suspended_now: &dyn Fn() -> bool,
dir: &Path,
cancel: &CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
use JvmHeapUnavailable as H;
use JvmMemoryUnavailable as U;
let Some(root) = tree.first().and_then(|&p| u32::try_from(p).ok()) else {
return Ok(memory(U::NoJvmProcess));
};
let Some(jvm) = pick_jvm_pid(tree, |p| probe.has_listener(p)) else {
return Ok(memory(gone(probe, root)));
};
let version = probe.version(jvm);
let java_major = version.as_ref().map_or(0, |v| v.major);
let safe = safe_while_suspended(version.as_ref());
let unsafe_stop = || Ok(memory(U::UnsafeWhileSuspended { java_major }));
if suspended && !safe {
return unsafe_stop();
}
// The program may stop during the histogram, and a stopped JDK below 17 hangs on the next
// request until it resumes.
let stopped_since = || !safe && suspended_now();
let text = match probe.histogram(jvm) {
Ok(text) => text,
Err(e) => return attach_failure(probe, jvm, e, "class histogram").map(memory),
};
let histogram: Vec<ide_heap::HistogramEntry> = ide_jvm_attach::parse_class_histogram(&text)
.rows
.into_iter()
.map(|r| ide_heap::HistogramEntry { name: r.name, instances: r.instances, bytes: r.bytes })
.collect();
let (instances, bytes) =
histogram.iter().fold((0u64, 0u64), |(i, b), e| (i.saturating_add(e.instances), b.saturating_add(e.bytes)));
let needed = ide_heap::capture_need(bytes, instances);
if let Some(available) = probe.free_bytes(dir)
&& available < needed
{
return Ok(JvmHeapOutcome::Unavailable(H::NotEnoughDiskSpace { needed, available }));
}
if stopped_since() {
return unsafe_stop();
}
// Without flags the index assumes the default object layout.
let vm_flags = probe.vm_flags(jvm).unwrap_or_else(|e| {
tracing::debug!("VM.flags of JVM {jvm}: {e}");
String::new()
});
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
if stopped_since() {
return unsafe_stop();
}
let path = dir.join(DUMP_FILE);
let timeout = ide_jvm_attach::dump_timeout(bytes);
let started = std::time::Instant::now();
let reply = probe.heap_dump(jvm, &path, timeout);
let dump_ms = u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX);
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
let failed = |message: String| Ok(JvmHeapOutcome::Unavailable(H::DumpFailed { message }));
match reply {
Ok(HeapDumpReply::Created { .. }) => {
if !std::fs::metadata(&path).is_ok_and(|m| m.is_file() && m.len() > 0) {
return failed(format!("the JVM reported a heap dump, but {} is missing or empty", path.display()));
}
Ok(JvmHeapOutcome::Ready(JvmHeapDump {
path,
histogram,
vm_flags,
java_major,
java_version: version.map(|v| v.text).unwrap_or_default(),
pid: jvm,
taken_while_suspended: suspended,
dump_ms,
}))
}
Ok(HeapDumpReply::Refused { message }) if message.is_empty() || message.starts_with("Dumping heap to") => {
failed(NO_REASON.into())
}
Ok(HeapDumpReply::Refused { message }) => failed(message),
Err(AttachError::Timeout) => {
let seconds = u32::try_from(timeout.as_secs()).unwrap_or(u32::MAX);
Ok(JvmHeapOutcome::Unavailable(H::DumpTimedOut { seconds }))
}
// A running JDK rejects an argument with a non-zero result code.
Err(AttachError::Command { message, .. }) => failed(message),
Err(e) => attach_failure(probe, jvm, e, "heap dump").map(memory),
}
}
fn memory(reason: JvmMemoryUnavailable) -> JvmHeapOutcome {
JvmHeapOutcome::Unavailable(JvmHeapUnavailable::Memory { reason })
}
fn discard(dir: &Path) {
if let Err(e) = std::fs::remove_dir_all(dir)
&& e.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not delete heap snapshot directory {}: {e}", dir.display());
}
}
/// A dump may be large, so it is deleted off the async threads.
async fn discard_later(dir: PathBuf) {
let _ = tokio::task::spawn_blocking(move || discard(&dir)).await;
}
+251
Переглянути файл
@@ -0,0 +1,251 @@
//! A suspended JVM's instances, fields and referrers, read through the session's
//! [`JvmInspector`]. An object is only valid within the stop it was read in.
use super::launch_pid;
use crate::session::Session;
use ide_core::event::{DebugEvent, JvmBackendState};
use ide_core::{IdeError, IdeResult};
use ide_jvm_inspect::{
DEFAULT_REFERRERS, InspectError, InspectFuture, JvmField, JvmInspector, JvmReferrer, JvmValue, MAX_INSTANCES,
MAX_REFERRERS, ObjectId, VmTarget,
};
use serde::{Deserialize, Serialize};
use specta::Type;
use std::sync::Arc;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct JvmObjectRef {
/// The `generation` of the [`JvmInstancesResult`] the object came from.
pub generation: u64,
pub id: ObjectId,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmInspectUnavailable {
/// The session has no inspector: not a JVM session, not ready yet, or turned off.
NoBackend,
/// The debug adapter lacks what the inspector needs; turned off for the session.
Unsupported {
detail: String,
},
NotSuspended,
/// The object is from an earlier stop, or the program stopped or resumed meanwhile.
Stale,
Collected,
ClassNotLoaded,
VmNotFound,
/// Transient: ask again.
TimedOut,
BackendClosed,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmInstancesResult {
Ready {
/// Goes into the [`JvmObjectRef`] of each instance.
generation: u64,
instances: Vec<JvmValue>,
/// The limit was reached; there may be more.
truncated: bool,
},
Unavailable {
reason: JvmInspectUnavailable,
},
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmFieldsResult {
Ready { fields: Vec<JvmField> },
Unavailable { reason: JvmInspectUnavailable },
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmReferrersResult {
Ready { referrers: Vec<JvmReferrer>, truncated: bool },
Unavailable { reason: JvmInspectUnavailable },
}
/// The session's JVM for the inspector: java-debug's VM is found by the launched program's pid
/// or by the port of an attach (the thread lists of two JVMs can be identical).
pub(crate) fn vm_target(s: &Session) -> VmTarget {
let (pid, attach) = if s.request == "launch" {
(u32::try_from(launch_pid(s)).ok().filter(|&p| p > 0), None)
} else {
(None, s.attach_target.get())
};
VmTarget {
pid,
attach_host: attach.map(|(host, _)| host.clone()).filter(|h| !h.is_empty()),
attach_port: attach.map(|&(_, port)| port),
generation: s.suspended.ticket(),
}
}
pub(crate) fn set_inspector(s: &Session, inspector: Arc<dyn JvmInspector>) {
*s.jvm_inspector.lock() = Some(inspector);
s.emit(DebugEvent::Capabilities);
}
pub(crate) async fn instances(s: &Session, class_name: &str, limit: u32) -> IdeResult<JvmInstancesResult> {
let limit = limit.clamp(1, MAX_INSTANCES);
let call = match Call::begin(s, None).await {
Ok(call) => call,
Err(reason) => return Ok(JvmInstancesResult::Unavailable { reason }),
};
let generation = call.target.generation;
let result = call.run(call.inspector.instances(&call.target, class_name, limit)).await;
Ok(match settle(call.finish(result))? {
Ok(instances) => {
JvmInstancesResult::Ready { generation, truncated: instances.len() >= limit as usize, instances }
}
Err(reason) => JvmInstancesResult::Unavailable { reason },
})
}
pub(crate) async fn fields(s: &Session, object: JvmObjectRef) -> IdeResult<JvmFieldsResult> {
let call = match Call::begin(s, Some(object.generation)).await {
Ok(call) => call,
Err(reason) => return Ok(JvmFieldsResult::Unavailable { reason }),
};
let result = call.run(call.inspector.fields(&call.target, object.id)).await;
Ok(match settle(call.finish(result))? {
Ok(fields) => JvmFieldsResult::Ready { fields },
Err(reason) => JvmFieldsResult::Unavailable { reason },
})
}
pub(crate) async fn referrers(s: &Session, object: JvmObjectRef, limit: u32) -> IdeResult<JvmReferrersResult> {
let limit = if limit == 0 { DEFAULT_REFERRERS } else { limit.min(MAX_REFERRERS) };
let call = match Call::begin(s, Some(object.generation)).await {
Ok(call) => call,
Err(reason) => return Ok(JvmReferrersResult::Unavailable { reason }),
};
let result = call.run(call.inspector.referrers(&call.target, object.id, limit)).await;
Ok(match settle(call.finish(result))? {
Ok(referrers) => JvmReferrersResult::Ready { truncated: referrers.len() >= limit as usize, referrers },
Err(reason) => JvmReferrersResult::Unavailable { reason },
})
}
/// One inspector request. Holds the session's inspect lock and a clone of its inspector, which
/// [`Call::run`] lets go of when the session ends.
pub(super) struct Call<'s> {
session: &'s Session,
pub(super) inspector: Arc<dyn JvmInspector>,
pub(super) target: VmTarget,
_turn: tokio::sync::MutexGuard<'s, ()>,
}
impl<'s> Call<'s> {
/// `object_generation`: the stop the request's object was read in.
pub(super) async fn begin(
s: &'s Session,
object_generation: Option<u64>,
) -> Result<Call<'s>, JvmInspectUnavailable> {
// Checked under the lock: a request queued behind another must see the state it runs in.
let turn = s.inspect_lock.lock().await;
let Some(inspector) = s.jvm_inspector.lock().clone() else {
return Err(JvmInspectUnavailable::NoBackend);
};
if !s.suspended.get() {
return Err(JvmInspectUnavailable::NotSuspended);
}
let target = vm_target(s);
if object_generation.is_some_and(|g| g != target.generation) {
return Err(JvmInspectUnavailable::Stale);
}
Ok(Call { session: s, inspector, target, _turn: turn })
}
pub(super) async fn run<T>(&self, request: InspectFuture<'_, T>) -> Result<T, InspectError> {
tokio::select! {
result = request => result,
() = self.session.ended.cancelled() => Err(InspectError::Closed),
}
}
/// The result unless the program stopped or resumed while it was read: a request of several
/// commands can straddle a stop. `Unsupported` turns the backend off.
pub(super) fn finish<T>(self, result: Result<T, InspectError>) -> Result<T, InspectError> {
let s = self.session;
match result {
Err(InspectError::Unsupported(detail)) => {
turn_off(s, &self.inspector, &detail);
Err(InspectError::Unsupported(detail))
}
_ if !s.suspended.get() || s.suspended.ticket() != self.target.generation => Err(InspectError::Stale),
result => result,
}
}
}
/// An adapter that lacks what the inspector needs will not gain it during the session.
fn turn_off(s: &Session, inspector: &Arc<dyn JvmInspector>, detail: &str) {
let removed = {
let mut slot = s.jvm_inspector.lock();
// Not one installed since this request began.
let current = slot.as_ref().is_some_and(|i| std::ptr::addr_eq(Arc::as_ptr(i), Arc::as_ptr(inspector)));
current && slot.take().is_some()
};
if removed {
s.emit(DebugEvent::JvmBackend { state: JvmBackendState::Unsupported { detail: detail.to_owned() } });
s.emit(DebugEvent::Capabilities);
}
}
/// A failure the inspector could not classify is an error, not a reason shown to the user.
fn settle<T>(result: Result<T, InspectError>) -> IdeResult<Result<T, JvmInspectUnavailable>> {
use JvmInspectUnavailable as U;
let error = match result {
Ok(value) => return Ok(Ok(value)),
Err(error) => error,
};
Ok(Err(match error {
InspectError::NotSuspended => U::NotSuspended,
InspectError::Stale => U::Stale,
InspectError::Collected => U::Collected,
InspectError::ClassNotLoaded => U::ClassNotLoaded,
InspectError::VmNotFound => U::VmNotFound,
InspectError::Unsupported(detail) => U::Unsupported { detail },
// Also the Kotlin link's `Busy`: the backend stays.
InspectError::TimedOut => U::TimedOut,
InspectError::Closed => U::BackendClosed,
InspectError::Failed(message) => return Err(IdeError::internal(format!("JVM inspector: {message}"))),
}))
}
/// Reports a Kotlin link that stops inspecting or closes. Holds neither the link nor the
/// session: it ends once the session drops the link, and never drops it itself (after
/// `VirtualMachine.Dispose` the link is closed but still forwarding the reply).
pub(crate) fn watch_link(s: &Arc<Session>, link: &ide_jdwp::JdwpLink) {
let mut states = link.subscribe();
let session = Arc::downgrade(s);
tokio::spawn(async move {
let mut reported = None;
loop {
let state = match &*states.borrow_and_update() {
ide_jdwp::LinkState::Degraded(detail) => Some(JvmBackendState::Degraded { detail: detail.clone() }),
ide_jdwp::LinkState::Closed => Some(JvmBackendState::Closed),
_ => None,
};
if let Some(state) = state
&& reported.as_ref() != Some(&state)
{
let Some(s) = session.upgrade() else { return };
if s.ended.is_cancelled() || s.terminated.load(std::sync::atomic::Ordering::Acquire) {
return;
}
s.emit(DebugEvent::JvmBackend { state: state.clone() });
reported = Some(state);
}
if states.changed().await.is_err() {
return;
}
}
});
}
+338
Переглянути файл
@@ -0,0 +1,338 @@
//! Class-table snapshots of a debugged JVM, read over the HotSpot attach listener or, where
//! that is unavailable, counted by the debugger; and the JVM's objects through the debugger.
mod fallback;
mod heap;
mod inspect;
mod scan;
mod snapshot;
#[cfg(test)]
pub(crate) mod test_support;
#[cfg(test)]
mod tests;
use crate::session::Session;
use crate::session::Suspension;
use ide_core::IdeResult;
use serde::{Deserialize, Serialize};
use specta::Type;
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicI64, Ordering};
use std::time::Duration;
pub use heap::*;
pub use inspect::{JvmFieldsResult, JvmInspectUnavailable, JvmInstancesResult, JvmObjectRef, JvmReferrersResult};
pub(crate) use inspect::{fields, instances, referrers, set_inspector, watch_link};
#[cfg(all(test, unix))]
pub(crate) use scan::watch_for_attached_jvm;
pub(crate) use scan::{JvmFinder, attach_target, find_attached_jvm, find_local_jvm, local_jdwp_port};
use snapshot::{AttachProbe, JdkProbe, take_snapshot};
pub const JVM_MEMORY_ADAPTERS: &[&str] = &["java-debug", crate::kotlin::ADAPTER];
pub const HISTOGRAM_TIMEOUT: Duration = Duration::from_secs(10);
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct JvmClassRow {
pub name: String,
pub module: Option<String>,
pub instances: u64,
pub bytes: u64,
/// Change since the previous snapshot; `None` on the first one and for ambiguous rows.
pub delta_instances: Option<i64>,
pub delta_bytes: Option<i64>,
/// Several classes share this name (different class loaders), so it has no delta.
pub ambiguous: bool,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct JvmMemorySnapshot {
/// Largest `bytes` first.
pub rows: Vec<JvmClassRow>,
pub total_instances: u64,
pub total_bytes: u64,
/// Empty when unknown.
pub java_version: String,
pub taken_while_suspended: bool,
/// A previous snapshot from the same source exists; deltas are against it.
pub has_baseline: bool,
/// Wall time of the histogram or count request.
pub elapsed_ms: u32,
pub source: JvmMemorySource,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub enum JvmMemorySource {
/// `GC.class_histogram -all`: sizes, and objects not yet collected.
Histogram,
/// JDWP `InstanceCounts`: reachable objects only, no sizes (`bytes` is 0).
Debugger,
}
#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmMemoryUnavailable {
/// An attach session whose JVM is remote, or local but not found by its JDWP port.
AttachedJvmNotFound,
/// An attach through the IDE's listen-mode relay: the JVM connected to the IDE, so it has
/// no JDWP listener to be found by.
ListenModeNotSupported,
UnsupportedPlatform,
NoJvmProcess,
ListenerNotRunning,
/// `java_major` is 0 when the version is unknown.
UnsafeWhileSuspended {
java_major: u32,
},
ProgramExited,
TimedOut,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmMemoryResult {
Ready {
snapshot: JvmMemorySnapshot,
},
Unavailable {
reason: JvmMemoryUnavailable,
/// The debugger could count the classes if the program were paused.
can_count_when_paused: bool,
},
}
/// Instances and bytes per class name, for the next snapshot's deltas; `None` for a name
/// several classes shared, so it gets no delta next time either.
type ClassCounts = HashMap<String, Option<(u64, u64)>>;
/// Per-session baseline; its mutex also keeps one class-table request per session at a time.
#[derive(Default)]
pub struct JvmMemoryState {
/// Shared with the blocking task, so a task that panics leaves it in place.
previous: Option<(JvmMemorySource, Arc<ClassCounts>)>,
}
impl JvmMemoryState {
/// The previous snapshot's counts if it came from `source`: counts from the other source
/// are not comparable (only reachable objects versus all of them).
pub(crate) fn baseline(&self, source: JvmMemorySource) -> Option<Arc<ClassCounts>> {
self.previous.as_ref().filter(|(s, _)| *s == source).map(|(_, counts)| counts.clone())
}
pub(crate) fn remember(&mut self, source: JvmMemorySource, counts: ClassCounts) {
self.previous = Some((source, Arc::new(counts)));
}
}
pub(crate) async fn session_classes(session: Arc<Session>) -> IdeResult<JvmMemoryResult> {
until_located(&session).await;
let launched = histogram_pid(&session);
// Detached, so a caller that gives up cannot release the lock while the JVM is still
// answering and let a second histogram request run next to it.
tokio::spawn(async move {
let target = session_target(&session, launched);
let probe = |suspended| -> Arc<dyn AttachProbe> { Arc::new(JdkProbe { suspended }) };
snapshot(&session.jvm_memory, &session.suspended, probe, target, Some(&session)).await
})
.await
.map_err(|e| ide_core::IdeError::internal(format!("JVM memory snapshot failed: {e}")))?
}
/// Bounds the wait of [`until_located`]; `configure` has its own timeouts, this only guards them.
const LOCATE_WAIT: Duration = Duration::from_secs(30);
/// Until `configure` has recorded where `session`'s JVM is. The Memory view asks as soon as the
/// session appears; looked up earlier, an attached JVM is reported as missing.
async fn until_located(session: &Session) {
let mut located = session.jvm_located.subscribe();
tokio::select! {
_ = located.wait_for(|v| *v) => {}
() = session.ended.cancelled() => {}
() = tokio::time::sleep(LOCATE_WAIT) => {}
}
}
/// `launched`: [`histogram_pid`] of `session`.
fn session_target(session: &Session, launched: Result<Option<i32>, JvmMemoryUnavailable>) -> JvmTarget<'_> {
match launched {
Ok(Some(pid)) => JvmTarget::Launched(pid),
Ok(None) => JvmTarget::Attached { pid: &session.jvm_pid, jdwp_port: session.jdwp_port.get().copied() },
Err(reason) => JvmTarget::Unavailable(reason),
}
}
/// The pid of a launch session's program (`None` for an attach session), or why no histogram
/// can be taken.
fn histogram_pid(session: &Session) -> Result<Option<i32>, JvmMemoryUnavailable> {
if !JVM_MEMORY_ADAPTERS.contains(&session.adapter.as_str()) {
return Err(JvmMemoryUnavailable::NoJvmProcess);
}
if !cfg!(unix) {
return Err(JvmMemoryUnavailable::UnsupportedPlatform);
}
if session.request == "launch" {
return match i32::try_from(launch_pid(session)).ok().filter(|&p| p > 0) {
Some(pid) => Ok(Some(pid)),
None => Err(JvmMemoryUnavailable::NoJvmProcess),
};
}
if session.jvm_pid.load(Ordering::Acquire) <= 0 {
if session.terminated.load(Ordering::Acquire) {
return Err(JvmMemoryUnavailable::ProgramExited);
}
if session.listen_relay.load(Ordering::Acquire) {
return Err(JvmMemoryUnavailable::ListenModeNotSupported);
}
}
Ok(None)
}
/// The JVM a launch session started: the one the IDE started for an app-launched (Kotlin)
/// session, else the program the adapter reported; 0 when neither is known.
pub(crate) fn launch_pid(session: &Session) -> i64 {
match session.jvm_pid.load(Ordering::Acquire) {
p if p > 0 => p,
_ => session.debuggee_pid.load(Ordering::Acquire),
}
}
#[derive(Clone, Copy)]
pub(crate) enum JvmTarget<'a> {
/// The program a launch session started: its pid and everything it started.
Launched(i32),
/// The JVM of an attach session. While `pid` is 0 it is looked up by `jdwp_port`, which is
/// `None` for a remote JVM.
Attached { pid: &'a AtomicI64, jdwp_port: Option<u16> },
/// Known without asking any JVM: no histogram can be taken, for this reason.
Unavailable(JvmMemoryUnavailable),
}
enum Lookup {
Tree(i32),
Known(i32),
Scan(u16),
}
/// `probe` is built from the suspension read once the lock is held: a request queued behind
/// another must not decide on a state from before the program hit a breakpoint. Where no
/// histogram can be taken, `debugger` (the session) may count the classes instead.
pub(crate) async fn snapshot(
state: &tokio::sync::Mutex<JvmMemoryState>,
suspended: &Suspension,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
debugger: Option<&Session>,
) -> IdeResult<JvmMemoryResult> {
let mut state = state.lock().await;
let outcome = histogram(&mut state, suspended, probe, target).await?;
fallback::or_debugger_counts(&mut state, outcome, debugger).await
}
async fn histogram(
state: &mut JvmMemoryState,
suspended: &Suspension,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
) -> IdeResult<Result<JvmMemorySnapshot, JvmMemoryUnavailable>> {
let suspended = suspended.get();
let lookup = match lookup(target) {
Ok(lookup) => lookup,
Err(reason) => return Ok(Err(reason)),
};
let probe = probe(suspended);
let previous = state.baseline(JvmMemorySource::Histogram);
let (found, outcome) = tokio::task::spawn_blocking(move || match resolve_jvm(lookup, &*probe, suspended) {
Ok((tree, found)) => (found, take_snapshot(&*probe, &tree, suspended, previous.as_deref())),
Err(reason) => (None, Ok(Err(reason))),
})
.await
.map_err(|e| ide_core::IdeError::internal(format!("JVM memory snapshot failed: {e}")))?;
keep_found(target, found);
Ok(outcome?.map(|(snapshot, counts)| {
state.remember(JvmMemorySource::Histogram, counts);
snapshot
}))
}
/// Where to look for `target`'s JVM, or why there is none to ask.
fn lookup(target: JvmTarget<'_>) -> Result<Lookup, JvmMemoryUnavailable> {
match target {
JvmTarget::Launched(pid) => Ok(Lookup::Tree(pid)),
JvmTarget::Attached { pid, jdwp_port } => match i32::try_from(pid.load(Ordering::Acquire)) {
Ok(p) if p > 0 => Ok(Lookup::Known(p)),
_ => jdwp_port.map(Lookup::Scan).ok_or(JvmMemoryUnavailable::AttachedJvmNotFound),
},
JvmTarget::Unavailable(reason) => Err(reason),
}
}
/// Blocking. The pids to pick the JVM from, and the pid a scan found (for [`keep_found`]).
fn resolve_jvm(
lookup: Lookup,
probe: &dyn AttachProbe,
suspended: bool,
) -> Result<(Vec<i32>, Option<i32>), JvmMemoryUnavailable> {
match lookup {
Lookup::Tree(pid) => Ok((pid_tree(pid), None)),
Lookup::Known(pid) => Ok((vec![pid], None)),
Lookup::Scan(port) => scan_for_jvm(probe, port, suspended).map(|pid| (vec![pid], Some(pid))),
}
}
fn keep_found(target: JvmTarget<'_>, found: Option<i32>) {
if let (Some(found), JvmTarget::Attached { pid, .. }) = (found, target) {
// The attach-time scan may have found it meanwhile; either answer is the same JVM.
let _ = pid.compare_exchange(0, i64::from(found), Ordering::AcqRel, Ordering::Acquire);
}
}
/// A suspended JDK below 17 hangs on an attach request, so while suspended only a JVM known to
/// be 17 or later is asked anything. Pass a version read without attaching (the `release` file).
fn safe_while_suspended(version: Option<&ide_jvm_attach::JavaVersion>) -> bool {
version.is_some_and(|v| ide_jvm_attach::histogram_safe_while_suspended(v.major))
}
/// The JVM listening on JDWP `port`. While suspended the scan reaches the session's own JVM
/// too, so JVMs unsafe to ask are skipped, and when one of them may be the session's the
/// answer is a refusal rather than "not found".
fn scan_for_jvm(probe: &dyn AttachProbe, port: u16, suspended: bool) -> Result<i32, JvmMemoryUnavailable> {
// Majors of the JVMs left out; 0 for an unknown version.
let skipped = std::cell::RefCell::new(std::collections::BTreeSet::new());
let may_query = |pid: u32| {
if !suspended {
return true;
}
let version = probe.version(pid);
let safe = safe_while_suspended(version.as_ref());
if !safe {
skipped.borrow_mut().insert(version.map_or(0, |v| v.major));
}
safe
};
match probe.find_by_jdwp_port(port, &may_query).and_then(|p| i32::try_from(p).ok()) {
Some(pid) if pid > 0 => Ok(pid),
_ => {
let skipped = skipped.into_inner();
Err(match skipped.first() {
None => JvmMemoryUnavailable::AttachedJvmNotFound,
Some(&major) => JvmMemoryUnavailable::UnsafeWhileSuspended {
java_major: if skipped.len() == 1 { major } else { 0 },
},
})
}
}
}
fn pid_tree(pid: i32) -> Vec<i32> {
#[cfg(unix)]
{
ide_core::process::process_tree(pid)
}
#[cfg(not(unix))]
{
vec![pid]
}
}
+146
Переглянути файл
@@ -0,0 +1,146 @@
//! Finding the JVM behind an attach session by its JDWP port.
use super::safe_while_suspended;
use crate::session::Session;
use serde_json::Value;
use std::net::IpAddr;
use std::sync::Arc;
use std::sync::atomic::Ordering;
use std::time::Duration;
/// One try of the scan at attach time, and the limit per socket within it: a silent socket's
/// query thread lives that long, so a longer per-socket limit piles threads up across tries.
const ATTACH_SCAN_TRY: Duration = Duration::from_millis(500);
const ATTACH_SCAN_INTERVAL: Duration = Duration::from_millis(150);
/// The JVM's attach socket can appear a little after the JDWP "Listening" banner, which is
/// when the IDE attaches.
const ATTACH_SCAN_WINDOW: Duration = Duration::from_secs(3);
/// The single scan a refresh makes while the attach-time one found nothing.
pub(super) const RESCAN_PER_SOCKET: Duration = Duration::from_secs(1);
pub(super) const RESCAN_BUDGET: Duration = Duration::from_secs(2);
/// JDWP port of an attach to this machine (`hostName` loopback or absent): only such a JVM has
/// an attach socket the IDE can find.
pub(crate) fn local_jdwp_port(arguments: &Value) -> Option<u16> {
let local = match arguments.get("hostName") {
None | Some(Value::Null) => true,
Some(Value::String(host)) => {
let host = host.trim();
let ip = host.strip_prefix('[').and_then(|h| h.strip_suffix(']')).unwrap_or(host);
host.eq_ignore_ascii_case("localhost") || ip.parse::<IpAddr>().is_ok_and(|ip| ip.is_loopback())
}
Some(_) => false,
};
jdwp_port(arguments).filter(|_| local)
}
/// `port` of an attach request: a number or a numeric string, never 0.
fn jdwp_port(arguments: &Value) -> Option<u16> {
let port = match arguments.get("port")? {
Value::Number(n) => n.as_u64().and_then(|p| u16::try_from(p).ok()),
Value::String(s) => s.trim().parse().ok(),
_ => None,
};
port.filter(|&p| p != 0)
}
/// `hostName` (empty when absent) and `port` of an attach request, local or not.
pub(crate) fn attach_target(arguments: &Value) -> Option<(String, u16)> {
let host = arguments.get("hostName").and_then(Value::as_str).unwrap_or_default();
Some((host.to_owned(), jdwp_port(arguments)?))
}
/// Pid of the local JVM listening on a JDWP port (first argument), given whether the session is
/// suspended (second). Replaceable so tests never send attach requests to real JVMs.
pub(crate) type JvmFinder = Arc<dyn Fn(u16, bool) -> Option<u32> + Send + Sync>;
/// The [`JvmFinder`] of the attach-time scan.
pub(crate) fn find_local_jvm(port: u16, suspended: bool) -> Option<u32> {
let may_query = |pid| !suspended || safe_while_suspended(ide_jvm_attach::java_version_of(pid).as_ref());
ide_jvm_attach::find_jvm_by_jdwp_port(port, ATTACH_SCAN_TRY, ATTACH_SCAN_TRY, may_query)
}
/// Looks for the JVM behind a local attach: one try now, the rest in the background, so a JVM
/// that never opens an attach socket holds the attach up by one try at most. Call it before the
/// debugger connects: the JVM clears its JDWP listener address then, and a scan can no longer
/// tell it by its port.
pub(crate) async fn find_attached_jvm(session: &Arc<Session>, port: u16, finder: JvmFinder) {
let _ = session.jdwp_port.set(port);
let find = move |suspended: bool| finder(port, suspended);
let first = find.clone();
if let Ok(Some(pid)) = tokio::task::spawn_blocking(move || first(false)).await {
let _ = session.jvm_pid.compare_exchange(0, i64::from(pid), Ordering::AcqRel, Ordering::Acquire);
return;
}
drop(watch_for_attached_jvm(session, port, find, ATTACH_SCAN_INTERVAL, ATTACH_SCAN_WINDOW));
}
/// Retries `find` for the session's `jvm_pid`, holding the session weakly so an ended session
/// is not kept alive. `find` is told whether the session is suspended.
pub(crate) fn watch_for_attached_jvm<F>(
session: &Arc<Session>,
port: u16,
find: F,
interval: Duration,
window: Duration,
) -> tokio::task::JoinHandle<()>
where
F: Fn(bool) -> Option<u32> + Clone + Send + 'static,
{
let weak = Arc::downgrade(session);
tokio::spawn(async move {
let gate = || match weak.upgrade() {
Some(s) if !s.terminated.load(Ordering::Acquire) && !s.conn.is_closed() => {
if s.jvm_pid.load(Ordering::Acquire) > 0 {
ScanGate::Ended
} else {
ScanGate::Scan { suspended: s.suspended.get() }
}
}
_ => ScanGate::Ended,
};
let Some(pid) = retry_find(find, gate, interval, window).await else {
tracing::debug!("no local JVM found for JDWP port {port} yet");
return;
};
if let Some(s) = weak.upgrade() {
let _ = s.jvm_pid.compare_exchange(0, i64::from(pid), Ordering::AcqRel, Ordering::Acquire);
}
})
}
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) enum ScanGate {
Scan { suspended: bool },
Ended,
}
/// Calls `find` on a blocking thread, on each try `gate` allows, until it returns a pid, `gate`
/// reports the end or `window` has passed.
pub(crate) async fn retry_find<F>(
find: F,
gate: impl Fn() -> ScanGate,
interval: Duration,
window: Duration,
) -> Option<u32>
where
F: Fn(bool) -> Option<u32> + Clone + Send + 'static,
{
let deadline = tokio::time::Instant::now() + window;
loop {
match gate() {
ScanGate::Ended => break,
ScanGate::Scan { suspended } => {
let find = find.clone();
if let Ok(Some(pid)) = tokio::task::spawn_blocking(move || find(suspended)).await {
return Some(pid);
}
}
}
if tokio::time::Instant::now() + interval >= deadline {
break;
}
tokio::time::sleep(interval).await;
}
None
}
+214
Переглянути файл
@@ -0,0 +1,214 @@
//! One class histogram of a JVM and its deltas against the previous one.
use super::scan::{RESCAN_BUDGET, RESCAN_PER_SOCKET};
use super::{
ClassCounts, HISTOGRAM_TIMEOUT, JvmClassRow, JvmMemorySnapshot, JvmMemorySource, JvmMemoryUnavailable,
safe_while_suspended,
};
use ide_core::IdeResult;
use ide_jvm_attach::{AttachError, HeapDumpReply, HistogramRow, JavaVersion};
use std::collections::HashMap;
use std::path::Path;
use std::time::Duration;
/// Version probe for a JVM whose `release` file could not be read.
const VERSION_TIMEOUT: Duration = Duration::from_secs(2);
const FLAGS_TIMEOUT: Duration = Duration::from_secs(2);
/// The attach calls a snapshot or a heap dump makes, so tests can stand in for a JVM.
pub(crate) trait AttachProbe: Send + Sync {
fn has_listener(&self, pid: u32) -> bool;
/// Asks only the JVMs whose pid `may_query` accepts.
fn find_by_jdwp_port(&self, port: u16, may_query: &dyn Fn(u32) -> bool) -> Option<u32>;
fn version(&self, pid: u32) -> Option<JavaVersion>;
fn histogram(&self, pid: u32) -> Result<String, AttachError>;
fn alive(&self, pid: u32) -> bool;
fn vm_flags(&self, pid: u32) -> Result<String, AttachError>;
fn heap_dump(&self, pid: u32, path: &Path, timeout: Duration) -> Result<HeapDumpReply, AttachError>;
/// `None` where the free space is not known.
fn free_bytes(&self, dir: &Path) -> Option<u64>;
}
pub(super) struct JdkProbe {
pub(super) suspended: bool,
}
impl AttachProbe for JdkProbe {
fn has_listener(&self, pid: u32) -> bool {
ide_jvm_attach::has_listener(pid)
}
fn find_by_jdwp_port(&self, port: u16, may_query: &dyn Fn(u32) -> bool) -> Option<u32> {
ide_jvm_attach::find_jvm_by_jdwp_port(port, RESCAN_PER_SOCKET, RESCAN_BUDGET, may_query)
}
fn version(&self, pid: u32) -> Option<JavaVersion> {
ide_jvm_attach::java_version_of(pid).or_else(|| {
// The version decides whether attaching to a suspended VM hangs it, so never
// attach to learn it while suspended.
if self.suspended {
return None;
}
let out = ide_jvm_attach::jcmd(pid, "VM.version", VERSION_TIMEOUT).ok()?;
ide_jvm_attach::parse_vm_version(&out)
})
}
fn histogram(&self, pid: u32) -> Result<String, AttachError> {
ide_jvm_attach::jcmd(pid, "GC.class_histogram -all", HISTOGRAM_TIMEOUT)
}
fn alive(&self, pid: u32) -> bool {
alive(pid)
}
fn vm_flags(&self, pid: u32) -> Result<String, AttachError> {
ide_jvm_attach::vm_flags(pid, FLAGS_TIMEOUT)
}
fn heap_dump(&self, pid: u32, path: &Path, timeout: Duration) -> Result<HeapDumpReply, AttachError> {
ide_jvm_attach::heap_dump(pid, path, timeout)
}
fn free_bytes(&self, dir: &Path) -> Option<u64> {
ide_heap::free_bytes(dir)
}
}
fn alive(pid: u32) -> bool {
#[cfg(unix)]
{
// 0 and negative pids address process groups.
let Ok(pid) = i32::try_from(pid) else { return false };
// SAFETY: signal 0 only checks that the process exists and may be signalled.
pid > 0
&& (unsafe { libc::kill(pid, 0) } == 0
|| std::io::Error::last_os_error().raw_os_error() == Some(libc::EPERM))
}
#[cfg(not(unix))]
{
let _ = pid;
false
}
}
/// Why `pid`'s attach listener cannot answer: it stopped listening or the program ended.
pub(super) fn gone(probe: &dyn AttachProbe, pid: u32) -> JvmMemoryUnavailable {
if probe.alive(pid) { JvmMemoryUnavailable::ListenerNotRunning } else { JvmMemoryUnavailable::ProgramExited }
}
/// The reason for an attach request (`what`) to `jvm` that failed, or an error where no reason
/// fits.
pub(super) fn attach_failure(
probe: &dyn AttachProbe,
jvm: u32,
error: AttachError,
what: &str,
) -> IdeResult<JvmMemoryUnavailable> {
use JvmMemoryUnavailable as U;
match error {
AttachError::Timeout => Ok(U::TimedOut),
AttachError::Refused | AttachError::NoListener => Ok(gone(probe, jvm)),
AttachError::ForeignListener => {
tracing::warn!("attach socket of JVM {jvm} is no longer private to this user; not using it");
Ok(U::ListenerNotRunning)
}
AttachError::Unsupported => Ok(U::UnsupportedPlatform),
e => Err(ide_core::IdeError::internal(format!("{what} of JVM {jvm}: {e}"))),
}
}
pub(crate) fn pick_jvm_pid(tree: &[i32], has_listener: impl Fn(u32) -> bool) -> Option<u32> {
tree.iter().filter_map(|&p| u32::try_from(p).ok()).filter(|&p| p > 0).find(|&p| has_listener(p))
}
pub(crate) fn diff_rows(rows: Vec<HistogramRow>, previous: Option<&ClassCounts>) -> (Vec<JvmClassRow>, ClassCounts) {
let mut seen: HashMap<&str, u32> = HashMap::with_capacity(rows.len());
for r in &rows {
*seen.entry(r.name.as_str()).or_default() += 1;
}
let ambiguous: Vec<bool> = rows.iter().map(|r| seen[r.name.as_str()] > 1).collect();
drop(seen);
let mut counts = ClassCounts::with_capacity(rows.len());
let mut out: Vec<JvmClassRow> = rows
.into_iter()
.zip(ambiguous)
.map(|(r, ambiguous)| {
counts.insert(r.name.clone(), (!ambiguous).then_some((r.instances, r.bytes)));
let delta = match previous {
_ if ambiguous => None,
None => None,
Some(p) => match p.get(&r.name) {
Some(None) => None,
// Absent: new since the previous snapshot, so it grew from nothing.
before => {
let (instances, bytes) = before.copied().flatten().unwrap_or_default();
Some((delta(r.instances, instances), delta(r.bytes, bytes)))
}
},
};
JvmClassRow {
name: r.name,
module: r.module,
instances: r.instances,
bytes: r.bytes,
delta_instances: delta.map(|d| d.0),
delta_bytes: delta.map(|d| d.1),
ambiguous,
}
})
.collect();
out.sort_by(|a, b| b.bytes.cmp(&a.bytes).then_with(|| a.name.cmp(&b.name)));
(out, counts)
}
/// Exact while the difference fits in an i64 (two's complement).
fn delta(now: u64, before: u64) -> i64 {
now.wrapping_sub(before) as i64
}
/// `pid_tree` starts with the session's reported pid, followed by its descendants.
pub(crate) fn take_snapshot(
probe: &dyn AttachProbe,
pid_tree: &[i32],
suspended: bool,
previous: Option<&ClassCounts>,
) -> IdeResult<Result<(JvmMemorySnapshot, ClassCounts), JvmMemoryUnavailable>> {
use JvmMemoryUnavailable as U;
let Some(root) = pid_tree.first().and_then(|&p| u32::try_from(p).ok()) else {
return Ok(Err(U::NoJvmProcess));
};
let Some(jvm) = pick_jvm_pid(pid_tree, |p| probe.has_listener(p)) else {
return Ok(Err(gone(probe, root)));
};
let version = probe.version(jvm);
if suspended && !safe_while_suspended(version.as_ref()) {
return Ok(Err(U::UnsafeWhileSuspended { java_major: version.map_or(0, |v| v.major) }));
}
let started = std::time::Instant::now();
let text = match probe.histogram(jvm) {
Ok(text) => text,
Err(e) => return attach_failure(probe, jvm, e, "class histogram").map(Err),
};
let elapsed_ms = u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX);
let histogram = ide_jvm_attach::parse_class_histogram(&text);
if histogram.skipped_lines > 0 {
tracing::debug!("class histogram of JVM {jvm}: {} unreadable lines", histogram.skipped_lines);
}
let (rows, counts) = diff_rows(histogram.rows, previous);
let snapshot = JvmMemorySnapshot {
total_instances: rows.iter().map(|r| r.instances).sum(),
total_bytes: rows.iter().map(|r| r.bytes).sum(),
rows,
java_version: version.map(|v| v.text).unwrap_or_default(),
taken_while_suspended: suspended,
has_baseline: previous.is_some(),
elapsed_ms,
source: JvmMemorySource::Histogram,
};
Ok(Ok((snapshot, counts)))
}
+119
Переглянути файл
@@ -0,0 +1,119 @@
//! A scripted [`JvmInspector`] and a session without an adapter behind it.
use crate::SessionOwner;
use crate::adapter::{AdapterSpec, Transport};
use crate::connection::Connection;
use crate::session::Session;
use ide_core::{DebugSessionId, IdeEvent, SharedSink};
use ide_jvm_inspect::{
ClassCount, InspectError, InspectFuture, JvmField, JvmInspector, JvmReferrer, JvmValue, ObjectId, VmTarget,
};
use parking_lot::Mutex;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use tokio::sync::Notify;
#[derive(Default)]
pub(crate) struct FakeInspector {
pub targets: Mutex<Vec<VmTarget>>,
/// The `limit` of each `instances` / `referrers` call.
pub limits: Mutex<Vec<u32>>,
pub calls: AtomicUsize,
/// Returned by every call when set.
pub error: Mutex<Option<InspectError>>,
pub counts: Mutex<Vec<ClassCount>>,
pub instances: Mutex<Vec<JvmValue>>,
/// Every call signals `entered`, then waits for `release`.
pub hold: bool,
pub entered: Notify,
pub release: Notify,
}
impl FakeInspector {
pub fn failing(error: InspectError) -> Self {
Self { error: Mutex::new(Some(error)), ..Self::default() }
}
pub fn calls(&self) -> usize {
self.calls.load(Ordering::SeqCst)
}
async fn answer<T: Send>(&self, target: &VmTarget, limit: Option<u32>, ok: T) -> Result<T, InspectError> {
self.calls.fetch_add(1, Ordering::SeqCst);
self.targets.lock().push(target.clone());
self.limits.lock().extend(limit);
if self.hold {
self.entered.notify_one();
self.release.notified().await;
}
match self.error.lock().clone() {
Some(e) => Err(e),
None => Ok(ok),
}
}
}
impl JvmInspector for FakeInspector {
fn class_counts<'a>(&'a self, target: &'a VmTarget) -> InspectFuture<'a, Vec<ClassCount>> {
Box::pin(self.answer(target, None, self.counts.lock().clone()))
}
fn instances<'a>(&'a self, target: &'a VmTarget, _: &'a str, limit: u32) -> InspectFuture<'a, Vec<JvmValue>> {
Box::pin(self.answer(target, Some(limit), self.instances.lock().clone()))
}
fn fields<'a>(&'a self, target: &'a VmTarget, _: ObjectId) -> InspectFuture<'a, Vec<JvmField>> {
Box::pin(self.answer(target, None, Vec::new()))
}
fn referrers<'a>(&'a self, target: &'a VmTarget, _: ObjectId, limit: u32) -> InspectFuture<'a, Vec<JvmReferrer>> {
Box::pin(self.answer(target, Some(limit), Vec::new()))
}
}
pub(crate) struct BareSession {
pub session: Arc<Session>,
pub events: Arc<Mutex<Vec<IdeEvent>>>,
/// The adapter's end of the connection, kept open.
_adapter: tokio::io::DuplexStream,
}
/// A java-debug session that no adapter answers; must be called inside a Tokio runtime.
pub(crate) fn bare_session(request: &str) -> BareSession {
bare_session_of("java-debug", request)
}
/// A session of adapter `name` that no adapter answers; must be called inside a Tokio runtime.
pub(crate) fn bare_session_of(name: &str, request: &str) -> BareSession {
let events: Arc<Mutex<Vec<IdeEvent>>> = Arc::default();
let sink_events = events.clone();
let sink: SharedSink = Arc::new(move |e: IdeEvent| sink_events.lock().push(e));
let (client, adapter) = tokio::io::duplex(1 << 10);
let (reader, writer) = tokio::io::split(client);
let (conn, _) = Connection::start(reader, writer);
let spec = AdapterSpec {
adapter: name.into(),
command: String::new(),
args: Vec::new(),
transport: Transport::Stdio,
install_hint: None,
cwd: std::env::temp_dir(),
env: Vec::new(),
server_host: None,
connect: None,
pinned: None,
};
let session = Session::new(
DebugSessionId(1),
SessionOwner::new("test", sink),
"test".into(),
name.into(),
None,
request.into(),
conn,
spec,
None,
None,
);
BareSession { session: Arc::new(session), events, _adapter: adapter }
}
File diff suppressed because it is too large. Load diff
+7
Переглянути файл
@@ -9,6 +9,13 @@ use std::{
pub const ADAPTER: &str = "kotlin-debug-adapter";
/// The adapter attaches to a JVM the IDE starts, through the JDWP link. It resumes the VM while
/// constructing its debuggee, before its breakpoint hooks exist, so the link holds resumes back
/// until the attach is done.
pub(crate) fn link_options() -> ide_jdwp::LinkOptions {
ide_jdwp::LinkOptions { hold_resumes: true }
}
pub struct SourceMirror {
dir: tempfile::TempDir,
paths: HashMap<PathBuf, PathBuf>,
-166
Переглянути файл
@@ -1,166 +0,0 @@
//! Keep a suspended JVM stopped until Kotlin's adapter has installed its listeners.
//! The official 0.4.4 adapter resumes VMStart while constructing its debuggee, before
//! installing breakpoint hooks. DAP attach completes only after those hooks exist.
use ide_core::IdeResult;
use std::{io, time::Duration};
use tokio::{
io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt},
net::{TcpListener, TcpStream},
sync::oneshot,
task::JoinHandle,
};
pub(crate) struct StartupRelay {
release: Option<oneshot::Sender<()>>,
task: JoinHandle<()>,
}
impl StartupRelay {
pub async fn start(host: String, port: u16) -> IdeResult<(Self, u16)> {
let listener = TcpListener::bind(("127.0.0.1", 0)).await?;
let local = listener.local_addr()?.port();
let (release, ready) = oneshot::channel();
let task = tokio::spawn(async move {
let connect = async {
let (adapter, _) = listener.accept().await?;
let jvm = TcpStream::connect((host.as_str(), port)).await?;
adapter.set_nodelay(true)?;
jvm.set_nodelay(true)?;
Ok::<_, io::Error>((adapter, jvm))
};
if let Ok(Ok((adapter, jvm))) = tokio::time::timeout(Duration::from_secs(60), connect).await {
let (mut ar, mut aw) = adapter.into_split();
let (mut jr, mut jw) = jvm.into_split();
let upstream = forward(&mut ar, &mut jw, ready);
let downstream = tokio::io::copy(&mut jr, &mut aw);
// Close both sockets as soon as either side disconnects.
tokio::select! {
r = upstream => if let Err(e) = r { tracing::debug!("Kotlin JDWP relay: {e}"); },
_ = downstream => {},
}
}
});
Ok((Self { release: Some(release), task }, local))
}
pub fn release(&mut self) {
if let Some(ready) = self.release.take() {
let _ = ready.send(());
}
}
}
impl Drop for StartupRelay {
fn drop(&mut self) {
self.task.abort();
}
}
async fn packet(reader: &mut (impl AsyncRead + Unpin)) -> io::Result<Vec<u8>> {
let mut header = [0; 11];
reader.read_exact(&mut header).await?;
let len = u32::from_be_bytes(header[..4].try_into().unwrap()) as usize;
if !(11..=16 * 1024 * 1024).contains(&len) {
return Err(io::Error::new(io::ErrorKind::InvalidData, "Invalid JDWP packet length"));
}
let mut packet = vec![0; len];
packet[..11].copy_from_slice(&header);
reader.read_exact(&mut packet[11..]).await?;
Ok(packet)
}
async fn forward(
reader: &mut (impl AsyncRead + Unpin),
writer: &mut (impl AsyncWrite + Unpin),
mut ready: oneshot::Receiver<()>,
) -> io::Result<()> {
let mut handshake = [0; 14];
reader.read_exact(&mut handshake).await?;
if &handshake != b"JDWP-Handshake" {
return Err(io::Error::new(io::ErrorKind::InvalidData, "Invalid JDWP handshake"));
}
writer.write_all(&handshake).await?;
let mut released = false;
let mut held: Vec<Vec<u8>> = Vec::new();
loop {
// Retain the in-flight read when attach completes, including partial packets.
let read = packet(reader);
tokio::pin!(read);
let packet = tokio::select! {
result = &mut read => result?,
result = &mut ready, if !released => {
result.map_err(|_| io::Error::new(io::ErrorKind::ConnectionAborted, "Kotlin attach cancelled"))?;
released = true;
for packet in held.drain(..) {
writer.write_all(&packet).await?;
}
read.await?
}
};
// JDWP VirtualMachine.Resume (1/9), ThreadReference.Resume (11/3).
// Other commands must proceed: attach itself queries threads and installs requests.
if !released && packet[8] == 0 && matches!((packet[9], packet[10]), (1, 9) | (11, 3)) {
if held.len() >= 1024 {
return Err(io::Error::new(io::ErrorKind::InvalidData, "Too many pending JDWP resumes"));
}
held.push(packet);
} else {
writer.write_all(&packet).await?;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn command(id: u8, set: u8, cmd: u8) -> Vec<u8> {
vec![0, 0, 0, 11, 0, 0, 0, id, 0, set, cmd]
}
#[tokio::test]
async fn holds_resumes_but_allows_setup_and_preserves_partial_reads() {
let (mut adapter, mut reader) = tokio::io::duplex(4096);
let (mut writer, mut jvm) = tokio::io::duplex(4096);
let (release, ready) = oneshot::channel();
let task = tokio::spawn(async move { forward(&mut reader, &mut writer, ready).await });
adapter.write_all(b"JDWP-Handshake").await.unwrap();
let mut handshake = [0; 14];
jvm.read_exact(&mut handshake).await.unwrap();
assert_eq!(&handshake, b"JDWP-Handshake");
let resume = command(1, 1, 9);
let thread_resume = command(2, 11, 3);
let setup = command(3, 1, 4);
adapter.write_all(&resume).await.unwrap();
adapter.write_all(&thread_resume).await.unwrap();
adapter.write_all(&setup).await.unwrap();
assert_eq!(packet(&mut jvm).await.unwrap(), setup);
let next = command(4, 1, 9);
adapter.write_all(&next[..6]).await.unwrap();
release.send(()).unwrap();
assert_eq!(packet(&mut jvm).await.unwrap(), resume);
assert_eq!(packet(&mut jvm).await.unwrap(), thread_resume);
adapter.write_all(&next[6..]).await.unwrap();
assert_eq!(packet(&mut jvm).await.unwrap(), next);
drop(adapter);
assert!(task.await.unwrap().is_err());
}
#[tokio::test]
async fn dropping_session_closes_relay() {
let (relay, port) = StartupRelay::start("127.0.0.1".into(), 1).await.unwrap();
let mut adapter = TcpStream::connect(("127.0.0.1", port)).await.unwrap();
drop(relay);
let result = tokio::time::timeout(Duration::from_secs(1), adapter.read_u8()).await.unwrap();
assert!(result.is_err());
}
#[tokio::test]
async fn rejects_invalid_packet_lengths() {
for len in [0_u32, 10, 16 * 1024 * 1024 + 1] {
let mut bytes = command(1, 1, 9);
bytes[..4].copy_from_slice(&len.to_be_bytes());
assert_eq!(packet(&mut bytes.as_slice()).await.unwrap_err().kind(), io::ErrorKind::InvalidData);
}
}
}
+129 -12
Переглянути файл
@@ -13,8 +13,8 @@ pub mod attach;
mod cargo;
pub mod connection;
pub mod jdwp;
mod jvm_memory;
pub mod kotlin;
mod kotlin_jdwp;
mod session;
use adapter::{AdapterLauncher, AdapterSpec, ProcessLauncher};
@@ -22,6 +22,7 @@ use connection::{Connection, DapError};
use futures::future::join_all;
use ide_core::{DebugSessionId, IdeError, IdeResult, SharedSink};
use ide_language::LanguageRegistry;
pub use jvm_memory::*;
use parking_lot::Mutex;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
@@ -126,6 +127,14 @@ pub struct DebugCapabilities {
pub log_points: bool,
/// HotSwap ("Reload Changed Classes"): java-debug's `redefineClasses` request.
pub hot_code_replace: bool,
/// The session debugs a JVM, so it has a Memory view; [`DebugManager::jvm_memory_classes`]
/// says why when it has no data (platform, remote JVM, ...).
pub jvm_memory: bool,
/// The session has a JVM inspector: instances, fields and referrers while suspended. Can
/// change during the session ([`ide_core::event::DebugEvent::Capabilities`]).
pub jvm_instances: bool,
/// The session can be asked for a heap dump ([`DebugManager::jvm_heap_dump`]).
pub jvm_heap: bool,
}
/// Result of [`DebugManager::redefine_classes`].
@@ -302,6 +311,7 @@ pub struct DebugManager {
data_breakpoints: Mutex<Vec<DataBreakpoint>>,
/// In-flight stops, by session: a second Stop sets `stop_force`, the waiting one kills.
stopping: Mutex<HashMap<DebugSessionId, Arc<Session>>>,
jvm_finder: Mutex<jvm_memory::JvmFinder>,
}
/// rdbg (Ruby's debug gem) starts the debuggee itself and ignores `launch`'s program,
@@ -566,9 +576,21 @@ pub(crate) fn capabilities_from(caps: &Value) -> DebugCapabilities {
hit_conditional_breakpoints: has("supportsHitConditionalBreakpoints"),
log_points: has("supportsLogPoints"),
hot_code_replace: false,
jvm_memory: false,
jvm_instances: false,
jvm_heap: false,
}
}
pub(crate) fn session_capabilities(session: &Session) -> DebugCapabilities {
let mut caps = capabilities_from(&session.caps.lock());
caps.hot_code_replace = HOT_CODE_REPLACE_ADAPTERS.contains(&session.adapter.as_str());
caps.jvm_memory = JVM_MEMORY_ADAPTERS.contains(&session.adapter.as_str());
caps.jvm_instances = session.jvm_inspector.lock().is_some();
caps.jvm_heap = JVM_MEMORY_ADAPTERS.contains(&session.adapter.as_str());
caps
}
/// States of a file's line breakpoints that no session has been sent.
fn unsent(breakpoints: &[Breakpoint]) -> Vec<BreakpointState> {
breakpoints
@@ -657,9 +679,15 @@ impl DebugManager {
function_breakpoints: Mutex::default(),
data_breakpoints: Mutex::default(),
stopping: Mutex::default(),
jvm_finder: Mutex::new(Arc::new(jvm_memory::find_local_jvm)),
})
}
#[cfg(test)]
pub(crate) fn set_jvm_finder(&self, finder: jvm_memory::JvmFinder) {
*self.jvm_finder.lock() = finder;
}
fn session(&self, s: DebugSessionId) -> IdeResult<Arc<Session>> {
self.sessions.lock().get(&s).cloned().ok_or_else(|| no_session(s))
}
@@ -979,13 +1007,31 @@ impl DebugManager {
.await?;
*s.caps.lock() = if caps.is_object() { caps } else { json!({}) };
// The Kotlin adapter ignores the resolved classpath and project runtime on launch.
// The Kotlin adapters ignore the resolved classpath and project runtime on launch.
let request = if s.adapter == kotlin::ADAPTER && s.request == "launch" {
session::start_kotlin_debuggee(s, &mut arguments).await?;
"attach"
} else {
&s.request
};
// Both blocks below run before the Kotlin link replaces the JVM's address with its own.
if request == "attach"
&& let Some(target) = jvm_memory::attach_target(&arguments)
{
let _ = s.attach_target.set(target);
}
if cfg!(unix)
&& s.request == "attach"
&& JVM_MEMORY_ADAPTERS.contains(&s.adapter.as_str())
&& let Some(port) = jvm_memory::local_jdwp_port(&arguments)
{
if jdwp::is_listen_relay(port) {
s.listen_relay.store(true, Ordering::Release);
} else {
let finder = self.jvm_finder.lock().clone();
jvm_memory::find_attached_jvm(s, port, finder).await;
}
}
if s.adapter == kotlin::ADAPTER {
let host = arguments
.get("hostName")
@@ -998,11 +1044,14 @@ impl DebugManager {
.and_then(|p| u16::try_from(p).ok())
.filter(|p| *p != 0)
.ok_or_else(|| IdeError::invalid("Invalid Kotlin JDWP port"))?;
let (relay, port) = kotlin_jdwp::StartupRelay::start(host, port).await?;
*s.kotlin_relay.lock() = Some(relay);
let (link, port) = ide_jdwp::JdwpLink::start(host, port, kotlin::link_options()).await?;
*s.jvm_inspector.lock() = Some(Arc::new(ide_jdwp::LinkInspector::new(link.clone())));
jvm_memory::watch_link(s, &link);
*s.jdwp_link.lock() = Some(link);
arguments["hostName"] = json!("127.0.0.1");
arguments["port"] = json!(port);
}
s.jvm_located.send_replace(true);
// Launch/attach may wait for a slow runtime or a remote target. Let the adapter report
// failure; Stop cancels configuration and a closed connection fails the pending reply.
// Many adapters answer only after `configurationDone`, so don't wait for the reply yet.
@@ -1097,8 +1146,8 @@ impl DebugManager {
reply.await.unwrap_or(Err(DapError::Closed))?;
}
}
if let Some(relay) = s.kotlin_relay.lock().as_mut() {
relay.release();
if let Some(link) = s.jdwp_link.lock().as_ref() {
link.release();
}
Ok(())
}
@@ -1331,8 +1380,9 @@ impl DebugManager {
}
pub async fn threads(&self, s: DebugSessionId) -> IdeResult<Vec<ThreadInfo>> {
let body = self.session(s)?.conn.request_timeout("threads", Value::Null, REQUEST_TIMEOUT).await?;
Ok(body
let session = self.session(s)?;
let body = session.conn.request_timeout("threads", Value::Null, REQUEST_TIMEOUT).await?;
let threads: Vec<ThreadInfo> = body
.get("threads")
.and_then(Value::as_array)
.map(|t| {
@@ -1343,7 +1393,8 @@ impl DebugManager {
})
.collect()
})
.unwrap_or_default())
.unwrap_or_default();
Ok(threads)
}
pub async fn stack_trace(&self, s: DebugSessionId, thread_id: i64) -> IdeResult<Vec<StackFrame>> {
@@ -1472,8 +1523,11 @@ impl DebugManager {
pub async fn resume(&self, s: DebugSessionId, thread_id: i64) -> IdeResult<()> {
let session = self.session(s)?;
let ticket = session.suspended.ticket();
let body = session.conn.request_timeout("continue", json!({ "threadId": thread_id }), REQUEST_TIMEOUT).await?;
// DAP: a response without the flag continued every thread.
let all = body.get("allThreadsContinued").and_then(Value::as_bool).unwrap_or(true);
session.suspended.resumed(ticket, thread_id, all);
session.emit(ide_core::event::DebugEvent::Continued { thread_id, all_threads_continued: all });
Ok(())
}
@@ -1513,7 +1567,10 @@ impl DebugManager {
StepKind::Out => "stepOut",
};
let session = self.session(s)?;
let ticket = session.suspended.ticket();
session.conn.request_timeout(command, json!({ "threadId": thread_id }), REQUEST_TIMEOUT).await?;
// A step runs its thread; other threads stopped by their own events stay stopped.
session.suspended.resumed(ticket, thread_id, false);
session.emit(ide_core::event::DebugEvent::Continued { thread_id, all_threads_continued: true });
Ok(())
}
@@ -1611,10 +1668,12 @@ impl DebugManager {
/// Step into a specific call of the current line (Smart Step Into).
pub async fn step_in_target(&self, s: DebugSessionId, thread_id: i64, target_id: i64) -> IdeResult<()> {
let session = self.session(s)?;
let ticket = session.suspended.ticket();
session
.conn
.request_timeout("stepIn", json!({ "threadId": thread_id, "targetId": target_id }), REQUEST_TIMEOUT)
.await?;
session.suspended.resumed(ticket, thread_id, false);
session.emit(ide_core::event::DebugEvent::Continued { thread_id, all_threads_continued: true });
Ok(())
}
@@ -1648,10 +1707,68 @@ impl DebugManager {
}
pub fn capabilities(&self, s: DebugSessionId) -> IdeResult<DebugCapabilities> {
Ok(session_capabilities(&*self.session(s)?))
}
/// Class table of the session's JVM (`GC.class_histogram -all`, or counted by the debugger
/// while suspended where no histogram can be taken), with deltas against the session's
/// previous snapshot from the same source.
pub async fn jvm_memory_classes(&self, s: DebugSessionId) -> IdeResult<JvmMemoryResult> {
let session = self.session(s)?;
let mut caps = capabilities_from(&session.caps.lock());
caps.hot_code_replace = HOT_CODE_REPLACE_ADAPTERS.contains(&session.adapter.as_str());
Ok(caps)
jvm_memory::session_classes(session).await
}
/// Writes a heap dump of the session's JVM to `dir/dump.hprof`, one class-table or dump
/// request per session at a time. Anything but `Ready` deletes `dir`, unless `s` is not a
/// session. On `cancel` the result is `Cancelled` at once, while a dump the JVM is writing
/// still holds the session's turn until the JVM answers.
pub async fn jvm_heap_dump(
&self,
s: DebugSessionId,
dir: PathBuf,
cancel: ide_core::CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let session = self.session(s)?;
jvm_memory::session_heap_dump(session, dir, cancel).await
}
/// Gives the session the backend for [`Self::jvm_memory_instances`] and the debugger's class
/// counts; the UI is told to reload the session's capabilities.
pub fn set_jvm_inspector(
&self,
s: DebugSessionId,
inspector: Arc<dyn ide_jvm_inspect::JvmInspector>,
) -> IdeResult<()> {
let session = self.session(s)?;
if session.ended.is_cancelled() {
return Err(no_session(s));
}
jvm_memory::set_inspector(&session, inspector);
Ok(())
}
/// Up to `limit` (clamped to `1..=MAX_INSTANCES`) instances of `class_name` while suspended.
pub async fn jvm_memory_instances(
&self,
s: DebugSessionId,
class_name: String,
limit: u32,
) -> IdeResult<JvmInstancesResult> {
jvm_memory::instances(&*self.session(s)?, &class_name, limit).await
}
pub async fn jvm_memory_fields(&self, s: DebugSessionId, object: JvmObjectRef) -> IdeResult<JvmFieldsResult> {
jvm_memory::fields(&*self.session(s)?, object).await
}
/// Objects that point at `object`; `limit` 0 means `DEFAULT_REFERRERS`, at most `MAX_REFERRERS`.
pub async fn jvm_memory_referrers(
&self,
s: DebugSessionId,
object: JvmObjectRef,
limit: u32,
) -> IdeResult<JvmReferrersResult> {
jvm_memory::referrers(&*self.session(s)?, object, limit).await
}
/// HotSwap: reloads the classes compiled since the session started (or since the
+315 -42
Переглянути файл
@@ -5,6 +5,8 @@ use crate::adapter::AdapterSpec;
use crate::connection::{Connection, DapError, Incoming};
use crate::{Breakpoint, BreakpointState, DataBreakpoint, DebugManager, FunctionBreakpoint, SessionOwner};
use ide_core::event::DebugEvent;
#[cfg(unix)]
use ide_core::process::process_tree;
use ide_core::{DebugSessionId, IdeEvent, TaskId};
use parking_lot::Mutex;
use serde_json::{Value, json};
@@ -62,6 +64,15 @@ pub(crate) struct Session {
pub stop_force: AtomicBool,
/// `systemProcessId` from the adapter's `process` event: the program being debugged.
pub debuggee_pid: AtomicI64,
/// The JVM an app-launched (Kotlin) session started, or the local JVM an attach session
/// found by its JDWP port. Separate from `debuggee_pid`, which decides how Stop signals the
/// program and is 0 for Kotlin.
pub jvm_pid: AtomicI64,
/// JDWP port of an attach to a local JVM, to look the JVM up again; unset for a remote one.
pub jdwp_port: std::sync::OnceLock<u16>,
/// The attach goes through the listen-mode relay of [`crate::jdwp::listen_for_jvm`].
pub listen_relay: AtomicBool,
pub suspended: Suspension,
/// Pids signalled for the current stop: the reported pid and everything it started (an
/// adapter may report the launcher shell, whose child is the program).
#[cfg_attr(not(unix), allow(dead_code))]
@@ -75,8 +86,105 @@ pub(crate) struct Session {
/// Resolved path → the path as the IDE knows it, for adapters given resolved paths.
resolved_paths: Mutex<HashMap<PathBuf, PathBuf>>,
progress: Mutex<HashMap<String, TaskId>>,
pub kotlin_relay: Mutex<Option<crate::kotlin_jdwp::StartupRelay>>,
/// Relay between a Kotlin adapter and its JVM. Dropping it closes both sockets.
pub jdwp_link: parking_lot::Mutex<Option<ide_jdwp::JdwpLink>>,
pub jvm_inspector: parking_lot::Mutex<Option<Arc<dyn ide_jvm_inspect::JvmInspector>>>,
/// One inspector request per session at a time.
pub inspect_lock: tokio::sync::Mutex<()>,
/// Cancelled when the session ends, so a request still waiting on the inspector lets go of
/// it (and of the JDWP link it holds).
pub ended: ide_core::CancellationToken,
/// `hostName` (empty when absent) and `port` of the session's `attach` request, as the IDE
/// was given them: Kotlin then replaces them with the link's. Unlike `jdwp_port`, also set
/// for a remote JVM.
pub attach_target: std::sync::OnceLock<(String, u16)>,
/// Set once `configure` has recorded where the program's JVM is (the fields above, the
/// Kotlin link), just before the `launch`/`attach` request goes out.
pub jvm_located: watch::Sender<bool>,
pub kotlin_mirror: Mutex<Option<crate::kotlin::SourceMirror>>,
pub jvm_memory: tokio::sync::Mutex<crate::JvmMemoryState>,
}
/// Which threads of the program are suspended: each from its `stopped` event until a
/// `continued` event or a successful resume/step request for it. Never cleared elsewhere, since
/// a stale "suspended" only makes a later check refuse. fwcd and java-debug stop a breakpoint's
/// thread alone, so a thread can run while another stays stopped.
///
/// A resume reply ends only the stops taken before the request was sent: a thread's next
/// `stopped` event can overtake the reply. The check and the removal happen under one lock, so
/// no stop can land between them and be lost.
#[derive(Default)]
pub(crate) struct Suspension(Mutex<Stops>);
#[derive(Default)]
struct Stops {
/// Counts stops, and moves back to an earlier stopped thread: an object read under one
/// ticket is stale under another.
ticket: u64,
/// Threads stopped alone by their event, oldest first, with the ticket of that stop.
threads: Vec<(i64, u64)>,
/// The ticket of a stop of every thread (`allThreadsStopped`, or no thread id): any resume
/// ends it.
all: Option<u64>,
}
impl Stops {
fn latest(&self) -> Option<u64> {
self.threads.last().map(|&(_, ticket)| ticket).max(self.all)
}
/// Ends the stops taken by ticket `upto` of `thread`, or of every thread.
fn release(&mut self, thread: Option<i64>, every: bool, upto: u64) {
let latest = self.latest();
if every || thread.is_none() || self.all.is_some_and(|t| t <= upto) {
self.all = self.all.filter(|&t| t > upto);
self.threads.retain(|&(_, t)| t > upto);
} else {
self.threads.retain(|&(id, t)| Some(id) != thread || t > upto);
}
// Back on an earlier thread's stop, as the JDWP link moves to it with a new generation.
let now = self.latest();
if now.is_some() && now != latest {
self.ticket += 1;
}
}
}
impl Suspension {
/// Whether some thread is stopped.
pub fn get(&self) -> bool {
let stops = self.0.lock();
stops.all.is_some() || !stops.threads.is_empty()
}
/// A `stopped` event; `all`: its `allThreadsStopped`.
pub fn stopped(&self, thread: Option<i64>, all: bool) {
let mut stops = self.0.lock();
stops.ticket += 1;
let ticket = stops.ticket;
match thread.filter(|_| !all) {
Some(thread) => {
stops.threads.retain(|&(id, _)| id != thread);
stops.threads.push((thread, ticket));
}
None => stops.all = Some(ticket),
}
}
/// A `continued` event; `all`: its `allThreadsContinued`.
pub fn continued(&self, thread: Option<i64>, all: bool) {
self.0.lock().release(thread, all, u64::MAX);
}
/// Take before sending a resume-like request and pass to [`Self::resumed`].
pub fn ticket(&self) -> u64 {
self.0.lock().ticket
}
/// After a successful resume-like request for `thread`; `all`: every thread ran.
pub fn resumed(&self, ticket: u64, thread: i64, all: bool) {
self.0.lock().release(Some(thread), all, ticket);
}
}
/// Adapters that only match files by their resolved path: Godot maps a script to
@@ -125,6 +233,10 @@ impl Session {
terminated: AtomicBool::new(false),
stop_force: AtomicBool::new(false),
debuggee_pid: AtomicI64::new(0),
jvm_pid: AtomicI64::new(0),
jdwp_port: std::sync::OnceLock::new(),
listen_relay: AtomicBool::new(false),
suspended: Suspension::default(),
stop_pids: parking_lot::Mutex::new(Vec::new()),
#[cfg(windows)]
stop_tree: Mutex::new(None),
@@ -132,8 +244,14 @@ impl Session {
breakpoint_ids: Mutex::new(HashMap::new()),
resolved_paths: Mutex::new(HashMap::new()),
progress: Mutex::new(HashMap::new()),
kotlin_relay: Mutex::new(None),
jdwp_link: parking_lot::Mutex::new(None),
jvm_inspector: parking_lot::Mutex::new(None),
inspect_lock: tokio::sync::Mutex::default(),
ended: ide_core::CancellationToken::new(),
attach_target: std::sync::OnceLock::new(),
jvm_located: watch::channel(false).0,
kotlin_mirror: Mutex::new(None),
jvm_memory: tokio::sync::Mutex::default(),
}
}
@@ -294,7 +412,10 @@ impl Session {
/// Kills the adapter and every debuggee we started.
pub fn kill_processes(&self) {
self.kotlin_relay.lock().take();
self.ended.cancel();
// The inspector holds a clone of the link, so both go.
self.jvm_inspector.lock().take();
self.jdwp_link.lock().take();
#[cfg(unix)]
if self.request == "launch" {
for group in launch_groups(self) {
@@ -351,32 +472,6 @@ pub(crate) async fn remember_debuggee(s: &Arc<Session>) {
let _ = s;
}
/// The program and everything it started: an adapter may report the launcher shell, whose
/// child is the debugged program, so stopping only the reported pid leaves the program alive.
#[cfg(unix)]
fn process_tree(pid: i32) -> Vec<i32> {
use std::collections::HashMap;
let Ok(out) = std::process::Command::new("ps").args(["-A", "-o", "pid=,ppid="]).output() else {
return vec![pid];
};
let mut children: HashMap<i32, Vec<i32>> = HashMap::new();
for line in String::from_utf8_lossy(&out.stdout).lines() {
let mut it = line.split_whitespace();
let (Some(p), Some(parent)) = (it.next(), it.next()) else { continue };
let (Ok(p), Ok(parent)) = (p.parse::<i32>(), parent.parse::<i32>()) else { continue };
children.entry(parent).or_default().push(p);
}
let mut tree = vec![pid];
let mut at = 0;
while at < tree.len() {
if let Some(kids) = children.get(&tree[at]) {
tree.extend(kids.iter().copied());
}
at += 1;
}
tree
}
/// Asks the program itself to exit (`force` = SIGKILL) without ending the adapter session:
/// its output keeps flowing over DAP while it shuts down. The first call remembers the pid
/// tree, because the reported pid may die before its program does.
@@ -600,18 +695,27 @@ async fn on_event(mgr: &Arc<DebugManager>, s: &Arc<Session>, event: &str, mut bo
s.emit(DebugEvent::Initialized);
}
"stopped" => {
let all = body.get("allThreadsStopped").and_then(Value::as_bool).unwrap_or(false);
s.suspended.stopped(int("threadId"), all);
let reason = str_("reason").unwrap_or_else(|| "pause".into());
mgr.restore_breakpoints(s).await;
s.emit(DebugEvent::Stopped {
thread_id: int("threadId").unwrap_or(0),
reason: str_("reason").unwrap_or_else(|| "pause".into()),
reason,
description: str_("description").or_else(|| str_("text")),
all_threads_stopped: body.get("allThreadsStopped").and_then(Value::as_bool).unwrap_or(false),
all_threads_stopped: all,
});
}
"continued" => {
let all = body.get("allThreadsContinued").and_then(Value::as_bool);
// DAP: an event without the flag continues its thread alone. The UI keeps treating
// it as all threads; the suspension, which holds only threads that stopped, need not.
s.suspended.continued(int("threadId"), all.unwrap_or(false));
s.emit(DebugEvent::Continued {
thread_id: int("threadId").unwrap_or(0),
all_threads_continued: all.unwrap_or(true),
});
}
"continued" => s.emit(DebugEvent::Continued {
thread_id: int("threadId").unwrap_or(0),
all_threads_continued: body.get("allThreadsContinued").and_then(Value::as_bool).unwrap_or(true),
}),
"output" => {
let category = str_("category").unwrap_or_else(|| "console".into());
if category == "telemetry" {
@@ -917,6 +1021,19 @@ fn forward_output<R: AsyncRead + Unpin + Send + 'static>(s: &Session, mut stream
});
}
/// The attach listener must be on the command line before the user's arguments, which may
/// override it; the memory view needs it because the JVM is started by the IDE, not attached to.
/// Not on Windows (`windows`), which the memory view does not support.
fn kotlin_jvm_args(user: Vec<String>, windows: bool) -> Vec<String> {
if windows || !ide_jvm_attach::wants_listener_flag(&user.join(" ")) {
return user;
}
let mut args = Vec::with_capacity(user.len() + 1);
args.push(ide_jvm_attach::START_LISTENER_FLAG.to_owned());
args.extend(user);
args
}
/// Launch with the IDE's resolved runtime and classpath; the adapter only attaches.
pub(crate) async fn start_kotlin_debuggee(s: &Arc<Session>, args: &mut Value) -> ide_core::IdeResult<()> {
use tokio::io::{AsyncBufReadExt, BufReader};
@@ -931,7 +1048,7 @@ pub(crate) async fn start_kotlin_debuggee(s: &Arc<Session>, args: &mut Value) ->
std::env::join_paths(strings("classPaths")).map_err(|e| ide_core::IdeError::invalid(e.to_string()))?;
let mut cmd = tokio::process::Command::new(java);
ide_core::process::prepare(cmd.as_std_mut());
cmd.args(strings("vmArguments"))
cmd.args(kotlin_jvm_args(strings("vmArguments"), cfg!(windows)))
// Let the JVM pick its own port (`:0`) and report it below: pre-picking a free
// port here races with other processes that may take it before the JVM binds.
.arg("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=127.0.0.1:0")
@@ -965,11 +1082,12 @@ pub(crate) async fn start_kotlin_debuggee(s: &Arc<Session>, args: &mut Value) ->
// forced stop), the session's job on Windows, so a stop reaches what it forks.
ide_core::process::own_group(cmd.as_std_mut());
let mut child = cmd.spawn()?;
#[cfg(windows)]
if let Some(pid) = child.id()
&& let Some(job) = s.job.lock().as_ref()
{
job.assign_pid(pid);
if let Some(pid) = child.id() {
s.jvm_pid.store(i64::from(pid), Ordering::Release);
#[cfg(windows)]
if let Some(job) = s.job.lock().as_ref() {
job.assign_pid(pid);
}
}
if let Some(err) = child.stderr.take() {
forward_output(s, err, "stderr");
@@ -1010,7 +1128,162 @@ fn parse_jdwp_port(line: &str) -> Option<u16> {
#[cfg(test)]
mod tests {
use super::parse_jdwp_port;
use super::{Suspension, kotlin_jvm_args, parse_jdwp_port};
#[test]
fn a_resume_reply_clears_the_stop_it_was_sent_for() {
let s = Suspension::default();
assert!(!s.get());
s.stopped(Some(1), true);
let ticket = s.ticket();
s.resumed(ticket, 1, true);
assert!(!s.get());
}
#[test]
fn a_stop_after_the_resume_request_keeps_the_session_suspended() {
let s = Suspension::default();
s.stopped(Some(1), true);
let ticket = s.ticket();
// The next breakpoint's `stopped` event overtakes the reply to `continue`.
s.continued(Some(1), true);
s.stopped(Some(1), true);
s.resumed(ticket, 1, true);
assert!(s.get());
// A reply for the new stop still clears it.
s.resumed(s.ticket(), 1, true);
assert!(!s.get());
}
#[test]
fn a_reply_after_a_continued_event_changes_nothing() {
let s = Suspension::default();
s.stopped(Some(1), true);
let ticket = s.ticket();
s.continued(Some(1), true);
s.resumed(ticket, 1, true);
assert!(!s.get());
s.stopped(Some(1), true);
assert!(s.get());
}
#[test]
fn concurrent_stops_are_never_lost() {
let s = std::sync::Arc::new(Suspension::default());
s.stopped(Some(1), true);
let ticket = s.ticket();
let stopping = {
let s = s.clone();
std::thread::spawn(move || (0..10_000).for_each(|_| s.stopped(Some(1), true)))
};
for _ in 0..10_000 {
s.resumed(ticket, 1, true);
}
stopping.join().unwrap();
// However they interleave, a resume for the old ticket never clears a later stop.
assert!(s.get());
assert_eq!(s.ticket(), ticket + 10_000);
}
/// fwcd and java-debug stop a breakpoint's thread alone (`SUSPEND_EVENT_THREAD`): 7, then 8.
fn two_threads_stopped() -> Suspension {
let s = Suspension::default();
s.stopped(Some(7), false);
s.stopped(Some(8), false);
s
}
#[test]
fn continuing_the_latest_stopped_thread_keeps_the_other_as_a_new_stop() {
let s = two_threads_stopped();
let ticket = s.ticket();
s.continued(Some(8), false);
assert!(s.get());
// Objects read on thread 8's stop are stale.
assert_eq!(s.ticket(), ticket + 1);
s.continued(Some(7), false);
assert!(!s.get());
}
#[test]
fn continuing_an_earlier_stopped_thread_keeps_the_latest_stop() {
let s = two_threads_stopped();
let ticket = s.ticket();
s.continued(Some(7), false);
assert!(s.get());
assert_eq!(s.ticket(), ticket);
}
#[test]
fn continuing_all_threads_ends_every_stop() {
let s = two_threads_stopped();
s.continued(Some(7), true);
assert!(!s.get());
let s = two_threads_stopped();
s.continued(None, false);
assert!(!s.get());
}
#[test]
fn continuing_a_thread_that_did_not_stop_changes_nothing() {
let s = two_threads_stopped();
let ticket = s.ticket();
s.continued(Some(5), false);
assert!(s.get());
assert_eq!(s.ticket(), ticket);
}
#[test]
fn a_step_reply_resumes_its_thread_alone() {
let s = two_threads_stopped();
let ticket = s.ticket();
s.resumed(ticket, 8, false);
assert!(s.get());
assert_eq!(s.ticket(), ticket + 1);
s.resumed(s.ticket(), 7, true);
assert!(!s.get());
}
#[test]
fn a_thread_stopping_again_before_its_resume_reply_stays_stopped() {
let s = two_threads_stopped();
let ticket = s.ticket();
// Thread 7's step ends, and its `stopped` event overtakes the reply to `next`.
s.stopped(Some(7), false);
s.resumed(ticket, 7, false);
s.continued(Some(8), false);
assert!(s.get());
}
#[test]
fn a_stop_of_every_thread_ends_with_any_resume() {
let s = Suspension::default();
s.stopped(Some(7), false);
s.stopped(Some(8), true);
s.continued(Some(8), false);
assert!(!s.get());
// A stop without a thread.
s.stopped(None, false);
s.resumed(s.ticket(), 5, false);
assert!(!s.get());
}
#[test]
fn kotlin_launch_starts_attach_listener_first() {
assert_eq!(
kotlin_jvm_args(vec!["-Xmx1g".into()], false),
vec!["-XX:+StartAttachListener".to_string(), "-Xmx1g".into()]
);
assert_eq!(
kotlin_jvm_args(vec!["-XX:+DisableAttachMechanism".into()], false),
vec!["-XX:+DisableAttachMechanism".to_string()]
);
}
#[test]
fn kotlin_launch_on_windows_gets_no_attach_listener() {
assert_eq!(kotlin_jvm_args(vec!["-Xmx1g".into()], true), vec!["-Xmx1g".to_string()]);
}
#[test]
fn reads_the_port_the_jvm_chose() {
+471 -17
Переглянути файл
@@ -1,5 +1,7 @@
//! Protocol flow against an in-memory fake adapter.
mod jvm_inspect;
use super::*;
use crate::adapter::{AdapterIo, LaunchFuture};
use ide_core::IdeEvent;
@@ -120,6 +122,14 @@ struct Script {
supports_terminate: bool,
/// Answer `setBreakpoints` only once this many are waiting (a client that waits for each never gets one).
hold_set_breakpoints: usize,
/// Send a `continued` event right after the first `stopped` one.
continued_after_stop: bool,
/// Close the connection when a `pause` request arrives (the adapter dies).
hang_up_on_pause: bool,
/// Answer `continue` with this body and keep the program running, instead of letting it exit.
continue_reply: Option<Value>,
/// Answer `initialize` only after this long (a slow adapter keeps the session connecting).
delay_initialize: Option<Duration>,
initialized_delay: Duration,
launch_delay: Duration,
/// Leave startup pending before `initialized` (false) or after configuration (true).
@@ -144,6 +154,10 @@ impl Default for Script {
output_after_terminate: false,
supports_terminate: true,
hold_set_breakpoints: 0,
continued_after_stop: false,
hang_up_on_pause: false,
continue_reply: None,
delay_initialize: None,
initialized_delay: Duration::ZERO,
launch_delay: Duration::ZERO,
pending_startup: None,
@@ -210,6 +224,9 @@ async fn fake_adapter<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>(stream
}
match msg["command"].as_str().unwrap_or_default() {
"initialize" => {
if let Some(delay) = script.delay_initialize {
tokio::time::sleep(delay).await;
}
ok!(
msg,
json!({
@@ -341,6 +358,9 @@ async fn fake_adapter<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>(stream
send!(
json!({"type": "event", "event": "stopped", "body": {"reason": "breakpoint", "threadId": 1, "allThreadsStopped": true}})
);
if script.continued_after_stop {
send!(json!({"type": "event", "event": "continued", "body": {"threadId": 1}}));
}
send!(
json!({"type": "event", "event": "breakpoint", "body": {"reason": "changed", "breakpoint": {"id": 1, "verified": false, "message": "moved", "line": 9, "source": {"path": script.source_path}}}})
);
@@ -372,11 +392,13 @@ async fn fake_adapter<S: AsyncRead + AsyncWrite + Unpin + Send + 'static>(stream
{"label": ""},
]})
),
"continue" if script.continue_reply.is_some() => ok!(msg, script.continue_reply.clone().unwrap()),
"continue" => {
ok!(msg, json!({"allThreadsContinued": true}));
send!(json!({"type": "event", "event": "exited", "body": {"exitCode": 3}}));
send!(json!({"type": "event", "event": "terminated"}));
}
"pause" if script.hang_up_on_pause => break,
"stepIn" | "stepOut" | "pause" => ok!(msg, Value::Null),
"redefineClasses" => {
send!(
@@ -430,6 +452,8 @@ fn harness(script: Script) -> Harness {
Arc::new(LanguageRegistry::from_configs(Vec::new())),
Arc::new(FakeLauncher { log: log.clone(), script }),
);
// A local attach must not send requests to the developer's own JVMs.
manager.set_jvm_finder(Arc::new(|_, _| None));
Harness { manager, log, events, owner }
}
@@ -1352,6 +1376,36 @@ async fn hot_swap_only_for_java_debug() {
assert_eq!(failed.error_message.as_deref(), Some("boom"));
}
/// A loopback port nothing listens on until `listen` is called on the socket, and that no parallel
/// test can bind meanwhile. A port picked by binding to 0 and closing the listener can be taken
/// by another test before the code under test connects to it.
fn reserved_port() -> (tokio::net::TcpSocket, u16) {
// Without SO_REUSEADDR, which every `TcpListener::bind` sets: no other socket may bind it.
let socket = tokio::net::TcpSocket::new_v4().unwrap();
socket.bind(([127, 0, 0, 1], 0).into()).unwrap();
let port = socket.local_addr().unwrap().port();
(socket, port)
}
/// An editor that starts listening on the reserved port after `delay` and serves one session.
#[cfg(unix)]
fn fake_editor(socket: tokio::net::TcpSocket, delay: Duration, log: Log) {
tokio::spawn(async move {
tokio::time::sleep(delay).await;
let listener = socket.listen(1).unwrap();
let (stream, _) = listener.accept().await.unwrap();
// Closed once its session is accepted, so a later start finds no editor open.
drop(listener);
fake_adapter(stream, log, Script::default()).await;
});
}
#[tokio::test]
async fn a_reserved_port_cannot_be_taken_by_a_parallel_test() {
let (_socket, port) = reserved_port();
assert!(tokio::net::TcpListener::bind(("127.0.0.1", port)).await.is_err(), "port {port} is still free to bind");
}
fn godot_registry(port: u16, command: &str, args: &[&str]) -> Arc<LanguageRegistry> {
let lang = ide_language::LanguageConfig {
id: ide_core::LanguageId::new("gdscript"),
@@ -1419,7 +1473,7 @@ async fn godot_connects_to_the_editor_adapter() {
#[tokio::test]
async fn godot_without_editor_or_executable_says_what_to_do() {
let port = std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap().local_addr().unwrap().port();
let (_no_editor, port) = reserved_port();
let (owner, _events) = recording_owner("main");
let manager = DebugManager::new(godot_registry(port, "openidea-no-such-godot", &[]));
let err = manager.start(godot_launch(&std::env::temp_dir()), owner).await.unwrap_err();
@@ -1432,15 +1486,9 @@ async fn godot_without_editor_or_executable_says_what_to_do() {
#[cfg(unix)]
#[tokio::test]
async fn godot_starts_a_headless_editor_when_none_is_open() {
let port = std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap().local_addr().unwrap().port();
let (editor, port) = reserved_port();
let log: Log = Arc::default();
let server_log = log.clone();
tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_millis(300)).await;
let listener = tokio::net::TcpListener::bind(("127.0.0.1", port)).await.unwrap();
let (stream, _) = listener.accept().await.unwrap();
fake_adapter(stream, server_log, Script::default()).await;
});
fake_editor(editor, Duration::from_millis(300), log.clone());
let (owner, _events) = recording_owner("main");
// `sleep` stands in for `godot --headless --editor --path ${root} …`.
let manager = DebugManager::new(godot_registry(port, "sleep", &["30"]));
@@ -1643,7 +1691,7 @@ async fn a_plugin_language_runs_its_debug_adapter_from_the_approved_program() {
std::fs::write(&program, "#!/bin/sh\nexec sleep 30\n").unwrap();
std::fs::set_permissions(&program, std::fs::Permissions::from_mode(0o755)).unwrap();
let port = std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap().local_addr().unwrap().port();
let (editor, port) = reserved_port();
let mut gdscript = parse_definition(include_str!("../../../languages/gdscript.toml")).unwrap();
let debugger = gdscript.debugger.as_mut().unwrap();
debugger.args.clear();
@@ -1660,13 +1708,7 @@ async fn a_plugin_language_runs_its_debug_adapter_from_the_approved_program() {
let _language = store.add("acme.godot".into(), gdscript);
let log: Log = Arc::default();
let server_log = log.clone();
tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_millis(300)).await;
let listener = tokio::net::TcpListener::bind(("127.0.0.1", port)).await.unwrap();
let (stream, _) = listener.accept().await.unwrap();
fake_adapter(stream, server_log, Script::default()).await;
});
fake_editor(editor, Duration::from_millis(300), log.clone());
let (owner, _events) = recording_owner("main");
let manager = DebugManager::new(registry.clone());
let id = manager.start(godot_launch(&dir), owner).await.unwrap();
@@ -2018,6 +2060,418 @@ fn delve_leftovers_are_removed_without_touching_other_files() {
std::fs::remove_dir_all(&dir).unwrap();
}
fn is_suspended(h: &Harness, id: DebugSessionId) -> bool {
h.manager.session(id).unwrap().suspended.get()
}
#[tokio::test]
async fn stopped_event_marks_session_suspended() {
let h = harness(Script::default());
let id = h.start(config()).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
assert!(is_suspended(&h, id));
}
#[tokio::test]
async fn resume_clears_suspended_without_continued_event() {
let h = harness(Script::default());
let id = h.start(config()).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
// The fake adapter ends the session after `continue`, so keep the session alive here.
let session = h.manager.session(id).unwrap();
assert!(session.suspended.get());
h.manager.resume(id, 1).await.unwrap();
assert!(!session.suspended.get());
}
/// After the fake adapter's stop of every thread ends, threads 7 then 8 stop alone, as a
/// breakpoint does on fwcd and java-debug (`SUSPEND_EVENT_THREAD`).
async fn threads_stopped(script: Script) -> (Harness, Arc<Session>, DebugSessionId) {
let h = harness(script);
let id = h.start(config()).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
let session = h.manager.session(id).unwrap();
session.suspended.continued(Some(1), true);
session.suspended.stopped(Some(7), false);
session.suspended.stopped(Some(8), false);
(h, session, id)
}
#[tokio::test]
async fn resume_of_one_thread_keeps_the_others_suspended() {
let only_it = Script { continue_reply: Some(json!({"allThreadsContinued": false})), ..Script::default() };
let (h, session, id) = threads_stopped(only_it).await;
h.manager.resume(id, 7).await.unwrap();
assert!(session.suspended.get());
h.manager.resume(id, 8).await.unwrap();
assert!(!session.suspended.get());
}
#[tokio::test]
async fn resume_reply_without_the_flag_continues_every_thread() {
let (h, session, id) = threads_stopped(Script { continue_reply: Some(json!({})), ..Script::default() }).await;
h.manager.resume(id, 7).await.unwrap();
assert!(!session.suspended.get());
}
#[tokio::test]
async fn step_of_one_thread_keeps_the_others_suspended() {
let (h, session, id) = threads_stopped(Script::default()).await;
let ticket = session.suspended.ticket();
h.manager.step(id, 8, StepKind::Into).await.unwrap();
assert!(session.suspended.get());
// Thread 8 was the latest stop: what was read on it is stale.
assert_eq!(session.suspended.ticket(), ticket + 1);
h.manager.step_in_target(id, 7, 1).await.unwrap();
assert!(!session.suspended.get());
}
#[tokio::test]
async fn continued_event_clears_suspended() {
let h = harness(Script { continued_after_stop: true, ..Script::default() });
let id = h.start(config()).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Continued { .. })).await;
assert!(!is_suspended(&h, id));
}
fn java_config(request: &str) -> LaunchConfig {
LaunchConfig { adapter: "java-debug".into(), request: request.into(), extra_json: None, ..config() }
}
fn remote_attach(adapter: &str, project: &std::path::Path) -> LaunchConfig {
LaunchConfig {
adapter: adapter.into(),
cwd: Some(project.to_owned().into()),
extra_json: Some(json!({"projectRoot": project, "hostName": "10.0.0.5", "port": 5005}).to_string()),
..java_config("attach")
}
}
#[tokio::test]
async fn attach_session_reports_capability() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
for adapter in ["java-debug", kotlin::ADAPTER] {
let id = h.start(remote_attach(adapter, project.path())).await.unwrap();
let caps = h.manager.capabilities(id).unwrap();
assert!(caps.jvm_memory, "{adapter}");
assert_eq!(caps.jvm_heap, caps.jvm_memory, "{adapter}");
}
let id = h.start(config()).await.unwrap();
let caps = h.manager.capabilities(id).unwrap();
assert!(!caps.jvm_memory);
assert_eq!(caps.jvm_heap, caps.jvm_memory);
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable { reason: JvmMemoryUnavailable::NoJvmProcess, can_count_when_paused: false }
);
}
#[cfg(unix)]
#[tokio::test]
async fn ended_attach_session_without_pid_reports_program_exited() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let id = h.start(remote_attach("java-debug", project.path())).await.unwrap();
h.manager.session(id).unwrap().terminated.store(true, Ordering::Release);
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable { reason: JvmMemoryUnavailable::ProgramExited, can_count_when_paused: false }
);
}
#[cfg(unix)]
#[tokio::test]
async fn listen_mode_attach_is_not_scanned() {
let h = harness(Script::default());
let scans = Arc::new(Mutex::new(Vec::new()));
let counted = scans.clone();
h.manager.set_jvm_finder(Arc::new(move |port, _| {
counted.lock().push(port);
None
}));
let (_jvm, relay) = jdwp::tests::connected_jvm().await;
let mut cfg = java_config("attach");
cfg.extra_json = Some(json!({"hostName": "127.0.0.1", "port": relay}).to_string());
let id = h.start(cfg).await.unwrap();
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable {
reason: JvmMemoryUnavailable::ListenModeNotSupported,
can_count_when_paused: false
}
);
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(scans.lock().is_empty());
}
#[cfg(unix)]
#[tokio::test]
async fn remote_host_is_not_scanned() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let id = h.start(remote_attach("java-debug", project.path())).await.unwrap();
let session = h.manager.session(id).unwrap();
assert_eq!(session.jdwp_port.get(), None);
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable {
reason: JvmMemoryUnavailable::AttachedJvmNotFound,
can_count_when_paused: false
}
);
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 0);
}
/// The Memory view asks as soon as the session appears, while its JVM is still being looked up:
/// answering then would claim there is no JVM to ask.
#[cfg(unix)]
#[tokio::test]
async fn memory_asked_while_an_attach_connects_waits_for_its_jvm() {
let h = harness(Script { delay_initialize: Some(Duration::from_millis(300)), ..Script::default() });
let project = tempfile::tempdir().unwrap();
let (manager, owner) = (h.manager.clone(), h.owner.clone());
let config = remote_attach("java-debug", project.path());
let start = tokio::spawn(async move { manager.start(config, owner).await });
h.wait_event(|e| matches!(e, DebugEvent::Started { .. })).await;
let id = DebugSessionId(*event_sessions(&h.events).first().unwrap());
let session = h.manager.session(id).unwrap();
assert!(session.attach_target.get().is_none(), "asked too late to cover the connecting session");
h.manager.jvm_memory_classes(id).await.unwrap();
assert!(session.attach_target.get().is_some(), "answered before the attach was looked at");
start.await.unwrap().unwrap();
}
/// A JVM clears its JDWP listener address once a debugger connects, so only a scan made before
/// the `attach` request can tell it by its port.
#[cfg(unix)]
#[tokio::test]
async fn local_attach_finds_its_jvm_before_the_debugger_connects() {
let h = harness(Script::default());
let log = h.log.clone();
h.manager.set_jvm_finder(Arc::new(move |port, _| {
std::thread::sleep(Duration::from_millis(50));
let attached = log.lock().iter().any(|m| m["command"] == "attach");
(port == 5005 && !attached).then_some(4242)
}));
let mut cfg = java_config("attach");
cfg.extra_json = Some(json!({"hostName": "127.0.0.1", "port": 5005}).to_string());
let id = h.start(cfg).await.unwrap();
let session = h.manager.session(id).unwrap();
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 4242);
}
#[cfg(unix)]
#[tokio::test]
async fn local_attach_remembers_its_jdwp_port() {
let h = harness(Script::default());
let ports = Arc::new(Mutex::new(Vec::new()));
let asked = ports.clone();
h.manager.set_jvm_finder(Arc::new(move |port, _| {
asked.lock().push(port);
Some(4242)
}));
let mut cfg = java_config("attach");
cfg.extra_json = Some(json!({"hostName": "localhost", "port": "5005"}).to_string());
let id = h.start(cfg).await.unwrap();
let session = h.manager.session(id).unwrap();
assert_eq!(session.jdwp_port.get(), Some(&5005));
assert_eq!(h.request("attach")["arguments"]["port"], "5005");
for _ in 0..200 {
if session.jvm_pid.load(Ordering::Acquire) != 0 {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 4242);
assert_eq!(ports.lock().first(), Some(&5005));
}
/// A fake `find_jvm_by_jdwp_port` for the JVM 4242, recording whether each scan ran while
/// suspended. A JVM not `safe_while_suspended` (JDK below 17) is skipped by a suspended scan.
#[cfg(unix)]
fn counting_find(
safe_while_suspended: bool,
) -> (Arc<Mutex<Vec<bool>>>, impl Fn(bool) -> Option<u32> + Clone + Send + 'static) {
let scans = Arc::new(Mutex::new(Vec::new()));
let counted = scans.clone();
(scans, move |suspended| {
counted.lock().push(suspended);
(!suspended || safe_while_suspended).then_some(4242)
})
}
#[cfg(unix)]
async fn suspended_remote_attach(h: &Harness) -> Arc<Session> {
let project = tempfile::tempdir().unwrap();
let id = h.start(remote_attach("java-debug", project.path())).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
let session = h.manager.session(id).unwrap();
assert!(session.suspended.get());
session
}
#[cfg(unix)]
#[tokio::test]
async fn background_scan_skips_an_old_jdk_while_suspended() {
let h = harness(Script::default());
let session = suspended_remote_attach(&h).await;
let (scans, find) = counting_find(false);
let task =
jvm_memory::watch_for_attached_jvm(&session, 5005, find, Duration::from_millis(5), Duration::from_secs(5));
tokio::time::sleep(Duration::from_millis(100)).await;
assert!(!scans.lock().is_empty() && scans.lock().iter().all(|&suspended| suspended));
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 0);
session.suspended.continued(None, true);
tokio::time::timeout(Duration::from_secs(2), task).await.unwrap().unwrap();
assert_eq!(scans.lock().last(), Some(&false));
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 4242);
}
#[cfg(unix)]
#[tokio::test]
async fn background_scan_finds_a_jdk_17_or_later_while_suspended() {
let h = harness(Script::default());
let session = suspended_remote_attach(&h).await;
let (scans, find) = counting_find(true);
let task =
jvm_memory::watch_for_attached_jvm(&session, 5005, find, Duration::from_millis(5), Duration::from_secs(5));
tokio::time::timeout(Duration::from_secs(2), task).await.unwrap().unwrap();
assert_eq!(*scans.lock(), [true]);
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 4242);
}
#[cfg(unix)]
#[tokio::test]
async fn background_scan_keeps_a_known_pid_and_stops_with_the_session() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let id = h.start(remote_attach("java-debug", project.path())).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
let session = h.manager.session(id).unwrap();
session.suspended.continued(None, true);
session.jvm_pid.store(7, Ordering::Release);
let watch = |find| {
jvm_memory::watch_for_attached_jvm(&session, 5005, find, Duration::from_millis(5), Duration::from_secs(5))
};
let (scans, find) = counting_find(true);
tokio::time::timeout(Duration::from_secs(2), watch(find)).await.unwrap().unwrap();
assert!(scans.lock().is_empty(), "a known pid needs no scan");
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 7);
// A refresh stores the pid while this scan is still running.
session.jvm_pid.store(0, Ordering::Release);
let racing = session.clone();
let find = move |_| {
racing.jvm_pid.store(7, Ordering::Release);
Some(4242)
};
let task =
jvm_memory::watch_for_attached_jvm(&session, 5005, find, Duration::from_millis(5), Duration::from_secs(5));
tokio::time::timeout(Duration::from_secs(2), task).await.unwrap().unwrap();
assert_eq!(session.jvm_pid.load(Ordering::Acquire), 7, "a pid stored first is kept");
h.manager.stop(id).await.unwrap();
let (scans, find) = counting_find(true);
tokio::time::timeout(Duration::from_secs(2), watch(find)).await.unwrap().unwrap();
assert!(scans.lock().is_empty());
}
#[cfg(unix)]
#[tokio::test]
async fn unknown_pid_reports_no_jvm_process() {
let h = harness(Script::default());
let id = h.start(java_config("launch")).await.unwrap();
assert!(h.manager.capabilities(id).unwrap().jvm_memory);
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable { reason: JvmMemoryUnavailable::NoJvmProcess, can_count_when_paused: false }
);
}
fn kotlin_attach(project: &std::path::Path, host: &str, port: u16) -> LaunchConfig {
LaunchConfig {
adapter: kotlin::ADAPTER.into(),
cwd: Some(project.to_owned().into()),
extra_json: Some(json!({"projectRoot": project, "hostName": host, "port": port}).to_string()),
..java_config("attach")
}
}
#[tokio::test]
async fn kotlin_session_installs_link_inspector() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let id = h.start(kotlin_attach(project.path(), "127.0.0.1", 5005)).await.unwrap();
let session = h.manager.session(id).unwrap();
assert!(session.jdwp_link.lock().is_some());
assert!(session.jvm_inspector.lock().is_some());
}
#[tokio::test]
async fn attach_target_recorded_before_kotlin_rewrite() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let id = h.start(kotlin_attach(project.path(), "10.0.0.5", 5005)).await.unwrap();
let session = h.manager.session(id).unwrap();
assert_eq!(session.attach_target.get(), Some(&("10.0.0.5".to_owned(), 5005)));
let sent = &h.request("attach")["arguments"];
assert_eq!(sent["hostName"], "127.0.0.1");
let link_port = sent["port"].as_u64().unwrap();
assert!(link_port != 0 && link_port != 5005);
}
#[tokio::test]
async fn java_attach_records_target() {
let h = harness(Script::default());
let mut cfg = java_config("attach");
cfg.extra_json = Some(json!({"hostName": "localhost", "port": 5005}).to_string());
let id = h.start(cfg).await.unwrap();
let session = h.manager.session(id).unwrap();
assert_eq!(session.attach_target.get(), Some(&("localhost".to_owned(), 5005)));
assert!(session.jdwp_link.lock().is_none());
assert!(session.jvm_inspector.lock().is_none());
}
#[tokio::test]
async fn attach_target_recorded_for_any_adapter() {
let h = harness(Script::default());
let mut cfg = java_config("attach");
// "x" is not a registered adapter; debugpy has no JVM link or memory support.
cfg.adapter = "debugpy".into();
cfg.extra_json = Some(json!({"hostName": "h", "port": 7}).to_string());
let id = h.start(cfg).await.unwrap();
assert_eq!(h.manager.session(id).unwrap().attach_target.get(), Some(&("h".to_owned(), 7)));
}
#[tokio::test]
async fn launch_sessions_have_no_attach_target() {
let h = harness(Script::default());
let id = h.start(java_config("launch")).await.unwrap();
assert_eq!(h.manager.session(id).unwrap().attach_target.get(), None);
}
#[tokio::test]
async fn closing_the_session_drops_the_link() {
let h = harness(Script { hang_up_on_pause: true, ..Script::default() });
let project = tempfile::tempdir().unwrap();
let id = h.start(kotlin_attach(project.path(), "127.0.0.1", 5005)).await.unwrap();
let session = h.manager.session(id).unwrap();
assert!(session.jdwp_link.lock().is_some() && session.jvm_inspector.lock().is_some());
let _ = session.conn.request_timeout("pause", json!({"threadId": 1}), Duration::from_secs(2)).await;
for _ in 0..200 {
if h.manager.session(id).is_err() {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(session.jdwp_link.lock().is_none());
assert!(session.jvm_inspector.lock().is_none());
}
#[cfg(unix)]
#[tokio::test]
async fn rdbg_drains_test_output_after_dap_closes_before_killing_the_process() {
+401
Переглянути файл
@@ -0,0 +1,401 @@
//! Instances, fields and referrers through a session's JVM inspector.
use super::*;
use crate::jvm_memory::test_support::FakeInspector;
use ide_core::event::JvmBackendState;
use ide_jvm_inspect::{InspectError, JvmValue, VmTarget};
use tokio::io::AsyncReadExt;
fn debug_events(h: &Harness) -> Vec<DebugEvent> {
h.events
.lock()
.iter()
.filter_map(|e| match e {
IdeEvent::Debug { event, .. } => Some(event.clone()),
_ => None,
})
.collect()
}
fn backend_events(h: &Harness) -> Vec<JvmBackendState> {
debug_events(h)
.into_iter()
.filter_map(|e| match e {
DebugEvent::JvmBackend { state } => Some(state),
_ => None,
})
.collect()
}
/// A java-debug session stopped at its breakpoint, inspected by `fake`.
async fn stopped_with(h: &Harness, config: LaunchConfig, fake: &Arc<FakeInspector>) -> DebugSessionId {
let id = h.start(config).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
h.manager.set_jvm_inspector(id, fake.clone()).unwrap();
id
}
fn object(id: u64) -> JvmValue {
JvmValue::Object { id, type_name: "A".into() }
}
const NOT_SUSPENDED: JvmInstancesResult =
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::NotSuspended };
#[tokio::test]
async fn instances_need_a_suspended_session() {
let h = harness(Script { continued_after_stop: true, ..Script::default() });
let id = h.start(java_config("launch")).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Continued { .. })).await;
let fake = Arc::new(FakeInspector::default());
h.manager.set_jvm_inspector(id, fake.clone()).unwrap();
assert_eq!(h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap(), NOT_SUSPENDED);
assert_eq!(fake.calls(), 0);
}
#[tokio::test]
async fn instances_without_an_inspector_have_no_backend() {
let h = harness(Script::default());
let id = h.start(java_config("launch")).await.unwrap();
h.wait_event(|e| matches!(e, DebugEvent::Stopped { .. })).await;
assert_eq!(
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::NoBackend }
);
}
#[tokio::test]
async fn object_from_previous_stop_is_stale() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::default());
fake.instances.lock().push(object(7));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let JvmInstancesResult::Ready { generation, instances, truncated } =
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap()
else {
panic!("instances unavailable");
};
assert_eq!((generation, instances, truncated), (1, vec![object(7)], false));
h.manager.session(id).unwrap().suspended.stopped(None, true);
let stale = JvmObjectRef { generation, id: 7 };
assert_eq!(
h.manager.jvm_memory_fields(id, stale).await.unwrap(),
JvmFieldsResult::Unavailable { reason: JvmInspectUnavailable::Stale }
);
assert_eq!(
h.manager.jvm_memory_referrers(id, stale, 10).await.unwrap(),
JvmReferrersResult::Unavailable { reason: JvmInspectUnavailable::Stale }
);
assert_eq!(fake.calls(), 1, "a stale object never reaches the inspector");
let current = JvmObjectRef { generation: 2, id: 7 };
assert_eq!(h.manager.jvm_memory_fields(id, current).await.unwrap(), JvmFieldsResult::Ready { fields: Vec::new() });
}
#[tokio::test]
async fn instances_stay_readable_while_another_thread_is_stopped() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::default());
fake.instances.lock().push(object(7));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let session = h.manager.session(id).unwrap();
// Threads 20, 21, then 22 stop alone (SUSPEND_EVENT_THREAD).
session.suspended.continued(Some(1), true);
for thread in [20, 21, 22] {
session.suspended.stopped(Some(thread), false);
}
let instances = || h.manager.jvm_memory_instances(id, "A".into(), 10);
let result = instances().await.unwrap();
let JvmInstancesResult::Ready { generation, .. } = result else { panic!("{result:?}") };
let read_on_22 = JvmObjectRef { generation, id: 7 };
// An earlier thread resumes: the stop on 22 goes on, and so does what was read on it.
session.suspended.continued(Some(20), false);
assert_eq!(
h.manager.jvm_memory_fields(id, read_on_22).await.unwrap(),
JvmFieldsResult::Ready { fields: Vec::new() }
);
// The latest one resumes: 21 is still stopped, but what was read on 22 is stale.
session.suspended.continued(Some(22), false);
assert_eq!(
h.manager.jvm_memory_fields(id, read_on_22).await.unwrap(),
JvmFieldsResult::Unavailable { reason: JvmInspectUnavailable::Stale }
);
let result = instances().await.unwrap();
assert!(matches!(result, JvmInstancesResult::Ready { .. }), "{result:?}");
}
/// Starts `instances` on a held fake inspector; returns once the inspector was entered.
async fn held_request(
h: &Harness,
id: DebugSessionId,
fake: &Arc<FakeInspector>,
) -> tokio::task::JoinHandle<IdeResult<JvmInstancesResult>> {
let manager = h.manager.clone();
let request = tokio::spawn(async move { manager.jvm_memory_instances(id, "A".into(), 10).await });
tokio::time::timeout(Duration::from_secs(2), fake.entered.notified()).await.unwrap();
request
}
#[tokio::test]
async fn stop_during_request_is_stale() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector { hold: true, ..FakeInspector::default() });
fake.instances.lock().push(object(7));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let request = held_request(&h, id, &fake).await;
h.manager.session(id).unwrap().suspended.stopped(None, true);
fake.release.notify_one();
assert_eq!(
request.await.unwrap().unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::Stale }
);
}
#[tokio::test]
async fn resume_during_request_is_stale() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector { hold: true, ..FakeInspector::default() });
let id = stopped_with(&h, java_config("launch"), &fake).await;
let request = held_request(&h, id, &fake).await;
h.manager.session(id).unwrap().suspended.continued(None, true);
fake.release.notify_one();
assert_eq!(
request.await.unwrap().unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::Stale }
);
}
#[tokio::test]
async fn one_request_at_a_time_per_session() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector { hold: true, ..FakeInspector::default() });
let id = stopped_with(&h, java_config("launch"), &fake).await;
let first = held_request(&h, id, &fake).await;
let manager = h.manager.clone();
let second = tokio::spawn(async move { manager.jvm_memory_instances(id, "B".into(), 10).await });
tokio::time::sleep(Duration::from_millis(50)).await;
assert_eq!(fake.calls(), 1, "the second request waits for the first");
fake.release.notify_one();
first.await.unwrap().unwrap();
tokio::time::timeout(Duration::from_secs(2), fake.entered.notified()).await.unwrap();
fake.release.notify_one();
second.await.unwrap().unwrap();
assert_eq!(fake.calls(), 2);
}
#[tokio::test]
async fn session_end_lets_go_of_a_waiting_inspector() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector { hold: true, ..FakeInspector::default() });
let id = stopped_with(&h, java_config("launch"), &fake).await;
let request = held_request(&h, id, &fake).await;
h.manager.session(id).unwrap().kill_processes();
assert_eq!(
tokio::time::timeout(Duration::from_secs(2), request).await.unwrap().unwrap().unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::BackendClosed }
);
assert_eq!(Arc::strong_count(&fake), 1, "no clone of the inspector outlives the session");
}
#[tokio::test]
async fn target_carries_pid_for_launch() {
let h = harness(Script { debuggee_pid: Some(4242), java_process_id: true, ..Script::default() });
let fake = Arc::new(FakeInspector::default());
let id = stopped_with(&h, java_config("launch"), &fake).await;
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap();
assert_eq!(
*fake.targets.lock(),
[VmTarget { pid: Some(4242), attach_host: None, attach_port: None, generation: 1 }]
);
}
#[tokio::test]
async fn target_carries_attach_port_and_host() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let fake = Arc::new(FakeInspector::default());
let id = stopped_with(&h, remote_attach("java-debug", project.path()), &fake).await;
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap();
assert_eq!(
*fake.targets.lock(),
[VmTarget { pid: None, attach_host: Some("10.0.0.5".into()), attach_port: Some(5005), generation: 1 }]
);
}
#[tokio::test]
async fn unsupported_backend_turns_instances_off() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::failing(InspectError::Unsupported("x".into())));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let before = debug_events(&h).len();
assert_eq!(
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::Unsupported { detail: "x".into() } }
);
let after: Vec<_> = debug_events(&h).split_off(before);
assert!(
matches!(
after.as_slice(),
[DebugEvent::JvmBackend { state: JvmBackendState::Unsupported { detail } }, DebugEvent::Capabilities]
if detail == "x"
),
"{after:?}"
);
assert!(!h.manager.capabilities(id).unwrap().jvm_instances);
assert_eq!(
h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap(),
JvmInstancesResult::Unavailable { reason: JvmInspectUnavailable::NoBackend }
);
assert_eq!(fake.calls(), 1);
}
/// `error` is reported but leaves the backend on.
async fn keeps_the_backend(error: InspectError, reason: JvmInspectUnavailable) {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::failing(error));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let object = JvmObjectRef { generation: 1, id: 7 };
assert_eq!(h.manager.jvm_memory_fields(id, object).await.unwrap(), JvmFieldsResult::Unavailable { reason });
assert!(h.manager.capabilities(id).unwrap().jvm_instances);
assert!(backend_events(&h).is_empty());
assert!(h.manager.session(id).unwrap().jvm_inspector.lock().is_some());
}
#[tokio::test]
async fn collected_keeps_the_backend() {
keeps_the_backend(InspectError::Collected, JvmInspectUnavailable::Collected).await;
}
#[tokio::test]
async fn timed_out_keeps_the_backend() {
keeps_the_backend(InspectError::TimedOut, JvmInspectUnavailable::TimedOut).await;
}
#[tokio::test]
async fn failed_inspection_is_an_error() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::failing(InspectError::Failed("boom".into())));
let id = stopped_with(&h, java_config("launch"), &fake).await;
let err = h.manager.jvm_memory_instances(id, "A".into(), 10).await.unwrap_err();
assert!(err.to_string().contains("boom"), "{err}");
assert!(h.manager.capabilities(id).unwrap().jvm_instances);
}
#[tokio::test]
async fn limits_are_clamped() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::default());
let id = stopped_with(&h, java_config("launch"), &fake).await;
for limit in [5000, 0] {
h.manager.jvm_memory_instances(id, "A".into(), limit).await.unwrap();
}
let object = JvmObjectRef { generation: 1, id: 7 };
for limit in [0, 900, 20] {
h.manager.jvm_memory_referrers(id, object, limit).await.unwrap();
}
assert_eq!(*fake.limits.lock(), [1000, 1, 50, 200, 20]);
}
#[tokio::test]
async fn a_full_page_is_truncated() {
let h = harness(Script::default());
let fake = Arc::new(FakeInspector::default());
fake.instances.lock().extend([object(1), object(2)]);
let id = stopped_with(&h, java_config("launch"), &fake).await;
let truncated = |r: JvmInstancesResult| match r {
JvmInstancesResult::Ready { truncated, .. } => truncated,
other => panic!("{other:?}"),
};
assert!(truncated(h.manager.jvm_memory_instances(id, "A".into(), 2).await.unwrap()));
assert!(!truncated(h.manager.jvm_memory_instances(id, "A".into(), 3).await.unwrap()));
}
#[tokio::test]
async fn capability_follows_the_inspector() {
let h = harness(Script::default());
let id = h.start(java_config("launch")).await.unwrap();
assert!(!h.manager.capabilities(id).unwrap().jvm_instances);
let before = debug_events(&h).len();
h.manager.set_jvm_inspector(id, Arc::new(FakeInspector::default())).unwrap();
assert!(h.manager.capabilities(id).unwrap().jvm_instances);
assert!(matches!(debug_events(&h).split_off(before).as_slice(), [DebugEvent::Capabilities]));
}
#[tokio::test]
async fn remote_attach_counts_classes_through_the_debugger() {
let h = harness(Script::default());
let project = tempfile::tempdir().unwrap();
let fake = Arc::new(FakeInspector::default());
fake.counts.lock().push(ide_jvm_inspect::ClassCount { name: "A".into(), instances: 3 });
let id = stopped_with(&h, remote_attach("java-debug", project.path()), &fake).await;
let result = h.manager.jvm_memory_classes(id).await.unwrap();
assert!(
matches!(&result, JvmMemoryResult::Ready { snapshot }
if snapshot.source == JvmMemorySource::Debugger && snapshot.rows.len() == 1),
"{result:?}"
);
assert_eq!(fake.targets.lock()[0].attach_port, Some(5005));
}
const HANDSHAKE: &[u8; 14] = b"JDWP-Handshake";
/// A Kotlin attach whose link reaches a fake JVM: (adapter end, JVM end) after both handshakes.
async fn linked_kotlin(h: &Harness) -> (DebugSessionId, tokio::net::TcpStream, tokio::net::TcpStream) {
let jvm = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let project = tempfile::tempdir().unwrap();
let id = h.start(kotlin_attach(project.path(), "127.0.0.1", jvm.local_addr().unwrap().port())).await.unwrap();
let link_port = h.request("attach")["arguments"]["port"].as_u64().unwrap() as u16;
let mut adapter = tokio::net::TcpStream::connect(("127.0.0.1", link_port)).await.unwrap();
adapter.write_all(HANDSHAKE).await.unwrap();
let (mut jvm, _) = tokio::time::timeout(Duration::from_secs(2), jvm.accept()).await.unwrap().unwrap();
let mut buf = [0; 14];
jvm.read_exact(&mut buf).await.unwrap();
jvm.write_all(HANDSHAKE).await.unwrap();
adapter.read_exact(&mut buf).await.unwrap();
(id, adapter, jvm)
}
/// A command the adapter sends with an id from the link's own range: the link then only forwards.
async fn degrade(adapter: &mut tokio::net::TcpStream) {
let mut packet = 11u32.to_be_bytes().to_vec();
packet.extend(ide_jdwp::OWN_IDS.start.to_be_bytes());
packet.extend([0, 1, 1]);
adapter.write_all(&packet).await.unwrap();
}
async fn wait_backend(h: &Harness, want: impl Fn(&JvmBackendState) -> bool) {
h.wait_event(|e| matches!(e, DebugEvent::JvmBackend { state } if want(state))).await;
}
#[tokio::test]
async fn kotlin_link_reports_degraded_then_closed() {
let h = harness(Script::default());
let (id, mut adapter, jvm) = linked_kotlin(&h).await;
degrade(&mut adapter).await;
wait_backend(&h, |s| matches!(s, JvmBackendState::Degraded { .. })).await;
drop(jvm);
wait_backend(&h, |s| *s == JvmBackendState::Closed).await;
assert_eq!(backend_events(&h).len(), 2, "{:?}", backend_events(&h));
let session = h.manager.session(id).unwrap();
assert!(session.jdwp_link.lock().is_some(), "a closed link is kept until the session ends");
}
#[tokio::test]
async fn kotlin_link_is_quiet_once_the_program_ended() {
let h = harness(Script::default());
let (id, _adapter, jvm) = linked_kotlin(&h).await;
let session = h.manager.session(id).unwrap();
// The adapter reported the end; the session keeps its link until it shuts down.
session.terminated.store(true, Ordering::Release);
drop(jvm);
for _ in 0..100 {
if session.jdwp_link.lock().as_ref().is_some_and(|l| l.state() == ide_jdwp::LinkState::Closed) {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(backend_events(&h).is_empty(), "{:?}", backend_events(&h));
}
+40
Переглянути файл
@@ -352,6 +352,22 @@ pub struct IndexEntry {
/// the IDE before showing it.
#[serde(default)]
pub icon: Option<PluginIcon>,
/// Where to read more: the manifest's homepage, else the registry's source repository.
#[serde(default)]
pub homepage: Option<String>,
/// This version's release notes (the manifest's `change-notes`).
#[serde(default)]
pub notes: Option<String>,
/// Downloads of the plugin's packages, as counted by whoever serves them.
#[serde(default)]
pub downloads: Option<u64>,
/// Picked by the marketplace's maintainers.
#[serde(default)]
pub featured: bool,
/// Project-relative paths whose presence makes the plugin worth suggesting
/// (`pom.xml`, `config/routes.rb`).
#[serde(default)]
pub suggest: Vec<String>,
}
impl IndexEntry {
@@ -685,9 +701,33 @@ mod tests {
engine: "^1.0".into(),
categories: Vec::new(),
icon: None,
homepage: None,
notes: None,
downloads: None,
featured: false,
suggest: Vec::new(),
}
}
#[test]
fn listing_details_are_optional() {
let mut json = serde_json::to_value(entry("acme.a", "acme")).unwrap();
for key in ["homepage", "notes", "downloads", "featured", "suggest"] {
json.as_object_mut().unwrap().remove(key);
}
assert_eq!(serde_json::from_value::<IndexEntry>(json).unwrap(), entry("acme.a", "acme"));
let full = IndexEntry {
homepage: Some("https://example.com".into()),
notes: Some("Fixes".into()),
downloads: Some(42),
featured: true,
suggest: vec!["pom.xml".into()],
..entry("acme.a", "acme")
};
let back: IndexEntry = serde_json::from_str(&serde_json::to_string(&full).unwrap()).unwrap();
assert_eq!(back, full);
}
fn version(id: &str, version: &str, engine: &str) -> IndexEntry {
IndexEntry { version: version.into(), engine: engine.into(), ..entry(id, "acme") }
}
+28
Переглянути файл
@@ -0,0 +1,28 @@
[package]
name = "ide-heap"
description = "HotSpot .hprof parser and heap-snapshot analysis for the debugger's JVM Memory view"
version.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
ide-core.workspace = true
memmap2.workspace = true
serde.workspace = true
serde_json.workspace = true
specta.workspace = true
thiserror.workspace = true
tracing.workspace = true
uuid.workspace = true
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[dev-dependencies]
criterion = { version = "0.5", default-features = false }
proptest.workspace = true
tempfile.workspace = true
[[bench]]
name = "heap"
harness = false
+100
Переглянути файл
@@ -0,0 +1,100 @@
//! `cargo bench -p ide-heap`: whole-dump analysis and the dominator tree at a million objects.
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use ide_core::CancellationToken;
use ide_heap::testing::{MiniHeap, XorShift};
use ide_heap::{AnalyzeOptions, BasicType, HeapError, HeapPhase, HeapRootKind, Store, analyze, dominators};
const CHAINS: u64 = 100_000;
const CHAIN_LEN: u64 = 5;
const NODES: u64 = CHAINS * CHAIN_LEN;
/// Every object takes one slot: node `k` at slot `2k`, its payload at `2k + 1`.
const BASE: u64 = 0x1_0000_0000;
const SLOT: u64 = 0x20;
fn node_addr(k: u64) -> u64 {
BASE + 2 * k * SLOT
}
fn payload_addr(k: u64) -> u64 {
node_addr(k) + SLOT
}
/// 100k rooted chains of five `Node { next, payload }`, each node with its own `byte[16]`. Every
/// seventh node points at the payload of the same position in the next chain instead, so that
/// payload has two owners and its own one is garbage: 1M objects in all.
fn million_objects() -> MiniHeap {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("payload", BasicType::Object)]);
for k in 0..NODES {
let next = if (k + 1) % CHAIN_LEN == 0 { 0 } else { node_addr(k + 1) };
let payload = if k % 7 == 0 { payload_addr((k + CHAIN_LEN) % NODES) } else { payload_addr(k) };
heap.object(node_addr(k), "Node", &[("next", next), ("payload", payload)]);
heap.prim(payload_addr(k), BasicType::Byte, 16);
if k % CHAIN_LEN == 0 {
heap.root(HeapRootKind::JniGlobal, node_addr(k));
}
}
heap
}
/// CSR of a random tree on `nodes` nodes rooted at 0 plus `extra` random edges, targets of each
/// node in scrambled order.
fn random_graph(nodes: u32, extra: u32, seed: u64) -> (Vec<u32>, Vec<u32>) {
let mut rng = XorShift::new(seed);
let mut edges: Vec<(u32, u32)> = (1..nodes).map(|v| (rng.below(v), v)).collect();
edges.extend((0..extra).map(|_| (rng.below(nodes), rng.below(nodes))));
for i in (1..edges.len()).rev() {
edges.swap(i, rng.below(i as u32 + 1) as usize);
}
let mut start = vec![0u32; nodes as usize + 1];
for &(from, _) in &edges {
start[from as usize + 1] += 1;
}
for v in 0..nodes as usize {
start[v + 1] += start[v];
}
let mut cursor = start.clone();
let mut out = vec![0u32; edges.len()];
for &(from, to) in &edges {
out[cursor[from as usize] as usize] = to;
cursor[from as usize] += 1;
}
(start, out)
}
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn bench(c: &mut Criterion) {
let dir = tempfile::tempdir().expect("temp dir");
let hprof = dir.path().join("dump.hprof");
million_objects().write_to(&hprof);
let (start, out) = random_graph(1_000_000, 500_000, 0x5EED_D0D0);
let cancel = CancellationToken::new();
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
let summary = analyze(&hprof, dir.path(), opts, &no_progress, &cancel).expect("analyze").summary;
assert_eq!(summary.objects, 2 * NODES, "{summary:?}");
assert_eq!(summary.unreachable_objects, NODES.div_ceil(7), "{summary:?}");
let mut g = c.benchmark_group("heap");
g.sample_size(10);
g.bench_function("index_1m_objects", |b| {
b.iter(|| {
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
let index = analyze(&hprof, dir.path(), opts, &no_progress, &cancel).expect("analyze");
black_box(index.summary.objects)
})
});
g.bench_function("dominators_1m_random", |b| {
b.iter(|| {
let d = dominators(&start, &out, 0, &Store::Ram, &no_progress, &cancel).expect("dominators");
black_box(d.reachable)
})
});
g.finish();
}
criterion_group!(benches, bench);
criterion_main!(benches);
+202
Переглянути файл
@@ -0,0 +1,202 @@
//! Ties the passes together: graph, dominators, shallow and retained sizes, the class table and
//! paths to GC roots, into a [`HeapIndex`].
mod classes;
mod paths;
mod retained;
use std::collections::HashMap;
use std::fs::File;
use std::path::Path;
use std::time::Instant;
use ide_core::CancellationToken;
use memmap2::Mmap;
use crate::column::Store;
use crate::dominators::{Dominators, dominators};
use crate::error::HeapError;
use crate::files::INDEX_FILES;
use crate::graph::{HeapGraph, build_graph};
use crate::index::{HeapIndex, HeapSummary, write_summary};
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
use crate::shallow::{HistogramEntry, ShallowModel, VmLayout};
#[derive(Clone, Debug, Default)]
pub struct AnalyzeOptions {
/// The JVM's class histogram from the same stop as the dump.
pub histogram: Option<Vec<HistogramEntry>>,
/// `VM.flags` output of the JVM that wrote the dump.
pub vm_flags: Option<String>,
pub java_major: Option<u32>,
/// Keep every array in RAM and write nothing to `index_dir`.
pub in_memory: bool,
}
/// Analyzes the dump at `hprof`, writing the index columns and `summary.json` into `index_dir`
/// (which must exist and may hold the dump itself). `progress` gets every [`HeapPhase`] in order;
/// an `Err` from it, a cancel or any other error removes every index file from `index_dir` and
/// leaves the rest of the directory alone.
pub fn analyze(
hprof: &Path,
index_dir: &Path,
opts: AnalyzeOptions,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapIndex, HeapError> {
let store = if opts.in_memory { Store::Ram } else { Store::Dir(index_dir.to_path_buf()) };
let result = run(hprof, &store, opts, progress, cancel);
// `run` has dropped every column by now: Windows cannot delete a mapped file.
if result.is_err() && !matches!(store, Store::Ram) {
remove_index_files(index_dir);
}
result
}
fn run(
hprof: &Path,
store: &Store,
opts: AnalyzeOptions,
report: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapIndex, HeapError> {
let started = Instant::now();
let progress = Progress::new(report, cancel);
let file = File::open(hprof)?;
// SAFETY: nothing writes the dump while it is analyzed or viewed: a capture finishes writing
// it before analysis and deletes it only with the snapshot that holds this index.
let dump = unsafe { Mmap::map(&file)? };
let graph = build_graph(&dump, store, report, cancel)?;
let Dominators { idom, preorder, .. } =
dominators(&graph.out_start, &graph.out, graph.root, store, report, cancel)?;
let nodes = u64::from(graph.root) + 1;
// Seven walks over the nodes, roughly: sizes, retained, child counts, child sorting, class
// counts and the tree walk, which visits every node twice.
let mut meter = Meter::start(&progress, HeapPhase::Retained, 7 * nodes)?;
let layout = VmLayout::from_vm_flags(opts.vm_flags.as_deref().unwrap_or(""), opts.java_major);
let model = ShallowModel::new(&graph, layout, opts.histogram.as_deref(), cancel)?;
let (shallow8, retained) = retained::sizes(&graph, &model, &idom, &preorder, store, &mut meter)?;
drop(preorder);
let (kids_start, kids) = retained::children(&idom, &retained, graph.root, store, &mut meter)?;
let tree = retained::Tree { idom, shallow8, retained, kids_start, kids };
let (classes, totals) = classes::table(&graph, &tree, store, &mut meter)?;
meter.finish()?;
let mut meter = Meter::start(&progress, HeapPhase::Paths, nodes + graph.out.len() as u64)?;
let gc_parent = paths::gc_parents(&graph, store, &mut meter)?;
meter.finish()?;
progress.report(HeapPhase::Saving, 0, 1)?;
let HeapGraph {
ids,
types,
offsets,
roots,
classes: class_info,
class_slot,
thread_objects,
weak_edges_skipped,
id_size,
..
} = graph;
let summary = HeapSummary {
objects: totals.reachable_objects + totals.unreachable_objects,
reachable_objects: totals.reachable_objects,
reachable_bytes: totals.reachable_bytes,
unreachable_objects: totals.unreachable_objects,
unreachable_bytes: totals.unreachable_bytes,
classes: class_info.iter().filter(|c| c.id != 0).count() as u32,
weak_edges_skipped,
size_source: model.source,
analysis_ms: u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX),
};
if let Store::Dir(dir) = store {
write_summary(dir, &summary)?;
}
let mut root_kinds = HashMap::with_capacity(roots.len());
for (object, kind, serial) in roots {
root_kinds.entry(object).or_insert((kind, serial));
}
progress.report(HeapPhase::Saving, 1, 1)?;
let retained::Tree { idom, shallow8, retained, kids_start, kids } = tree;
Ok(HeapIndex {
hprof: dump,
ids,
types,
offsets,
shallow8,
retained,
idom,
kids_start,
kids,
gc_parent,
classes,
class_info,
class_slot,
roots: root_kinds,
thread_objects,
id_size,
summary,
})
}
/// Removes what an analysis may have written; a file that is not there is fine.
fn remove_index_files(dir: &Path) {
for name in INDEX_FILES {
let path = dir.join(name);
if let Err(err) = std::fs::remove_file(&path)
&& err.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not remove {}: {err}", path.display());
}
}
}
/// Steps of one phase: reports progress every [`CHECK_EVERY`] steps, and each loop looks at the
/// token before its first step.
pub(super) struct Meter<'a> {
progress: &'a Progress<'a>,
phase: HeapPhase,
/// An estimate; `done` is clamped to it.
total: u64,
done: u64,
next_report: u64,
}
impl<'a> Meter<'a> {
fn start(progress: &'a Progress<'a>, phase: HeapPhase, total: u64) -> Result<Self, HeapError> {
progress.report(phase, 0, total)?;
Ok(Self { progress, phase, total, done: 0, next_report: CHECK_EVERY })
}
pub(super) fn begin_loop(&self) -> Result<(), HeapError> {
self.progress.check()
}
pub(super) fn token(&self) -> &'a CancellationToken {
self.progress.token()
}
#[inline]
pub(super) fn tick(&mut self) -> Result<(), HeapError> {
self.advance(1)
}
pub(super) fn advance(&mut self, steps: u64) -> Result<(), HeapError> {
self.done += steps;
if self.done < self.next_report {
return Ok(());
}
self.next_report = self.done + CHECK_EVERY;
self.progress.report(self.phase, self.done.min(self.total), self.total)
}
fn finish(self) -> Result<(), HeapError> {
self.progress.report(self.phase, self.total, self.total)
}
}
#[cfg(test)]
mod tests;
+117
Переглянути файл
@@ -0,0 +1,117 @@
//! The class table: counts and shallow sizes per type slot, retained sizes per type from a walk of
//! the dominator tree, and the reachable / unreachable totals of the summary.
use super::Meter;
use super::retained::Tree;
use crate::column::{NONE, Store};
use crate::error::HeapError;
use crate::graph::{HeapGraph, PRIM_SLOTS};
use crate::hprof::BasicType;
use crate::index::ClassStat;
use crate::names::display_name;
/// Instances and arrays only: class objects are left out.
#[derive(Default)]
pub(super) struct Totals {
pub(super) reachable_objects: u64,
pub(super) reachable_bytes: u64,
pub(super) unreachable_objects: u64,
pub(super) unreachable_bytes: u64,
}
pub(super) fn table(
graph: &HeapGraph,
tree: &Tree,
store: &Store,
meter: &mut Meter,
) -> Result<(Vec<ClassStat>, Totals), HeapError> {
let slots = graph.classes.len() + PRIM_SLOTS as usize;
let mut classes: Vec<ClassStat> = (0..slots as u32)
.map(|slot| ClassStat { name: slot_name(graph, slot), count: 0, shallow: 0, retained: 0, unreachable_count: 0 })
.collect();
let mut totals = Totals::default();
meter.begin_loop()?;
for v in 0..graph.root as usize {
meter.tick()?;
let slot = graph.types[v];
let bytes = u64::from(tree.shallow8[v]) * 8;
let stat = &mut classes[slot as usize];
let instance = slot != graph.class_slot;
if tree.idom[v] == NONE {
stat.unreachable_count += 1;
if instance {
totals.unreachable_objects += 1;
totals.unreachable_bytes += bytes;
}
} else {
stat.count += 1;
stat.shallow += bytes;
if instance {
totals.reachable_objects += 1;
totals.reachable_bytes += bytes;
}
}
}
retained_by_type(graph, tree, &mut classes, store, meter)?;
Ok((classes, totals))
}
/// Adds to each type the retained size of its instances that have no instance of the same type
/// above them in the dominator tree. The walk keeps no stack: it goes up through `idom` and
/// resumes a node's children from `cursor`, while `active[t]` counts the instances of type `t` on
/// the path from the root.
fn retained_by_type(
graph: &HeapGraph,
tree: &Tree,
classes: &mut [ClassStat],
store: &Store,
meter: &mut Meter,
) -> Result<(), HeapError> {
let root = graph.root;
let mut cursor = store.scratch::<u32>(root as usize + 1, 0)?;
let mut active = vec![0u32; classes.len()];
cursor[root as usize] = tree.kids_start[root as usize];
let mut v = root;
meter.begin_loop()?;
loop {
meter.tick()?;
let next = cursor[v as usize];
if next < tree.kids_start[v as usize + 1] {
cursor[v as usize] = next + 1;
let child = tree.kids[next as usize];
let slot = graph.types[child as usize] as usize;
if active[slot] == 0 {
classes[slot].retained += tree.retained[child as usize];
}
active[slot] += 1;
cursor[child as usize] = tree.kids_start[child as usize];
v = child;
} else if v == root {
return Ok(());
} else {
active[graph.types[v as usize] as usize] -= 1;
v = tree.idom[v as usize];
}
}
}
fn slot_name(graph: &HeapGraph, slot: u32) -> String {
match graph.classes.get(slot as usize) {
Some(class) => display_name(&class.name),
None => graph.prim_type(slot).map_or_else(String::new, |ty| prim_array_name(ty).to_owned()),
}
}
fn prim_array_name(ty: BasicType) -> &'static str {
match ty {
BasicType::Object => "java.lang.Object[]",
BasicType::Boolean => "boolean[]",
BasicType::Char => "char[]",
BasicType::Float => "float[]",
BasicType::Double => "double[]",
BasicType::Byte => "byte[]",
BasicType::Short => "short[]",
BasicType::Int => "int[]",
BasicType::Long => "long[]",
}
}
+32
Переглянути файл
@@ -0,0 +1,32 @@
//! Parents on shortest paths from the GC roots, by breadth-first search over strong edges.
use super::Meter;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::GC_PARENT_FILE;
use crate::graph::HeapGraph;
pub(super) fn gc_parents(graph: &HeapGraph, store: &Store, meter: &mut Meter) -> Result<Column<u32>, HeapError> {
let root = graph.root;
let nodes = root as usize + 1;
let mut parent = store.column::<u32>(GC_PARENT_FILE, nodes, NONE)?;
let mut queue = store.scratch::<u32>(nodes, 0)?;
parent[root as usize] = root;
queue[0] = root;
let (mut head, mut tail) = (0, 1);
meter.begin_loop()?;
while head < tail {
meter.tick()?;
let v = queue[head];
head += 1;
for &w in graph.out_edges(v) {
meter.tick()?;
if parent[w as usize] == NONE {
parent[w as usize] = v;
queue[tail] = w;
tail += 1;
}
}
}
Ok(parent)
}
+209
Переглянути файл
@@ -0,0 +1,209 @@
//! Shallow and retained sizes, and the dominator tree as child lists sorted by retained size.
use std::cmp::Reverse;
use super::Meter;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::{KIDS_FILE, KIDS_START_FILE, RETAINED_FILE, SHALLOW8_FILE};
use crate::graph::{HeapGraph, accumulate};
use crate::progress::CHECK_EVERY;
use crate::shallow::ShallowModel;
/// The dominator tree with sizes; see [`crate::HeapIndex`] for each column.
pub(super) struct Tree {
pub(super) idom: Column<u32>,
pub(super) shallow8: Column<u32>,
pub(super) retained: Column<u64>,
pub(super) kids_start: Column<u32>,
pub(super) kids: Column<u32>,
}
/// `(shallow8, retained)` of every node, the root included. `preorder` is the DFS preorder of the
/// reachable nodes, from the root.
pub(super) fn sizes(
graph: &HeapGraph,
model: &ShallowModel,
idom: &[u32],
preorder: &[u32],
store: &Store,
meter: &mut Meter,
) -> Result<(Column<u32>, Column<u64>), HeapError> {
let nodes = idom.len();
let mut shallow8 = store.column::<u32>(SHALLOW8_FILE, nodes, 0)?;
let mut retained = store.column::<u64>(RETAINED_FILE, nodes, 0)?;
meter.begin_loop()?;
for v in 0..graph.root {
meter.tick()?;
let units = u32::try_from(model.size_of(graph, v).div_ceil(8)).unwrap_or(u32::MAX);
shallow8[v as usize] = units;
if idom[v as usize] != NONE {
retained[v as usize] = u64::from(units) * 8;
}
}
// A dominator is a DFS ancestor, so in reverse preorder every node is complete before it is
// added to its dominator.
meter.begin_loop()?;
for &v in preorder.iter().skip(1).rev() {
meter.tick()?;
let own = retained[v as usize];
retained[idom[v as usize] as usize] += own;
}
Ok((shallow8, retained))
}
/// `(kids_start, kids)`: the children of every reachable node in the dominator tree, as CSR.
pub(super) fn children(
idom: &[u32],
retained: &[u64],
root: u32,
store: &Store,
meter: &mut Meter,
) -> Result<(Column<u32>, Column<u32>), HeapError> {
let nodes = idom.len();
// Counted into `starts[d + 1]`; filled through `starts[d]` as a cursor, which leaves it at d's
// end until the column is shifted one slot.
let mut starts = store.column::<u32>(KIDS_START_FILE, nodes + 1, 0)?;
let mut longest = 0;
meter.begin_loop()?;
for (v, &d) in idom.iter().enumerate() {
meter.tick()?;
if d != NONE && v != root as usize {
let count = &mut starts[d as usize + 1];
*count += 1;
longest = longest.max(*count);
}
}
let total = accumulate(&mut starts, meter.token())?;
let mut kids = store.column::<u32>(KIDS_FILE, total as usize, 0)?;
meter.begin_loop()?;
for (v, &d) in idom.iter().enumerate() {
meter.tick()?;
if d != NONE && v != root as usize {
let cursor = &mut starts[d as usize];
kids[*cursor as usize] = v as u32;
*cursor += 1;
}
}
starts.copy_within(0..nodes, 1);
starts[0] = 0;
sort_children(&mut kids, &starts, retained, longest as usize, CHECK_EVERY as usize, store, meter)?;
Ok((starts, kids))
}
/// Sorts each child list by retained size descending, then by node. A list longer than `run` is
/// sorted in runs of `run` that are then merged, so no single step takes longer than one run.
fn sort_children(
kids: &mut [u32],
starts: &[u32],
retained: &[u64],
longest: usize,
run: usize,
store: &Store,
meter: &mut Meter,
) -> Result<(), HeapError> {
let key = |&k: &u32| (Reverse(retained[k as usize]), k);
let mut buffer = if longest > run { store.scratch::<u32>(longest, 0)? } else { Column::ram(0, 0) };
meter.begin_loop()?;
for v in 0..starts.len() - 1 {
meter.tick()?;
let list = &mut kids[starts[v] as usize..starts[v + 1] as usize];
if list.len() <= run {
list.sort_unstable_by_key(key);
meter.advance(list.len() as u64)?;
continue;
}
for chunk in list.chunks_mut(run) {
meter.begin_loop()?;
chunk.sort_unstable_by_key(key);
meter.advance(chunk.len() as u64)?;
}
let len = list.len();
let (mut from, mut to) = (list, &mut buffer[..len]);
let mut in_buffer = false;
let mut width = run;
while width < len {
meter.begin_loop()?;
for start in (0..len).step_by(2 * width) {
let mid = (start + width).min(len);
let end = (start + 2 * width).min(len);
merge(&from[start..mid], &from[mid..end], &mut to[start..end], &key, meter)?;
}
std::mem::swap(&mut from, &mut to);
in_buffer = !in_buffer;
width *= 2;
}
if in_buffer {
to.copy_from_slice(from);
}
}
Ok(())
}
fn merge<K: Ord>(
left: &[u32],
right: &[u32],
out: &mut [u32],
key: &impl Fn(&u32) -> K,
meter: &mut Meter,
) -> Result<(), HeapError> {
let (mut i, mut j) = (0, 0);
for slot in out.iter_mut() {
meter.tick()?;
let take_left = j == right.len() || (i < left.len() && key(&left[i]) <= key(&right[j]));
if take_left {
*slot = left[i];
i += 1;
} else {
*slot = right[j];
j += 1;
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use ide_core::CancellationToken;
use super::*;
use crate::error::HeapError;
use crate::progress::{HeapPhase, Progress};
use crate::testing::XorShift;
fn sorted_with_run(lists: &[Vec<u32>], retained: &[u64], run: usize) -> Vec<u32> {
let report = |_, _, _| Ok::<(), HeapError>(());
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
let mut meter = Meter::start(&progress, HeapPhase::Retained, 1).unwrap();
let mut starts = vec![0u32];
for list in lists {
starts.push(starts.last().unwrap() + list.len() as u32);
}
let mut kids: Vec<u32> = lists.concat();
let longest = lists.iter().map(Vec::len).max().unwrap_or(0);
sort_children(&mut kids, &starts, retained, longest, run, &Store::Ram, &mut meter).unwrap();
kids
}
#[test]
fn long_child_lists_are_merged_from_sorted_runs() {
let mut rng = XorShift::new(7);
let retained: Vec<u64> = (0..500).map(|_| u64::from(rng.below(40))).collect();
let mut nodes: Vec<u32> = (0..500).collect();
for i in (1..nodes.len()).rev() {
nodes.swap(i, rng.below(i as u32 + 1) as usize);
}
// Lists below, at and above the run length, and one that ends on a partial run.
let lists = vec![nodes[..3].to_vec(), nodes[3..7].to_vec(), nodes[7..400].to_vec(), nodes[400..].to_vec()];
for run in [1, 4, 7, 64, 1000] {
let mut expected = Vec::new();
for list in &lists {
let mut list = list.clone();
list.sort_by_key(|&k| (Reverse(retained[k as usize]), k));
expected.extend(list);
}
assert_eq!(sorted_with_run(&lists, &retained, run), expected, "run {run}");
}
}
}
+265
Переглянути файл
@@ -0,0 +1,265 @@
use std::cell::Cell;
use std::collections::BTreeSet;
use std::path::PathBuf;
use super::*;
use crate::column::{Column, NONE};
use crate::files::INDEX_FILES;
use crate::hprof::{BasicType, HeapRootKind};
use crate::index::{ClassStat, HeapSummary};
use crate::shallow::SizeSource;
use crate::testing::{MiniHeap, sample};
const DUMP: &str = "dump.hprof";
const ROOT: Option<&str> = Some("java.lang.Object");
const NODE_FIELDS: [(&str, BasicType); 2] = [("next", BasicType::Object), ("v", BasicType::Long)];
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn in_memory() -> AnalyzeOptions {
AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() }
}
/// A temp dir holding `heap` as `dump.hprof`, the way a capture lays out its snapshot dir.
fn dump_dir(heap: &MiniHeap) -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().unwrap();
let dump = dir.path().join(DUMP);
heap.write_to(&dump);
(dir, dump)
}
fn analyze_with(heap: &MiniHeap, opts: AnalyzeOptions) -> HeapIndex {
let (dir, dump) = dump_dir(heap);
analyze(&dump, dir.path(), opts, &no_progress, &CancellationToken::new()).unwrap()
}
fn file_names(dir: &Path) -> BTreeSet<String> {
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect()
}
fn node(index: &HeapIndex, addr: u64) -> u32 {
index.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the dump"))
}
fn shallow(index: &HeapIndex, addr: u64) -> u64 {
u64::from(index.shallow8[node(index, addr) as usize]) * 8
}
fn retained(index: &HeapIndex, addr: u64) -> u64 {
index.retained[node(index, addr) as usize]
}
fn idom(index: &HeapIndex, addr: u64) -> u32 {
index.idom[node(index, addr) as usize]
}
fn class<'a>(index: &'a HeapIndex, name: &str) -> &'a ClassStat {
let mut rows = index.classes.iter().filter(|c| c.name == name);
let row = rows.next().unwrap_or_else(|| panic!("no class {name}"));
assert!(rows.next().is_none(), "two classes named {name}");
row
}
fn stat(c: &ClassStat) -> (u64, u64, u64, u64) {
(c.count, c.shallow, c.retained, c.unreachable_count)
}
fn kids_of(index: &HeapIndex, v: u32) -> &[u32] {
&index.kids[index.kids_start[v as usize] as usize..index.kids_start[v as usize + 1] as usize]
}
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N2: u64 = 0x1200;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const R: u64 = 0x1500;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
#[test]
fn sample_retained_sizes() {
let index = analyze_with(&sample(), in_memory());
let shallows: Vec<u64> = [H, N1, L, W, R].iter().map(|&a| shallow(&index, a)).collect();
assert_eq!(shallows, [24, 24, 32, 24, 32]);
let retained_sizes: Vec<u64> = [L, N3, N1, N2, H, W].iter().map(|&a| retained(&index, a)).collect();
assert_eq!(retained_sizes, [32, 56, 24, 24, 72, 24]);
let root = index.root();
assert_eq!(
[idom(&index, N1), idom(&index, N2), idom(&index, N3), idom(&index, L)],
[node(&index, H), node(&index, H), root, node(&index, N3)]
);
assert_eq!((idom(&index, R), idom(&index, N4)), (NONE, NONE));
assert_eq!((retained(&index, R), retained(&index, N4)), (0, 0));
// Children by retained size descending, ties by node.
assert_eq!(kids_of(&index, node(&index, H)), [node(&index, N1), node(&index, N2)]);
let top = kids_of(&index, root);
assert_eq!(top[..2], [node(&index, H), node(&index, N3)]);
assert!(top.windows(2).all(|p| (index.retained[p[0] as usize], p[1]) > (index.retained[p[1] as usize], p[0])));
assert_eq!(index.retained[root as usize], top.iter().map(|&k| index.retained[k as usize]).sum::<u64>());
}
#[test]
fn sample_class_table() {
let index = analyze_with(&sample(), in_memory());
assert_eq!(stat(class(&index, "Node")), (3, 72, 104, 1));
assert_eq!(stat(class(&index, "Holder")), (1, 24, 72, 0));
assert_eq!(stat(class(&index, "long[]")), (1, 32, 32, 0));
assert_eq!(stat(class(&index, "java.lang.ref.WeakReference")), (1, 24, 24, 0));
let objects = class(&index, "java.lang.Object[]");
assert_eq!((objects.count, objects.unreachable_count), (0, 1));
// Indexed by type slot, primitive array slots included.
assert_eq!(index.classes.len(), index.class_info.len() + crate::graph::PRIM_SLOTS as usize);
let slot = index.types[node(&index, H) as usize] as usize;
assert_eq!(index.classes[slot].name, "Holder");
assert_eq!(index.class_info[slot].name, "Holder");
}
#[test]
fn sample_summary() {
let index = analyze_with(&sample(), in_memory());
let s = &index.summary;
assert_eq!(
(s.reachable_objects, s.reachable_bytes, s.unreachable_objects, s.unreachable_bytes, s.weak_edges_skipped),
(6, 152, 2, 56, 1)
);
assert_eq!(s.objects, 8);
assert_eq!(s.classes, 7);
assert_eq!(s.size_source, SizeSource::Layout);
}
#[test]
fn sample_gc_parents() {
let index = analyze_with(&sample(), in_memory());
let parent = |addr| index.gc_parent[node(&index, addr) as usize];
let root = index.root();
assert_eq!(index.gc_parent[root as usize], root);
assert_eq!([parent(H), parent(N3), parent(W)], [root; 3]);
assert_eq!([parent(N1), parent(L)], [node(&index, H), node(&index, N3)]);
assert_eq!((parent(R), parent(N4)), (NONE, NONE));
assert_eq!(index.roots[&node(&index, N3)], (HeapRootKind::JavaFrame, Some(1)));
}
#[test]
fn cycle_is_retained_once() {
let mut heap = MiniHeap::new();
heap.class("Node", ROOT, &NODE_FIELDS)
.object(0xA000, "Node", &[("next", 0xB000)])
.object(0xB000, "Node", &[("next", 0xA000)])
.root(HeapRootKind::JniGlobal, 0xA000);
let index = analyze_with(&heap, in_memory());
assert_eq!((retained(&index, 0xA000), retained(&index, 0xB000)), (48, 24));
assert_eq!(class(&index, "Node").retained, 48);
}
#[test]
fn deep_chain_retained_by_class() {
// A small stack proves no pass recurses along the chain.
std::thread::Builder::new()
.stack_size(256 << 10)
.spawn(|| {
const NODES: u64 = 1_000_000;
let addr = |i: u64| 0x10_0000 + i * 0x100;
let mut heap = MiniHeap::new();
heap.class("Node", ROOT, &NODE_FIELDS);
for i in 0..NODES {
let next = if i + 1 < NODES { addr(i + 1) } else { 0 };
heap.object(addr(i), "Node", &[("next", next)]);
}
heap.root(HeapRootKind::JniGlobal, addr(0));
let index = analyze_with(&heap, in_memory());
assert_eq!(retained(&index, addr(0)), 24_000_000);
assert_eq!(retained(&index, addr(NODES - 1)), 24);
assert_eq!(class(&index, "Node").retained, 24_000_000);
assert_eq!(index.summary.reachable_objects, NODES);
})
.unwrap()
.join()
.unwrap();
}
#[test]
fn histogram_sizes_flow_into_retained() {
let histogram = vec![HistogramEntry { name: "Node".to_owned(), instances: 3, bytes: 96 }];
let index = analyze_with(&sample(), AnalyzeOptions { histogram: Some(histogram), ..in_memory() });
assert_eq!(shallow(&index, N3), 32);
assert_eq!(retained(&index, N3), 64);
assert_eq!(index.summary.size_source, SizeSource::Jvm);
}
#[test]
fn vm_flags_of_an_unknown_jdk_set_the_layout() {
// Without compressed oops a Node is 12 + 8 + 8 = 28 → 32.
let opts = AnalyzeOptions { vm_flags: Some("-XX:MaxHeapSize=42949672960".to_owned()), ..in_memory() };
let index = analyze_with(&sample(), opts);
assert_eq!(shallow(&index, N4), 32);
}
#[test]
fn cancel_stops_analysis() {
let (dir, dump) = dump_dir(&sample());
let cancel = CancellationToken::new();
let phases = Cell::new(Vec::new());
let progress = |phase, _, _| {
let mut seen = phases.take();
seen.push(phase);
phases.set(seen);
if phase == HeapPhase::Dominators {
cancel.cancel();
}
Ok(())
};
let result = analyze(&dump, dir.path(), AnalyzeOptions::default(), &progress, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)), "{:?}", result.err());
assert_eq!(phases.take().last(), Some(&HeapPhase::Dominators), "stops at the first check");
// The dump the capture analyzes in place stays; every file the analysis wrote is gone.
assert_eq!(file_names(dir.path()), BTreeSet::from([DUMP.to_owned()]));
}
#[test]
fn progress_error_stops_analysis_and_removes_its_files() {
let (dir, dump) = dump_dir(&sample());
// The last report, once every index file is written.
let progress = |phase, done, _| match (phase, done) {
(HeapPhase::Saving, 1) => Err(HeapError::NotEnoughDiskSpace { needed: 2, available: 1 }),
_ => Ok(()),
};
let result = analyze(&dump, dir.path(), AnalyzeOptions::default(), &progress, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::NotEnoughDiskSpace { .. })), "{:?}", result.err());
assert_eq!(file_names(dir.path()), BTreeSet::from([DUMP.to_owned()]));
}
#[test]
fn writes_index_files() {
let (dir, dump) = dump_dir(&sample());
let index = analyze(&dump, dir.path(), AnalyzeOptions::default(), &no_progress, &CancellationToken::new()).unwrap();
let names = file_names(dir.path());
for name in [
"types.u32",
"shallow8.u32",
"retained.u64",
"idom.u32",
"kids_start.u32",
"kids.u32",
"gc_parent.u32",
"offsets.u64",
"summary.json",
] {
assert!(names.contains(name), "{name} missing from {names:?}");
}
assert!(names.iter().any(|n| n.starts_with("ids.")), "{names:?}");
// Nothing but the index files and the dump: scratch arrays are gone.
let unexpected: Vec<_> = names.iter().filter(|n| *n != DUMP && !INDEX_FILES.contains(&n.as_str())).collect();
assert!(unexpected.is_empty(), "{unexpected:?}");
let summary: HeapSummary =
serde_json::from_slice(&std::fs::read(dir.path().join("summary.json")).unwrap()).unwrap();
assert_eq!(summary, index.summary);
drop(index);
let retained = Column::<u64>::open(&dir.path().join("retained.u64")).unwrap();
let in_ram = analyze_with(&sample(), in_memory());
assert_eq!(&*retained, &*in_ram.retained);
}
+306
Переглянути файл
@@ -0,0 +1,306 @@
//! Fixed-width arrays that live in RAM or in memory-mapped files, so the analysis arrays of a
//! large heap do not count against the process's anonymous memory.
use std::fs::{File, OpenOptions};
use std::io;
use std::marker::PhantomData;
use std::mem::size_of;
use std::ops::{Deref, DerefMut};
use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
/// Index value meaning "no object" in every `u32` column.
pub const NONE: u32 = u32::MAX;
/// # Safety
/// Every bit pattern of `Self` is a valid value and `Self` has no padding, so a mapped file's
/// bytes can be viewed as `[Self]` and a file can be written from one.
pub unsafe trait Pod: Copy + 'static {}
unsafe impl Pod for u8 {}
unsafe impl Pod for u32 {}
unsafe impl Pod for u64 {}
enum Backing<T> {
Ram(Vec<T>),
Map(MmapMut),
ReadOnly(Mmap),
}
/// A fixed-length array of `T`. File contents use the machine's byte order: the arrays are a
/// cache for this machine and are never moved to another one.
pub struct Column<T: Pod> {
backing: Backing<T>,
/// Scratch file to delete on drop where it could not be unlinked while mapped.
delete_on_drop: Option<PathBuf>,
_marker: PhantomData<T>,
}
fn byte_len<T: Pod>(len: usize) -> io::Result<usize> {
len.checked_mul(size_of::<T>())
.filter(|&bytes| bytes <= isize::MAX as usize)
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "column is too long"))
}
fn is_zero<T: Pod>(value: &T) -> bool {
// SAFETY: `Pod` types have no padding, so all `size_of::<T>()` bytes are initialized.
let bytes = unsafe { std::slice::from_raw_parts((value as *const T).cast::<u8>(), size_of::<T>()) };
bytes.iter().all(|&b| b == 0)
}
impl<T: Pod> Column<T> {
fn new(backing: Backing<T>, delete_on_drop: Option<PathBuf>) -> Self {
Self { backing, delete_on_drop, _marker: PhantomData }
}
pub fn ram(len: usize, fill: T) -> Self {
Self::new(Backing::Ram(vec![fill; len]), None)
}
/// Creates (or truncates) `path` and maps it; the file stays after the column is dropped.
pub fn file(path: &Path, len: usize, fill: T) -> io::Result<Self> {
let bytes = byte_len::<T>(len)?;
let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?;
match Self::map_new(&file, len, bytes, fill) {
Ok(column) => Ok(column),
Err(err) => {
drop(file);
let _ = std::fs::remove_file(path);
Err(err)
}
}
}
/// A temporary column in `dir` that never outlives the process. On unix the file is unlinked
/// right after mapping, so even a crash leaves nothing behind; elsewhere (Windows cannot delete
/// a mapped file) it is deleted on drop.
pub fn scratch(dir: &Path, len: usize, fill: T) -> io::Result<Self> {
let bytes = byte_len::<T>(len)?;
if bytes == 0 {
return Ok(Self::ram(0, fill));
}
let path = dir.join(format!("scratch-{}.tmp", uuid::Uuid::new_v4().simple()));
let file = OpenOptions::new().read(true).write(true).create_new(true).open(&path)?;
let mut column = match Self::map_new(&file, len, bytes, fill) {
Ok(column) => column,
Err(err) => {
drop(file);
let _ = std::fs::remove_file(&path);
return Err(err);
}
};
// If the unlink fails the drop-time delete is the fallback, so the file is never left behind.
if !cfg!(unix) || std::fs::remove_file(&path).is_err() {
column.delete_on_drop = Some(path);
}
Ok(column)
}
/// Maps an existing file read-only; writing through the column panics.
pub fn open(path: &Path) -> io::Result<Self> {
let file = File::open(path)?;
let bytes = file.metadata()?.len();
if bytes % size_of::<T>() as u64 != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("{} is {bytes} bytes, not a multiple of {}", path.display(), size_of::<T>()),
));
}
if bytes == 0 {
return Ok(Self::ram(0, zero_value()));
}
// SAFETY: the file is private to the snapshot directory; nothing truncates it while mapped.
let map = unsafe { Mmap::map(&file)? };
Ok(Self::new(Backing::ReadOnly(map), None))
}
fn map_new(file: &File, len: usize, bytes: usize, fill: T) -> io::Result<Self> {
if bytes == 0 {
return Ok(Self::ram(0, fill));
}
file.set_len(bytes as u64)?;
// SAFETY: as in `open`; the file was just created by this process.
let map = unsafe { MmapMut::map_mut(file)? };
let mut column = Self::new(Backing::Map(map), None);
// A fresh file reads as zeros, and writing a zero fill would fault in every page.
if !is_zero(&fill) {
debug_assert_eq!(column.len(), len);
column.fill(fill);
}
Ok(column)
}
}
fn zero_value<T: Pod>() -> T {
// SAFETY: every bit pattern of a `Pod` type is a valid value.
unsafe { std::mem::zeroed() }
}
impl<T: Pod> Deref for Column<T> {
type Target = [T];
fn deref(&self) -> &[T] {
match &self.backing {
Backing::Ram(v) => v,
// SAFETY: maps are page-aligned, `T` is `Pod` and the length is a multiple of its width.
Backing::Map(m) => unsafe { std::slice::from_raw_parts(m.as_ptr().cast(), m.len() / size_of::<T>()) },
Backing::ReadOnly(m) => unsafe { std::slice::from_raw_parts(m.as_ptr().cast(), m.len() / size_of::<T>()) },
}
}
}
impl<T: Pod> DerefMut for Column<T> {
fn deref_mut(&mut self) -> &mut [T] {
match &mut self.backing {
Backing::Ram(v) => v,
// SAFETY: as in `deref`; `&mut self` makes the access exclusive.
Backing::Map(m) => unsafe {
std::slice::from_raw_parts_mut(m.as_mut_ptr().cast(), m.len() / size_of::<T>())
},
Backing::ReadOnly(_) => panic!("column opened from a file is read-only"),
}
}
}
impl<T: Pod> Drop for Column<T> {
fn drop(&mut self) {
if let Some(path) = self.delete_on_drop.take() {
// Unmap before deleting: Windows refuses to delete a mapped file.
self.backing = Backing::Ram(Vec::new());
let _ = std::fs::remove_file(path);
}
}
}
/// Where a computation puts its arrays: result columns are named files in `Dir`, temporary arrays
/// are scratch columns in the same directory.
pub enum Store {
Ram,
Dir(PathBuf),
}
impl Store {
/// `name` is the whole file name (`types.u32`); it is used as given.
pub fn column<T: Pod>(&self, name: &str, len: usize, fill: T) -> io::Result<Column<T>> {
match self {
Store::Ram => Ok(Column::ram(len, fill)),
Store::Dir(dir) => Column::file(&dir.join(name), len, fill),
}
}
pub fn scratch<T: Pod>(&self, len: usize, fill: T) -> io::Result<Column<T>> {
match self {
Store::Ram => Ok(Column::ram(len, fill)),
Store::Dir(dir) => Column::scratch(dir, len, fill),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn dir_is_empty(dir: &Path) -> bool {
std::fs::read_dir(dir).unwrap().next().is_none()
}
#[test]
fn file_column_round_trips() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("x.u32");
{
let mut col = Column::<u32>::file(&path, 4, 7).unwrap();
assert_eq!(&*col, &[7, 7, 7, 7]);
col.copy_from_slice(&[1, 2, 3, 4]);
}
let col = Column::<u32>::open(&path).unwrap();
assert_eq!(&*col, &[1, 2, 3, 4]);
}
#[test]
fn scratch_column_leaves_no_file() {
let dir = tempfile::tempdir().unwrap();
{
let mut col = Column::<u64>::scratch(dir.path(), 1024, 9).unwrap();
col[1023] = 5;
assert_eq!((col[0], col[1023]), (9, 5));
// Windows cannot delete a mapped file, so only unix unlinks while the map is alive.
#[cfg(unix)]
assert!(dir_is_empty(dir.path()));
}
assert!(dir_is_empty(dir.path()));
}
#[test]
fn zero_fill_and_non_zero_fill() {
let dir = tempfile::tempdir().unwrap();
let zeros = Column::<u64>::file(&dir.path().join("z.u64"), 3000, 0).unwrap();
assert!(zeros.iter().all(|&v| v == 0));
let ones = Column::<u8>::file(&dir.path().join("o.u8"), 10_000, 0xAB).unwrap();
assert!(ones.iter().all(|&v| v == 0xAB));
let ram = Column::<u32>::ram(5, NONE);
assert_eq!(&*ram, &[NONE; 5]);
}
#[test]
fn empty_columns_work_everywhere() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("e.u32");
assert_eq!(Column::<u32>::ram(0, 1).len(), 0);
assert_eq!(Column::<u32>::file(&path, 0, 1).unwrap().len(), 0);
assert_eq!(Column::<u32>::open(&path).unwrap().len(), 0);
assert_eq!(Column::<u32>::scratch(dir.path(), 0, 1).unwrap().len(), 0);
}
#[test]
fn open_rejects_a_length_that_is_not_a_multiple_of_the_width() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("odd.u32");
std::fs::write(&path, [0u8; 6]).unwrap();
let err = Column::<u32>::open(&path).err().unwrap();
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
}
#[test]
fn file_rejects_a_byte_length_that_overflows() {
let dir = tempfile::tempdir().unwrap();
let err = Column::<u64>::file(&dir.path().join("big.u64"), usize::MAX, 0).err().unwrap();
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
assert!(dir_is_empty(dir.path()));
}
#[test]
#[should_panic(expected = "read-only")]
fn opened_columns_are_read_only() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("r.u32");
drop(Column::<u32>::file(&path, 2, 0).unwrap());
let mut col = Column::<u32>::open(&path).unwrap();
col[0] = 1;
}
#[test]
fn store_names_files_exactly_as_given() {
let dir = tempfile::tempdir().unwrap();
let store = Store::Dir(dir.path().to_path_buf());
let mut col = store.column::<u32>("types.u32", 3, 0).unwrap();
col[2] = 8;
drop(col);
let names: Vec<_> =
std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
assert_eq!(names, ["types.u32"]);
assert_eq!(&*Column::<u32>::open(&dir.path().join("types.u32")).unwrap(), &[0, 0, 8]);
}
#[test]
fn store_scratch_is_anonymous_and_ram_store_touches_no_disk() {
let dir = tempfile::tempdir().unwrap();
let store = Store::Dir(dir.path().to_path_buf());
drop(store.scratch::<u32>(16, 1).unwrap());
assert!(dir_is_empty(dir.path()));
let ram = Store::Ram.column::<u8>("ignored.u8", 4, 2).unwrap();
assert_eq!(&*ram, &[2; 4]);
assert_eq!(&*Store::Ram.scratch::<u8>(2, 3).unwrap(), &[3, 3]);
}
}
+59
Переглянути файл
@@ -0,0 +1,59 @@
//! Class-by-class differences between two snapshots.
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use crate::index::HeapIndex;
/// One class name in two snapshots; a side without the name has zeros.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapClassDiff {
pub name: String,
pub base_count: u64,
pub count: u64,
pub base_shallow: u64,
pub shallow: u64,
pub base_retained: u64,
pub retained: u64,
/// Several classes have this name on one side or both, so the sides cannot be matched: each
/// side shows the sum over its classes, and there is no difference to show.
pub ambiguous: bool,
}
/// Joined by class name, the only key two dumps share. By the change in retained size
/// descending (none for an ambiguous name), then by name.
pub fn compare(base: &HeapIndex, current: &HeapIndex) -> Vec<HeapClassDiff> {
let mut by_name: HashMap<String, HeapClassDiff> = HashMap::new();
for (index, is_base) in [(base, true), (current, false)] {
for row in index.class_rows() {
let diff = by_name.entry(row.name).or_insert_with_key(|name| HeapClassDiff {
name: name.clone(),
base_count: 0,
count: 0,
base_shallow: 0,
shallow: 0,
base_retained: 0,
retained: 0,
ambiguous: false,
});
let (count, shallow, retained) = if is_base {
(&mut diff.base_count, &mut diff.base_shallow, &mut diff.base_retained)
} else {
(&mut diff.count, &mut diff.shallow, &mut diff.retained)
};
*count = count.saturating_add(row.count);
*shallow = shallow.saturating_add(row.shallow);
*retained = retained.saturating_add(row.retained);
diff.ambiguous |= row.ambiguous;
}
}
let change = |d: &HeapClassDiff| if d.ambiguous { 0 } else { d.retained.abs_diff(d.base_retained) };
let mut diffs: Vec<HeapClassDiff> = by_name.into_values().collect();
diffs.sort_by(|a, b| change(b).cmp(&change(a)).then_with(|| a.name.cmp(&b.name)));
diffs
}
#[cfg(test)]
mod tests;
+78
Переглянути файл
@@ -0,0 +1,78 @@
use super::*;
use crate::hprof::HeapRootKind;
use crate::test_index::analyzed;
use crate::testing::{MiniHeap, sample, sample_with_two_more_nodes};
fn diff<'a>(diffs: &'a [HeapClassDiff], name: &str) -> &'a HeapClassDiff {
let mut rows = diffs.iter().filter(|d| d.name == name);
let row = rows.next().unwrap_or_else(|| panic!("no diff for {name}"));
assert!(rows.next().is_none(), "two diffs for {name}");
row
}
fn sort_key(d: &HeapClassDiff) -> u64 {
if d.ambiguous { 0 } else { d.retained.abs_diff(d.base_retained) }
}
fn assert_sorted(diffs: &[HeapClassDiff]) {
assert!(diffs.windows(2).all(|p| (sort_key(&p[1]), &p[0].name) < (sort_key(&p[0]), &p[1].name)), "{diffs:#?}");
}
#[test]
fn compare_counts_by_class() {
let base = analyzed(&sample());
let mut current = sample_with_two_more_nodes();
current
.class("Extra", Some("java.lang.Object"), &[])
.object(0x4000, "Extra", &[])
.root(HeapRootKind::JniGlobal, 0x4000);
let diffs = compare(&base, &analyzed(&current));
let node = diff(&diffs, "Node");
assert_eq!((node.base_count, node.count), (3, 5));
assert_eq!(node.shallow - node.base_shallow, 48);
assert_eq!((node.base_retained, node.retained), (104, 152));
assert!(!node.ambiguous);
let extra = diff(&diffs, "Extra");
assert_eq!((extra.base_count, extra.base_shallow, extra.base_retained, extra.count), (0, 0, 0, 1));
assert_eq!(diffs[0].name, "Node");
let holder = diff(&diffs, "Holder");
assert_eq!((holder.base_count, holder.count, holder.base_retained, holder.retained), (1, 1, 72, 72));
assert_sorted(&diffs);
// A class only in the base snapshot.
let gone = compare(&analyzed(&current), &base);
let extra = diff(&gone, "Extra");
assert_eq!((extra.base_count, extra.count, extra.shallow, extra.retained), (1, 0, 0, 0));
assert_sorted(&gone);
}
fn two_foos(extra: bool) -> MiniHeap {
let mut heap = sample();
heap.class("Foo", Some("java.lang.Object"), &[])
.object(0x3000, "Foo", &[])
.class("Foo", Some("java.lang.Object"), &[])
.object(0x3100, "Foo", &[]);
if extra {
heap.object(0x3200, "Foo", &[]);
}
for addr in [0x3000, 0x3100, 0x3200] {
heap.root(HeapRootKind::JniGlobal, addr);
}
heap
}
#[test]
fn compare_gives_no_diff_for_a_name_of_several_classes() {
let diffs = compare(&analyzed(&two_foos(false)), &analyzed(&two_foos(true)));
let foo = diff(&diffs, "Foo");
assert!(foo.ambiguous);
assert_eq!((foo.base_count, foo.count), (2, 3));
assert!(diffs.iter().filter(|d| d.name != "Foo").all(|d| !d.ambiguous));
assert_sorted(&diffs);
// Ambiguous on one side only is still ambiguous.
let diffs = compare(&analyzed(&sample()), &analyzed(&two_foos(false)));
let foo = diff(&diffs, "Foo");
assert!(foo.ambiguous);
assert_eq!((foo.base_count, foo.count), (0, 2));
}
+137
Переглянути файл
@@ -0,0 +1,137 @@
//! Disk-space estimates and the per-snapshot directories under the app's cache.
//!
//! A snapshot directory holds the index columns and, for a capture, the dump itself. Its `owner`
//! file names the host process that created it, so a host that starts later can remove the
//! directories of hosts that died without cleaning up.
use std::fs;
use std::io::{self, Read};
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime};
/// Space to leave free on top of an estimate.
pub const DISK_MARGIN: u64 = 1 << 30;
/// hprof ≈ `bytes + 16 × instances`; index columns and scratch arrays ≈ `90 × instances`.
const CAPTURE_BYTES_PER_INSTANCE: u64 = 106;
const INDEX_BYTES_PER_OBJECT: u64 = 90;
const OWNER_FILE: &str = "owner";
const TMP_SUFFIX: &str = ".tmp";
/// A directory still being created by a live host has no `owner` for a moment; one that stays
/// unfinished for longer than this was abandoned.
const TMP_GRACE: Duration = Duration::from_secs(60 * 60);
/// Bytes of free space that a dump of `heap_bytes` bytes holding `instances` instances needs,
/// including its index.
pub fn capture_need(heap_bytes: u64, instances: u64) -> u64 {
heap_bytes.saturating_add(CAPTURE_BYTES_PER_INSTANCE.saturating_mul(instances)).saturating_add(DISK_MARGIN)
}
/// Bytes of free space that indexing a dump of `objects` objects needs.
pub fn index_need(objects: u64) -> u64 {
INDEX_BYTES_PER_OBJECT.saturating_mul(objects).saturating_add(DISK_MARGIN)
}
/// Bytes available to this process on the volume holding `dir`; `None` where it is not checked
/// (not unix) or cannot be read, so callers go ahead rather than refuse.
pub fn free_bytes(dir: &Path) -> Option<u64> {
#[cfg(unix)]
{
use std::os::unix::ffi::OsStrExt;
let path = std::ffi::CString::new(dir.as_os_str().as_bytes()).ok()?;
// SAFETY: `path` is NUL-terminated and `stat` is only read after `statvfs` filled it in.
let stat = unsafe {
let mut stat = std::mem::MaybeUninit::<libc::statvfs>::zeroed();
if libc::statvfs(path.as_ptr(), stat.as_mut_ptr()) != 0 {
return None;
}
stat.assume_init()
};
// The field widths differ between unix targets.
#[allow(clippy::unnecessary_cast)]
Some((stat.f_bavail as u64).saturating_mul(stat.f_frsize as u64))
}
#[cfg(not(unix))]
{
let _ = dir;
None
}
}
/// `<local_data_dir>/cache/heap-snapshots`.
pub fn snapshots_root(local_data_dir: &Path) -> PathBuf {
local_data_dir.join("cache").join("heap-snapshots")
}
/// Creates `<root>/<uuid>/` holding an `owner` file with this process's pid. The directory is
/// built as `<uuid>.tmp` and renamed, so another host scanning `root` never sees it without its
/// owner and removes it as abandoned.
pub fn create_snapshot_dir(root: &Path) -> io::Result<PathBuf> {
fs::create_dir_all(root)?;
let name = uuid::Uuid::new_v4().to_string();
let building = root.join(format!("{name}{TMP_SUFFIX}"));
let dir = root.join(&name);
create_private_dir(&building)?;
let published =
fs::write(building.join(OWNER_FILE), std::process::id().to_string()).and_then(|()| fs::rename(&building, &dir));
if let Err(err) = published {
let _ = fs::remove_dir_all(&building);
return Err(err);
}
Ok(dir)
}
/// A dump holds whatever was in the JVM's memory, passwords and tokens included, so on unix only
/// the owner may enter the directory. Windows keeps the ACL inherited from the user's data dir.
fn create_private_dir(path: &Path) -> io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::DirBuilderExt;
fs::DirBuilder::new().mode(0o700).create(path)
}
#[cfg(not(unix))]
{
fs::DirBuilder::new().create(path)
}
}
/// Removes the directories under `root` whose owner file is missing, unreadable or names a pid
/// for which `alive` is false, and unfinished `*.tmp` ones older than an hour. Only directories
/// directly under `root` are touched; a symlink there is left alone.
pub fn remove_stale(root: &Path, alive: impl Fn(u32) -> bool) {
remove_stale_at(root, alive, SystemTime::now());
}
fn remove_stale_at(root: &Path, alive: impl Fn(u32) -> bool, now: SystemTime) {
let Ok(entries) = fs::read_dir(root) else { return };
for entry in entries.flatten() {
let path = entry.path();
// `symlink_metadata` does not follow: a link out of `root` is not a directory here.
let Ok(meta) = fs::symlink_metadata(&path) else { continue };
if !meta.file_type().is_dir() {
continue;
}
let stale = if path.extension().is_some_and(|ext| ext == "tmp") {
meta.modified()
.ok()
.and_then(|modified| now.duration_since(modified).ok())
.is_some_and(|age| age >= TMP_GRACE)
} else {
read_owner(&path).is_none_or(|pid| !alive(pid))
};
if stale && let Err(err) = fs::remove_dir_all(&path) {
tracing::warn!(dir = %path.display(), %err, "could not remove a stale heap snapshot directory");
}
}
}
fn read_owner(dir: &Path) -> Option<u32> {
let mut text = String::new();
fs::File::open(dir.join(OWNER_FILE)).ok()?.take(32).read_to_string(&mut text).ok()?;
text.trim().parse().ok()
}
#[cfg(test)]
mod tests;
+137
Переглянути файл
@@ -0,0 +1,137 @@
use std::time::{Duration, SystemTime};
use super::*;
fn names(dir: &Path) -> Vec<String> {
let mut names: Vec<_> =
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
names
}
fn dir_with_owner(root: &Path, name: &str, owner: Option<&str>) {
let dir = root.join(name);
std::fs::create_dir(&dir).unwrap();
std::fs::write(dir.join("dump.hprof"), b"x").unwrap();
if let Some(owner) = owner {
std::fs::write(dir.join(OWNER_FILE), owner).unwrap();
}
}
#[test]
fn estimates() {
assert_eq!(capture_need(664_018_520, 20_087_506), 664_018_520 + 106 * 20_087_506 + (1 << 30));
assert_eq!(index_need(10), 900 + (1 << 30));
}
#[test]
fn estimates_saturate_instead_of_wrapping() {
assert_eq!(capture_need(u64::MAX, u64::MAX), u64::MAX);
assert_eq!(index_need(u64::MAX), u64::MAX);
}
#[cfg(unix)]
#[test]
fn not_enough_space_for_estimate() {
let dir = tempfile::tempdir().unwrap();
let free = free_bytes(dir.path()).expect("statvfs works on a temp dir");
assert!(free > 0);
assert!(capture_need(u64::MAX / 4, 0) > free);
}
#[cfg(unix)]
#[test]
fn free_bytes_of_a_missing_dir_is_unknown() {
let dir = tempfile::tempdir().unwrap();
assert_eq!(free_bytes(&dir.path().join("missing")), None);
}
#[cfg(not(unix))]
#[test]
fn free_space_is_not_checked_off_unix() {
assert_eq!(free_bytes(Path::new(".")), None);
}
#[test]
fn snapshots_root_is_under_the_cache() {
let local = Path::new("local");
assert_eq!(snapshots_root(local), local.join("cache").join("heap-snapshots"));
}
#[test]
fn snapshot_dir_records_its_owner_and_leaves_no_tmp() {
let root = tempfile::tempdir().unwrap();
let missing_parent = root.path().join("cache").join("heap-snapshots");
let dir = create_snapshot_dir(&missing_parent).unwrap();
assert_eq!(dir.parent(), Some(missing_parent.as_path()));
assert!(uuid::Uuid::parse_str(dir.file_name().unwrap().to_str().unwrap()).is_ok());
assert_eq!(std::fs::read_to_string(dir.join(OWNER_FILE)).unwrap(), std::process::id().to_string());
assert_eq!(names(&missing_parent), [dir.file_name().unwrap().to_str().unwrap()]);
assert_ne!(create_snapshot_dir(&missing_parent).unwrap(), dir);
}
#[cfg(unix)]
#[test]
fn snapshot_dir_is_private_to_the_owner() {
use std::os::unix::fs::PermissionsExt;
let root = tempfile::tempdir().unwrap();
let dir = create_snapshot_dir(root.path()).unwrap();
assert_eq!(std::fs::metadata(&dir).unwrap().permissions().mode() & 0o777, 0o700);
}
#[test]
fn stale_dirs_removed_on_start() {
let root = tempfile::tempdir().unwrap();
dir_with_owner(root.path(), "a", Some("999999"));
dir_with_owner(root.path(), "b", Some(&std::process::id().to_string()));
dir_with_owner(root.path(), "c", None);
remove_stale(root.path(), |pid| pid == std::process::id());
assert_eq!(names(root.path()), ["b"]);
}
#[test]
fn owner_that_is_not_a_pid_is_stale() {
let root = tempfile::tempdir().unwrap();
dir_with_owner(root.path(), "empty", Some(""));
dir_with_owner(root.path(), "word", Some("not-a-pid"));
dir_with_owner(root.path(), "negative", Some("-1"));
dir_with_owner(root.path(), "padded", Some(&format!(" {}\n", std::process::id())));
remove_stale(root.path(), |pid| pid == std::process::id());
assert_eq!(names(root.path()), ["padded"]);
}
#[test]
fn stale_scan_tolerates_a_missing_root_and_leaves_loose_files_alone() {
let root = tempfile::tempdir().unwrap();
remove_stale(&root.path().join("missing"), |_| false);
std::fs::write(root.path().join("notes.txt"), b"keep").unwrap();
remove_stale(root.path(), |_| false);
assert_eq!(names(root.path()), ["notes.txt"]);
}
#[test]
fn unfinished_dir_is_left_alone_for_an_hour() {
let root = tempfile::tempdir().unwrap();
// Another host may be between `create_dir` and the rename of its own snapshot dir.
dir_with_owner(root.path(), "young.tmp", None);
let now = SystemTime::now();
remove_stale_at(root.path(), |_| false, now);
assert_eq!(names(root.path()), ["young.tmp"]);
remove_stale_at(root.path(), |_| false, now + TMP_GRACE - Duration::from_secs(60));
assert_eq!(names(root.path()), ["young.tmp"]);
remove_stale_at(root.path(), |_| false, now + TMP_GRACE + Duration::from_secs(60));
assert!(names(root.path()).is_empty());
}
#[cfg(unix)]
#[test]
fn stale_scan_never_follows_a_symlink_out_of_the_root() {
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("precious"), b"x").unwrap();
let root = tempfile::tempdir().unwrap();
std::os::unix::fs::symlink(outside.path(), root.path().join("link")).unwrap();
remove_stale(root.path(), |_| false);
remove_stale_at(root.path(), |_| false, SystemTime::now() + 2 * TMP_GRACE);
assert_eq!(names(outside.path()), ["precious"]);
}
+247
Переглянути файл
@@ -0,0 +1,247 @@
//! Dominator tree of the object graph, by Lengauer–Tarjan with path compression and no recursion:
//! the DFS "stack" is the chain of DFS parents, and `eval` reverses the links of the path it
//! compresses instead of keeping a stack.
//!
//! Between the DFS and the final translation every array is indexed by DFS number, so the root is
//! 0 and a vertex's DFS ancestors all have smaller numbers.
use ide_core::CancellationToken;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::IDOM_FILE;
use crate::graph::accumulate;
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
const PHASE: HeapPhase = HeapPhase::Dominators;
pub struct Dominators {
/// Immediate dominator of each node, saved as `idom.u32` in the store; [`NONE`] for
/// unreachable nodes, `root` for `root`.
pub idom: Column<u32>,
/// Nodes in DFS preorder from `root`, `reachable` entries.
pub preorder: Column<u32>,
pub reachable: u32,
}
/// The dominators of the graph whose node `v` has targets `out[out_start[v]..out_start[v + 1]]`,
/// from `root` (which must be a node). `progress` gets `Dominators` from `(0, total)` to
/// `(total, total)`; an `Err` from it stops the computation with that error.
pub fn dominators(
out_start: &[u32],
out: &[u32],
root: u32,
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<Dominators, HeapError> {
compute(out_start, out, root, store, progress, cancel).map(|(dominators, _)| dominators)
}
/// [`dominators`], plus the number of steps of work it took.
fn compute(
out_start: &[u32],
out: &[u32],
root: u32,
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<(Dominators, u64), HeapError> {
let nodes = out_start.len().saturating_sub(1);
if nodes >= NONE as usize {
return Err(HeapError::TooLarge);
}
assert!((root as usize) < nodes, "root {root} is not one of the {nodes} nodes");
// Three walks of at most `nodes` vertices each: DFS, semidominators, immediate dominators.
let span = nodes as u64;
let mut work = Work::new(Progress::new(progress, cancel), 3 * span);
work.report(0)?;
// DFS: `pre` maps nodes to DFS numbers and `vertex` back; `cursor[d]` is the next edge of
// DFS vertex d to follow.
let mut pre = store.scratch::<u32>(nodes, NONE)?;
let mut vertex = store.scratch::<u32>(nodes, 0)?;
let mut parent = store.scratch::<u32>(nodes, 0)?;
let mut cursor = store.scratch::<u32>(nodes, 0)?;
pre[root as usize] = 0;
vertex[0] = root;
cursor[0] = out_start[root as usize];
let (mut reachable, mut d) = (1u32, 0u32);
loop {
work.tick()?;
let edge = cursor[d as usize];
if edge < out_start[vertex[d as usize] as usize + 1] {
cursor[d as usize] = edge + 1;
let w = out[edge as usize] as usize;
if pre[w] == NONE {
let next = reachable as usize;
pre[w] = reachable;
vertex[next] = w as u32;
parent[next] = d;
cursor[next] = out_start[w];
d = reachable;
reachable += 1;
work.done = u64::from(reachable);
}
} else if d == 0 {
break;
} else {
d = parent[d as usize];
}
}
let r = reachable as usize;
work.report(span)?;
// Predecessors by DFS number, as CSR: counted into `starts[w + 1]`, then filled through
// `starts[w]` as a cursor, which leaves it at w's end until the column is shifted one slot.
let mut semi = cursor;
let mut label = store.scratch::<u32>(r, 0)?;
let mut starts = store.scratch::<u32>(r + 1, 0)?;
for d in 0..r {
work.tick()?;
semi[d] = d as u32;
label[d] = d as u32;
for &w in targets(out_start, out, vertex[d]) {
work.tick()?;
starts[pre[w as usize] as usize + 1] += 1;
}
}
let edges = accumulate(&mut starts, cancel)?;
let mut preds = store.scratch::<u32>(edges as usize, 0)?;
for d in 0..r {
work.tick()?;
for &w in targets(out_start, out, vertex[d]) {
work.tick()?;
let slot = &mut starts[pre[w as usize] as usize];
preds[*slot as usize] = d as u32;
*slot += 1;
}
}
starts.copy_within(0..r, 1);
starts[0] = 0;
drop(pre);
work.report(span)?;
// Semidominators in reverse preorder, linking each finished vertex to its parent. Vertex w
// waits in its semidominator's bucket until a child of that semidominator is linked; then
// `eval` gives w's idom, or a vertex whose idom is also w's, fixed up in preorder below.
// `idom[v]` is the next vertex in v's bucket until v leaves the bucket.
let mut ancestor = store.scratch::<u32>(r, NONE)?;
let mut bucket = store.scratch::<u32>(r, NONE)?;
let mut idom = store.scratch::<u32>(r, NONE)?;
for w in (1..r).rev() {
work.done = span + (r - w) as u64;
work.tick()?;
let mut s = semi[w];
for &v in &preds[starts[w] as usize..starts[w + 1] as usize] {
work.tick()?;
let u = eval(v, &mut ancestor, &mut label, &semi, &mut work)?;
s = s.min(semi[u as usize]);
}
semi[w] = s;
idom[w] = bucket[s as usize];
bucket[s as usize] = w as u32;
let p = parent[w];
ancestor[w] = p;
let mut v = std::mem::replace(&mut bucket[p as usize], NONE);
while v != NONE {
work.tick()?;
let next = idom[v as usize];
let u = eval(v, &mut ancestor, &mut label, &semi, &mut work)?;
idom[v as usize] = if semi[u as usize] < semi[v as usize] { u } else { p };
v = next;
}
}
drop((label, starts, preds, ancestor, bucket, parent));
work.report(2 * span)?;
// Immediate dominators, translated back to nodes; in preorder, so a deferred vertex's
// stand-in already has its final idom.
let mut node_idom = store.column::<u32>(IDOM_FILE, nodes, NONE)?;
let mut preorder = store.scratch::<u32>(r, 0)?;
idom[0] = 0;
node_idom[root as usize] = root;
preorder[0] = root;
for w in 1..r {
work.done = 2 * span + w as u64;
work.tick()?;
if idom[w] != semi[w] {
idom[w] = idom[idom[w] as usize];
}
node_idom[vertex[w] as usize] = vertex[idom[w] as usize];
preorder[w] = vertex[w];
}
work.report(3 * span)?;
Ok((Dominators { idom: node_idom, preorder, reachable }, work.steps))
}
fn targets<'a>(out_start: &[u32], out: &'a [u32], v: u32) -> &'a [u32] {
let v = v as usize;
&out[out_start[v] as usize..out_start[v + 1] as usize]
}
/// Counts steps of work and reports progress, which also looks at the cancellation token, at
/// every phase boundary and after every [`CHECK_EVERY`] steps, whichever loop they are in.
struct Work<'a> {
progress: Progress<'a>,
total: u64,
/// Progress to report at the next check; the loops keep it current.
done: u64,
steps: u64,
next_report: u64,
}
impl<'a> Work<'a> {
fn new(progress: Progress<'a>, total: u64) -> Self {
Self { progress, total, done: 0, steps: 0, next_report: CHECK_EVERY }
}
fn report(&mut self, done: u64) -> Result<(), HeapError> {
self.done = done;
self.next_report = self.steps + CHECK_EVERY;
self.progress.report(PHASE, done, self.total)
}
#[inline]
fn tick(&mut self) -> Result<(), HeapError> {
self.steps += 1;
if self.steps < self.next_report { Ok(()) } else { self.report(self.done) }
}
}
/// The vertex of smallest semidominator on the forest path from `v` up to, not including, its
/// tree's root (`v` itself when it is a root), compressing that path so the root becomes the
/// ancestor of every vertex on it. The walk up reverses the `ancestor` links so the walk down can
/// fold labels nearest-root first, as the recursive compress does, without a stack.
fn eval(v: u32, ancestor: &mut [u32], label: &mut [u32], semi: &[u32], work: &mut Work) -> Result<u32, HeapError> {
let at = |x: u32| x as usize;
if ancestor[at(v)] == NONE {
return Ok(v);
}
let (mut x, mut below) = (v, NONE);
while ancestor[at(ancestor[at(x)])] != NONE {
work.tick()?;
let up = ancestor[at(x)];
ancestor[at(x)] = below;
below = x;
x = up;
}
// `x` is the root's child, already compressed; `below` is the vertex under it.
let root = ancestor[at(x)];
let mut best = label[at(x)];
while below != NONE {
work.tick()?;
let y = at(below);
if semi[at(best)] < semi[at(label[y])] {
label[y] = best;
} else {
best = label[y];
}
below = ancestor[y];
ancestor[y] = root;
}
Ok(label[at(v)])
}
#[cfg(test)]
mod tests;
+348
Переглянути файл
@@ -0,0 +1,348 @@
use std::cell::RefCell;
use std::time::Instant;
use proptest::collection::vec;
use proptest::prelude::*;
use super::*;
use crate::NONE;
use crate::graph::build_graph;
use crate::testing::{XorShift, sample};
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
/// CSR of `edges` over `n` nodes, targets in edge order.
fn csr(n: usize, edges: &[(u32, u32)]) -> (Vec<u32>, Vec<u32>) {
let mut start = vec![0u32; n + 1];
for &(from, _) in edges {
start[from as usize + 1] += 1;
}
for v in 0..n {
start[v + 1] += start[v];
}
let mut cursor = start.clone();
let mut out = vec![0u32; edges.len()];
for &(from, to) in edges {
out[cursor[from as usize] as usize] = to;
cursor[from as usize] += 1;
}
(start, out)
}
fn run(n: usize, edges: &[(u32, u32)], root: u32) -> Dominators {
let (start, out) = csr(n, edges);
dominators(&start, &out, root, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap()
}
fn preds(n: usize, edges: &[(u32, u32)]) -> Vec<Vec<u32>> {
let mut preds = vec![Vec::new(); n];
for &(from, to) in edges {
preds[to as usize].push(from);
}
preds
}
/// Immediate dominators from iterated `Dom` sets (one `u64` bitset per node, `n <= 64`).
fn dataflow_idom(n: usize, edges: &[(u32, u32)], root: u32) -> Vec<u32> {
let (start, out) = csr(n, edges);
let mut reachable = vec![false; n];
let mut todo = vec![root];
reachable[root as usize] = true;
while let Some(v) = todo.pop() {
for &w in &out[start[v as usize] as usize..start[v as usize + 1] as usize] {
if !std::mem::replace(&mut reachable[w as usize], true) {
todo.push(w);
}
}
}
let preds = preds(n, edges);
let mut dom = vec![u64::MAX; n];
dom[root as usize] = 1 << root;
let mut changed = true;
while changed {
changed = false;
for v in (0..n).filter(|&v| reachable[v] && v != root as usize) {
let meet =
preds[v].iter().filter(|&&p| reachable[p as usize]).fold(u64::MAX, |acc, &p| acc & dom[p as usize]);
let next = meet | 1 << v;
if next != dom[v] {
dom[v] = next;
changed = true;
}
}
}
(0..n)
.map(|v| {
if !reachable[v] {
NONE
} else if v == root as usize {
root
} else {
// The strict dominator dominated by all the others has the most dominators itself.
let strict = dom[v] & !(1 << v);
(0..n as u32).filter(|&d| strict & 1 << d != 0).max_by_key(|&d| dom[d as usize].count_ones()).unwrap()
}
})
.collect()
}
/// Cooper–Harvey–Kennedy iteration over reverse postorder from an explicit-stack DFS.
fn chk_idom(n: usize, edges: &[(u32, u32)], root: u32) -> Vec<u32> {
let (start, out) = csr(n, edges);
let mut post_number = vec![NONE; n];
let mut postorder = Vec::new();
let mut seen = vec![false; n];
let mut stack = vec![(root, start[root as usize])];
seen[root as usize] = true;
while let Some(&mut (v, ref mut next)) = stack.last_mut() {
if *next < start[v as usize + 1] {
let w = out[*next as usize];
*next += 1;
if !std::mem::replace(&mut seen[w as usize], true) {
stack.push((w, start[w as usize]));
}
} else {
post_number[v as usize] = postorder.len() as u32;
postorder.push(v);
stack.pop();
}
}
let preds = preds(n, edges);
let mut idom = vec![NONE; n];
idom[root as usize] = root;
let intersect = |idom: &[u32], mut a: u32, mut b: u32| {
while a != b {
while post_number[a as usize] < post_number[b as usize] {
a = idom[a as usize];
}
while post_number[b as usize] < post_number[a as usize] {
b = idom[b as usize];
}
}
a
};
let mut changed = true;
while changed {
changed = false;
for &v in postorder.iter().rev().filter(|&&v| v != root) {
let mut new = NONE;
for &p in &preds[v as usize] {
if idom[p as usize] != NONE {
new = if new == NONE { p } else { intersect(&idom, p, new) };
}
}
if new != idom[v as usize] {
idom[v as usize] = new;
changed = true;
}
}
}
idom
}
#[test]
fn diamond() {
let d = run(5, &[(0, 1), (0, 2), (1, 3), (2, 3), (3, 4)], 0);
assert_eq!(&*d.idom, &[0, 0, 0, 0, 3]);
assert_eq!(d.reachable, 5);
assert_eq!(&*d.preorder, &[0, 1, 3, 4, 2]);
}
#[test]
fn unreachable_is_none() {
let d = run(6, &[(0, 1), (0, 2), (1, 3), (2, 3), (3, 4), (5, 3)], 0);
assert_eq!(&*d.idom, &[0, 0, 0, 0, 3, NONE]);
assert_eq!(d.reachable, 5);
assert_eq!(d.preorder.len(), 5);
}
#[test]
fn cycle_back_to_root() {
let d = run(3, &[(0, 1), (1, 2), (2, 1), (2, 0)], 0);
assert_eq!(&*d.idom, &[0, 0, 1]);
}
#[test]
fn root_need_not_be_node_zero() {
let d = run(4, &[(3, 1), (1, 0), (3, 2), (2, 0), (0, 3)], 3);
assert_eq!(&*d.idom, &[3, 3, 3, 3]);
assert_eq!(d.preorder[0], 3);
}
#[test]
fn sample_heap_dominators() {
let bytes = sample().bytes();
let g = build_graph(&bytes, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let d = dominators(&g.out_start, &g.out, g.root, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let node = |id: u64| g.ids.rank(id).unwrap();
let idom_of = |id: u64| d.idom[node(id) as usize];
assert_eq!(idom_of(0x1100), node(0x1000), "N1 → H");
assert_eq!(idom_of(0x1200), node(0x1000), "N2 → H");
assert_eq!(idom_of(0x1300), g.root, "N3 → root");
assert_eq!(idom_of(0x1400), node(0x1300), "L → N3");
assert_eq!(idom_of(0x1500), NONE, "R is unreachable");
assert_eq!(idom_of(0x1700), NONE, "N4 only through the weak referent");
assert_eq!(d.idom[g.root as usize], g.root);
}
#[test]
fn idom_is_written_to_the_store() {
let dir = tempfile::tempdir().unwrap();
let (start, out) = csr(3, &[(0, 1), (1, 2)]);
let store = Store::Dir(dir.path().to_path_buf());
let d = dominators(&start, &out, 0, &store, &no_progress, &CancellationToken::new()).unwrap();
drop(d);
assert_eq!(&*Column::<u32>::open(&dir.path().join("idom.u32")).unwrap(), &[0, 0, 1]);
let names: Vec<_> = std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name()).collect();
assert_eq!(names, ["idom.u32"], "scratch columns leave no files");
}
#[test]
fn reports_dominators_progress_and_stops_on_cancel() {
let (start, out) = csr(3, &[(0, 1), (1, 2)]);
let calls = RefCell::new(Vec::new());
let record = |phase, done, total| {
calls.borrow_mut().push((phase, done, total));
Ok(())
};
dominators(&start, &out, 0, &Store::Ram, &record, &CancellationToken::new()).unwrap();
let calls = calls.into_inner();
let total = calls[0].2;
assert_eq!(calls.first(), Some(&(HeapPhase::Dominators, 0, total)));
assert_eq!(calls.last(), Some(&(HeapPhase::Dominators, total, total)));
assert!(calls.windows(2).all(|w| w[0].1 <= w[1].1), "progress never goes back: {calls:?}");
let cancel = CancellationToken::new();
let cancel_on_first = |_, _, _| {
cancel.cancel();
Ok(())
};
let result = dominators(&start, &out, 0, &Store::Ram, &cancel_on_first, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)));
let refuse = |_, _, _| Err(HeapError::TooLarge);
let result = dominators(&start, &out, 0, &Store::Ram, &refuse, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::TooLarge)));
}
#[test]
fn cancel_during_idom_phase() {
// Long enough that the last phase alone passes a check interval.
const NODES: u32 = (CHECK_EVERY + CHECK_EVERY / 2) as u32;
let edges: Vec<_> = (1..NODES).map(|v| (v - 1, v)).collect();
let (start, out) = csr(NODES as usize, &edges);
let span = u64::from(NODES);
let cancel = CancellationToken::new();
let calls = RefCell::new(Vec::new());
let cancel_past_semidominators = |_, done, _| {
calls.borrow_mut().push(done);
if done > 2 * span {
cancel.cancel();
}
Ok(())
};
let result = dominators(&start, &out, 0, &Store::Ram, &cancel_past_semidominators, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)));
let calls = calls.into_inner();
assert!(calls.contains(&(2 * span)), "the semidominator phase finished: {calls:?}");
let last = *calls.last().unwrap();
assert!(last > 2 * span && last < 3 * span, "stopped inside the idom phase: {calls:?}");
}
proptest! {
#[test]
fn matches_dataflow_on_random_graphs(n in 2usize..64, edges in vec((0u32..64, 0u32..64), 0..192)) {
let edges: Vec<_> = edges.into_iter().map(|(a, b)| (a % n as u32, b % n as u32)).collect();
let d = run(n, &edges, 0);
prop_assert_eq!(d.idom.to_vec(), dataflow_idom(n, &edges, 0));
prop_assert_eq!(d.preorder.len(), d.reachable as usize);
}
}
#[test]
fn matches_chk_on_a_large_random_graph() {
const NODES: u32 = 200_000;
const EXTRA_EDGES: u32 = 100_000;
let mut rng = XorShift::new(0x5EED_D0D0);
let mut edges: Vec<_> = (1..NODES).map(|v| (rng.below(v), v)).collect();
edges.extend((0..EXTRA_EDGES).map(|_| (rng.below(NODES), rng.below(NODES))));
// Visit edges in a scrambled order so the DFS tree differs from the generating tree.
for i in (1..edges.len()).rev() {
edges.swap(i, rng.below(i as u32 + 1) as usize);
}
let started = Instant::now();
let d = run(NODES as usize, &edges, 0);
let elapsed = started.elapsed();
let expected = chk_idom(NODES as usize, &edges, 0);
let mismatches = d.idom.iter().zip(&expected).filter(|(a, b)| a != b).count();
assert_eq!(mismatches, 0);
assert_eq!(d.reachable, NODES);
eprintln!("dominators of {NODES} random nodes: {elapsed:?}");
}
/// A doubly linked list whose items an array list also holds: super-root 0 → list 1 → node 0,
/// 0 → array list 2 → element array 3 → every item; node j → item j, next, prev.
fn shared_list_and_array(k: u32) -> (usize, Vec<(u32, u32)>) {
let node = |j: u32| 4 + 2 * j;
let item = |j: u32| 5 + 2 * j;
let mut edges = vec![(0, 1), (0, 2), (2, 3), (1, node(0))];
for j in 0..k {
edges.push((node(j), item(j)));
if j + 1 < k {
edges.push((node(j), node(j + 1)));
}
if j > 0 {
edges.push((node(j), node(j - 1)));
}
edges.push((3, item(j)));
}
((4 + 2 * k) as usize, edges)
}
#[test]
fn shared_list_and_array_is_near_linear() {
let (n, edges) = shared_list_and_array(29);
assert_eq!(run(n, &edges, 0).idom.to_vec(), dataflow_idom(n, &edges, 0));
let (n, edges) = shared_list_and_array(3_000);
assert_eq!(run(n, &edges, 0).idom.to_vec(), chk_idom(n, &edges, 0));
const K: u32 = 200_000;
let (n, edges) = shared_list_and_array(K);
let (start, out) = csr(n, &edges);
let started = Instant::now();
let (d, steps) = compute(&start, &out, 0, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let elapsed = started.elapsed();
let item = |j: u32| (5 + 2 * j) as usize;
assert!((0..K).all(|j| d.idom[item(j)] == 0), "every item is held by both collections");
let m = edges.len() as u64;
let bound = (n as u64 + m) * u64::from(n.ilog2());
eprintln!("shared list k={K}: {steps} steps (n={n}, m={m}, bound {bound}), {elapsed:?}");
assert!(steps <= bound, "{steps} steps for n={n}, m={m}");
}
#[test]
fn deep_chain_has_linear_dominators() {
// A small stack proves the algorithm does not recurse along the chain.
std::thread::Builder::new()
.stack_size(256 << 10)
.spawn(|| {
const NODES: u32 = 1_000_000;
let mut edges: Vec<_> = (1..NODES).map(|v| (v - 1, v)).collect();
edges.push((0, NODES - 1));
// Makes `eval` compress the whole chain at once.
edges.push((NODES - 1, 1));
let started = Instant::now();
let d = run(NODES as usize, &edges, 0);
eprintln!("dominators of a {NODES}-node chain: {:?}", started.elapsed());
assert_eq!(d.reachable, NODES);
assert_eq!(d.idom[0], 0);
assert_eq!(d.idom[NODES as usize - 1], 0);
let wrong = (1..NODES - 1).filter(|&v| d.idom[v as usize] != v - 1).count();
assert_eq!(wrong, 0);
})
.unwrap()
.join()
.unwrap();
}
+26
Переглянути файл
@@ -0,0 +1,26 @@
/// Offsets (`at`) are byte positions in the `.hprof` file.
#[derive(Debug, thiserror::Error)]
pub enum HeapError {
#[error("the dump ends early at byte {at}")]
Truncated { at: u64 },
#[error("not a heap dump (no JAVA PROFILE header)")]
NotHprof,
#[error("unsupported identifier size {0}")]
UnsupportedIdSize(u32),
#[error("unknown heap sub-record tag {tag:#04x} at byte {at}")]
UnknownSubRecord { tag: u8, at: u64 },
#[error("unknown value type {ty} at byte {at}")]
UnknownType { ty: u8, at: u64 },
#[error("class {id:#x} used at byte {at} is not in the dump")]
MissingClass { id: u64, at: u64 },
#[error("the heap has too many objects to analyze")]
TooLarge,
#[error("the dump's class and field names are too long; it is not a dump a JVM wrote")]
NamesTooLarge,
#[error("not enough disk space: {needed} bytes needed, {available} free")]
NotEnoughDiskSpace { needed: u64, available: u64 },
#[error("cancelled")]
Cancelled,
#[error("{0}")]
Io(#[from] std::io::Error),
}
+34
Переглянути файл
@@ -0,0 +1,34 @@
//! Names of the files an analysis writes into its index directory. They live in one list so a
//! failed analysis removes exactly these and never the dump or anything else sharing the directory.
pub(crate) const IDS_BITS_FILE: &str = "ids.bits.u64";
pub(crate) const IDS_WRANK_FILE: &str = "ids.wrank.u32";
pub(crate) const IDS_TABLE_FILE: &str = "ids.table.u32";
pub(crate) const IDS_CHUNKS_FILE: &str = "ids.chunks.u32";
pub(crate) const IDS_SORTED_FILE: &str = "ids.sorted.u64";
pub(crate) const TYPES_FILE: &str = "types.u32";
pub(crate) const OFFSETS_FILE: &str = "offsets.u64";
pub(crate) const IDOM_FILE: &str = "idom.u32";
pub(crate) const SHALLOW8_FILE: &str = "shallow8.u32";
pub(crate) const RETAINED_FILE: &str = "retained.u64";
pub(crate) const KIDS_START_FILE: &str = "kids_start.u32";
pub(crate) const KIDS_FILE: &str = "kids.u32";
pub(crate) const GC_PARENT_FILE: &str = "gc_parent.u32";
pub(crate) const SUMMARY_FILE: &str = "summary.json";
pub(crate) const INDEX_FILES: &[&str] = &[
IDS_BITS_FILE,
IDS_WRANK_FILE,
IDS_TABLE_FILE,
IDS_CHUNKS_FILE,
IDS_SORTED_FILE,
TYPES_FILE,
OFFSETS_FILE,
IDOM_FILE,
SHALLOW8_FILE,
RETAINED_FILE,
KIDS_START_FILE,
KIDS_FILE,
GC_PARENT_FILE,
SUMMARY_FILE,
];
+274
Переглянути файл
@@ -0,0 +1,274 @@
//! The object graph of a heap dump in CSR form: one node per object, plus a super-root.
mod classes;
mod link;
use std::collections::hash_map::Entry;
use std::collections::{HashMap, HashSet};
use ide_core::CancellationToken;
use self::classes::{ClassTable, IdHash};
use self::link::Linker;
use crate::column::{Column, Store};
use crate::error::HeapError;
use crate::files::{OFFSETS_FILE, TYPES_FILE};
use crate::hprof::{BasicType, HeapRecord, HeapRootKind, Record, parse_header, scan, scan_segment};
use crate::ids::{IdMap, IdMapBuilder};
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
/// Number of primitive array type slots, one per non-object [`BasicType`].
pub const PRIM_SLOTS: u32 = 8;
pub struct ClassInfo {
pub id: u64,
/// The hprof name with `/` turned into `.` (`java.lang.String`, `[Ljava.lang.Object;`).
pub name: String,
/// Index in [`HeapGraph::classes`]. Chains always end: a super cycle in the dump is cut where it closes.
pub super_index: Option<u32>,
/// This class's own instance fields, in dump order.
pub fields: Vec<(String, BasicType)>,
/// `(name, object id)` of reference statics; a null static has object id 0.
pub statics: Vec<(String, u64)>,
pub static_bytes: u64,
/// Index in `fields` of `java.lang.ref.Reference.referent`, on that class only.
pub weak_field: Option<usize>,
}
pub struct HeapGraph {
pub ids: IdMap,
/// Type slot of each object: class index for instances and object arrays,
/// [`Self::class_slot`] for class objects, [`Self::prim_slot`] for primitive arrays.
pub types: Column<u32>,
/// Array length, 0 for instances and classes.
pub lens: Column<u32>,
/// File offset of each object's record, for [`crate::record_at`].
pub offsets: Column<u64>,
/// `n + 2` entries: node `v`'s targets are `out[out_start[v]..out_start[v + 1]]`.
pub out_start: Column<u32>,
pub out: Column<u32>,
/// The super-root, node `n`, with an edge to every rooted object.
pub root: u32,
/// `(object, kind, thread serial)` of every root record whose object is in the dump.
pub roots: Vec<(u32, HeapRootKind, Option<u32>)>,
/// Class dumps in dump order, then a synthetic `java.lang.Class` when the dump has none.
pub classes: Vec<ClassInfo>,
/// Type slot of class objects (`java.lang.Class`).
pub class_slot: u32,
/// Thread serial → thread object, from `ROOT THREAD OBJECT` records.
pub thread_objects: HashMap<u32, u32>,
/// `Reference.referent` edges left out of the graph.
pub weak_edges_skipped: u64,
pub id_size: u8,
pub timestamp_ms: u64,
}
impl HeapGraph {
pub fn prim_slot(&self, ty: BasicType) -> u32 {
self.classes.len() as u32 + (ty as u32 - BasicType::Boolean as u32)
}
/// The element type of primitive array slot `slot`, or `None` for a class slot.
pub fn prim_type(&self, slot: u32) -> Option<BasicType> {
let offset = slot.checked_sub(self.classes.len() as u32).filter(|&o| o < PRIM_SLOTS)?;
BasicType::from_u8(offset as u8 + BasicType::Boolean as u8)
}
pub fn out_edges(&self, v: u32) -> &[u32] {
let v = v as usize;
&self.out[self.out_start[v] as usize..self.out_start[v + 1] as usize]
}
}
/// Builds the graph of the dump in `buf`. Strong edges only: `Reference.referent` is counted in
/// [`HeapGraph::weak_edges_skipped`] instead. `progress` gets `Reading` by bytes, then
/// `Linking` by objects, starting with `(Linking, 0, objects)` once the object count is known;
/// an `Err` from it stops the build with that error.
pub fn build_graph(
buf: &[u8],
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapGraph, HeapError> {
let progress = Progress::new(progress, cancel);
let header = parse_header(buf)?;
let id_size = header.id_size;
let len = buf.len() as u64;
// Pass 1: names, classes, roots and every object id.
progress.report(HeapPhase::Reading, 0, len)?;
let mut names: IdHash<&[u8]> = IdHash::default();
let mut class_names: IdHash<u64> = IdHash::default();
let mut dumps = Vec::new();
let mut class_by_id: IdHash<u32> = IdHash::default();
let mut raw_roots = Vec::new();
let mut segments = Vec::new();
let mut builder = IdMapBuilder::new();
let (mut step, mut pos) = (0, header.body_start as u64);
scan(buf, &header, &mut |record| {
progress.step(step, HeapPhase::Reading, pos, len)?;
step += 1;
match record {
Record::Utf8 { id, text } => {
names.insert(id, text);
}
Record::LoadClass { class_id, name_id } => {
class_names.insert(class_id, name_id);
}
Record::Segment(range) => {
pos = range.start as u64;
segments.push(range);
}
Record::Heap { at, record } => {
pos = at as u64;
match record {
HeapRecord::Root { kind, id, thread_serial } => raw_roots.push((id, kind, thread_serial)),
HeapRecord::Class(dump) => {
builder.insert(dump.id);
if let Entry::Vacant(slot) = class_by_id.entry(dump.id) {
slot.insert(dumps.len() as u32);
dumps.push(dump);
}
}
HeapRecord::Instance { id, .. }
| HeapRecord::ObjectArray { id, .. }
| HeapRecord::PrimitiveArray { id, .. } => builder.insert(id),
}
}
}
Ok(())
})?;
progress.report(HeapPhase::Reading, len, len)?;
let ids = builder.build(store, cancel)?;
let table = classes::resolve(dumps, class_by_id, &names, &class_names, id_size, buf.len())?;
drop((names, class_names));
let n = ids.len();
let objects = u64::from(n);
progress.report(HeapPhase::Linking, 0, objects)?;
let mut roots = Vec::new();
let mut root_targets = Vec::new();
let mut thread_objects = HashMap::new();
let (mut seen_records, mut seen_targets) = (HashSet::new(), HashSet::new());
for (i, &(id, kind, serial)) in raw_roots.iter().enumerate() {
progress.step(i as u64, HeapPhase::Linking, 0, objects)?;
let Some(object) = ids.rank(id).filter(|_| id != 0) else { continue };
if seen_records.insert((object, kind as u8, serial)) {
roots.push((object, kind, serial));
}
if seen_targets.insert(object) {
root_targets.push(object);
}
if let (HeapRootKind::ThreadObject, Some(serial)) = (kind, serial) {
thread_objects.entry(serial).or_insert(object);
}
}
drop((raw_roots, seen_records, seen_targets));
let linker = Linker::new(&ids, &table, id_size);
let nodes = n as usize;
let mut types = store.column::<u32>(TYPES_FILE, nodes, 0)?;
let mut lens = store.scratch::<u32>(nodes, 0)?;
let mut offsets = store.column::<u64>(OFFSETS_FILE, nodes, 0)?;
let mut out_start = store.scratch::<u32>(nodes + 2, 0)?;
// Both passes visit every object record once; report them as one walk over the objects.
let mut visited = 0u64;
let tick = |visited: &mut u64| {
let result = progress.step(*visited, HeapPhase::Linking, (*visited / 2).min(objects), objects);
*visited += 1;
result
};
// Pass 2: node columns and out-degrees, counted into `out_start[v + 1]`.
let mut weak_edges_skipped = 0;
for segment in &segments {
scan_segment(buf, segment.clone(), id_size, &mut |at, record| {
let Some(node) = linker.node(at, &record)? else { return Ok(()) };
tick(&mut visited)?;
let v = node.index as usize;
types[v] = node.ty;
lens[v] = node.len;
offsets[v] = at as u64;
let mut degree = 0u64;
linker.edges(&record, &mut |_, weak| {
if weak {
weak_edges_skipped += 1;
} else {
degree += 1;
}
});
let count = &mut out_start[v + 1];
*count = u32::try_from(u64::from(*count) + degree).map_err(|_| HeapError::TooLarge)?;
Ok(())
})?;
}
out_start[nodes + 1] = u32::try_from(root_targets.len()).map_err(|_| HeapError::TooLarge)?;
let edges = accumulate(&mut out_start, cancel)?;
// Pass 3: `out_start[v]` is node v's write cursor; once filled it holds v's end, so the
// column is shifted one slot right to become the starts again.
let mut out = store.scratch::<u32>(edges as usize, 0)?;
for segment in &segments {
scan_segment(buf, segment.clone(), id_size, &mut |at, record| {
let Some(node) = linker.node(at, &record)? else { return Ok(()) };
tick(&mut visited)?;
let v = node.index as usize;
let mut cursor = out_start[v] as usize;
linker.edges(&record, &mut |target, weak| {
if !weak && let Some(slot) = out.get_mut(cursor) {
*slot = target;
cursor += 1;
}
});
out_start[v] = cursor as u32;
Ok(())
})?;
}
let first_root_edge = out_start[nodes] as usize;
for (slot, &target) in out[first_root_edge..].iter_mut().zip(&root_targets) {
*slot = target;
}
out_start[nodes] = edges;
out_start.copy_within(0..nodes + 1, 1);
out_start[0] = 0;
progress.report(HeapPhase::Linking, objects, objects)?;
drop(linker);
let ClassTable { classes, class_slot, .. } = table;
Ok(HeapGraph {
ids,
types,
lens,
offsets,
out_start,
out,
root: n,
roots,
classes,
class_slot,
thread_objects,
weak_edges_skipped,
id_size,
timestamp_ms: header.timestamp_ms,
})
}
/// Turns per-node counts into running totals in place (`counts[k]` becomes the sum of
/// `counts[..=k]`) and returns the last one; past `u32::MAX` the graph is `TooLarge`.
pub(crate) fn accumulate(counts: &mut [u32], cancel: &CancellationToken) -> Result<u32, HeapError> {
let mut sum = 0u32;
for chunk in counts.chunks_mut(CHECK_EVERY as usize) {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
for count in chunk {
sum = sum.checked_add(*count).ok_or(HeapError::TooLarge)?;
*count = sum;
}
}
Ok(sum)
}
#[cfg(test)]
mod tests;
+188
Переглянути файл
@@ -0,0 +1,188 @@
//! Class dumps resolved into [`ClassInfo`]s, plus where the references sit in instance data.
use std::collections::HashMap;
use std::hash::BuildHasherDefault;
use super::{ClassInfo, PRIM_SLOTS};
use crate::column::NONE;
use crate::error::HeapError;
use crate::hprof::{BasicType, ClassDump};
use crate::ids::ChunkHasher;
/// Keyed by hprof id (an address).
pub(super) type IdHash<V> = HashMap<u64, V, BuildHasherDefault<ChunkHasher>>;
const REFERENCE: &str = "java.lang.ref.Reference";
const JAVA_LANG_CLASS: &str = "java.lang.Class";
/// Not computed yet, in [`ClassTable::fielded`].
const UNSET: u32 = NONE - 1;
pub(super) struct ClassTable {
pub(super) classes: Vec<ClassInfo>,
/// Class id → class index.
pub(super) by_id: IdHash<u32>,
pub(super) class_slot: u32,
/// Per class: `(offset, is the referent)` of each reference field in the class's own part of
/// the instance data.
pub(super) refs: Vec<Vec<(usize, bool)>>,
/// Per class: bytes of its own part of the instance data.
pub(super) own_bytes: Vec<usize>,
/// Per class: the nearest class up its chain (itself included) whose own part is not empty, or
/// `NONE`. Walking these instead of every super keeps a walk bounded by the data it reads.
pub(super) fielded: Vec<u32>,
}
/// The longest name a JVM gives a class or field (a constant pool UTF8 has a `u2` length).
const MAX_NAME_BYTES: usize = 65_535;
/// Name bytes a dump may decode beyond 4 per byte of the file.
const NAME_BUDGET_FLOOR: usize = 64 << 20;
/// Names decoded from the UTF8 records within a budget: every field of every class may name the
/// same record, so a small crafted file could otherwise ask for terabytes and abort the IDE.
struct Names<'m, 'b> {
records: &'m IdHash<&'b [u8]>,
left: usize,
}
impl Names<'_, '_> {
/// Empty for an id with no record.
fn text(&mut self, id: u64) -> Result<String, HeapError> {
let Some(bytes) = self.records.get(&id) else { return Ok(String::new()) };
if bytes.len() > MAX_NAME_BYTES || bytes.len() > self.left {
return Err(HeapError::NamesTooLarge);
}
self.left -= bytes.len();
Ok(String::from_utf8_lossy(bytes).into_owned())
}
}
/// `dumps` holds one dump per class id, and `by_id` its position; `names` are the UTF8 records
/// and `class_names` the LOAD CLASS class id → name id pairs. `file_len` bounds the name bytes.
pub(super) fn resolve(
dumps: Vec<ClassDump>,
by_id: IdHash<u32>,
names: &IdHash<&[u8]>,
class_names: &IdHash<u64>,
id_size: u8,
file_len: usize,
) -> Result<ClassTable, HeapError> {
// Room for the synthetic java.lang.Class and the primitive slots in a `u32` type id.
if dumps.len() > (u32::MAX - PRIM_SLOTS - 1) as usize {
return Err(HeapError::TooLarge);
}
let mut names = Names { records: names, left: file_len.saturating_mul(4).saturating_add(NAME_BUDGET_FLOOR) };
let mut supers: Vec<Option<u32>> =
dumps.iter().map(|d| if d.super_id == 0 { None } else { by_id.get(&d.super_id).copied() }).collect();
cut_super_cycles(&mut supers);
let mut classes = Vec::with_capacity(dumps.len() + 1);
let mut refs = Vec::with_capacity(dumps.len() + 1);
let mut own_bytes = Vec::with_capacity(dumps.len() + 1);
for (dump, super_index) in dumps.into_iter().zip(supers) {
let name = match class_names.get(&dump.id) {
Some(&name_id) if names.records.contains_key(&name_id) => names.text(name_id)?.replace('/', "."),
_ => format!("{:#x}", dump.id),
};
let fields =
dump.fields.iter().map(|&(n, ty)| Ok((names.text(n)?, ty))).collect::<Result<Vec<_>, HeapError>>()?;
let weak_field = if name == REFERENCE {
fields.iter().position(|(n, ty)| n == "referent" && *ty == BasicType::Object)
} else {
None
};
let mut offset = 0;
let mut class_refs = Vec::new();
for (k, &(_, ty)) in fields.iter().enumerate() {
if ty == BasicType::Object {
class_refs.push((offset, weak_field == Some(k)));
}
offset += ty.hprof_size(id_size);
}
refs.push(class_refs);
own_bytes.push(offset);
classes.push(ClassInfo {
id: dump.id,
name,
super_index,
fields,
statics: dump
.statics
.iter()
.map(|&(n, object)| Ok((names.text(n)?, object)))
.collect::<Result<_, HeapError>>()?,
static_bytes: dump.static_bytes,
weak_field,
});
}
let class_slot = match classes.iter().position(|c| c.name == JAVA_LANG_CLASS) {
Some(slot) => slot as u32,
None => {
classes.push(ClassInfo {
id: 0,
name: JAVA_LANG_CLASS.to_owned(),
super_index: None,
fields: Vec::new(),
statics: Vec::new(),
static_bytes: 0,
weak_field: None,
});
refs.push(Vec::new());
own_bytes.push(0);
classes.len() as u32 - 1
}
};
let fielded = nearest_fielded(&classes, &own_bytes);
Ok(ClassTable { classes, by_id, class_slot, refs, own_bytes, fielded })
}
/// A dump may claim `A extends B extends A`; the edge that closes a cycle is dropped so every
/// walk up the chain ends. Iterative, so a long chain cannot overflow the stack.
fn cut_super_cycles(supers: &mut [Option<u32>]) {
const NEW: u8 = 0;
const ON_PATH: u8 = 1;
const DONE: u8 = 2;
let mut state = vec![NEW; supers.len()];
let mut path = Vec::new();
for start in 0..supers.len() {
let mut c = start;
while state[c] == NEW {
state[c] = ON_PATH;
path.push(c);
match supers[c] {
Some(s) if state[s as usize] == ON_PATH => {
supers[c] = None;
break;
}
Some(s) => c = s as usize,
None => break,
}
}
for p in path.drain(..) {
state[p] = DONE;
}
}
}
fn nearest_fielded(classes: &[ClassInfo], own_bytes: &[usize]) -> Vec<u32> {
let mut fielded: Vec<u32> =
own_bytes.iter().enumerate().map(|(c, &b)| if b > 0 { c as u32 } else { UNSET }).collect();
let mut path = Vec::new();
for start in 0..classes.len() {
let mut c = start;
let found = loop {
if fielded[c] != UNSET {
break fielded[c];
}
path.push(c);
match classes[c].super_index {
Some(s) => c = s as usize,
None => break NONE,
}
};
for p in path.drain(..) {
fielded[p] = found;
}
}
fielded
}
+107
Переглянути файл
@@ -0,0 +1,107 @@
//! Turns one heap sub-record into its node (index, type, length) and its outgoing edges. Both
//! linking passes go through here, so the degrees counted in the first match the edges written
//! in the second.
use super::classes::ClassTable;
use crate::column::NONE;
use crate::error::HeapError;
use crate::hprof::{BasicType, HeapRecord};
use crate::ids::IdMap;
pub(super) struct Node {
pub(super) index: u32,
pub(super) ty: u32,
pub(super) len: u32,
}
pub(super) struct Linker<'a> {
ids: &'a IdMap,
table: &'a ClassTable,
/// Per class: the node of its class object, `NONE` for the synthetic `java.lang.Class`.
class_objects: Vec<u32>,
id_size: u8,
}
impl<'a> Linker<'a> {
pub(super) fn new(ids: &'a IdMap, table: &'a ClassTable, id_size: u8) -> Self {
let class_objects =
table.classes.iter().map(|c| if c.id == 0 { NONE } else { ids.rank(c.id).unwrap_or(NONE) }).collect();
Self { ids, table, class_objects, id_size }
}
/// `None` for root records. `at` is the record's offset, for the error of an instance or array
/// whose class is not in the dump.
pub(super) fn node(&self, at: usize, record: &HeapRecord<'_>) -> Result<Option<Node>, HeapError> {
let class_index = |class_id: u64| {
self.table.by_id.get(&class_id).copied().ok_or(HeapError::MissingClass { id: class_id, at: at as u64 })
};
let (id, ty, len) = match record {
HeapRecord::Root { .. } => return Ok(None),
HeapRecord::Class(c) => (c.id, self.table.class_slot, 0),
HeapRecord::Instance { id, class_id, .. } => (*id, class_index(*class_id)?, 0),
HeapRecord::ObjectArray { id, class_id, len, .. } => (*id, class_index(*class_id)?, *len),
HeapRecord::PrimitiveArray { id, ty, len, .. } => {
let prim_base = self.table.classes.len() as u32;
(*id, prim_base + (*ty as u32 - BasicType::Boolean as u32), *len)
}
};
Ok(self.ids.rank(id).map(|index| Node { index, ty, len }))
}
/// Calls `edge(target, is_referent)` for every reference to an object in the dump, in field
/// order; `is_referent` marks `Reference.referent`, which callers leave out of the graph.
pub(super) fn edges(&self, record: &HeapRecord<'_>, edge: &mut impl FnMut(u32, bool)) {
let mut to = |id: u64, weak: bool| {
if let Some(target) = self.target(id) {
edge(target, weak);
}
};
match record {
HeapRecord::Root { .. } | HeapRecord::PrimitiveArray { .. } => {}
HeapRecord::Class(c) => {
to(c.super_id, false);
to(c.loader_id, false);
for &(_, object) in &c.statics {
to(object, false);
}
}
HeapRecord::ObjectArray { data, .. } => {
for item in data.chunks_exact(usize::from(self.id_size)) {
to(read_id(item, 0, self.id_size).unwrap_or(0), false);
}
}
HeapRecord::Instance { class_id, data, .. } => {
let Some(&class) = self.table.by_id.get(class_id) else { return };
let mut base = 0;
let mut next = self.table.fielded[class as usize];
'chain: while next != NONE && base < data.len() {
let c = next as usize;
for &(offset, weak) in &self.table.refs[c] {
let Some(id) = read_id(data, base + offset, self.id_size) else { break 'chain };
to(id, weak);
}
base += self.table.own_bytes[c];
next = self.table.classes[c].super_index.map_or(NONE, |s| self.table.fielded[s as usize]);
}
let object = self.class_objects[class as usize];
if object != NONE {
edge(object, false);
}
}
}
}
/// Id 0 is null; ids not in the dump have no node.
fn target(&self, id: u64) -> Option<u32> {
if id == 0 { None } else { self.ids.rank(id) }
}
}
/// The big-endian id at `at`, or `None` when `data` ends first.
fn read_id(data: &[u8], at: usize, id_size: u8) -> Option<u64> {
let bytes = data.get(at..at.checked_add(usize::from(id_size))?)?;
Some(match *bytes {
[a, b, c, d] => u64::from(u32::from_be_bytes([a, b, c, d])),
_ => u64::from_be_bytes(bytes.try_into().ok()?),
})
}
+479
Переглянути файл
@@ -0,0 +1,479 @@
use std::cell::RefCell;
use proptest::prelude::*;
use super::*;
use crate::hprof::{HeapRecord, record_at};
use crate::testing::{HprofWriter, MiniHeap, sample};
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N2: u64 = 0x1200;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const R: u64 = 0x1500;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn build(bytes: &[u8]) -> Result<HeapGraph, HeapError> {
build_graph(bytes, &Store::Ram, &no_progress, &CancellationToken::new())
}
fn index(g: &HeapGraph, id: u64) -> u32 {
g.ids.rank(id).unwrap_or_else(|| panic!("{id:#x} is not in the graph"))
}
fn class(g: &HeapGraph, name: &str) -> (u32, u64) {
let slot = g.classes.iter().position(|c| c.name == name).unwrap_or_else(|| panic!("no class {name}"));
(slot as u32, g.classes[slot].id)
}
/// Targets of `id`'s edges, as ids, in edge order.
fn out_of(g: &HeapGraph, id: u64) -> Vec<u64> {
g.out_edges(index(g, id)).iter().map(|&t| g.ids.id_at(t)).collect()
}
fn root_targets(g: &HeapGraph) -> Vec<u64> {
g.out_edges(g.root).iter().map(|&t| g.ids.id_at(t)).collect()
}
fn assert_csr(g: &HeapGraph) {
let n = g.ids.len();
assert_eq!(g.root, n);
assert_eq!(g.out_start.len(), n as usize + 2);
assert_eq!(g.out_start[0], 0);
assert!(g.out_start.windows(2).all(|w| w[0] <= w[1]), "out_start is not monotone");
assert_eq!(g.out_start[n as usize + 1] as usize, g.out.len());
assert!(g.out.iter().all(|&t| t < n), "edge to a node outside the objects");
assert_eq!((g.types.len(), g.lens.len(), g.offsets.len()), (n as usize, n as usize, n as usize));
let slots = g.classes.len() as u32 + PRIM_SLOTS;
assert!(g.types.iter().all(|&t| t < slots), "type outside the slots");
for c in &g.classes {
let mut steps = 0;
let mut cursor = c.super_index;
while let Some(s) = cursor {
steps += 1;
assert!(steps <= g.classes.len(), "super chain of {} does not end", c.name);
cursor = g.classes[s as usize].super_index;
}
}
}
#[test]
fn edges_follow_fields_and_skip_weak_referents() {
let g = build(&sample().bytes()).unwrap();
assert_csr(&g);
let (_, holder) = class(&g, "Holder");
let (_, node) = class(&g, "Node");
let (_, weak) = class(&g, "java.lang.ref.WeakReference");
assert_eq!(out_of(&g, H), [N1, N2, holder]);
assert_eq!(out_of(&g, N1), [N3, node]);
assert_eq!(out_of(&g, N3), [L, node]);
assert_eq!(out_of(&g, W), [weak]);
assert_eq!(out_of(&g, N4), [node]);
assert_eq!(out_of(&g, L), Vec::<u64>::new());
assert_eq!(g.weak_edges_skipped, 1);
let (reference, _) = class(&g, "java.lang.ref.Reference");
assert_eq!(g.classes[reference as usize].weak_field, Some(0));
assert!(g.classes.iter().filter(|c| c.name != "java.lang.ref.Reference").all(|c| c.weak_field.is_none()));
}
#[test]
fn object_arrays_skip_null_and_unknown() {
let mut heap = sample();
heap.array_of(0x1800, "[Ljava.lang.Object;", &[N2, 0, 0x9990, N1, 0xDEAD_BEEF]);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, R), [N1]);
assert_eq!(out_of(&g, 0x1800), [N2, N1]);
}
#[test]
fn super_root_points_at_each_root_once() {
let mut heap = sample();
heap.root(HeapRootKind::JniGlobal, H).root(HeapRootKind::MonitorUsed, H).root(HeapRootKind::JniLocal, 0x9990);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
let targets = root_targets(&g);
assert_eq!(targets.iter().filter(|&&t| t == H).count(), 1);
assert!(targets.contains(&N3) && targets.contains(&W));
assert!(!targets.contains(&N1) && !targets.contains(&R) && !targets.contains(&N4));
let classes = g.classes.iter().filter(|c| c.id != 0).count();
assert_eq!(targets.len(), 3 + classes, "H, N3, W and every sticky class");
let mut sorted = targets.clone();
sorted.sort_unstable();
sorted.dedup();
assert_eq!(sorted.len(), targets.len());
let h = index(&g, H);
let of_h: Vec<_> = g.roots.iter().filter(|r| r.0 == h).map(|r| (r.1, r.2)).collect();
assert_eq!(of_h, [(HeapRootKind::JniGlobal, None), (HeapRootKind::MonitorUsed, None)]);
assert!(g.roots.contains(&(index(&g, N3), HeapRootKind::JavaFrame, Some(1))));
assert!(g.roots.iter().all(|r| r.0 < g.root));
}
#[test]
fn class_dump_edges_cover_statics() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object")
.load_class(0x100, 1)
.class(0x100, 0, &[], &[])
.string(2, "Config")
.string(3, "INSTANCE")
.string(4, "EMPTY")
.string(5, "LOST")
.load_class(0x200, 2)
.class_with_loader(0x200, 0x100, 0x1200, &[], &[(3, N1), (4, 0), (5, 0x9990)])
.instance(N1, 0x100, &[])
.instance(0x1200, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x200), [0x100, 0x1200, N1]);
assert_eq!(out_of(&g, 0x100), Vec::<u64>::new());
let (config, _) = class(&g, "Config");
let info = &g.classes[config as usize];
let statics: Vec<_> = info.statics.iter().map(|(n, id)| (n.as_str(), *id)).collect();
assert_eq!(statics, [("INSTANCE", N1), ("EMPTY", 0), ("LOST", 0x9990)]);
assert_eq!(info.super_index, Some(class(&g, "java.lang.Object").0));
}
#[test]
fn missing_class_is_an_error() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]).instance(N1, 0x999, &[]);
let bytes = w.finish();
let at = instance_offset(&bytes, N1);
let err = build(&bytes).err().unwrap();
assert!(matches!(err, HeapError::MissingClass { id: 0x999, at: a } if a == at as u64), "{err:?}");
let mut w = HprofWriter::new();
w.obj_array(0x10, 0x999, &[]);
assert!(matches!(build(&w.finish()), Err(HeapError::MissingClass { id: 0x999, .. })));
}
fn instance_offset(bytes: &[u8], id: u64) -> usize {
let header = crate::hprof::parse_header(bytes).unwrap();
let mut found = None;
crate::hprof::scan(bytes, &header, &mut |r| {
if let crate::hprof::Record::Heap { at, record: HeapRecord::Instance { id: i, .. } } = r
&& i == id
{
found = Some(at);
}
Ok(())
})
.unwrap();
found.unwrap()
}
#[test]
fn cancel_during_linking_returns_cancelled() {
let bytes = sample().bytes();
let cancelled = CancellationToken::new();
cancelled.cancel();
let result = build_graph(&bytes, &Store::Ram, &no_progress, &cancelled);
assert!(matches!(result, Err(HeapError::Cancelled)));
let cancel = CancellationToken::new();
let phases = RefCell::new(Vec::new());
let result = build_graph(
&bytes,
&Store::Ram,
&|phase, done, total| {
phases.borrow_mut().push((phase, done, total));
if phase == HeapPhase::Linking {
cancel.cancel();
}
Ok(())
},
&cancel,
);
assert!(matches!(result, Err(HeapError::Cancelled)));
let phases = phases.into_inner();
assert_eq!(phases.first(), Some(&(HeapPhase::Reading, 0, bytes.len() as u64)));
assert_eq!(phases.last(), Some(&(HeapPhase::Linking, 0, 15)), "stops at the first Linking report");
}
#[test]
fn progress_error_stops_the_build() {
let bytes = sample().bytes();
let fail = |phase, _, total| {
if phase == HeapPhase::Linking {
Err(HeapError::NotEnoughDiskSpace { needed: total * 90, available: 0 })
} else {
Ok(())
}
};
let result = build_graph(&bytes, &Store::Ram, &fail, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::NotEnoughDiskSpace { needed: 1350, available: 0 })));
}
#[test]
fn progress_reads_bytes_then_links_objects() {
let bytes = sample().bytes();
let phases = RefCell::new(Vec::new());
let g = build_graph(
&bytes,
&Store::Ram,
&|phase, done, total| {
phases.borrow_mut().push((phase, done, total));
Ok(())
},
&CancellationToken::new(),
)
.unwrap();
let phases = phases.into_inner();
let len = bytes.len() as u64;
let objects = u64::from(g.root);
assert_eq!(objects, 15, "7 classes and 8 objects");
assert_eq!(phases.first(), Some(&(HeapPhase::Reading, 0, len)));
assert!(phases.contains(&(HeapPhase::Reading, len, len)));
let linking: Vec<_> = phases.iter().filter(|p| p.0 == HeapPhase::Linking).collect();
assert_eq!(linking.first(), Some(&&(HeapPhase::Linking, 0, objects)));
assert_eq!(linking.iter().filter(|p| ***p == (HeapPhase::Linking, 0, objects)).count(), 1, "{linking:?}");
assert_eq!(linking.last(), Some(&&(HeapPhase::Linking, objects, objects)));
assert!(phases.iter().all(|&(_, done, total)| done <= total));
}
#[test]
fn types_lengths_and_offsets() {
let bytes = sample().bytes();
let g = build(&bytes).unwrap();
let (holder, holder_id) = class(&g, "Holder");
let (node, _) = class(&g, "Node");
let (array, _) = class(&g, "[Ljava.lang.Object;");
let (java_lang_class, _) = class(&g, "java.lang.Class");
assert_eq!(g.class_slot, java_lang_class);
let ty = |id| g.types[index(&g, id) as usize];
let len = |id| g.lens[index(&g, id) as usize];
assert_eq!([ty(H), ty(N1), ty(R), ty(holder_id)], [holder, node, array, java_lang_class]);
assert_eq!(ty(L), g.prim_slot(BasicType::Long));
assert_eq!(g.prim_type(ty(L)), Some(BasicType::Long));
assert_eq!(g.prim_type(holder), None);
assert_eq!(g.prim_type(g.classes.len() as u32 + PRIM_SLOTS), None);
assert_eq!(g.prim_slot(BasicType::Boolean), g.classes.len() as u32);
assert_eq!(g.prim_slot(BasicType::Long), g.classes.len() as u32 + PRIM_SLOTS - 1);
assert_eq!([len(H), len(L), len(R), len(holder_id)], [0, 2, 3, 0]);
for i in 0..g.root {
let id = match record_at(&bytes, g.offsets[i as usize] as usize, g.id_size).unwrap() {
HeapRecord::Class(c) => c.id,
HeapRecord::Instance { id, .. } | HeapRecord::ObjectArray { id, .. } => id,
HeapRecord::PrimitiveArray { id, .. } => id,
other => panic!("offset of object {i} points at {other:?}"),
};
assert_eq!(id, g.ids.id_at(i));
}
assert_eq!((g.id_size, g.timestamp_ms), (8, 1_700_000_000_000));
let holder_info = &g.classes[holder as usize];
let fields: Vec<_> = holder_info.fields.iter().map(|(n, t)| (n.as_str(), *t)).collect();
assert_eq!(fields, [("a", BasicType::Object), ("b", BasicType::Object)]);
assert_eq!(holder_info.super_index, Some(class(&g, "java.lang.Object").0));
assert_eq!(g.classes[class(&g, "java.lang.Object").0 as usize].super_index, None);
}
#[test]
fn dump_without_java_lang_class_gets_a_synthetic_slot() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]).instance(N1, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(g.classes.len(), 2);
let synthetic = &g.classes[g.class_slot as usize];
assert_eq!((synthetic.id, synthetic.name.as_str(), synthetic.super_index), (0, "java.lang.Class", None));
assert_eq!(g.types[index(&g, 0x100) as usize], g.class_slot);
assert_eq!(g.types[index(&g, N1) as usize], 0);
}
#[test]
fn classes_without_a_name_record_get_their_address() {
let mut w = HprofWriter::new();
w.class(0x100, 0, &[(9, BasicType::Object)], &[])
.instance(N1, 0x100, &N2.to_be_bytes())
.instance(N2, 0x100, &[0; 8]);
let g = build(&w.finish()).unwrap();
assert_eq!(g.classes[0].name, "0x100");
assert_eq!(g.classes[0].fields, [(String::new(), BasicType::Object)]);
assert_eq!(out_of(&g, N1), [N2, 0x100]);
}
#[test]
fn thread_objects_map_serials_to_threads() {
let mut heap = sample();
heap.class("java.lang.Thread", Some("java.lang.Object"), &[])
.object(0x2000, "java.lang.Thread", &[])
.object(0x2100, "java.lang.Thread", &[])
.root_in_thread(HeapRootKind::ThreadObject, 0x2000, 1)
.root_in_thread(HeapRootKind::ThreadObject, 0x2100, 7)
.root_in_thread(HeapRootKind::ThreadObject, 0x9990, 9);
let g = build(&heap.bytes()).unwrap();
let expected: HashMap<u32, u32> = [(1, index(&g, 0x2000)), (7, index(&g, 0x2100))].into();
assert_eq!(g.thread_objects, expected);
assert!(g.roots.contains(&(index(&g, 0x2100), HeapRootKind::ThreadObject, Some(7))));
}
#[test]
fn inherited_fields_follow_the_class_first_layout() {
let mut heap = MiniHeap::new();
heap.class("A", Some("java.lang.Object"), &[("a", BasicType::Object), ("n", BasicType::Int)])
.class("Empty", Some("A"), &[])
.class("B", Some("Empty"), &[("flag", BasicType::Byte), ("b", BasicType::Object)])
.object(0x10, "B", &[("a", 0x20), ("b", 0x30), ("flag", 1)])
.object(0x20, "A", &[])
.object(0x30, "A", &[]);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x30, 0x20, class(&g, "B").1]);
}
#[test]
fn super_class_cycles_are_cut() {
let mut w = HprofWriter::new();
w.string(1, "A")
.string(2, "B")
.string(3, "S")
.string(4, "f")
.load_class(0x100, 1)
.load_class(0x200, 2)
.load_class(0x300, 3)
.class(0x100, 0x200, &[(4, BasicType::Object)], &[])
.class(0x200, 0x100, &[(4, BasicType::Object)], &[])
.class(0x300, 0x300, &[], &[])
.instance(0x10, 0x100, &[[0; 7].as_slice(), &[0x20], &[0; 7], &[0x30]].concat())
.instance(0x20, 0x300, &[])
.instance(0x30, 0x300, &[0; 64]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x20, 0x30, 0x100]);
assert_eq!(out_of(&g, 0x30), [0x300]);
}
#[test]
fn short_instance_data_stops_at_the_last_whole_field() {
let mut w = HprofWriter::new();
w.string(1, "P")
.string(2, "x")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (2, BasicType::Object)], &[])
.instance(0x10, 0x100, &[[0; 7].as_slice(), &[0x20], &[0; 5]].concat())
.instance(0x20, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x20, 0x100]);
}
#[test]
fn id_size_4_dump_links() {
let mut w = HprofWriter::with_id_size(4);
w.string(1, "Node")
.string(2, "next")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object)], &[(2, 0x20)])
.instance(0x10, 0x100, &0x20u32.to_be_bytes())
.instance(0x20, 0x100, &[0; 4])
.obj_array(0x30, 0x100, &[0x10, 0, 0x20])
.root(HeapRootKind::JniGlobal, 0x30);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(g.id_size, 4);
assert_eq!(out_of(&g, 0x10), [0x20, 0x100]);
assert_eq!(out_of(&g, 0x30), [0x10, 0x20]);
assert_eq!(out_of(&g, 0x100), [0x20]);
assert_eq!(root_targets(&g), [0x30]);
}
#[test]
fn dir_store_keeps_types_and_offsets_only() {
let dir = tempfile::tempdir().unwrap();
// The id map's own files (`ids.*`) depend on its layout; MiniHeap's class ids are unaligned.
let names = || {
let mut names: Vec<_> = std::fs::read_dir(dir.path())
.unwrap()
.map(|e| e.unwrap().file_name().into_string().unwrap())
.filter(|name| !name.starts_with("ids."))
.collect();
names.sort();
names
};
let g =
build_graph(&sample().bytes(), &Store::Dir(dir.path().to_path_buf()), &no_progress, &CancellationToken::new())
.unwrap();
assert_eq!(out_of(&g, H).len(), 3);
let kept = ["offsets.u64", "types.u32"];
assert!(dir.path().join("ids.sorted.u64").exists());
#[cfg(unix)]
assert_eq!(names(), kept);
drop(g);
assert_eq!(names(), kept);
}
#[test]
fn empty_dump_has_only_the_super_root() {
let g = build(&HprofWriter::new().finish()).unwrap();
assert_csr(&g);
assert_eq!((g.root, g.out.len(), g.classes.len()), (0, 0, 1));
assert_eq!(g.class_slot, 0);
}
#[test]
fn edge_count_past_u32_is_too_large() {
let cancel = CancellationToken::new();
let mut counts = [0, 2, 0, 3, 1];
assert_eq!(accumulate(&mut counts, &cancel).unwrap(), 6);
assert_eq!(counts, [0, 2, 2, 5, 6]);
let mut full = [0, u32::MAX - 1, 1];
assert_eq!(accumulate(&mut full, &cancel).unwrap(), u32::MAX);
let mut over = [0, u32::MAX, 1];
assert!(matches!(accumulate(&mut over, &cancel), Err(HeapError::TooLarge)));
cancel.cancel();
assert!(matches!(accumulate(&mut [0, 1], &cancel), Err(HeapError::Cancelled)));
}
/// A class with `fields` instance fields, every one named by the same `name_len`-byte UTF8 record.
fn shared_long_field_names(name_len: usize, fields: usize) -> Vec<u8> {
let mut w = HprofWriter::new();
let long = "f".repeat(name_len);
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]);
w.string(2, "Wide").string(3, &long).load_class(0x200, 2);
w.class(0x200, 0x100, &vec![(3, BasicType::Int); fields], &[]);
w.finish()
}
#[test]
fn names_that_would_decode_far_larger_than_the_file_are_refused() {
// 128 MiB of names from a file of about 70 KB.
let bytes = shared_long_field_names(32 << 10, 4000);
assert!(bytes.len() < 100 << 10, "{}", bytes.len());
let err = build(&bytes).err().expect("refused");
assert!(matches!(err, HeapError::NamesTooLarge), "{err:?}");
}
#[test]
fn a_name_longer_than_the_jvm_allows_is_refused() {
let mut w = HprofWriter::new();
w.string(1, &"C".repeat(70_000)).load_class(0x100, 1).class(0x100, 0, &[], &[]);
assert!(matches!(build(&w.finish()), Err(HeapError::NamesTooLarge)));
}
#[test]
fn real_sized_names_shared_by_many_fields_are_fine() {
let g = build(&shared_long_field_names(40, 4000)).unwrap();
assert_eq!(g.classes[class(&g, "Wide").0 as usize].fields.len(), 4000);
}
proptest! {
#[test]
fn build_graph_never_panics_on_mutated_dumps(flips in proptest::collection::vec((0usize..4096, any::<u8>()), 1..12)) {
let mut bytes = sample().bytes();
for (pos, value) in flips {
let at = pos % bytes.len();
bytes[at] = value;
}
if let Ok(g) = build(&bytes) {
assert_csr(&g);
}
}
}
+945
Переглянути файл
@@ -0,0 +1,945 @@
//! Reader for HotSpot `.hprof` heap dumps (format 1.0.1 / 1.0.2, 4- or 8-byte ids).
use std::ops::Range;
use serde::{Deserialize, Serialize};
use crate::error::HeapError;
use crate::reader::Reader;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum BasicType {
Object = 2,
Boolean = 4,
Char = 5,
Float = 6,
Double = 7,
Byte = 8,
Short = 9,
Int = 10,
Long = 11,
}
impl BasicType {
pub fn from_u8(v: u8) -> Option<Self> {
Some(match v {
2 => Self::Object,
4 => Self::Boolean,
5 => Self::Char,
6 => Self::Float,
7 => Self::Double,
8 => Self::Byte,
9 => Self::Short,
10 => Self::Int,
11 => Self::Long,
_ => return None,
})
}
/// Bytes one value takes in the dump; `id_size` is the dump's identifier size.
pub fn hprof_size(self, id_size: u8) -> usize {
match self {
Self::Object => usize::from(id_size),
Self::Boolean | Self::Byte => 1,
Self::Char | Self::Short => 2,
Self::Float | Self::Int => 4,
Self::Double | Self::Long => 8,
}
}
/// Bytes one value takes in the JVM heap; `ref_size` is 4 with compressed oops, else 8.
pub fn heap_size(self, ref_size: u8) -> u64 {
match self {
Self::Object => u64::from(ref_size),
other => other.hprof_size(ref_size) as u64,
}
}
/// JVM class name of an array of this type (`[I`); `Object` arrays have no single name, so
/// they report the most general one.
pub fn array_name(self) -> &'static str {
match self {
Self::Object => "[Ljava.lang.Object;",
Self::Boolean => "[Z",
Self::Char => "[C",
Self::Float => "[F",
Self::Double => "[D",
Self::Byte => "[B",
Self::Short => "[S",
Self::Int => "[I",
Self::Long => "[J",
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum HeapRootKind {
Unknown,
JniGlobal,
JniLocal,
JavaFrame,
NativeStack,
StickyClass,
ThreadBlock,
MonitorUsed,
ThreadObject,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct HprofHeader {
pub version: String,
pub id_size: u8,
pub timestamp_ms: u64,
/// Offset of the first record.
pub body_start: usize,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClassDump {
pub id: u64,
pub super_id: u64,
pub loader_id: u64,
/// `(name id, object id)` of reference statics; a null static has object id 0.
pub statics: Vec<(u64, u64)>,
/// Heap bytes taken by the primitive statics.
pub static_bytes: u64,
/// Instance fields in dump order, `(name id, type)`.
pub fields: Vec<(u64, BasicType)>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum HeapRecord<'a> {
/// `thread_serial` is present for the kinds that carry one.
Root {
kind: HeapRootKind,
id: u64,
thread_serial: Option<u32>,
},
Class(ClassDump),
/// `data` is the field values, class first, then each superclass.
Instance {
id: u64,
class_id: u64,
data: &'a [u8],
},
/// `data` is `len` ids.
ObjectArray {
id: u64,
class_id: u64,
len: u32,
data: &'a [u8],
},
/// `data` is `len` big-endian elements.
PrimitiveArray {
id: u64,
ty: BasicType,
len: u32,
data: &'a [u8],
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Record<'a> {
Utf8 {
id: u64,
text: &'a [u8],
},
LoadClass {
class_id: u64,
name_id: u64,
},
/// Byte range of a heap-dump segment's sub-records; its contents follow as `Heap` records.
Segment(Range<usize>),
/// `at` is the offset of the sub-record's tag, for [`record_at`].
Heap {
at: usize,
record: HeapRecord<'a>,
},
}
const MAGIC_1_0_2: &[u8] = b"JAVA PROFILE 1.0.2\0";
const MAGIC_1_0_1: &[u8] = b"JAVA PROFILE 1.0.1\0";
const TAG_UTF8: u8 = 0x01;
const TAG_LOAD_CLASS: u8 = 0x02;
const TAG_HEAP_DUMP: u8 = 0x0C;
const TAG_HEAP_DUMP_SEGMENT: u8 = 0x1C;
const TAG_HEAP_DUMP_END: u8 = 0x2C;
/// Tag (u1), timestamp (u4) and body length (u4).
const RECORD_HEADER_LEN: usize = 9;
fn check_id_size(id_size: u8) -> Result<(), HeapError> {
match id_size {
4 | 8 => Ok(()),
other => Err(HeapError::UnsupportedIdSize(u32::from(other))),
}
}
fn truncated(buf: &[u8]) -> HeapError {
HeapError::Truncated { at: buf.len() as u64 }
}
pub fn parse_header(buf: &[u8]) -> Result<HprofHeader, HeapError> {
let magic =
[MAGIC_1_0_2, MAGIC_1_0_1].into_iter().find(|magic| buf.starts_with(magic)).ok_or(HeapError::NotHprof)?;
let mut r = Reader::new(buf, magic.len(), 8);
let id_size = r.u4()?;
let id_size = u8::try_from(id_size).map_err(|_| HeapError::UnsupportedIdSize(id_size))?;
check_id_size(id_size)?;
let timestamp_ms = r.u8()?;
let version = String::from_utf8_lossy(&magic[..magic.len() - 1]).into_owned();
Ok(HprofHeader { version, id_size, timestamp_ms, body_start: r.pos() })
}
/// Walks every top-level record and, for heap-dump segments, the sub-records inside them.
/// A record whose length runs past the end of `buf` is `Truncated`, and so are segments that no
/// HEAP DUMP END follows: HotSpot writes whole segments, so a dump it did not finish ends after one.
pub fn scan<'a>(
buf: &'a [u8],
header: &HprofHeader,
visit: &mut dyn FnMut(Record<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
let id_size = header.id_size;
check_id_size(id_size)?;
if header.body_start > buf.len() {
return Err(truncated(buf));
}
let mut pos = header.body_start;
let mut unended_segments = false;
while pos < buf.len() {
let mut head = Reader::new(buf, pos, id_size);
let tag = head.u1()?;
head.skip(4)?; // timestamp
let len = head.u4()? as usize;
let start = pos + RECORD_HEADER_LEN;
let end = start.checked_add(len).filter(|&end| end <= buf.len()).ok_or_else(|| truncated(buf))?;
let body = &buf[..end];
let mut r = Reader::new(body, start, id_size);
match tag {
TAG_UTF8 => {
let id = r.id()?;
let text = r.bytes(end - r.pos())?;
visit(Record::Utf8 { id, text })?;
}
TAG_LOAD_CLASS => {
r.skip(4)?; // class serial
let class_id = r.id()?;
r.skip(4)?; // stack trace serial
let name_id = r.id()?;
visit(Record::LoadClass { class_id, name_id })?;
}
TAG_HEAP_DUMP | TAG_HEAP_DUMP_SEGMENT => {
// A single HEAP DUMP (JDK 8, under 2 GB) has no end record.
unended_segments |= tag == TAG_HEAP_DUMP_SEGMENT;
visit(Record::Segment(start..end))?;
walk_segment(buf, start..end, id_size, &mut |at, record| visit(Record::Heap { at, record }))?;
}
TAG_HEAP_DUMP_END => unended_segments = false,
_ => {}
}
pos = end;
}
if unended_segments {
return Err(truncated(buf));
}
Ok(())
}
/// Visits the sub-records of one segment (the range from [`Record::Segment`]) with their offsets.
/// The last sub-record must end exactly at the segment's end.
pub fn scan_segment<'a>(
buf: &'a [u8],
segment: Range<usize>,
id_size: u8,
visit: &mut dyn FnMut(usize, HeapRecord<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
check_id_size(id_size)?;
walk_segment(buf, segment, id_size, visit)
}
fn walk_segment<'a>(
buf: &'a [u8],
segment: Range<usize>,
id_size: u8,
visit: &mut dyn FnMut(usize, HeapRecord<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
if segment.start > segment.end || segment.end > buf.len() {
return Err(truncated(buf));
}
let mut r = Reader::new(&buf[..segment.end], segment.start, id_size);
while r.pos() < segment.end {
let at = r.pos();
let record = read_sub_record(&mut r)?;
visit(at, record)?;
}
Ok(())
}
/// Reads the sub-record whose tag is at `at`, as given by [`Record::Heap`].
pub fn record_at(buf: &[u8], at: usize, id_size: u8) -> Result<HeapRecord<'_>, HeapError> {
check_id_size(id_size)?;
read_sub_record(&mut Reader::new(buf, at, id_size))
}
fn read_sub_record<'a>(r: &mut Reader<'a>) -> Result<HeapRecord<'a>, HeapError> {
let at = r.pos();
let tag = r.u1()?;
// (kind, carries a thread serial, bytes after it)
let root = match tag {
0xFF => Some((HeapRootKind::Unknown, false, 0)),
0x01 => Some((HeapRootKind::JniGlobal, false, usize::from(r.id_size()))),
0x02 => Some((HeapRootKind::JniLocal, true, 4)),
0x03 => Some((HeapRootKind::JavaFrame, true, 4)),
0x04 => Some((HeapRootKind::NativeStack, true, 0)),
0x05 => Some((HeapRootKind::StickyClass, false, 0)),
0x06 => Some((HeapRootKind::ThreadBlock, true, 0)),
0x07 => Some((HeapRootKind::MonitorUsed, false, 0)),
0x08 => Some((HeapRootKind::ThreadObject, true, 4)),
_ => None,
};
if let Some((kind, has_serial, trailing)) = root {
let id = r.id()?;
let thread_serial = if has_serial { Some(r.u4()?) } else { None };
r.skip(trailing)?;
return Ok(HeapRecord::Root { kind, id, thread_serial });
}
match tag {
0x20 => read_class_dump(r).map(HeapRecord::Class),
0x21 => {
let id = r.id()?;
r.skip(4)?; // stack trace serial
let class_id = r.id()?;
let len = r.u4()? as usize;
Ok(HeapRecord::Instance { id, class_id, data: r.bytes(len)? })
}
0x22 => {
let id = r.id()?;
r.skip(4)?;
let len = r.u4()?;
let class_id = r.id()?;
let size = (len as usize).checked_mul(usize::from(r.id_size())).ok_or_else(|| truncated(r.buf()))?;
Ok(HeapRecord::ObjectArray { id, class_id, len, data: r.bytes(size)? })
}
0x23 => {
let id = r.id()?;
r.skip(4)?;
let len = r.u4()?;
let ty = read_type(r)?;
if ty == BasicType::Object {
return Err(HeapError::UnknownType { ty: ty as u8, at: (r.pos() - 1) as u64 });
}
let size = (len as usize).checked_mul(ty.hprof_size(r.id_size())).ok_or_else(|| truncated(r.buf()))?;
Ok(HeapRecord::PrimitiveArray { id, ty, len, data: r.bytes(size)? })
}
tag => Err(HeapError::UnknownSubRecord { tag, at: at as u64 }),
}
}
fn read_type(r: &mut Reader<'_>) -> Result<BasicType, HeapError> {
let at = r.pos();
let ty = r.u1()?;
BasicType::from_u8(ty).ok_or(HeapError::UnknownType { ty, at: at as u64 })
}
fn read_class_dump(r: &mut Reader<'_>) -> Result<ClassDump, HeapError> {
let id_size = r.id_size();
let id = r.id()?;
r.skip(4)?; // stack trace serial
let super_id = r.id()?;
let loader_id = r.id()?;
r.skip(4 * usize::from(id_size))?; // signers, protection domain, 2 reserved
r.skip(4)?; // instance size
for _ in 0..r.u2()? {
r.skip(2)?; // constant pool index
let ty = read_type(r)?;
r.skip(ty.hprof_size(id_size))?;
}
let mut statics = Vec::new();
let mut static_bytes = 0;
for _ in 0..r.u2()? {
let name = r.id()?;
let ty = read_type(r)?;
if ty == BasicType::Object {
statics.push((name, r.id()?));
} else {
r.skip(ty.hprof_size(id_size))?;
static_bytes += ty.heap_size(id_size);
}
}
let mut fields = Vec::new();
for _ in 0..r.u2()? {
let name = r.id()?;
fields.push((name, read_type(r)?));
}
Ok(ClassDump { id, super_id, loader_id, statics, static_bytes, fields })
}
#[cfg(test)]
mod tests {
use super::*;
use crate::testing::{HprofWriter, MiniHeap};
use proptest::prelude::*;
fn collect(bytes: &[u8]) -> Result<Vec<Record<'_>>, HeapError> {
let header = parse_header(bytes)?;
let mut out = Vec::new();
scan(bytes, &header, &mut |r| {
out.push(r);
Ok(())
})?;
Ok(out)
}
fn segment_of(bytes: &[u8]) -> Range<usize> {
collect(bytes)
.unwrap()
.into_iter()
.find_map(|r| if let Record::Segment(s) = r { Some(s) } else { None })
.expect("a segment")
}
fn all_kinds(w: &mut HprofWriter) {
w.string(1, "Node")
.string(2, "next")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (3, BasicType::Long)], &[(2, 0x10)])
.instance(0x10, 0x100, &[0u8; 16])
.obj_array(0x20, 0x100, &[0x10, 0])
.prim_array(0x30, BasicType::Int, 3)
.root(HeapRootKind::JavaFrame, 0x10)
.root(HeapRootKind::StickyClass, 0x100);
}
#[test]
fn parses_header() {
let header = parse_header(&MiniHeap::new().bytes()).unwrap();
assert_eq!((header.version.as_str(), header.id_size, header.body_start), ("JAVA PROFILE 1.0.2", 8, 31));
let mut old = b"JAVA PROFILE 1.0.1\0".to_vec();
old.extend_from_slice(&[0, 0, 0, 4, 0, 0, 0, 1, 0, 0, 0, 2]);
let header = parse_header(&old).unwrap();
assert_eq!(
(header.version.as_str(), header.id_size, header.timestamp_ms),
("JAVA PROFILE 1.0.1", 4, (1 << 32) + 2)
);
let bad_size = b"JAVA PROFILE 1.0.2\0\0\0\0\x03\0\0\0\0\0\0\0\0";
assert!(matches!(parse_header(bad_size), Err(HeapError::UnsupportedIdSize(3))));
assert!(matches!(parse_header(b"GIF89a"), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b""), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b"JAVA PROFILE 1.0.3\0\0\0\0\x08"), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b"JAVA PROFILE 1.0.2\0\0\0"), Err(HeapError::Truncated { at: 21 })));
}
#[test]
fn scans_every_record_kind() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let records = collect(&bytes).unwrap();
let (mut utf8, mut loads, mut segments) = (0, 0, 0);
let (mut classes, mut instances, mut obj_arrays, mut prim_arrays, mut roots) = (0, 0, 0, 0, 0);
for record in &records {
match record {
Record::Utf8 { .. } => utf8 += 1,
Record::LoadClass { class_id, name_id } => {
loads += 1;
assert_eq!((*class_id, *name_id), (0x100, 1));
}
Record::Segment(_) => segments += 1,
Record::Heap { record, .. } => match record {
HeapRecord::Class(c) => {
classes += 1;
assert_eq!((c.id, c.super_id, c.loader_id), (0x100, 0, 0));
assert_eq!(c.statics, [(2, 0x10)]);
assert_eq!(c.fields, [(2, BasicType::Object), (3, BasicType::Long)]);
}
HeapRecord::Instance { id, class_id, data } => {
instances += 1;
assert_eq!((*id, *class_id, data.len()), (0x10, 0x100, 16));
}
HeapRecord::ObjectArray { id, class_id, len, data } => {
obj_arrays += 1;
assert_eq!((*id, *class_id, *len, data.len()), (0x20, 0x100, 2, 16));
}
HeapRecord::PrimitiveArray { id, ty, len, data } => {
prim_arrays += 1;
assert_eq!((*id, *ty, *len, data.len()), (0x30, BasicType::Int, 3, 12));
}
HeapRecord::Root { kind, id, thread_serial } => {
roots += 1;
match kind {
HeapRootKind::JavaFrame => assert_eq!((*id, *thread_serial), (0x10, Some(1))),
HeapRootKind::StickyClass => assert_eq!((*id, *thread_serial), (0x100, None)),
other => panic!("unexpected root {other:?}"),
}
}
},
}
}
assert_eq!((utf8, loads, segments), (2, 1, 1));
assert_eq!((classes, instances, obj_arrays, prim_arrays, roots), (1, 1, 1, 1, 2));
assert!(matches!(&records[3], Record::Segment(s) if s.end <= bytes.len()), "segment precedes its contents");
}
#[test]
fn root_kinds_carry_the_thread_serial_they_have() {
let kinds = [
(HeapRootKind::Unknown, false),
(HeapRootKind::JniGlobal, false),
(HeapRootKind::JniLocal, true),
(HeapRootKind::JavaFrame, true),
(HeapRootKind::NativeStack, true),
(HeapRootKind::StickyClass, false),
(HeapRootKind::ThreadBlock, true),
(HeapRootKind::MonitorUsed, false),
(HeapRootKind::ThreadObject, true),
];
let mut w = HprofWriter::new();
for (i, (kind, _)) in kinds.iter().enumerate() {
w.root_in_thread(*kind, 0x40 + i as u64, 7);
w.root(*kind, 0x80 + i as u64);
}
let bytes = w.finish();
let mut seen = Vec::new();
for r in collect(&bytes).unwrap() {
if let Record::Heap { record: HeapRecord::Root { kind, id, thread_serial }, .. } = r {
seen.push((kind, id, thread_serial));
}
}
assert_eq!(seen.len(), kinds.len() * 2);
for (i, (kind, carries)) in kinds.iter().enumerate() {
let explicit = seen[2 * i];
let default = seen[2 * i + 1];
assert_eq!(explicit, (*kind, 0x40 + i as u64, carries.then_some(7)));
assert_eq!(default, (*kind, 0x80 + i as u64, carries.then_some(1)));
}
}
#[test]
fn primitive_arrays_expose_their_contents() {
let mut w = HprofWriter::new();
w.prim_bytes(0x1, BasicType::Char, &[0, b'h', 0, 0xE9]).prim_bytes(0x2, BasicType::Byte, b"v42");
let bytes = w.finish();
let arrays: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::PrimitiveArray { ty, len, data, .. }, .. } => {
Some((ty, len, data.to_vec()))
}
_ => None,
})
.collect();
assert_eq!(arrays, [(BasicType::Char, 2, vec![0, b'h', 0, 0xE9]), (BasicType::Byte, 3, b"v42".to_vec())]);
}
#[test]
fn mini_heap_builds_strings_and_defines_object_and_class() {
let mut heap = MiniHeap::new();
heap.string(0x10, "v42", true).string(0x20, "hé", false);
let bytes = heap.bytes();
let records = collect(&bytes).unwrap();
let names: Vec<_> = records
.iter()
.filter_map(|r| {
if let Record::Utf8 { text, .. } = r { Some(String::from_utf8_lossy(text).into_owned()) } else { None }
})
.collect();
assert!(names.iter().any(|n| n == "java/lang/Object") && names.iter().any(|n| n == "java/lang/Class"));
let arrays: Vec<_> = records
.iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::PrimitiveArray { data, .. }, .. } => Some(data.to_vec()),
_ => None,
})
.collect();
assert_eq!(arrays, [b"v42".to_vec(), vec![b'h', 0, 0xE9, 0]]);
let coders: Vec<_> = records
.iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::Instance { id, data, .. }, .. } => Some((*id, data[8])),
_ => None,
})
.collect();
assert_eq!(coders, [(0x10, 0), (0x20, 1)]);
}
#[test]
fn mini_heap_lays_out_fields_class_first_then_super() {
let mut heap = MiniHeap::new();
heap.class("A", Some("java.lang.Object"), &[("a", BasicType::Int)])
.class("B", Some("A"), &[("b", BasicType::Object), ("c", BasicType::Byte)])
.object(0x10, "B", &[("a", 7), ("b", 0x99), ("c", 0x1FF)]);
let bytes = heap.bytes();
let data = collect(&bytes)
.unwrap()
.into_iter()
.find_map(|r| match r {
Record::Heap { record: HeapRecord::Instance { data, .. }, .. } => Some(data.to_vec()),
_ => None,
})
.unwrap();
assert_eq!(data, [0, 0, 0, 0, 0, 0, 0, 0x99, 0xFF, 0, 0, 0, 7]);
}
#[test]
fn record_at_reads_back_an_instance() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let at = collect(&bytes)
.unwrap()
.into_iter()
.find_map(|r| match r {
Record::Heap { at, record: HeapRecord::Instance { .. } } => Some(at),
_ => None,
})
.unwrap();
match record_at(&bytes, at, 8).unwrap() {
HeapRecord::Instance { id, class_id, data } => assert_eq!((id, class_id, data.len()), (0x10, 0x100, 16)),
other => panic!("expected an instance, got {other:?}"),
}
assert!(matches!(record_at(&bytes, bytes.len(), 8), Err(HeapError::Truncated { .. })));
assert!(matches!(record_at(&bytes, at, 3), Err(HeapError::UnsupportedIdSize(3))));
}
#[test]
fn scan_segment_visits_the_same_records_scan_does() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let from_scan: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| if let Record::Heap { at, record } = r { Some((at, record)) } else { None })
.collect();
let mut from_segment = Vec::new();
scan_segment(&bytes, segment_of(&bytes), 8, &mut |at, record| {
from_segment.push((at, record));
Ok(())
})
.unwrap();
assert_eq!(from_scan.len(), 6);
assert_eq!(from_scan, from_segment);
}
#[test]
fn visitor_errors_stop_the_scan() {
let bytes = MiniHeap::new().bytes();
let header = parse_header(&bytes).unwrap();
let mut calls = 0;
let result = scan(&bytes, &header, &mut |_| {
calls += 1;
Err(HeapError::Cancelled)
});
assert!(matches!(result, Err(HeapError::Cancelled)));
assert_eq!(calls, 1);
}
#[test]
fn both_heap_dump_tags_are_segments() {
let bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
let mut old_style = bytes.clone();
old_style[segment.start - 9] = 0x0C;
assert_eq!(segment_of(&old_style), segment);
}
#[test]
fn id_size_4_dump_parses() {
let mut w = HprofWriter::with_id_size(4);
w.string(1, "Node")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (3, BasicType::Long)], &[(2, 0x10)])
.instance(0x10, 0x100, &[0u8; 12])
.obj_array(0x20, 0x100, &[0x10, 0, 0xFFFF_FFFF])
.prim_array(0x30, BasicType::Short, 5)
.root(HeapRootKind::ThreadObject, 0x10);
let bytes = w.finish();
let header = parse_header(&bytes).unwrap();
assert_eq!((header.id_size, header.body_start), (4, 31));
let records: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| if let Record::Heap { record, .. } = r { Some(record) } else { None })
.collect();
assert_eq!(records.len(), 5);
match &records[0] {
HeapRecord::Class(c) => {
assert_eq!((c.id, c.statics.as_slice(), c.fields.len()), (0x100, &[(2, 0x10)][..], 2));
}
other => panic!("expected the class, got {other:?}"),
}
assert!(matches!(&records[1], HeapRecord::Instance { id: 0x10, class_id: 0x100, data } if data.len() == 12));
assert!(
matches!(&records[2], HeapRecord::ObjectArray { len: 3, data, .. } if data.len() == 12 && data[8..] == [0xFF; 4])
);
assert!(
matches!(&records[3], HeapRecord::PrimitiveArray { ty: BasicType::Short, len: 5, data, .. } if data.len() == 10)
);
assert!(matches!(
&records[4],
HeapRecord::Root { kind: HeapRootKind::ThreadObject, id: 0x10, thread_serial: Some(1) }
));
}
fn push_id(out: &mut Vec<u8>, v: u64) {
out.extend_from_slice(&v.to_be_bytes());
}
/// A class dump written by hand: loader, a constant-pool entry, and statics of each shape.
fn class_dump_bytes(last_field_type: u8) -> Vec<u8> {
let mut b = vec![0x20];
push_id(&mut b, 0x500);
b.extend_from_slice(&0u32.to_be_bytes());
for id in [0x400, 0x77, 0, 0, 0, 0] {
push_id(&mut b, id); // super, loader, signers, domain, 2 reserved
}
b.extend_from_slice(&24u32.to_be_bytes());
b.extend_from_slice(&[0, 1, 0, 7, 10, 0, 0, 0, 99]); // constant pool: one int
b.extend_from_slice(&[0, 3]);
push_id(&mut b, 0xA);
b.push(2);
push_id(&mut b, 0xBEEF); // object static
push_id(&mut b, 0xB);
b.push(10);
b.extend_from_slice(&5u32.to_be_bytes()); // int static
push_id(&mut b, 0xC);
b.push(11);
b.extend_from_slice(&6u64.to_be_bytes()); // long static
b.extend_from_slice(&[0, 2]);
push_id(&mut b, 0xD);
b.push(2);
push_id(&mut b, 0xE);
b.push(last_field_type);
b
}
#[test]
fn class_dump_reads_loader_statics_and_skips_the_constant_pool() {
let bytes = class_dump_bytes(11);
let HeapRecord::Class(c) = record_at(&bytes, 0, 8).unwrap() else { panic!("expected a class") };
assert_eq!((c.id, c.super_id, c.loader_id), (0x500, 0x400, 0x77));
assert_eq!(c.statics, [(0xA, 0xBEEF)]);
assert_eq!(c.static_bytes, 12);
assert_eq!(c.fields, [(0xD, BasicType::Object), (0xE, BasicType::Long)]);
}
#[test]
fn unknown_value_type_reports_its_offset() {
let bytes = class_dump_bytes(0x63);
let err = record_at(&bytes, 0, 8).unwrap_err();
assert!(matches!(err, HeapError::UnknownType { ty: 0x63, at } if at == bytes.len() as u64 - 1), "{err:?}");
}
#[test]
fn truncated_dump_reports_offset() {
let mut bytes = MiniHeap::new().bytes();
let body_start = parse_header(&bytes).unwrap().body_start as u64;
bytes.truncate(bytes.len() - 5);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at > body_start && at == bytes.len() as u64), "{err:?}");
assert!(err.to_string().contains(&bytes.len().to_string()));
}
/// HotSpot flushes whole segments, so a JVM killed while dumping leaves a file that ends
/// right after one, without HEAP DUMP END.
#[test]
fn a_dump_the_jvm_did_not_finish_ends_early() {
let mut bytes = MiniHeap::new().bytes();
bytes.truncate(bytes.len() - RECORD_HEADER_LEN);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at == bytes.len() as u64), "{err:?}");
assert!(err.to_string().starts_with("the dump ends early"), "{err}");
}
#[test]
fn every_prefix_is_ok_or_truncated_at_its_length() {
let mut heap = MiniHeap::new();
heap.class("N", Some("java.lang.Object"), &[("next", BasicType::Object)])
.object(0x10, "N", &[("next", 0x20)])
.string(0x20, "x", true)
.root(HeapRootKind::ThreadObject, 0x10);
let bytes = heap.bytes();
let header = parse_header(&bytes).unwrap();
let segment = segment_of(&bytes);
for cut in header.body_start..bytes.len() {
let prefix = &bytes[..cut];
match scan(prefix, &header, &mut |_| Ok(())) {
// Only a prefix without heap data is a complete (empty) dump.
Ok(()) => assert!(cut < segment.start, "cut at {cut} has heap data but no end"),
Err(HeapError::Truncated { at }) => assert_eq!(at, cut as u64, "cut at {cut}"),
Err(other) => panic!("cut at {cut}: {other:?}"),
}
}
let past_end = HprofHeader { body_start: bytes.len() + 1, ..header };
assert!(matches!(scan(&bytes, &past_end, &mut |_| Ok(())), Err(HeapError::Truncated { .. })));
}
#[test]
fn sub_record_overrunning_its_segment_is_truncated() {
let mut bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
let len_at = segment.start - 4;
let shrunk = (segment.len() as u32 - 3).to_be_bytes();
bytes[len_at..segment.start].copy_from_slice(&shrunk);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at == segment.end as u64 - 3), "{err:?}");
}
#[test]
fn unknown_sub_record_is_an_error() {
let mut bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
bytes[segment.start] = 0x99;
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::UnknownSubRecord { tag: 0x99, at } if at == segment.start as u64), "{err:?}");
assert!(err.to_string().contains("0x99"));
}
#[test]
fn array_lengths_larger_than_the_dump_do_not_allocate_or_panic() {
let mut b = vec![0x23];
push_id(&mut b, 1);
b.extend_from_slice(&0u32.to_be_bytes());
b.extend_from_slice(&u32::MAX.to_be_bytes());
b.push(11);
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::Truncated { .. })));
b[0] = 0x22;
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::Truncated { .. })));
}
#[test]
fn primitive_array_of_objects_is_an_unknown_type() {
let mut b = vec![0x23];
push_id(&mut b, 1);
b.extend_from_slice(&[0, 0, 0, 0, 0, 0, 0, 0]);
b.push(2);
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::UnknownType { ty: 2, .. })));
}
#[test]
fn basic_type_sizes_and_names() {
assert_eq!(BasicType::from_u8(10), Some(BasicType::Int));
assert_eq!([0, 1, 3, 12, 255].map(BasicType::from_u8), [None; 5]);
assert_eq!((BasicType::Object.hprof_size(4), BasicType::Object.hprof_size(8)), (4, 8));
assert_eq!((BasicType::Object.heap_size(4), BasicType::Long.heap_size(4)), (4, 8));
assert_eq!((BasicType::Char.hprof_size(8), BasicType::Boolean.heap_size(8)), (2, 1));
assert_eq!(
(BasicType::Int.array_name(), BasicType::Boolean.array_name(), BasicType::Long.array_name()),
("[I", "[Z", "[J")
);
}
fn sample() -> Vec<u8> {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("v", BasicType::Long)])
.class("[LNode;", Some("java.lang.Object"), &[])
.object(0x10, "Node", &[("next", 0x20), ("v", 5)])
.object(0x20, "Node", &[])
.array_of(0x30, "[LNode;", &[0x10, 0x20, 0])
.prim(0x40, BasicType::Int, 3)
.string(0x50, "hello", true)
.root(HeapRootKind::ThreadObject, 0x10)
.root(HeapRootKind::JniGlobal, 0x30);
heap.bytes()
}
fn with_header(id_size: u8, body: &[u8]) -> Vec<u8> {
let mut bytes = MAGIC_1_0_2.to_vec();
bytes.extend_from_slice(&u32::from(id_size).to_be_bytes());
bytes.extend_from_slice(&1_700_000_000_000u64.to_be_bytes());
bytes.extend_from_slice(body);
bytes
}
proptest! {
#[test]
fn parser_never_panics(bytes in proptest::collection::vec(any::<u8>(), 0..4096)) {
let _ = parse_header(&bytes).and_then(|h| scan(&bytes, &h, &mut |_| Ok(())));
}
/// Random bytes alone fail the magic check, so this puts a valid header in front to make
/// `scan` and `scan_segment` run on arbitrary record streams.
#[test]
fn random_bytes_after_a_valid_header_never_panic(
id_size in prop_oneof![Just(4u8), Just(8u8)],
body in proptest::collection::vec(any::<u8>(), 0..4096),
) {
let bytes = with_header(id_size, &body);
let header = parse_header(&bytes).unwrap();
prop_assert_eq!(header.id_size, id_size);
let _ = scan(&bytes, &header, &mut |_| Ok(()));
let _ = scan_segment(&bytes, header.body_start..bytes.len(), id_size, &mut |_, _| Ok(()));
for at in header.body_start..bytes.len() {
let _ = record_at(&bytes, at, id_size);
}
}
/// Same, but every record has a correct length so the random bytes reach the heap-dump
/// segments and the sub-record parsers instead of stopping at the first bad length.
#[test]
fn random_records_after_a_valid_header_never_panic(
id_size in prop_oneof![Just(4u8), Just(8u8)],
records in proptest::collection::vec(
(
prop_oneof![Just(0x01u8), Just(0x02), Just(0x0C), Just(0x1C), Just(0x2C), any::<u8>()],
prop_oneof![
Just(0x20u8), Just(0x21), Just(0x22), Just(0x23), Just(0x01), Just(0x02), Just(0x03),
Just(0x04), Just(0x05), Just(0x06), Just(0x07), Just(0x08), Just(0xFF), any::<u8>()
],
proptest::collection::vec(any::<u8>(), 0..96),
),
0..12,
),
) {
let mut body = Vec::new();
for (tag, first, rest) in records {
let mut record_body = vec![first];
record_body.extend_from_slice(&rest);
body.push(tag);
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&(record_body.len() as u32).to_be_bytes());
body.extend_from_slice(&record_body);
}
let bytes = with_header(id_size, &body);
let header = parse_header(&bytes).unwrap();
let mut segments = Vec::new();
let _ = scan(&bytes, &header, &mut |r| {
if let Record::Segment(s) = r {
segments.push(s);
}
Ok(())
});
for s in segments {
let _ = scan_segment(&bytes, s, id_size, &mut |_, _| Ok(()));
}
}
#[test]
fn mutated_dump_never_panics(flips in proptest::collection::vec((0usize..2048, any::<u8>()), 1..16)) {
let pristine = sample();
let header = parse_header(&pristine).unwrap();
let segment = segment_of(&pristine);
let mut bytes = pristine.clone();
for (pos, value) in flips {
let at = pos % bytes.len();
bytes[at] = value;
}
let _ = scan(&bytes, &header, &mut |_| Ok(()));
for id_size in [4, 8] {
let _ = scan_segment(&bytes, segment.clone(), id_size, &mut |_, _| Ok(()));
}
for at in 0..bytes.len() {
let _ = record_at(&bytes, at, 8);
}
}
}
}
+324
Переглянути файл
@@ -0,0 +1,324 @@
//! Maps hprof object ids (addresses) to dense `u32` indices in ascending address order.
//!
//! Object ids are addresses and objects are at least 8 bytes apart, so one bit per 8-byte slot
//! ranks ids at about a byte per object. Addresses are grouped in 1 MiB chunks of
//! [`WORDS_PER_CHUNK`] words; only chunks holding an id get words, and a dense table maps a chunk
//! to its first word. The rank of an id is the number of set bits before its word plus the set
//! bits below it in the word. Ids that are not multiples of 8, or that span too wide an address
//! range for the table, use a sorted array and binary search instead.
mod builder;
pub(crate) use builder::ChunkHasher;
pub use builder::IdMapBuilder;
use crate::column::{Column, NONE};
const SLOT_SHIFT: u32 = 3;
const CHUNK_SLOT_BITS: u32 = 17;
const WORD_BITS: u64 = 64;
const WORDS_PER_CHUNK: usize = (1 << CHUNK_SLOT_BITS) / WORD_BITS as usize;
enum Repr {
Bits {
/// Chunk number of the first table entry.
min_chunk: u64,
words: Column<u64>,
/// Set bits before each word.
wrank: Column<u32>,
/// Per chunk from `min_chunk` up: index of its first word, or `NONE` when it has no id.
table: Column<u32>,
/// Per chunk that has words, in word order: its offset from `min_chunk`.
chunk_offsets: Column<u32>,
len: u32,
},
Sorted(Column<u64>),
}
/// Ids ranked in ascending address order: the index of an object is its position in the order.
pub struct IdMap {
repr: Repr,
}
impl IdMap {
pub fn len(&self) -> u32 {
match &self.repr {
Repr::Bits { len, .. } => *len,
Repr::Sorted(ids) => ids.len() as u32,
}
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// The index of `id`, or `None` when it is not in the map.
pub fn rank(&self, id: u64) -> Option<u32> {
match &self.repr {
Repr::Sorted(ids) => ids.binary_search(&id).ok().map(|r| r as u32),
Repr::Bits { min_chunk, words, wrank, table, .. } => {
if id & 7 != 0 {
return None;
}
let slot = id >> SLOT_SHIFT;
let offset = (slot >> CHUNK_SLOT_BITS).checked_sub(*min_chunk)?;
let first = *table.get(usize::try_from(offset).ok()?)?;
if first == NONE {
return None;
}
let w = first as usize + (slot >> 6) as usize % WORDS_PER_CHUNK;
let bit = slot & 63;
let word = words[w];
if word >> bit & 1 == 0 {
return None;
}
Some(wrank[w] + (word & ((1u64 << bit) - 1)).count_ones())
}
}
}
/// The id with index `rank`. Panics when `rank >= len()`.
pub fn id_at(&self, rank: u32) -> u64 {
match &self.repr {
Repr::Sorted(ids) => ids[rank as usize],
Repr::Bits { words, wrank, len, .. } => {
assert!(rank < *len, "rank {rank} out of range for {len} ids");
let w = wrank.partition_point(|&before| before <= rank) - 1;
let mut word = words[w];
for _ in 0..rank - wrank[w] {
word &= word - 1;
}
(self.word_first_slot(w) + u64::from(word.trailing_zeros())) << SLOT_SHIFT
}
}
}
/// All ids in ascending order.
pub fn ids(&self) -> impl Iterator<Item = u64> + '_ {
Ids { map: self, next: 0, slot_base: 0, rest: 0 }
}
/// Bytes of all the columns, in RAM or on disk.
pub fn column_bytes(&self) -> u64 {
match &self.repr {
Repr::Sorted(ids) => std::mem::size_of_val(&**ids) as u64,
Repr::Bits { words, wrank, table, chunk_offsets, .. } => {
(std::mem::size_of_val(&**words)
+ std::mem::size_of_val(&**wrank)
+ std::mem::size_of_val(&**table)
+ std::mem::size_of_val(&**chunk_offsets)) as u64
}
}
}
/// Slot number of bit 0 of word `w` (bitmap layout only).
fn word_first_slot(&self, w: usize) -> u64 {
let Repr::Bits { min_chunk, chunk_offsets, .. } = &self.repr else {
unreachable!("sorted layout has no words")
};
let chunk = min_chunk + u64::from(chunk_offsets[w / WORDS_PER_CHUNK]);
(chunk << CHUNK_SLOT_BITS) + (w % WORDS_PER_CHUNK) as u64 * WORD_BITS
}
}
struct Ids<'a> {
map: &'a IdMap,
/// Next word (bitmap layout) or next id (sorted layout).
next: usize,
slot_base: u64,
/// Bits of the current word not yielded yet.
rest: u64,
}
impl Iterator for Ids<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
match &self.map.repr {
Repr::Sorted(ids) => {
let id = *ids.get(self.next)?;
self.next += 1;
Some(id)
}
Repr::Bits { words, .. } => loop {
if self.rest != 0 {
let bit = u64::from(self.rest.trailing_zeros());
self.rest &= self.rest - 1;
return Some((self.slot_base + bit) << SLOT_SHIFT);
}
self.rest = *words.get(self.next)?;
self.slot_base = self.map.word_first_slot(self.next);
self.next += 1;
},
}
}
}
#[cfg(test)]
mod tests {
use ide_core::CancellationToken;
use super::*;
use crate::column::Store;
/// Deterministic generator so failures replay; the crate has no `rand`.
struct XorShift(u64);
impl XorShift {
fn next(&mut self) -> u64 {
self.0 ^= self.0 << 13;
self.0 ^= self.0 >> 7;
self.0 ^= self.0 << 17;
self.0
}
}
fn build(ids: &[u64], store: &Store) -> IdMap {
let mut builder = IdMapBuilder::new();
for &id in ids {
builder.insert(id);
}
builder.build(store, &CancellationToken::new()).unwrap()
}
fn assert_matches_model(map: &IdMap, model: &[u64]) {
assert_eq!(map.len() as usize, model.len());
assert_eq!(map.ids().collect::<Vec<_>>(), model);
for (rank, &id) in model.iter().enumerate() {
assert_eq!(map.rank(id), Some(rank as u32), "rank of {id:#x}");
assert_eq!(map.id_at(rank as u32), id, "id at {rank}");
for near in [id.wrapping_sub(8), id.wrapping_add(8), id.wrapping_add(1), id.wrapping_add(4)] {
if model.binary_search(&near).is_err() {
assert_eq!(map.rank(near), None, "{near:#x} was never inserted");
}
}
}
}
#[test]
fn rank_follows_address_order() {
let map = build(&[0x2000, 0x1000, 0x1010, 0x7_0000_0000], &Store::Ram);
assert_eq!(map.len(), 4);
assert_eq!(
[0x1000, 0x1010, 0x2000, 0x7_0000_0000].map(|id| map.rank(id)),
[Some(0), Some(1), Some(2), Some(3)]
);
assert_eq!(map.rank(0x1008), None);
assert_eq!(map.rank(0), None);
assert_eq!(map.id_at(2), 0x2000);
assert_eq!(map.ids().collect::<Vec<_>>(), [0x1000, 0x1010, 0x2000, 0x7_0000_0000]);
}
#[test]
fn unaligned_ids_use_sorted_fallback() {
let map = build(&[0x2000, 0x1004, 0x1000, 0x1010, 0x7_0000_0000], &Store::Ram);
assert_eq!(
[0x1000, 0x1004, 0x1010, 0x2000, 0x7_0000_0000].map(|id| map.rank(id)),
[Some(0), Some(1), Some(2), Some(3), Some(4)]
);
assert_eq!(map.rank(0x1008), None);
assert_eq!(map.id_at(1), 0x1004);
}
#[test]
fn sparse_wide_span_uses_fallback() {
let map = build(&[0x1000 + (1 << 50), 0x1000], &Store::Ram);
assert_eq!(map.rank(0x1000), Some(0));
assert_eq!(map.rank(0x1000 + (1 << 50)), Some(1));
assert_eq!(map.rank(0x1008), None);
assert!(map.column_bytes() < 64, "a two-id map must not allocate a chunk table");
}
#[test]
fn ten_million_ids_cost_about_a_byte_each() {
let mut builder = IdMapBuilder::new();
for i in 0..10_000_000u64 {
builder.insert(0x7_0000_0000 + i * 32);
}
let map = builder.build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(map.len(), 10_000_000);
assert!(map.column_bytes() < 16_000_000, "{} bytes", map.column_bytes());
assert_eq!(map.rank(0x7_0000_0000 + 9_999_999 * 32), Some(9_999_999));
assert_eq!(map.id_at(5_000_000), 0x7_0000_0000 + 5_000_000 * 32);
}
#[test]
fn empty_map_and_duplicates() {
let map = build(&[], &Store::Ram);
assert!(map.is_empty());
assert_eq!(map.rank(0x1000), None);
assert_eq!(map.ids().count(), 0);
let map = build(&[0x1000, 0x1000, 0x1008, 0x1000], &Store::Ram);
assert_eq!(map.len(), 2);
assert_eq!(map.rank(0x1008), Some(1));
}
#[test]
fn chunk_boundaries_and_gaps_between_chunks() {
let ids = [0, 0xF_FFF8, 0x10_0000, 0x10_0008, 0x50_0000, 0x1_0000_0000];
let mut model = ids.to_vec();
model.sort_unstable();
let map = build(&ids, &Store::Ram);
assert_matches_model(&map, &model);
}
#[test]
fn ids_outside_the_populated_chunks_are_absent() {
let ids = [0x30_0000, 0x30_0008, 0x70_0000];
let map = build(&ids, &Store::Ram);
assert_matches_model(&map, &ids);
// Chunk 5 lies between the lowest and highest chunk but holds nothing.
assert_eq!(map.rank(0x50_0000), None);
// Chunk 1 is below the lowest populated chunk.
assert_eq!(map.rank(0x10_0000), None);
// Chunk 9 is past the last table entry.
assert_eq!(map.rank(0x900_0000), None);
assert_eq!(map.rank(u64::MAX - 7), None);
}
#[test]
fn highest_address_round_trips() {
let map = build(&[0xFFFF_FFFF_FFFF_FFF8], &Store::Ram);
assert_eq!(map.rank(0xFFFF_FFFF_FFFF_FFF8), Some(0));
assert_eq!(map.id_at(0), 0xFFFF_FFFF_FFFF_FFF8);
let map = build(&[0, 0xFFFF_FFFF_FFFF_FFF8], &Store::Ram);
assert_matches_model(&map, &[0, 0xFFFF_FFFF_FFFF_FFF8]);
}
#[test]
fn random_ids_match_a_sorted_model_in_ram_and_on_disk() {
for seed in 1..=8u64 {
let mut rng = XorShift(seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1);
let aligned = seed % 4 != 0;
let wide = seed % 3 == 0;
let ids: Vec<u64> = (0..3000)
.map(|_| {
let r = rng.next();
let base = if wide { (r >> 20) << 12 } else { 0x6_0000_0000 + (r % (6 << 20)) };
if aligned { base & !7 } else { base }
})
.collect();
let mut model = ids.clone();
model.sort_unstable();
model.dedup();
let dir = tempfile::tempdir().unwrap();
for store in [Store::Ram, Store::Dir(dir.path().to_path_buf())] {
assert_matches_model(&build(&ids, &store), &model);
}
}
}
#[test]
fn dir_store_keeps_the_index_columns_under_stable_names() {
let dir = tempfile::tempdir().unwrap();
let map = build(&[0x1000, 0x2000], &Store::Dir(dir.path().to_path_buf()));
let mut names: Vec<_> =
std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
assert_eq!(names, ["ids.bits.u64", "ids.chunks.u32", "ids.table.u32", "ids.wrank.u32"]);
assert_eq!(map.rank(0x2000), Some(1));
let dir = tempfile::tempdir().unwrap();
build(&[0x1004], &Store::Dir(dir.path().to_path_buf()));
assert!(dir.path().join("ids.sorted.u64").exists());
}
}
+409
Переглянути файл
@@ -0,0 +1,409 @@
//! Collects object ids in any order and lays them out as an [`IdMap`].
use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};
use std::hash::{BuildHasherDefault, Hasher};
use ide_core::CancellationToken;
use super::{CHUNK_SLOT_BITS, IdMap, Repr, SLOT_SHIFT, WORD_BITS, WORDS_PER_CHUNK};
use crate::column::{NONE, Store};
use crate::error::HeapError;
use crate::files::{IDS_BITS_FILE, IDS_CHUNKS_FILE, IDS_SORTED_FILE, IDS_TABLE_FILE, IDS_WRANK_FILE};
/// Widest chunk range the dense table covers (256 MiB of table at most).
const MAX_CHUNK_SPAN: u64 = 1 << 26;
/// Chunks whose first word index still fits in the table's `u32`.
const MAX_CHUNKS: usize = (NONE as usize - 1) / WORDS_PER_CHUNK;
/// Largest object count: `NONE` and the one after it stay free as sentinels for the columns above.
const MAX_IDS: u64 = NONE as u64 - 2;
const CHUNK_BYTES: u64 = (WORDS_PER_CHUNK * size_of::<u64>()) as u64;
/// The bitmaps may take this much before any id is inserted, then this many bytes per distinct id.
/// A heap that fits in 4 GiB of address space never reaches the base allowance whatever the
/// insertion order; sparse input (one id per chunk) is moved to plain `u64` ids, 8 bytes each,
/// after at most `BITMAP_BASE_BYTES + BITMAP_BYTES_PER_ID * ids` of bitmaps, so builder memory is
/// bounded by the number of ids rather than by the address range they spread over.
const BITMAP_BASE_BYTES: u64 = 64 << 20;
const BITMAP_BYTES_PER_ID: u64 = 32;
/// Longest stretch of work between two looks at the cancellation token.
const CHECK_EVERY: usize = 1 << 20;
/// Chunk numbers are already well spread by a multiplicative hash; SipHash would dominate the
/// per-insert cost on a 100M-object heap. The product keeps the low zero bits of its input, so
/// the high half is folded down for the bucket index. The graph builder keys class and string
/// ids (addresses) with it for the same reason.
#[derive(Default)]
pub(crate) struct ChunkHasher(u64);
impl Hasher for ChunkHasher {
fn finish(&self) -> u64 {
self.0 ^ (self.0 >> 32)
}
fn write(&mut self, bytes: &[u8]) {
for &b in bytes {
self.write_u64(self.0.rotate_left(8) ^ u64::from(b));
}
}
fn write_u64(&mut self, n: u64) {
self.0 = n.wrapping_mul(0x9E37_79B9_7F4A_7C15);
}
}
struct BitChunks {
/// Chunk number → position in `words`.
index: HashMap<u64, usize, BuildHasherDefault<ChunkHasher>>,
words: Vec<Box<[u64]>>,
min: u64,
max: u64,
/// The chunk of the previous insert; ids usually arrive in runs.
last: Option<(u64, usize)>,
distinct: u64,
}
impl BitChunks {
fn new() -> Self {
Self { index: HashMap::default(), words: Vec::new(), min: u64::MAX, max: 0, last: None, distinct: 0 }
}
/// `false` when the id's chunk does not fit the dense table or the bitmap budget; the id is
/// then not recorded and the caller moves to plain ids.
fn insert(&mut self, id: u64) -> bool {
let slot = id >> SLOT_SHIFT;
let chunk = slot >> CHUNK_SLOT_BITS;
let at = match self.last {
Some((c, at)) if c == chunk => at,
_ => match self.index.get(&chunk) {
Some(&at) => at,
None => {
let (min, max) = (self.min.min(chunk), self.max.max(chunk));
let bitmap_bytes = (self.words.len() as u64 + 1) * CHUNK_BYTES;
if self.words.len() >= MAX_CHUNKS
|| max - min >= MAX_CHUNK_SPAN
|| bitmap_bytes > BITMAP_BASE_BYTES + BITMAP_BYTES_PER_ID * self.distinct
{
return false;
}
(self.min, self.max) = (min, max);
self.words.push(vec![0u64; WORDS_PER_CHUNK].into_boxed_slice());
self.index.insert(chunk, self.words.len() - 1);
self.words.len() - 1
}
},
};
self.last = Some((chunk, at));
let word = &mut self.words[at][(slot >> 6) as usize % WORDS_PER_CHUNK];
let bit = 1u64 << (slot & 63);
if *word & bit == 0 {
*word |= bit;
self.distinct += 1;
}
true
}
/// `(chunk number, position in words)` in ascending address order.
fn sorted_chunks(&self) -> Vec<(u64, usize)> {
let mut chunks: Vec<_> = self.index.iter().map(|(&chunk, &at)| (chunk, at)).collect();
chunks.sort_unstable();
chunks
}
/// The recorded ids, ascending.
fn ids(&self) -> BitIds<'_> {
BitIds { chunks: self.sorted_chunks(), words: &self.words, chunk: 0, word: 0, base: 0, rest: 0 }
}
}
struct BitIds<'a> {
chunks: Vec<(u64, usize)>,
words: &'a [Box<[u64]>],
chunk: usize,
word: usize,
/// Slot number of bit 0 of the word `rest` came from.
base: u64,
rest: u64,
}
impl Iterator for BitIds<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
loop {
if self.rest != 0 {
let bit = u64::from(self.rest.trailing_zeros());
self.rest &= self.rest - 1;
return Some((self.base + bit) << SLOT_SHIFT);
}
let &(chunk, at) = self.chunks.get(self.chunk)?;
self.rest = self.words[at][self.word];
self.base = (chunk << CHUNK_SLOT_BITS) + self.word as u64 * WORD_BITS;
self.word += 1;
if self.word == WORDS_PER_CHUNK {
self.word = 0;
self.chunk += 1;
}
}
}
}
pub struct IdMapBuilder {
bits: BitChunks,
/// Set once an id cannot go into `bits`; every later id goes here, and `bits` keeps what it
/// had. Unsorted, with duplicates.
plain: Option<Vec<u64>>,
max_ids: u64,
}
impl Default for IdMapBuilder {
fn default() -> Self {
Self::new()
}
}
impl IdMapBuilder {
pub fn new() -> Self {
Self::with_limit(MAX_IDS)
}
/// `max_ids` is a parameter so tests can reach the too-large path without billions of inserts.
fn with_limit(max_ids: u64) -> Self {
Self { bits: BitChunks::new(), plain: None, max_ids }
}
/// Inserting an id twice is the same as once.
pub fn insert(&mut self, id: u64) {
match &mut self.plain {
Some(plain) => plain.push(id),
None => {
if id & 7 != 0 || !self.bits.insert(id) {
self.plain = Some(vec![id]);
}
}
}
}
/// Fails with [`HeapError::TooLarge`] when there are more than `u32::MAX - 2` distinct ids,
/// and with [`HeapError::Cancelled`] when `cancel` fires (checked at least every `1 << 20`
/// steps).
pub fn build(self, store: &Store, cancel: &CancellationToken) -> Result<IdMap, HeapError> {
match self.plain {
None if self.bits.words.is_empty() => sorted_map(store, &[]),
None => bit_map(&self.bits, store, self.max_ids, cancel),
Some(plain) => merged_map(&self.bits, plain, store, self.max_ids, cancel),
}
}
#[cfg(test)]
fn allocated_bytes(&self) -> u64 {
self.bits.words.len() as u64 * CHUNK_BYTES
+ self.plain.as_ref().map_or(0, |p| (p.capacity() * size_of::<u64>()) as u64)
}
}
fn check(cancel: &CancellationToken) -> Result<(), HeapError> {
if cancel.is_cancelled() { Err(HeapError::Cancelled) } else { Ok(()) }
}
fn sorted_map(store: &Store, ids: &[u64]) -> Result<IdMap, HeapError> {
let mut column = store.column(IDS_SORTED_FILE, ids.len(), 0u64)?;
column.copy_from_slice(ids);
Ok(IdMap { repr: Repr::Sorted(column) })
}
fn bit_map(bits: &BitChunks, store: &Store, max_ids: u64, cancel: &CancellationToken) -> Result<IdMap, HeapError> {
if bits.distinct > max_ids {
return Err(HeapError::TooLarge);
}
let chunks = bits.sorted_chunks();
let span = (bits.max - bits.min + 1) as usize;
let mut words = store.column(IDS_BITS_FILE, chunks.len() * WORDS_PER_CHUNK, 0u64)?;
let mut wrank = store.column(IDS_WRANK_FILE, chunks.len() * WORDS_PER_CHUNK, 0u32)?;
let mut table = store.column(IDS_TABLE_FILE, span, NONE)?;
let mut chunk_offsets = store.column(IDS_CHUNKS_FILE, chunks.len(), 0u32)?;
let mut before = 0u32;
for (ordinal, &(chunk, at)) in chunks.iter().enumerate() {
check(cancel)?;
let first = ordinal * WORDS_PER_CHUNK;
let offset = (chunk - bits.min) as usize;
table[offset] = first as u32;
chunk_offsets[ordinal] = offset as u32;
for (w, &word) in bits.words[at].iter().enumerate() {
words[first + w] = word;
wrank[first + w] = before;
before += word.count_ones();
}
}
Ok(IdMap { repr: Repr::Bits { min_chunk: bits.min, words, wrank, table, chunk_offsets, len: before } })
}
enum Source<'a> {
Bits(BitIds<'a>),
Run(std::iter::Copied<std::slice::Iter<'a, u64>>),
}
impl Iterator for Source<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
match self {
Source::Bits(it) => it.next(),
Source::Run(it) => it.next(),
}
}
}
/// Sorts `plain` in runs and merges the runs with the ids already in `bits` (itself a sorted
/// source), dropping duplicates. A single `sort_unstable` of a very large vector could not be
/// interrupted.
fn merged_map(
bits: &BitChunks,
mut plain: Vec<u64>,
store: &Store,
max_ids: u64,
cancel: &CancellationToken,
) -> Result<IdMap, HeapError> {
for run in plain.chunks_mut(CHECK_EVERY) {
check(cancel)?;
run.sort_unstable();
}
let mut sources = vec![Source::Bits(bits.ids())];
sources.extend(plain.chunks(CHECK_EVERY).map(|run| Source::Run(run.iter().copied())));
let mut heap = BinaryHeap::with_capacity(sources.len());
for (i, source) in sources.iter_mut().enumerate() {
if let Some(id) = source.next() {
heap.push(Reverse((id, i)));
}
}
let upper_bound = bits.distinct as usize + plain.len();
let mut merged = store.scratch::<u64>(upper_bound, 0)?;
let mut count = 0usize;
let mut steps = 0usize;
while let Some(Reverse((id, i))) = heap.pop() {
steps += 1;
if steps.is_multiple_of(CHECK_EVERY) {
check(cancel)?;
}
if let Some(next) = sources[i].next() {
heap.push(Reverse((next, i)));
}
if count > 0 && merged[count - 1] == id {
continue;
}
if count as u64 >= max_ids {
return Err(HeapError::TooLarge);
}
merged[count] = id;
count += 1;
}
let mut column = store.column(IDS_SORTED_FILE, count, 0u64)?;
for (to, from) in column.chunks_mut(CHECK_EVERY).zip(merged[..count].chunks(CHECK_EVERY)) {
check(cancel)?;
to.copy_from_slice(from);
}
Ok(IdMap { repr: Repr::Sorted(column) })
}
#[cfg(test)]
mod tests {
use super::*;
fn insert_all(builder: &mut IdMapBuilder, ids: &[u64]) {
for &id in ids {
builder.insert(id);
}
}
#[test]
fn too_many_ids_is_too_large() {
// Dense layout, then the merged layout (one unaligned id); the duplicate does not count.
for unaligned in [false, true] {
let filled = |limit| {
let mut builder = IdMapBuilder::with_limit(limit);
insert_all(&mut builder, &[0x1000, 0x1008, 0x1010, 0x1010]);
if unaligned {
builder.insert(0x1_0004);
}
builder
};
let distinct = if unaligned { 4 } else { 3 };
let map = filled(distinct).build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(u64::from(map.len()), distinct);
let err = filled(distinct - 1).build(&Store::Ram, &CancellationToken::new()).err().unwrap();
assert!(matches!(err, HeapError::TooLarge), "{err}");
}
}
#[test]
fn cancelled_build_returns_cancelled() {
let cancelled = CancellationToken::new();
cancelled.cancel();
let mut dense = IdMapBuilder::new();
insert_all(&mut dense, &[0x1000, 0x1008, 0x20_0000]);
assert!(matches!(dense.build(&Store::Ram, &cancelled), Err(HeapError::Cancelled)));
let mut merged = IdMapBuilder::new();
insert_all(&mut merged, &[0x1000, 0x1004, 0x20_0000]);
assert!(matches!(merged.build(&Store::Ram, &cancelled), Err(HeapError::Cancelled)));
let mut ok = IdMapBuilder::new();
insert_all(&mut ok, &[0x1000, 0x1004]);
assert_eq!(ok.build(&Store::Ram, &CancellationToken::new()).unwrap().len(), 2);
}
#[test]
fn sparse_ids_do_not_amplify_builder_memory() {
const IDS: u64 = 2_000_000;
let mut builder = IdMapBuilder::new();
let mut peak = 0;
for i in 0..IDS {
builder.insert(0x1_0000_0000 + (i << 20));
if i % 1024 == 0 {
peak = peak.max(builder.allocated_bytes());
}
}
peak = peak.max(builder.allocated_bytes());
// Bitmaps up to the budget, plus the plain ids' vector, which may have doubled its capacity.
let budget = BITMAP_BASE_BYTES + (BITMAP_BYTES_PER_ID + 2 * size_of::<u64>() as u64) * IDS;
assert!(peak <= budget, "peak {peak} bytes, budget {budget}");
// One bitmap per id would have taken 16 KiB each.
assert!(peak < IDS * CHUNK_BYTES / 100, "peak {peak} bytes");
assert!(builder.plain.is_some(), "sparse input must have left the bitmaps");
let map = builder.build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(map.len() as u64, IDS);
assert_eq!(map.rank(0x1_0000_0000 + (1_234_567 << 20)), Some(1_234_567));
assert_eq!(map.rank(0x1_0000_0000 + (1_234_567 << 20) + 8), None);
}
#[test]
fn dense_input_in_any_order_stays_in_bitmaps() {
// 4 GiB of address space is the base allowance; random order must not trip the budget.
let mut state = 0x2545_F491_4F6C_DD1Du64;
let mut builder = IdMapBuilder::new();
for _ in 0..300_000 {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
builder.insert((0x7_0000_0000 + (state % (4 << 30))) & !7);
}
assert!(builder.plain.is_none());
assert!(builder.allocated_bytes() <= BITMAP_BASE_BYTES);
}
#[test]
fn chunk_hasher_spreads_chunks_that_share_low_zero_bits() {
let buckets: std::collections::HashSet<u64> = (0..1024u64)
.map(|i| {
let mut hasher = ChunkHasher::default();
hasher.write_u64(i << 12);
hasher.finish() & 1023
})
.collect();
assert!(buckets.len() > 500, "{} buckets for 1024 keys", buckets.len());
}
}
+109
Переглянути файл
@@ -0,0 +1,109 @@
//! The result of analyzing a heap dump: per-object columns, the class table and the summary.
mod fields;
mod path;
mod queries;
mod rows;
use std::collections::HashMap;
use std::path::Path;
use memmap2::Mmap;
use serde::{Deserialize, Serialize};
use crate::column::Column;
use crate::error::HeapError;
use crate::files::SUMMARY_FILE;
use crate::graph::ClassInfo;
use crate::hprof::HeapRootKind;
use crate::ids::IdMap;
use crate::shallow::SizeSource;
pub(crate) use self::fields::{id_value, instance_fields};
pub use self::rows::{
HeapClassRow, HeapEdge, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapPathStep, HeapRootPath, MAX_PAGE,
};
/// Counts over instances and arrays; class objects (class dumps) are left out.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapSummary {
/// `reachable_objects + unreachable_objects`.
pub objects: u64,
pub reachable_objects: u64,
pub reachable_bytes: u64,
pub unreachable_objects: u64,
pub unreachable_bytes: u64,
/// Class dumps in the dump.
pub classes: u32,
pub weak_edges_skipped: u64,
pub size_source: SizeSource,
pub analysis_ms: u32,
}
/// One row of the class table, for one type slot.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClassStat {
/// [`crate::display_name`] of the class (`long[]` for a primitive array slot).
pub name: String,
/// Reachable instances.
pub count: u64,
/// Shallow bytes of the reachable instances.
pub shallow: u64,
/// Bytes the reachable instances retain together: the sum of the retained sizes of those
/// with no instance of the same type above them in the dominator tree.
pub retained: u64,
pub unreachable_count: u64,
}
/// Columns are indexed by node: objects `0..root`, then the super-root `root`. Columns of the
/// objects only (`types`, `offsets`) have no entry for the root.
pub struct HeapIndex {
/// The dump, for [`crate::record_at`] at `offsets[v]`.
pub hprof: Mmap,
pub ids: IdMap,
/// Type slot of each object, as in [`crate::HeapGraph::types`].
pub types: Column<u32>,
pub offsets: Column<u64>,
/// Shallow size in units of 8 bytes, rounded up.
pub shallow8: Column<u32>,
/// Retained bytes; 0 for unreachable objects, every reachable byte for the root.
pub retained: Column<u64>,
/// Immediate dominator; [`crate::NONE`] for unreachable objects, `root` for the root.
pub idom: Column<u32>,
/// Children of node `v` in the dominator tree are `kids[kids_start[v]..kids_start[v + 1]]`,
/// by retained size descending, then by node.
pub kids_start: Column<u32>,
pub kids: Column<u32>,
/// Parent on a shortest path from the root (BFS over strong edges); [`crate::NONE`] for
/// unreachable objects, `root` for the root.
pub gc_parent: Column<u32>,
/// By type slot: class slots, then the primitive array slots.
pub classes: Vec<ClassStat>,
/// By class slot, as in [`crate::HeapGraph::classes`].
pub class_info: Vec<ClassInfo>,
/// Type slot of class objects.
pub class_slot: u32,
/// Object → kind and thread serial of its first root record.
pub roots: HashMap<u32, (HeapRootKind, Option<u32>)>,
/// Thread serial → thread object.
pub thread_objects: HashMap<u32, u32>,
pub id_size: u8,
pub summary: HeapSummary,
}
impl HeapIndex {
/// The super-root, whose children in both trees are the rooted objects.
pub fn root(&self) -> u32 {
self.ids.len()
}
}
pub(crate) fn write_summary(dir: &Path, summary: &HeapSummary) -> Result<(), HeapError> {
let json = serde_json::to_vec_pretty(summary).map_err(std::io::Error::from)?;
std::fs::write(dir.join(SUMMARY_FILE), json)?;
Ok(())
}
#[cfg(test)]
mod tests;
+41
Переглянути файл
@@ -0,0 +1,41 @@
//! Named field values of an instance record.
use super::HeapIndex;
use crate::hprof::BasicType;
pub(crate) struct FieldValue<'a> {
pub(crate) name: &'a str,
pub(crate) ty: BasicType,
/// Big-endian, as in the dump.
pub(crate) bytes: &'a [u8],
/// `java.lang.ref.Reference.referent`, which the graph leaves out.
pub(crate) weak: bool,
}
/// The fields of an instance of class slot `class` whose record holds `data`: the class's own,
/// then each superclass's, as HotSpot lays them out. Stops where `data` ends.
pub(crate) fn instance_fields<'a>(index: &'a HeapIndex, class: u32, data: &'a [u8]) -> Vec<FieldValue<'a>> {
let mut out = Vec::new();
let mut base = 0usize;
let mut cursor = Some(class);
// Super chains end (their cycles are cut), so this is only a bound on corrupt input.
for _ in 0..index.class_info.len() {
let Some(info) = cursor.and_then(|c| index.class_info.get(c as usize)) else { break };
for (k, (name, ty)) in info.fields.iter().enumerate() {
let size = ty.hprof_size(index.id_size);
let Some(bytes) = base.checked_add(size).and_then(|end| data.get(base..end)) else { return out };
out.push(FieldValue { name, ty: *ty, bytes, weak: info.weak_field == Some(k) });
base += size;
}
cursor = info.super_index;
}
out
}
/// A 4- or 8-byte big-endian id.
pub(crate) fn id_value(bytes: &[u8]) -> Option<u64> {
match *bytes {
[a, b, c, d] => Some(u64::from(u32::from_be_bytes([a, b, c, d]))),
_ => Some(u64::from_be_bytes(bytes.try_into().ok()?)),
}
}
+105
Переглянути файл
@@ -0,0 +1,105 @@
//! Paths to GC roots, along the BFS tree of `gc_parent`.
use super::{HeapEdge, HeapIndex, HeapObjectRef, HeapPathStep, HeapRootPath, id_value, instance_fields};
use crate::column::NONE;
use crate::hprof::{BasicType, HeapRecord, HeapRootKind};
use crate::strings::{char_array_text, string_preview};
const THREAD_NAME_CHARS: usize = 100;
/// Steps a long path keeps at each end: one deep in a long linked list would otherwise send
/// every node over IPC.
const PATH_END_STEPS: u32 = 500;
impl HeapIndex {
/// From `object` up to its GC root, each step with how the next one holds it; past 1000
/// steps, only the 500 at each end. `None` for an unreachable object or a ref that names no
/// object.
pub fn path_to_root(&self, object: HeapObjectRef) -> Option<HeapRootPath> {
let root = self.root();
let mut v = object.0;
if v >= root || self.gc_parent[v as usize] == NONE {
return None;
}
let len = self.path_len(v)?;
let end = PATH_END_STEPS;
let cut = len > 2 * end;
let mut steps = Vec::with_capacity(len.min(2 * end) as usize);
for i in 0..len {
let parent = self.gc_parent[v as usize];
if !cut || i < end || i >= len - end {
let via = if parent == root { self.root_edge(v) } else { self.edge(parent, v) };
steps.push(HeapPathStep { object: self.object_row(HeapObjectRef(v))?, via });
}
v = parent;
}
let omitted = if cut { len - 2 * end } else { 0 };
Some(HeapRootPath { steps, omitted, omitted_after: if cut { end } else { 0 } })
}
/// Steps from `v` to the root; `None` when `gc_parent` leads nowhere.
fn path_len(&self, mut v: u32) -> Option<u32> {
let root = self.root();
// A BFS tree has no cycles; the bound only stops a corrupt column from looping.
for len in 1..=root {
let parent = self.gc_parent[v as usize];
if parent == root {
return Some(len);
}
if parent > root {
return None;
}
v = parent;
}
None
}
fn root_edge(&self, v: u32) -> HeapEdge {
let (kind, serial) = self.roots.get(&v).copied().unwrap_or((HeapRootKind::Unknown, None));
let thread = serial.and_then(|s| self.thread_objects.get(&s)).and_then(|&t| self.thread_name(t));
HeapEdge::Root { root: kind, thread }
}
/// `Thread.name`: a String, or a `char[]` before JDK 9.
fn thread_name(&self, thread: u32) -> Option<String> {
let Some(HeapRecord::Instance { data, .. }) = self.record(thread) else { return None };
let fields = instance_fields(self, *self.types.get(thread as usize)?, data);
let name = fields.iter().find(|f| f.name == "name" && f.ty == BasicType::Object)?;
let target = self.ids.rank(id_value(name.bytes)?)?;
string_preview(self, target, THREAD_NAME_CHARS).or_else(|| char_array_text(self, target, THREAD_NAME_CHARS))
}
/// How `parent` refers to `child`, checked in the order the graph's edges were made. A strong
/// edge from `parent` always exists; the empty field name only covers a dump that changed.
fn edge(&self, parent: u32, child: u32) -> HeapEdge {
self.find_edge(parent, self.ids.id_at(child)).unwrap_or(HeapEdge::Field { name: String::new() })
}
fn find_edge(&self, parent: u32, child_id: u64) -> Option<HeapEdge> {
match self.record(parent)? {
HeapRecord::Instance { data, .. } => {
let class = self.types[parent as usize];
let field = instance_fields(self, class, data)
.into_iter()
.find(|f| !f.weak && f.ty == BasicType::Object && id_value(f.bytes) == Some(child_id));
match field {
Some(f) => Some(HeapEdge::Field { name: f.name.to_owned() }),
None => (self.class_info.get(class as usize)?.id == child_id).then_some(HeapEdge::ClassOf),
}
}
HeapRecord::ObjectArray { data, .. } => {
let index =
data.chunks_exact(usize::from(self.id_size)).position(|item| id_value(item) == Some(child_id))?;
Some(HeapEdge::Element { index: u32::try_from(index).ok()? })
}
HeapRecord::Class(dump) if dump.super_id == child_id => Some(HeapEdge::SuperClass),
HeapRecord::Class(dump) if dump.loader_id == child_id => Some(HeapEdge::ClassLoader),
HeapRecord::Class(_) => {
let info = self.class_info.get(self.class_of_object(parent)?)?;
let (name, _) = info.statics.iter().find(|(_, object)| *object == child_id)?;
Some(HeapEdge::Static { name: name.clone() })
}
HeapRecord::Root { .. } | HeapRecord::PrimitiveArray { .. } => None,
}
}
}
+164
Переглянути файл
@@ -0,0 +1,164 @@
//! Class table rows, pages of the dominator tree and single object rows.
use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};
use super::{HeapClassRow, HeapIndex, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapSummary, MAX_PAGE};
use crate::column::NONE;
use crate::hprof::HeapRecord;
use crate::strings::string_preview;
const PREVIEW_CHARS: usize = 100;
impl HeapIndex {
pub fn summary(&self) -> &HeapSummary {
&self.summary
}
/// Types that have objects, by retained size descending, then by name.
pub fn class_rows(&self) -> Vec<HeapClassRow> {
let class_slots = &self.classes[..self.class_info.len().min(self.classes.len())];
let mut names: HashMap<&str, u32> = HashMap::with_capacity(class_slots.len());
for class in class_slots {
*names.entry(class.name.as_str()).or_default() += 1;
}
let mut rows: Vec<HeapClassRow> = self
.classes
.iter()
.enumerate()
.filter(|(_, c)| c.count > 0 || c.unreachable_count > 0)
.map(|(slot, c)| HeapClassRow {
class_index: slot as u32,
name: c.name.clone(),
count: c.count,
shallow: c.shallow,
retained: c.retained,
unreachable_count: c.unreachable_count,
// Primitive array slots have no class dump and so no second loader.
ambiguous: slot < class_slots.len() && names[c.name.as_str()] > 1,
})
.collect();
rows.sort_by(|a, b| b.retained.cmp(&a.retained).then_with(|| a.name.cmp(&b.name)));
rows
}
/// A page of objects by retained size descending, then by index:
///
/// - no `parent`, no `class_index`: the children of the super-root, class objects left out;
/// - `parent`: its children in the dominator tree, of type `class_index` if given;
/// - `class_index` only: every reachable object of that type.
///
/// `limit` is clamped to [`MAX_PAGE`]. A `parent` or `class_index` that names nothing gives an
/// empty page.
pub fn dominated(
&self,
parent: Option<HeapObjectRef>,
class_index: Option<u32>,
offset: u32,
limit: u32,
) -> HeapObjectPage {
let limit = limit.min(MAX_PAGE);
let root = self.root();
if class_index.is_some_and(|c| c as usize >= self.classes.len()) {
return HeapObjectPage { rows: Vec::new(), total: 0 };
}
match (parent, class_index) {
(Some(HeapObjectRef(p)), _) if p >= root => HeapObjectPage { rows: Vec::new(), total: 0 },
(Some(HeapObjectRef(p)), class) => self.page(
self.children(p).filter(|&v| class.is_none_or(|c| self.types[v as usize] == c)),
offset,
limit,
),
(None, None) => {
self.page(self.children(root).filter(|&v| self.types[v as usize] != self.class_slot), offset, limit)
}
(None, Some(class)) => self.largest_of(class, offset, limit),
}
}
/// `None` for a ref that names no object, the super-root included.
pub fn object_row(&self, object: HeapObjectRef) -> Option<HeapObjectRow> {
let v = object.0;
if v >= self.root() {
return None;
}
let i = v as usize;
let slot = self.types[i];
let preview = if slot == self.class_slot {
self.class_of_object(v).and_then(|c| self.classes.get(c)).map(|c| c.name.clone())
} else {
string_preview(self, v, PREVIEW_CHARS)
};
Some(HeapObjectRow {
object,
class_name: self.classes.get(slot as usize).map(|c| c.name.clone()).unwrap_or_default(),
address: format!("{:#x}", self.ids.id_at(v)),
shallow: u64::from(self.shallow8[i]) * 8,
retained: self.retained[i],
has_children: self.kids_start[i + 1] > self.kids_start[i],
preview,
})
}
/// The record of object `v`, or `None` when it cannot be read.
pub(crate) fn record(&self, v: u32) -> Option<HeapRecord<'_>> {
let at = usize::try_from(*self.offsets.get(v as usize)?).ok()?;
crate::hprof::record_at(&self.hprof, at, self.id_size).ok()
}
/// The class slot a class object stands for.
pub(crate) fn class_of_object(&self, v: u32) -> Option<usize> {
if v >= self.root() {
return None;
}
let id = self.ids.id_at(v);
self.class_info.iter().position(|c| c.id == id && c.id != 0)
}
/// Children of node `v` in the dominator tree, already in page order.
fn children(&self, v: u32) -> impl Iterator<Item = u32> + '_ {
let v = v as usize;
self.kids[self.kids_start[v] as usize..self.kids_start[v + 1] as usize].iter().copied()
}
fn page(&self, objects: impl Iterator<Item = u32>, offset: u32, limit: u32) -> HeapObjectPage {
let mut rows = Vec::new();
let mut total = 0u32;
for v in objects {
if total >= offset && total - offset < limit {
rows.extend(self.object_row(HeapObjectRef(v)));
}
total = total.saturating_add(1);
}
HeapObjectPage { rows, total }
}
/// Scans every object: the top `offset + limit` stay in a heap whose top is the smallest.
fn largest_of(&self, class: u32, offset: u32, limit: u32) -> HeapObjectPage {
let count = self.classes[class as usize].count;
let total = u32::try_from(count).unwrap_or(u32::MAX);
let keep = match offset.checked_add(limit) {
Some(keep) if offset < total && limit > 0 => keep.min(total) as usize,
_ => return HeapObjectPage { rows: Vec::new(), total },
};
let mut top: BinaryHeap<Reverse<(u64, Reverse<u32>)>> = BinaryHeap::with_capacity(keep + 1);
for v in 0..self.root() {
let i = v as usize;
if self.types[i] != class || self.idom[i] == NONE {
continue;
}
top.push(Reverse((self.retained[i], Reverse(v))));
if top.len() > keep {
top.pop();
}
}
// Ascending `Reverse` order is descending retained, then ascending index.
let rows = top
.into_sorted_vec()
.into_iter()
.skip(offset as usize)
.filter_map(|Reverse((_, Reverse(v)))| self.object_row(HeapObjectRef(v)))
.collect();
HeapObjectPage { rows, total }
}
}
+95
Переглянути файл
@@ -0,0 +1,95 @@
//! What the index queries return over IPC.
use serde::{Deserialize, Serialize};
use crate::hprof::HeapRootKind;
/// Most rows one page of [`crate::HeapIndex::dominated`] returns.
pub const MAX_PAGE: u32 = 500;
/// An object of one snapshot: its index in the [`crate::HeapIndex`] columns. Any value may
/// arrive over IPC; queries answer an index that names no object with nothing.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, specta::Type)]
#[serde(transparent)]
pub struct HeapObjectRef(pub u32);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapClassRow {
/// Type slot, as `classIndex` in [`crate::HeapIndex::dominated`].
pub class_index: u32,
pub name: String,
pub count: u64,
pub shallow: u64,
pub retained: u64,
pub unreachable_count: u64,
/// Another class has the same name (several class loaders): nothing joined by name, such as a
/// second snapshot, can tell them apart.
pub ambiguous: bool,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapObjectRow {
pub object: HeapObjectRef,
pub class_name: String,
/// The object's hprof id in hex, `0x1300`.
pub address: String,
pub shallow: u64,
pub retained: u64,
/// Has children in the dominator tree.
pub has_children: bool,
/// The text of a `java.lang.String`, or the name of the class a class object stands for.
pub preview: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapObjectPage {
pub rows: Vec<HeapObjectRow>,
/// Rows of every page together.
pub total: u32,
}
/// How an object is held by the next step of a path to a GC root.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapEdge {
/// An instance field; the name is empty when the dump does not show which one.
Field {
name: String,
},
Element {
index: u32,
},
Static {
name: String,
},
/// From an instance to its class.
ClassOf,
SuperClass,
ClassLoader,
/// `thread` is the name of the thread whose root it is, when the dump has it.
Root {
root: HeapRootKind,
thread: Option<String>,
},
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapPathStep {
pub object: HeapObjectRow,
/// How the next step holds this object; on the last step, the root.
pub via: HeapEdge,
}
/// A path from an object up to its GC root. A long one keeps only the steps nearest each end.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapRootPath {
pub steps: Vec<HeapPathStep>,
/// Steps left out between `steps[omitted_after - 1]` and `steps[omitted_after]`; 0 for none.
pub omitted: u32,
pub omitted_after: u32,
}
+336
Переглянути файл
@@ -0,0 +1,336 @@
use super::*;
use crate::hprof::BasicType;
use crate::test_index::{analyzed, analyzed_bytes, obj, slot};
use crate::testing::{HprofWriter, MiniHeap, sample, sample_with_string};
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
fn addresses(page: &HeapObjectPage) -> Vec<&str> {
page.rows.iter().map(|r| r.address.as_str()).collect()
}
fn retained(page: &HeapObjectPage) -> Vec<u64> {
page.rows.iter().map(|r| r.retained).collect()
}
fn path(index: &HeapIndex, addr: u64) -> Option<Vec<(String, HeapEdge)>> {
let path = index.path_to_root(obj(index, addr))?;
assert_eq!((path.omitted, path.omitted_after), (0, 0));
Some(path.steps.into_iter().map(|s| (s.object.address, s.via)).collect())
}
fn field(name: &str) -> HeapEdge {
HeapEdge::Field { name: name.to_owned() }
}
fn root(root: HeapRootKind, thread: Option<&str>) -> HeapEdge {
HeapEdge::Root { root, thread: thread.map(str::to_owned) }
}
#[test]
fn top_level_dominators_sorted_by_retained() {
let index = analyzed(&sample());
let page = index.dominated(None, None, 0, 10);
assert_eq!(addresses(&page), ["0x1000", "0x1300", "0x1600"]);
assert_eq!(retained(&page), [72, 56, 24]);
// The class objects, rooted as sticky classes, are left out.
assert_eq!(page.total, 3);
assert_eq!(addresses(&index.dominated(None, None, 1, 1)), ["0x1300"]);
}
#[test]
fn children_of_holder() {
let index = analyzed(&sample());
let page = index.dominated(Some(obj(&index, H)), None, 0, 10);
assert_eq!(addresses(&page), ["0x1100", "0x1200"]);
assert_eq!(retained(&page), [24, 24]);
assert_eq!(page.total, 2);
assert!(page.rows.iter().all(|r| !r.has_children && r.class_name == "Node"));
let n3 = index.object_row(obj(&index, N3)).unwrap();
assert!(n3.has_children);
assert_eq!(index.dominated(Some(n3.object), None, 0, 10).rows[0].class_name, "long[]");
}
#[test]
fn largest_of_class() {
let index = analyzed(&sample());
let node = slot(&index, "Node");
let page = index.dominated(None, Some(node), 0, 2);
assert_eq!(addresses(&page), ["0x1300", "0x1100"]);
assert_eq!(retained(&page), [56, 24]);
assert_eq!(page.total, 3);
assert_eq!(addresses(&index.dominated(None, Some(node), 2, 2)), ["0x1200"]);
assert!(index.dominated(None, Some(node), 3, 2).rows.is_empty());
// Within a parent's children.
assert_eq!(addresses(&index.dominated(Some(obj(&index, H)), Some(node), 1, 5)), ["0x1200"]);
assert_eq!(index.dominated(Some(obj(&index, H)), Some(slot(&index, "Holder")), 0, 5).total, 0);
}
#[test]
fn object_row_describes_object() {
let index = analyzed(&sample());
let row = index.object_row(obj(&index, N3)).unwrap();
assert_eq!(
row,
HeapObjectRow {
object: obj(&index, N3),
class_name: "Node".to_owned(),
address: "0x1300".to_owned(),
shallow: 24,
retained: 56,
has_children: true,
preview: None,
}
);
let unreachable = index.object_row(obj(&index, N4)).unwrap();
assert_eq!((unreachable.retained, unreachable.has_children), (0, false));
let holder = &index.class_info[slot(&index, "Holder") as usize];
let class_object = obj(&index, holder.id);
let row = index.object_row(class_object).unwrap();
assert_eq!((row.class_name.as_str(), row.preview.as_deref()), ("java.lang.Class", Some("Holder")));
}
#[test]
fn refs_that_name_no_object_answer_nothing() {
let index = analyzed(&sample());
let node = slot(&index, "Node");
for bad in [index.root(), index.root() + 1, u32::MAX] {
let bad = HeapObjectRef(bad);
assert_eq!(index.object_row(bad), None);
assert_eq!(index.path_to_root(bad), None);
assert_eq!(index.dominated(Some(bad), None, 0, 10), HeapObjectPage { rows: Vec::new(), total: 0 });
assert_eq!(index.dominated(Some(bad), Some(node), 0, 10).total, 0);
}
for bad in [index.classes.len() as u32, u32::MAX] {
assert_eq!(index.dominated(None, Some(bad), 0, 10).total, 0);
}
}
#[test]
fn path_to_root_names_fields_and_root() {
let index = analyzed(&sample());
assert_eq!(
path(&index, L),
Some(vec![("0x1400".to_owned(), field("next")), ("0x1300".to_owned(), root(HeapRootKind::JavaFrame, None))])
);
assert_eq!(
path(&index, N1),
Some(vec![("0x1100".to_owned(), field("a")), ("0x1000".to_owned(), root(HeapRootKind::JniGlobal, None))])
);
assert_eq!(path(&index, W), Some(vec![("0x1600".to_owned(), root(HeapRootKind::JniGlobal, None))]));
assert_eq!(path(&index, N4), None);
}
/// A rooted linked list of `len` Nodes from `0x100000`, each held by the next one's `next`.
fn chain(len: u64) -> HeapIndex {
let mut heap = sample();
for i in 0..len {
let next = if i == 0 { 0 } else { 0x10_0000 + (i - 1) * 0x20 };
heap.object(0x10_0000 + i * 0x20, "Node", &[("next", next)]);
}
heap.root(HeapRootKind::JniGlobal, 0x10_0000 + (len - 1) * 0x20);
analyzed(&heap)
}
#[test]
fn path_of_an_object_deep_in_a_linked_list_keeps_both_ends() {
let index = chain(3000);
let path = index.path_to_root(obj(&index, 0x10_0000)).unwrap();
assert_eq!((path.steps.len(), path.omitted, path.omitted_after), (1000, 2000, 500));
let address = |i: usize| path.steps[i].object.address.clone();
assert_eq!([address(0), address(499)], ["0x100000", "0x103e60"]);
// The step after the gap is the one 2000 nodes further up.
assert_eq!([address(500), address(999)], ["0x113880", "0x1176e0"]);
assert_eq!(path.steps[999].via, root(HeapRootKind::JniGlobal, None));
assert!(path.steps[..999].iter().all(|s| s.via == field("next")));
}
#[test]
fn path_of_exactly_a_thousand_steps_is_whole() {
let index = chain(1000);
let path = index.path_to_root(obj(&index, 0x10_0000)).unwrap();
assert_eq!((path.steps.len(), path.omitted, path.omitted_after), (1000, 0, 0));
}
#[test]
fn path_skips_the_weak_referent() {
let mut heap = sample();
heap.object(0x1c00, "java.lang.ref.WeakReference", &[("referent", 0x1d00), ("queue", 0x1d00)])
.object(0x1d00, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1c00);
let index = analyzed(&heap);
assert_eq!(path(&index, 0x1d00).unwrap()[0].1, field("queue"));
}
/// Kid extends Base, loaded by loader `0x300`; Base's static `cache` holds X; K is a rooted Kid;
/// `0x700` is a rooted `Object[] { null, Y }`. No class is rooted on its own.
fn class_edges_heap() -> Vec<u8> {
let mut w = HprofWriter::new();
let names = [(1, "Base"), (2, "Kid"), (3, "Ldr"), (4, "[Ljava/lang/Object;"), (5, "cache")];
for (id, name) in names {
w.string(id, name);
}
for (class, name) in [(0x100, 1), (0x200, 2), (0x400, 3), (0x800, 4)] {
w.load_class(class, name);
}
w.class(0x100, 0, &[], &[(5, 0x500)])
.class_with_loader(0x200, 0x100, 0x300, &[], &[])
.class(0x400, 0, &[], &[])
.class(0x800, 0, &[], &[])
.instance(0x300, 0x400, &[])
.instance(0x500, 0x400, &[])
.instance(0x600, 0x200, &[])
.obj_array(0x700, 0x800, &[0, 0x900])
.instance(0x900, 0x400, &[])
.root(HeapRootKind::JniGlobal, 0x600)
.root(HeapRootKind::JniGlobal, 0x700);
w.finish()
}
#[test]
fn path_names_statics_supers_loaders_classes_and_elements() {
let index = analyzed_bytes(&class_edges_heap());
let k = ("0x600".to_owned(), root(HeapRootKind::JniGlobal, None));
assert_eq!(
path(&index, 0x500),
Some(vec![
("0x500".to_owned(), HeapEdge::Static { name: "cache".to_owned() }),
("0x100".to_owned(), HeapEdge::SuperClass),
("0x200".to_owned(), HeapEdge::ClassOf),
k.clone(),
])
);
assert_eq!(
path(&index, 0x300),
Some(vec![("0x300".to_owned(), HeapEdge::ClassLoader), ("0x200".to_owned(), HeapEdge::ClassOf), k])
);
assert_eq!(
path(&index, 0x900),
Some(vec![
("0x900".to_owned(), HeapEdge::Element { index: 1 }),
("0x700".to_owned(), root(HeapRootKind::JniGlobal, None))
])
);
}
fn with_main_thread(mut heap: MiniHeap, name: impl FnOnce(&mut MiniHeap, u64)) -> MiniHeap {
heap.class("java.lang.Thread", Some("java.lang.Object"), &[("name", BasicType::Object)]);
name(&mut heap, 0x2000);
heap.object(0x2100, "java.lang.Thread", &[("name", 0x2000)]).root_in_thread(HeapRootKind::ThreadObject, 0x2100, 1);
heap
}
#[test]
fn thread_name_on_java_frame_root() {
let heap = with_main_thread(sample_with_string(), |h, at| {
h.string(at, "main", true);
});
let index = analyzed(&heap);
let steps = path(&index, L).unwrap();
assert_eq!(steps.last().unwrap().1, root(HeapRootKind::JavaFrame, Some("main")));
// A JNI global carries no thread.
assert_eq!(path(&index, H).unwrap()[0].1, root(HeapRootKind::JniGlobal, None));
}
#[test]
fn jdk8_thread_name_is_a_char_array() {
let heap = with_main_thread(sample(), |h, at| {
let utf16: Vec<u8> = "main".encode_utf16().flat_map(u16::to_be_bytes).collect();
h.prim_bytes(at, BasicType::Char, &utf16);
});
let index = analyzed(&heap);
assert_eq!(path(&index, N3).unwrap()[0].1, root(HeapRootKind::JavaFrame, Some("main")));
}
#[test]
fn page_limit_is_clamped() {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object)]);
for k in 0..600u64 {
let addr = 0x10_0000 + k * 0x100;
heap.object(addr, "Node", &[]).root(HeapRootKind::JniGlobal, addr);
}
let index = analyzed(&heap);
let node = slot(&index, "Node");
let page = index.dominated(None, Some(node), 0, 10_000);
assert_eq!((page.rows.len(), page.total), (MAX_PAGE as usize, 600));
let top = index.dominated(None, None, 0, 10_000);
assert_eq!((top.rows.len(), top.total), (MAX_PAGE as usize, 600));
assert_eq!(index.dominated(None, None, 550, 10_000).rows.len(), 50);
for (offset, limit) in [(u32::MAX, u32::MAX), (u32::MAX - 10, 100), (599, u32::MAX)] {
let page = index.dominated(None, Some(node), offset, limit);
assert_eq!((page.rows.len(), page.total), (usize::from(offset == 599), 600), "{offset} {limit}");
assert!(index.dominated(None, None, offset, limit).rows.len() <= 1);
}
}
#[test]
fn class_rows_skip_empty_slots() {
let index = analyzed(&sample());
let rows = index.class_rows();
let node = rows.iter().find(|r| r.name == "Node").unwrap();
assert_eq!(
(node.class_index, node.count, node.shallow, node.retained, node.unreachable_count, node.ambiguous),
(slot(&index, "Node"), 3, 72, 104, 1, false)
);
assert!(rows.iter().all(|r| r.count + r.unreachable_count > 0));
assert!(!rows.iter().any(|r| r.name == "boolean[]"));
assert!(rows.windows(2).all(|p| p[0].retained >= p[1].retained));
assert_eq!(index.summary().objects, 8);
}
#[test]
fn class_rows_mark_names_of_several_classes() {
let mut heap = sample();
// `object` takes the class defined last under a name, so each Foo gets one instance.
heap.class("Foo", Some("java.lang.Object"), &[])
.object(0x3000, "Foo", &[])
.class("Foo", Some("java.lang.Object"), &[])
.object(0x3100, "Foo", &[])
.object(0x3200, "Foo", &[]);
let index = analyzed(&heap);
let rows = index.class_rows();
let foos: Vec<_> = rows.iter().filter(|r| r.name == "Foo").collect();
assert_eq!(foos.len(), 2);
assert!(foos.iter().all(|r| r.ambiguous));
let mut counts: Vec<u64> = foos.iter().map(|r| r.unreachable_count).collect();
counts.sort_unstable();
assert_eq!(counts, [1, 2]);
assert!(rows.iter().filter(|r| r.name != "Foo").all(|r| !r.ambiguous));
}
#[test]
fn ipc_shapes() {
assert_eq!(json(&HeapObjectRef(7)), "7");
assert_eq!(json(&HeapEdge::ClassOf), r#"{"kind":"classOf"}"#);
assert_eq!(json(&HeapEdge::Element { index: 2 }), r#"{"kind":"element","index":2}"#);
assert_eq!(
json(&root(HeapRootKind::JavaFrame, Some("main"))),
r#"{"kind":"root","root":"javaFrame","thread":"main"}"#
);
let row = HeapClassRow {
class_index: 1,
name: "A".to_owned(),
count: 2,
shallow: 3,
retained: 4,
unreachable_count: 5,
ambiguous: false,
};
assert_eq!(
json(&row),
r#"{"classIndex":1,"name":"A","count":2,"shallow":3,"retained":4,"unreachableCount":5,"ambiguous":false}"#
);
let back: HeapObjectRef = serde_json::from_str("4294967295").unwrap();
assert_eq!(back, HeapObjectRef(u32::MAX));
}
fn json(value: &impl serde::Serialize) -> String {
serde_json::to_string(value).unwrap()
}
+44
Переглянути файл
@@ -0,0 +1,44 @@
//! HotSpot `.hprof` heap-dump reading and analysis for the debugger's JVM Memory view.
mod analysis;
mod column;
mod compare;
mod disk;
mod dominators;
mod error;
mod files;
mod graph;
mod hprof;
mod ids;
mod index;
mod names;
mod progress;
mod reader;
mod registry;
mod shallow;
mod strings;
#[cfg(test)]
mod test_index;
/// Fixture builders shared by this crate's tests and by dependants' tests and benches; not API.
#[doc(hidden)]
pub mod testing;
pub use analysis::{AnalyzeOptions, analyze};
pub use column::{Column, NONE, Pod, Store};
pub use compare::{HeapClassDiff, compare};
pub use disk::{DISK_MARGIN, capture_need, create_snapshot_dir, free_bytes, index_need, remove_stale, snapshots_root};
pub use dominators::{Dominators, dominators};
pub use error::HeapError;
pub use graph::{ClassInfo, HeapGraph, PRIM_SLOTS, build_graph};
pub use hprof::{
BasicType, ClassDump, HeapRecord, HeapRootKind, HprofHeader, Record, parse_header, record_at, scan, scan_segment,
};
pub use ids::{IdMap, IdMapBuilder};
pub use index::{
ClassStat, HeapClassRow, HeapEdge, HeapIndex, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapPathStep,
HeapRootPath, HeapSummary, MAX_PAGE,
};
pub use names::display_name;
pub use progress::HeapPhase;
pub use registry::{HeapOrigin, HeapRegistry, HeapSnapshotId, HeapSnapshotInfo, OpenSnapshot};
pub use shallow::{HistogramEntry, ShallowModel, SizeSource, VmLayout};
+66
Переглянути файл
@@ -0,0 +1,66 @@
//! Class names in the form the JVM's class histogram and the UI use.
/// The Java source form of an hprof class name (`java/util/HashMap$Node`, `[Ljava/lang/String;`),
/// as `ide_jvm_attach::display_class_name` writes it, plus `+0x` → `/0x` for hidden classes,
/// which the histogram spells with a slash. Names that already use dots pass through unchanged.
pub fn display_name(hprof_name: &str) -> String {
let dotted = hprof_name.replace('/', ".");
let dims = dotted.bytes().take_while(|&b| b == b'[').count();
let shown = if dims == 0 { dotted } else { array_name(&dotted, dims) };
// After the slash replacement, or the new slash would become a dot.
shown.replace("+0x", "/0x")
}
/// `[[I` → `int[][]`; a descriptor that is not one stays as it is.
fn array_name(dotted: &str, dims: usize) -> String {
let element = &dotted[dims..];
let base = match element {
"Z" => "boolean",
"B" => "byte",
"C" => "char",
"S" => "short",
"I" => "int",
"J" => "long",
"F" => "float",
"D" => "double",
_ => match element.strip_prefix('L').and_then(|e| e.strip_suffix(';')) {
Some(class) => class,
None => return dotted.to_owned(),
},
};
format!("{base}{}", "[]".repeat(dims))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn display_names() {
for (hprof, shown) in [
("[Z", "boolean[]"),
("[J", "long[]"),
("[[D", "double[][]"),
("[Ljava/lang/String;", "java.lang.String[]"),
("java/util/HashMap$Node", "java.util.HashMap$Node"),
("Gen$$Lambda+0x0000007800000a00", "Gen$$Lambda/0x0000007800000a00"),
] {
assert_eq!(display_name(hprof), shown, "{hprof}");
}
}
#[test]
fn dotted_names_and_hidden_class_arrays() {
for (hprof, shown) in [
("java.lang.String", "java.lang.String"),
("[Ljava.lang.Object;", "java.lang.Object[]"),
("[[Lpkg.Gen$$Lambda+0x0000007800000a00;", "pkg.Gen$$Lambda/0x0000007800000a00[][]"),
("[Ljava.lang.String", "[Ljava.lang.String"),
("[Q", "[Q"),
("0x7f00", "0x7f00"),
("", ""),
] {
assert_eq!(display_name(hprof), shown, "{hprof}");
}
}
}
+103
Переглянути файл
@@ -0,0 +1,103 @@
//! Phases of a heap analysis and the progress/cancellation checks every long pass makes.
use ide_core::CancellationToken;
use serde::{Deserialize, Serialize};
use crate::error::HeapError;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum HeapPhase {
Reading,
Linking,
Dominators,
Retained,
Paths,
Saving,
}
/// Longest stretch of work between two progress reports (and looks at the cancellation token).
pub(crate) const CHECK_EVERY: u64 = 1 << 20;
/// The caller's progress callback plus the token, looked at right after every report so a callback
/// that cancels stops the work before the next step.
pub(crate) struct Progress<'a> {
report: &'a dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &'a CancellationToken,
}
impl<'a> Progress<'a> {
pub(crate) fn new(
report: &'a dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &'a CancellationToken,
) -> Self {
Self { report, cancel }
}
pub(crate) fn report(&self, phase: HeapPhase, done: u64, total: u64) -> Result<(), HeapError> {
(self.report)(phase, done, total)?;
self.check()
}
pub(crate) fn token(&self) -> &'a CancellationToken {
self.cancel
}
/// Looks at the token without reporting.
pub(crate) fn check(&self) -> Result<(), HeapError> {
if self.cancel.is_cancelled() { Err(HeapError::Cancelled) } else { Ok(()) }
}
/// Looks at the token at step 0 and reports every [`CHECK_EVERY`] steps after it. Step 0 does
/// not report: a loop starts right after its phase's own report.
pub(crate) fn step(&self, step: u64, phase: HeapPhase, done: u64, total: u64) -> Result<(), HeapError> {
if step == 0 {
self.check()
} else if step.is_multiple_of(CHECK_EVERY) {
self.report(phase, done, total)
} else {
Ok(())
}
}
}
#[cfg(test)]
mod tests {
use std::cell::RefCell;
use super::*;
#[test]
fn steps_check_at_zero_and_report_every_check_interval() {
let calls = RefCell::new(Vec::new());
let report = |phase, done, total| {
calls.borrow_mut().push((phase, done, total));
Ok(())
};
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
for step in [0, 1, CHECK_EVERY - 1, CHECK_EVERY, CHECK_EVERY + 1, 3 * CHECK_EVERY] {
progress.step(step, HeapPhase::Linking, step, 9).unwrap();
}
assert_eq!(*calls.borrow(), [(HeapPhase::Linking, CHECK_EVERY, 9), (HeapPhase::Linking, 3 * CHECK_EVERY, 9)]);
cancel.cancel();
assert!(matches!(progress.step(0, HeapPhase::Linking, 0, 9), Err(HeapError::Cancelled)));
assert!(matches!(progress.report(HeapPhase::Saving, 1, 1), Err(HeapError::Cancelled)));
assert_eq!(calls.borrow().last(), Some(&(HeapPhase::Saving, 1, 1)), "reports before it looks at the token");
}
#[test]
fn report_errors_pass_through() {
let report = |_, _, _| Err(HeapError::NotEnoughDiskSpace { needed: 2, available: 1 });
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
assert!(matches!(progress.report(HeapPhase::Linking, 0, 1), Err(HeapError::NotEnoughDiskSpace { .. })));
}
#[test]
fn phases_serialize_in_camel_case() {
assert_eq!(serde_json::to_string(&HeapPhase::Dominators).unwrap(), "\"dominators\"");
assert_eq!(serde_json::from_str::<HeapPhase>("\"saving\"").unwrap(), HeapPhase::Saving);
}
}
+92
Переглянути файл
@@ -0,0 +1,92 @@
use crate::error::HeapError;
/// Big-endian cursor over a dump. Every read is bounds-checked; running off `buf` yields
/// `Truncated` at `buf.len()`, so callers bound a record by slicing `buf` to its end.
pub(crate) struct Reader<'a> {
buf: &'a [u8],
pos: usize,
id_size: u8,
}
impl<'a> Reader<'a> {
pub(crate) fn new(buf: &'a [u8], pos: usize, id_size: u8) -> Self {
Self { buf, pos, id_size }
}
pub(crate) fn pos(&self) -> usize {
self.pos
}
pub(crate) fn id_size(&self) -> u8 {
self.id_size
}
pub(crate) fn buf(&self) -> &'a [u8] {
self.buf
}
pub(crate) fn bytes(&mut self, n: usize) -> Result<&'a [u8], HeapError> {
let end = self.pos.checked_add(n).filter(|&end| end <= self.buf.len());
let Some(end) = end else {
return Err(HeapError::Truncated { at: self.buf.len() as u64 });
};
let slice = &self.buf[self.pos..end];
self.pos = end;
Ok(slice)
}
pub(crate) fn skip(&mut self, n: usize) -> Result<(), HeapError> {
self.bytes(n).map(|_| ())
}
fn array<const N: usize>(&mut self) -> Result<[u8; N], HeapError> {
let mut out = [0u8; N];
out.copy_from_slice(self.bytes(N)?);
Ok(out)
}
pub(crate) fn u1(&mut self) -> Result<u8, HeapError> {
Ok(self.array::<1>()?[0])
}
pub(crate) fn u2(&mut self) -> Result<u16, HeapError> {
Ok(u16::from_be_bytes(self.array()?))
}
pub(crate) fn u4(&mut self) -> Result<u32, HeapError> {
Ok(u32::from_be_bytes(self.array()?))
}
pub(crate) fn u8(&mut self) -> Result<u64, HeapError> {
Ok(u64::from_be_bytes(self.array()?))
}
/// `id_size` is 4 or 8; the public entry points reject anything else before building a reader.
pub(crate) fn id(&mut self) -> Result<u64, HeapError> {
if self.id_size == 4 { self.u4().map(u64::from) } else { self.u8() }
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reads_big_endian_and_ids_of_both_sizes() {
let buf = [1, 0, 2, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 5];
let mut r = Reader::new(&buf, 0, 8);
assert_eq!((r.u1().unwrap(), r.u2().unwrap(), r.u4().unwrap(), r.u8().unwrap()), (1, 2, 3, 4));
assert_eq!(r.pos(), 15);
assert_eq!(Reader::new(&buf[15..], 0, 4).id().unwrap(), 5);
}
#[test]
fn running_off_the_end_reports_the_buffer_end() {
let buf = [0u8; 6];
let mut r = Reader::new(&buf, 4, 8);
assert!(matches!(r.u4(), Err(HeapError::Truncated { at: 6 })));
assert_eq!(r.pos(), 4, "a failed read does not advance");
assert!(matches!(r.skip(usize::MAX), Err(HeapError::Truncated { at: 6 })));
assert!(matches!(Reader::new(&buf, 99, 8).u1(), Err(HeapError::Truncated { at: 6 })));
}
}
+163
Переглянути файл
@@ -0,0 +1,163 @@
//! The heap snapshots that are open in this process.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use ide_core::process::process_alive;
use ide_core::{DebugSessionId, IpcPath};
use serde::{Deserialize, Serialize};
use crate::disk::remove_stale;
use crate::index::{HeapIndex, HeapSummary};
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, specta::Type)]
#[serde(transparent)]
pub struct HeapSnapshotId(pub u32);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapOrigin {
/// Dumped from a live debug session.
Captured { session: DebugSessionId, pid: u32, java_version: String, taken_while_suspended: bool, dump_ms: u32 },
/// An `.hprof` the user opened; it is never deleted.
File { path: IpcPath },
}
impl HeapOrigin {
/// Text identifying the snapshot in the UI: the process id of a capture, the file name of a
/// file. The capture time is `created_ms`, which the UI formats in local time.
pub fn label(&self) -> String {
match self {
HeapOrigin::Captured { pid, .. } => pid.to_string(),
HeapOrigin::File { path } => path.file_name().unwrap_or(path.as_os_str()).to_string_lossy().into_owned(),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapSnapshotInfo {
pub id: HeapSnapshotId,
pub label: String,
pub origin: HeapOrigin,
pub summary: HeapSummary,
/// Unix time in milliseconds when the snapshot was opened.
pub created_ms: u64,
}
/// An analyzed snapshot together with the directory that holds its files. Dropping it unmaps the
/// index and then removes the directory, dump included for a capture; a dump opened from a file
/// is outside the directory and stays.
pub struct OpenSnapshot {
pub info: HeapSnapshotInfo,
/// `None` only while dropping, so the mappings are gone before the directory is deleted
/// (Windows cannot delete a mapped file).
index: Option<HeapIndex>,
/// Holds the index files and the dump of a capture.
pub dir: PathBuf,
pub dump: PathBuf,
/// Set by [`HeapRegistry::insert`] once `dir` is known to be one of the registry's own.
remove_dir: bool,
}
impl OpenSnapshot {
/// The directory is not removed on drop until the snapshot is in a [`HeapRegistry`] whose
/// root holds it.
pub fn new(info: HeapSnapshotInfo, index: HeapIndex, dir: PathBuf, dump: PathBuf) -> Self {
Self { info, index: Some(index), dir, dump, remove_dir: false }
}
pub fn index(&self) -> &HeapIndex {
self.index.as_ref().expect("the index is only taken while dropping")
}
}
impl Drop for OpenSnapshot {
fn drop(&mut self) {
drop(self.index.take());
if !self.remove_dir {
return;
}
// The user's own dump that happens to sit inside `dir` (an opened file is analyzed from a
// private copy, which goes with the directory).
if matches!(&self.info.origin, HeapOrigin::File { path } if **path == self.dump)
&& self.dump.starts_with(&self.dir)
{
return;
}
// A link is not followed: only a real directory is deleted.
if std::fs::symlink_metadata(&self.dir).is_ok_and(|meta| meta.file_type().is_dir())
&& let Err(err) = std::fs::remove_dir_all(&self.dir)
{
tracing::warn!(dir = %self.dir.display(), %err, "could not remove a heap snapshot directory");
}
}
}
struct State {
next_id: u32,
open: BTreeMap<u32, Arc<OpenSnapshot>>,
}
/// The open snapshots, shared between threads. Ids are never reused within a process.
pub struct HeapRegistry {
root: PathBuf,
state: Mutex<State>,
}
impl HeapRegistry {
/// Registry for snapshot directories under `root`; first removes the directories there that
/// belong to processes that no longer exist.
pub fn new(root: PathBuf) -> Self {
remove_stale(&root, process_alive);
Self { root, state: Mutex::new(State { next_id: 1, open: BTreeMap::new() }) }
}
pub fn root(&self) -> &Path {
&self.root
}
fn state(&self) -> MutexGuard<'_, State> {
self.state.lock().unwrap_or_else(PoisonError::into_inner)
}
/// Registers the snapshot that `make` builds for a fresh id. `make` runs outside the lock.
pub fn insert(&self, make: impl FnOnce(HeapSnapshotId) -> OpenSnapshot) -> HeapSnapshotInfo {
let id = {
let mut state = self.state();
let id = state.next_id;
state.next_id = id.saturating_add(1);
HeapSnapshotId(id)
};
let mut snapshot = make(id);
snapshot.info.id = id;
snapshot.remove_dir = snapshot.dir.file_name().is_some() && snapshot.dir.parent() == Some(self.root.as_path());
if !snapshot.remove_dir {
tracing::warn!(dir = %snapshot.dir.display(), "heap snapshot directory is outside the registry root; keeping it");
}
let info = snapshot.info.clone();
self.state().open.insert(id.0, Arc::new(snapshot));
info
}
pub fn get(&self, id: HeapSnapshotId) -> Option<Arc<OpenSnapshot>> {
self.state().open.get(&id.0).cloned()
}
/// Forgets the snapshot; its directory goes once the last query using it lets go. `false` for
/// an id that is not open.
pub fn close(&self, id: HeapSnapshotId) -> bool {
// Bound first so the lock is released before the snapshot may be dropped: that deletes files.
let removed = self.state().open.remove(&id.0);
removed.is_some()
}
/// In the order the snapshots were opened.
pub fn list(&self) -> Vec<HeapSnapshotInfo> {
self.state().open.values().map(|snapshot| snapshot.info.clone()).collect()
}
}
#[cfg(test)]
mod tests;
+265
Переглянути файл
@@ -0,0 +1,265 @@
use std::sync::Arc;
use ide_core::{CancellationToken, DebugSessionId};
use super::*;
use crate::analysis::{AnalyzeOptions, analyze};
use crate::disk::create_snapshot_dir;
use crate::error::HeapError;
use crate::progress::HeapPhase;
use crate::testing::sample;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn captured() -> HeapOrigin {
HeapOrigin::Captured {
session: DebugSessionId(7),
pid: 4242,
java_version: "21.0.2".into(),
taken_while_suspended: true,
dump_ms: 35,
}
}
fn external(path: &Path) -> HeapOrigin {
HeapOrigin::File { path: IpcPath(path.to_path_buf()) }
}
/// A snapshot whose index files are mapped from `dir`, like a real one: `dump` is analyzed in
/// place and `dir` receives the columns.
fn open_in(dir: &Path, dump: &Path, origin: HeapOrigin) -> impl FnOnce(HeapSnapshotId) -> OpenSnapshot {
let dir = dir.to_path_buf();
let dump = dump.to_path_buf();
move |id| {
std::fs::write(&dump, sample().bytes()).unwrap();
let index = analyze(&dump, &dir, AnalyzeOptions::default(), &no_progress, &CancellationToken::new()).unwrap();
let info = HeapSnapshotInfo {
id,
label: origin.label(),
origin,
summary: index.summary.clone(),
created_ms: 1_700_000_000_000,
};
OpenSnapshot::new(info, index, dir, dump)
}
}
/// Inserts a captured snapshot with its own directory under the registry's root.
fn insert_captured(registry: &HeapRegistry) -> (HeapSnapshotInfo, PathBuf) {
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("dump.hprof");
let info = registry.insert(open_in(&dir, &dump, captured()));
(info, dir)
}
fn names(dir: &Path) -> Vec<String> {
let mut names: Vec<_> =
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
names
}
#[test]
fn closing_snapshot_removes_dir() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (info, dir) = insert_captured(&registry);
assert!(dir.join("dump.hprof").exists() && dir.join("summary.json").exists());
assert!(registry.close(info.id));
assert!(!dir.exists());
assert!(!registry.close(info.id));
assert!(registry.get(info.id).is_none());
}
#[test]
fn external_dump_is_never_deleted() {
let root = tempfile::tempdir().unwrap();
let outside = tempfile::tempdir().unwrap();
let dump = outside.path().join("app.hprof");
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let info = registry.insert(open_in(&dir, &dump, external(&dump)));
assert_eq!(info.label, "app.hprof");
assert!(dir.join("summary.json").exists());
assert!(registry.close(info.id));
assert!(!dir.exists());
assert!(dump.exists());
}
#[test]
fn a_dump_inside_the_dir_of_a_file_snapshot_keeps_the_dir() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("opened.hprof");
let info = registry.insert(open_in(&dir, &dump, external(&dump)));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[test]
fn a_dir_outside_the_root_is_never_deleted() {
let root = tempfile::tempdir().unwrap();
let elsewhere = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dump = elsewhere.path().join("dump.hprof");
let info = registry.insert(open_in(elsewhere.path(), &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
// Neither the root itself nor a path that climbs out of it counts as a snapshot dir.
let sneaky = root.path().join("..").join(elsewhere.path().file_name().unwrap());
let info = registry.insert(open_in(&sneaky, &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[cfg(unix)]
#[test]
fn a_symlinked_dir_is_not_followed() {
let root = tempfile::tempdir().unwrap();
let outside = tempfile::tempdir().unwrap();
let dump = outside.path().join("dump.hprof");
let link = root.path().join("link");
std::os::unix::fs::symlink(outside.path(), &link).unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let info = registry.insert(open_in(&link, &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[test]
fn ids_are_never_reused() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (first, _) = insert_captured(&registry);
assert!(registry.close(first.id));
let (second, _) = insert_captured(&registry);
let (third, _) = insert_captured(&registry);
assert!(first.id.0 < second.id.0 && second.id.0 < third.id.0);
}
#[test]
fn the_registry_decides_the_id_in_the_info() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("dump.hprof");
let make = open_in(&dir, &dump, captured());
let info = registry.insert(|id| {
let mut snapshot = make(id);
snapshot.info.id = HeapSnapshotId(u32::MAX);
snapshot
});
assert_ne!(info.id, HeapSnapshotId(u32::MAX));
assert_eq!(registry.get(info.id).unwrap().info.id, info.id);
}
#[test]
fn list_is_in_opening_order_and_get_shares_the_snapshot() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (a, _) = insert_captured(&registry);
let (b, _) = insert_captured(&registry);
let (c, _) = insert_captured(&registry);
assert!(registry.close(b.id));
assert_eq!(registry.list(), [a.clone(), c.clone()]);
let one = registry.get(a.id).unwrap();
let two = registry.get(a.id).unwrap();
assert!(Arc::ptr_eq(&one, &two));
assert_eq!(one.info, a);
assert_eq!(one.index().summary, a.summary);
}
#[test]
fn a_snapshot_in_use_keeps_its_dir_until_the_last_holder_lets_go() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (info, dir) = insert_captured(&registry);
let held = registry.get(info.id).unwrap();
assert!(registry.close(info.id));
assert!(registry.get(info.id).is_none() && registry.list().is_empty());
assert!(dir.exists());
drop(held);
assert!(!dir.exists());
}
#[test]
fn dropping_the_registry_removes_the_open_snapshots() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (_, dir) = insert_captured(&registry);
drop(registry);
assert!(!dir.exists());
assert!(names(root.path()).is_empty());
}
#[cfg(unix)]
#[test]
fn new_clears_dirs_whose_owner_died() {
let root = tempfile::tempdir().unwrap();
let mut child = std::process::Command::new("true").spawn().unwrap();
let dead = child.id();
child.wait().unwrap();
let dead_dir = root.path().join("dead");
std::fs::create_dir(&dead_dir).unwrap();
std::fs::write(dead_dir.join("owner"), dead.to_string()).unwrap();
let live_dir = create_snapshot_dir(root.path()).unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
assert!(!dead_dir.exists());
assert!(live_dir.exists());
assert_eq!(registry.root(), root.path());
}
#[test]
fn concurrent_inserts_get_distinct_ids() {
let root = tempfile::tempdir().unwrap();
let registry = Arc::new(HeapRegistry::new(root.path().to_path_buf()));
let handles: Vec<_> = (0..6)
.map(|_| {
let registry = registry.clone();
std::thread::spawn(move || (0..3).map(|_| insert_captured(&registry).0.id).collect::<Vec<_>>())
})
.collect();
let mut ids: Vec<_> = handles.into_iter().flat_map(|h| h.join().unwrap()).collect();
ids.sort_by_key(|id| id.0);
ids.dedup();
assert_eq!(ids.len(), 18);
assert_eq!(registry.list().len(), 18);
for id in ids {
assert!(registry.close(id));
}
assert!(names(root.path()).is_empty());
}
#[test]
fn labels_name_the_pid_or_the_file() {
assert_eq!(captured().label(), "4242");
assert_eq!(external(Path::new("/tmp/dumps/app.hprof")).label(), "app.hprof");
assert_eq!(external(Path::new("/")).label(), "/");
}
#[test]
fn origin_and_info_serialize_as_the_ui_expects() {
let value = serde_json::to_value(captured()).unwrap();
assert_eq!(
value,
serde_json::json!({
"kind": "captured", "session": 7, "pid": 4242, "javaVersion": "21.0.2",
"takenWhileSuspended": true, "dumpMs": 35
})
);
assert_eq!(serde_json::to_value(external(Path::new("/a/b.hprof"))).unwrap()["path"], "/a/b.hprof");
assert_eq!(serde_json::to_value(HeapSnapshotId(3)).unwrap(), serde_json::json!(3));
let back: HeapOrigin = serde_json::from_value(value).unwrap();
assert_eq!(back, captured());
}
// `HeapRegistry::get` hands out `Arc<OpenSnapshot>` to query threads.
const _: fn() = || {
fn check<T: Send + Sync>() {}
check::<HeapIndex>();
check::<OpenSnapshot>();
check::<HeapRegistry>();
};
+265
Переглянути файл
@@ -0,0 +1,265 @@
//! Shallow sizes: the bytes each object takes in the JVM heap, from the object layout rules, the
//! JVM's own class histogram, or the distances between object addresses in the dump.
mod layout;
mod mirrors;
use std::collections::HashMap;
use ide_core::CancellationToken;
use serde::{Deserialize, Serialize};
pub use self::layout::VmLayout;
use self::mirrors::{Mirrors, injected_by_table};
use crate::error::HeapError;
use crate::graph::{ClassInfo, HeapGraph};
use crate::hprof::BasicType;
use crate::names::display_name;
use crate::progress::CHECK_EVERY;
/// How far past the layout size an address gap may be and still count as an object size.
const GAP_SLACK: u64 = 64;
/// Share of the instance types that can have a gap (two or more instances), in tenths, that must
/// take their size from address gaps for the sizes to be reported as coming from them.
const GAP_TRUST_TENTHS: usize = 9;
/// Where the sizes of instances came from.
///
/// Class objects (mirrors) are approximate in every case: HotSpot leaves the fields it injects
/// into `java.lang.Class` out of the dump, so their layout size is short by a fixed amount per
/// mirror. With a histogram that amount is fitted to its `java.lang.Class` row; otherwise (or when
/// the fit is rejected) it comes from a table measured per JDK and reference size, within a few
/// bytes per mirror, or is 32 when the JDK is unknown.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum SizeSource {
/// The JVM's class histogram from the capture.
Jvm,
/// The smallest distance between neighbouring objects of a type, in a dump the IDE did not
/// capture.
AddressGaps,
Layout,
}
/// A row of the JVM's class histogram.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct HistogramEntry {
/// Display form: `java.lang.String`, `int[]`, `Gen$$Lambda/0x…`.
pub name: String,
pub instances: u64,
pub bytes: u64,
}
enum TypeSize {
Fixed(u64),
ObjectArray,
/// Class objects are sized one by one; this is for an instance of `java.lang.Class` that is
/// not a class object.
ClassObject(u64),
}
/// Shallow size of every object of a [`HeapGraph`]. Instances of a type share one size; arrays
/// follow the layout; class objects add their statics to the size of `java.lang.Class`.
pub struct ShallowModel {
pub source: SizeSource,
layout: VmLayout,
/// Per type slot of a class; primitive array slots follow and have no entry.
types: Vec<TypeSize>,
/// `(node, size)` of each class object, by node.
class_objects: Vec<(u32, u64)>,
/// Per class slot: another class has the same display name.
ambiguous: Vec<bool>,
}
impl ShallowModel {
/// `histogram` is the JVM's class histogram of the same stop as the dump, when the IDE captured
/// it. Without it, sizes come from address gaps where they agree with the layout.
///
/// A name shared by several classes (several class loaders) never takes a histogram size,
/// since the name is the only link between the dump and the histogram; see
/// [`Self::is_ambiguous`]. `java.lang.Class` never does either, because mirror sizes vary with
/// their statics; the histogram only tells how many bytes HotSpot adds to each mirror.
pub fn new(
graph: &HeapGraph,
layout: VmLayout,
histogram: Option<&[HistogramEntry]>,
cancel: &CancellationToken,
) -> Result<Self, HeapError> {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
let classes = &graph.classes;
let field_bytes = field_bytes(classes, layout);
let by_layout: Vec<u64> = field_bytes.iter().map(|&bytes| layout.instance_size(bytes)).collect();
let names: Vec<String> = classes.iter().map(|c| display_name(&c.name)).collect();
let ambiguous = ambiguous_names(&names);
let class_slot = graph.class_slot as usize;
let mirrors = Mirrors::collect(graph, by_layout[class_slot], layout);
let mut fitted = None;
if let Some(rows) = histogram
&& !ambiguous[class_slot]
{
let mut class_rows = rows.iter().filter(|row| row.name == names[class_slot]);
if let (Some(row), None) = (class_rows.next(), class_rows.next()) {
fitted = mirrors.fit_injected(graph, layout, by_layout[class_slot], row, cancel)?;
}
}
let injected = fitted.unwrap_or_else(|| injected_by_table(layout));
let mut types: Vec<TypeSize> = classes
.iter()
.enumerate()
.map(|(slot, class)| {
if class.name.starts_with('[') {
TypeSize::ObjectArray
} else if slot == class_slot {
TypeSize::ClassObject(layout.align(by_layout[slot] + injected))
} else {
TypeSize::Fixed(by_layout[slot])
}
})
.collect();
let source = match histogram {
Some(rows) => {
apply_histogram(&mut types, &names, &ambiguous, rows);
SizeSource::Jvm
}
None => apply_address_gaps(&mut types, graph, &by_layout, cancel)?,
};
Ok(Self { source, layout, types, class_objects: mirrors.sizes(layout, injected), ambiguous })
}
/// Shallow size of `object` (a node of the graph); 0 for the super-root.
pub fn size_of(&self, graph: &HeapGraph, object: u32) -> u64 {
let Some(&slot) = graph.types.get(object as usize) else { return 0 };
match self.types.get(slot as usize) {
Some(&TypeSize::Fixed(size)) => size,
Some(TypeSize::ObjectArray) => self.array_size(BasicType::Object, graph.lens[object as usize]),
Some(&TypeSize::ClassObject(fallback)) => self
.class_objects
.binary_search_by_key(&object, |&(node, _)| node)
.map_or(fallback, |at| self.class_objects[at].1),
None => match graph.prim_type(slot) {
Some(ty) => self.array_size(ty, graph.lens[object as usize]),
None => 0,
},
}
}
/// Whether another class has the same display name as the class in `slot`. Sizes of such
/// classes come from the layout, and anything joined to them by name (histogram rows, a
/// second snapshot) cannot tell them apart.
pub fn is_ambiguous(&self, slot: u32) -> bool {
self.ambiguous.get(slot as usize).copied().unwrap_or(false)
}
fn array_size(&self, ty: BasicType, len: u32) -> u64 {
self.layout.array_size(ty, u64::from(len))
}
}
/// Bytes of instance fields per class, its superclasses included. Super chains always end
/// (`ClassInfo::super_index` has its cycles cut).
fn field_bytes(classes: &[ClassInfo], layout: VmLayout) -> Vec<u64> {
let mut total: Vec<Option<u64>> = vec![None; classes.len()];
let mut path = Vec::new();
for start in 0..classes.len() {
let mut cursor = Some(start);
let mut below = 0;
while let Some(c) = cursor {
if let Some(known) = total[c] {
below = known;
break;
}
path.push(c);
cursor = classes[c].super_index.map(|s| s as usize);
}
while let Some(c) = path.pop() {
let own: u64 = classes[c].fields.iter().map(|(_, ty)| ty.heap_size(layout.ref_size)).sum();
below += own;
total[c] = Some(below);
}
}
total.into_iter().map(Option::unwrap_or_default).collect()
}
fn ambiguous_names(names: &[String]) -> Vec<bool> {
let mut counts: HashMap<&str, u32> = HashMap::with_capacity(names.len());
for name in names {
*counts.entry(name).or_default() += 1;
}
names.iter().map(|name| counts[name.as_str()] > 1).collect()
}
/// `bytes / instances` for each plain class with a histogram row of its name. A name that is
/// ambiguous in the dump or repeated in the histogram, or a row that is not a whole number of
/// objects, keeps its layout size.
fn apply_histogram(types: &mut [TypeSize], names: &[String], ambiguous: &[bool], rows: &[HistogramEntry]) {
let mut per_object: HashMap<&str, Option<u64>> = HashMap::with_capacity(rows.len());
for row in rows {
let size = row.bytes.checked_div(row.instances).filter(|&s| s > 0 && row.bytes % row.instances == 0);
per_object.entry(&row.name).and_modify(|seen| *seen = None).or_insert(size);
}
for (slot, ty) in types.iter_mut().enumerate() {
if let TypeSize::Fixed(size) = ty
&& !ambiguous[slot]
&& let Some(&Some(from_histogram)) = per_object.get(names[slot].as_str())
{
*size = from_histogram;
}
}
}
/// Takes the smallest distance from an instance to the next object in address order as the size
/// of its type, for types with two or more instances where that distance is no smaller than the
/// layout size and at most [`GAP_SLACK`] bigger. The result says [`SizeSource::AddressGaps`] when
/// nine in ten of the types with two or more instances did; types with one instance have no gap
/// to measure, so they neither count for nor against.
fn apply_address_gaps(
types: &mut [TypeSize],
graph: &HeapGraph,
by_layout: &[u64],
cancel: &CancellationToken,
) -> Result<SizeSource, HeapError> {
let slots = types.len();
let mut instances = vec![0u32; slots];
let mut smallest_gap = vec![u64::MAX; slots];
let mut ids = graph.ids.ids();
let mut current = ids.next();
let mut object = 0usize;
while let Some(id) = current {
if (object as u64).is_multiple_of(CHECK_EVERY) && cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
let next = ids.next();
let slot = graph.types[object] as usize;
if matches!(types.get(slot), Some(TypeSize::Fixed(_))) {
instances[slot] = instances[slot].saturating_add(1);
if let Some(next) = next.filter(|&n| n > id) {
smallest_gap[slot] = smallest_gap[slot].min(next - id);
}
}
current = next;
object += 1;
}
let (mut measurable, mut by_gap) = (0usize, 0usize);
for (slot, ty) in types.iter_mut().enumerate() {
let TypeSize::Fixed(size) = ty else { continue };
if instances[slot] < 2 {
continue;
}
measurable += 1;
let gap = smallest_gap[slot];
if (by_layout[slot]..=by_layout[slot] + GAP_SLACK).contains(&gap) {
*size = gap;
by_gap += 1;
}
}
let trusted = measurable > 0 && by_gap * 10 >= measurable * GAP_TRUST_TENTHS;
Ok(if trusted { SizeSource::AddressGaps } else { SizeSource::Layout })
}
#[cfg(test)]
mod tests;
+113
Переглянути файл
@@ -0,0 +1,113 @@
//! The object layout rules of the JVM that wrote a dump.
use std::collections::HashMap;
use crate::hprof::BasicType;
/// Object layout of the JVM that wrote the dump.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct VmLayout {
pub header: u8,
/// Bytes of a reference: 4 with compressed oops.
pub ref_size: u8,
/// Offset of the length field of an array.
pub length_offset: u8,
/// JDK 21 and older put array elements at the next 8-byte boundary after the length, later
/// JDKs at the next boundary of the element size.
pub legacy_array_base: bool,
pub alignment: u16,
/// The JDK the flags came from; `None` when nothing is known about the JVM.
pub java_major: Option<u32>,
}
impl VmLayout {
pub const DEFAULT: VmLayout = VmLayout {
header: 12,
ref_size: 4,
length_offset: 12,
legacy_array_base: false,
alignment: 8,
java_major: None,
};
/// `flags` is the output of `VM.flags`, which lists the flags set on this run. Compressed oops
/// are off by default and only switched on by ergonomics, so `-XX:+UseCompressedOops` is
/// printed exactly when references take 4 bytes; a heap too big for them prints no such flag.
/// Before JDK 15 compressed class pointers follow the same rule; with `java_major: None` (JDK
/// unknown) they follow the JDK 15+ rule and the array base that of the newest JDKs. Flags
/// without any `-XX:` option (an external dump) give [`Self::DEFAULT`] with the array base of
/// `java_major` and an unknown JDK, since the reference size is a guess.
/// An `ObjectAlignmentInBytes` that is not a power of two in `8..=256` is ignored.
pub fn from_vm_flags(flags: &str, java_major: Option<u32>) -> Self {
let legacy_array_base = java_major.map_or(Self::DEFAULT.legacy_array_base, |major| major <= 21);
let flags = parse_flags(flags);
if flags.is_empty() {
return VmLayout { legacy_array_base, ..Self::DEFAULT };
}
let on = |name: &str| flags.get(name) == Some(&"true");
let compressed_classes = if java_major.is_none_or(|major| major >= 15) {
flags.get("UseCompressedClassPointers") != Some(&"false")
} else {
on("UseCompressedClassPointers")
};
let (header, length_offset) = if on("UseCompactObjectHeaders") {
(8, 8)
} else if compressed_classes {
(12, 12)
} else {
(16, 16)
};
let alignment = flags
.get("ObjectAlignmentInBytes")
.and_then(|v| v.parse::<u16>().ok())
.filter(|a| (8..=256).contains(a) && a.is_power_of_two())
.unwrap_or(Self::DEFAULT.alignment);
VmLayout {
header,
ref_size: if on("UseCompressedOops") { 4 } else { 8 },
length_offset,
legacy_array_base,
alignment,
java_major,
}
}
/// Size of an instance whose fields (its class and every superclass) take `field_bytes`.
pub fn instance_size(&self, field_bytes: u64) -> u64 {
let smallest = if self.header == 8 { 8 } else { 16 };
self.align(u64::from(self.header) + field_bytes).max(smallest)
}
pub fn array_size(&self, ty: BasicType, len: u64) -> u64 {
let element = ty.heap_size(self.ref_size);
let base_alignment = if self.legacy_array_base { 8 } else { element };
let base = align_to(u64::from(self.length_offset) + 4, base_alignment);
self.align(base + element * len)
}
/// `value` rounded up to the object alignment.
pub(super) fn align(&self, value: u64) -> u64 {
align_to(value, u64::from(self.alignment))
}
}
fn align_to(value: u64, to: u64) -> u64 {
value.div_ceil(to.max(1)) * to.max(1)
}
/// `-XX:+Name` / `-XX:-Name` / `-XX:Name=value` tokens by name; booleans become `"true"` / `"false"`.
/// A flag given twice keeps its last value, as in the JVM.
fn parse_flags(flags: &str) -> HashMap<&str, &str> {
let mut parsed = HashMap::new();
for token in flags.split_whitespace() {
let Some(body) = token.strip_prefix("-XX:") else { continue };
if let Some(name) = body.strip_prefix('+') {
parsed.insert(name, "true");
} else if let Some(name) = body.strip_prefix('-') {
parsed.insert(name, "false");
} else if let Some((name, value)) = body.split_once('=') {
parsed.insert(name, value);
}
}
parsed
}
+119
Переглянути файл
@@ -0,0 +1,119 @@
//! Sizes of class objects (mirrors): the `java.lang.Class` instance plus the class's statics, plus
//! the fields HotSpot injects into every mirror and leaves out of the class dump.
use ide_core::CancellationToken;
use super::{HistogramEntry, VmLayout};
use crate::error::HeapError;
use crate::graph::HeapGraph;
use crate::progress::CHECK_EVERY;
/// Largest injected byte count (per mirror, a multiple of 4) the histogram fit tries.
const MAX_INJECTED: u64 = 128;
/// How far, in percent of the histogram's total, the best fit may be off before the row is taken
/// for something else than the mirrors of this dump.
const MAX_FIT_ERROR_PERCENT: u128 = 5;
/// Bytes HotSpot injects into every mirror (`klass`, `array_klass`, `oop_size`,
/// `static_oop_field_count`, and oops such as `protection_domain`) without listing them in the class
/// dump, as measured by comparing the dump with `GC.class_histogram` on JDK 8, 11, 21, 23 and 25,
/// with and without compressed oops. Each X is the best fit in steps of 4; the real value is
/// within a few bytes (statics padding). `None` for a reference size the table does not cover.
pub(super) fn injected_mirror_bytes(java_major: u32, ref_size: u8) -> Option<u64> {
match (ref_size, java_major) {
// 8, 11, 21 and 23 measure 36; 25 is the first JDK that shows 28.
(4, ..=24) => Some(36),
(4, 25..) => Some(28),
// 8, 11 and 21 measure 44; 23 is the first to show 40 and 25 the first to show 32.
(8, ..=22) => Some(44),
(8, 23..=24) => Some(40),
(8, 25..) => Some(32),
_ => None,
}
}
/// Used when the JVM is unknown (a dump without `VM.flags`): the midpoint of the measured values
/// with 4-byte references, 36 and 28.
const UNKNOWN_JVM_INJECTED: u64 = 32;
/// The injected bytes per mirror the layout implies, without looking at a histogram.
pub(super) fn injected_by_table(layout: VmLayout) -> u64 {
layout.java_major.and_then(|major| injected_mirror_bytes(major, layout.ref_size)).unwrap_or(UNKNOWN_JVM_INJECTED)
}
pub(super) struct Mirrors {
/// `(node, size before alignment)` of every class dump that is an object of the graph, by node.
dumped: Vec<(u32, u64)>,
}
impl Mirrors {
/// `instance` is the layout size of `java.lang.Class`, the base of every mirror.
pub(super) fn collect(graph: &HeapGraph, instance: u64, layout: VmLayout) -> Self {
let mut dumped: Vec<(u32, u64)> = graph
.classes
.iter()
.filter(|class| class.id != 0)
.filter_map(|class| {
let node = graph.ids.rank(class.id)?;
let references = class.statics.len() as u64 * u64::from(layout.ref_size);
Some((node, instance + class.static_bytes + references))
})
.collect();
dumped.sort_unstable_by_key(|&(node, _)| node);
Self { dumped }
}
/// `(node, size)` of each class object when HotSpot adds `injected` bytes to every mirror.
pub(super) fn sizes(&self, layout: VmLayout, injected: u64) -> Vec<(u32, u64)> {
self.dumped.iter().map(|&(node, bytes)| (node, layout.align(bytes + injected))).collect()
}
/// The injected bytes per mirror (a multiple of 4, at most 128) that make the mirrors of this
/// dump add up to `class_row`'s share of the histogram, or `None` when no value gets within 5%:
/// a histogram of other mirrors than these (a different JVM stop) must not skew the sizes.
///
/// The fit compares averages, since the histogram also counts mirrors the dump has no class
/// dump for (primitive types, classes only archived by CDS). Every mirror still gets the same
/// extra bytes, never the average size, because statics make mirror sizes differ.
pub(super) fn fit_injected(
&self,
graph: &HeapGraph,
layout: VmLayout,
instance: u64,
class_row: &HistogramEntry,
cancel: &CancellationToken,
) -> Result<Option<u64>, HeapError> {
if class_row.instances == 0 {
return Ok(None);
}
// Mirrors without a class dump (primitive types) have no statics.
let class_typed = count_slot(graph, graph.class_slot, cancel)?;
let bare = class_typed.saturating_sub(self.dumped.len() as u64);
let mirrors = self.dumped.len() as u64 + bare;
if mirrors == 0 {
return Ok(None);
}
let target = u128::from(class_row.bytes) * u128::from(mirrors) / u128::from(class_row.instances);
let total = |injected: u64| -> u128 {
let dumped: u128 = self.dumped.iter().map(|&(_, bytes)| u128::from(layout.align(bytes + injected))).sum();
dumped + u128::from(bare) * u128::from(layout.align(instance + injected))
};
let (best, error) = (0..=MAX_INJECTED)
.step_by(4)
.map(|injected| (injected, total(injected).abs_diff(target)))
.min_by_key(|&(injected, error)| (error, injected))
.unwrap_or((0, u128::MAX));
Ok((error * 100 <= target * MAX_FIT_ERROR_PERCENT).then_some(best))
}
}
fn count_slot(graph: &HeapGraph, slot: u32, cancel: &CancellationToken) -> Result<u64, HeapError> {
let mut count = 0;
for chunk in graph.types.chunks(CHECK_EVERY as usize) {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
count += chunk.iter().filter(|&&ty| ty == slot).count() as u64;
}
Ok(count)
}
+593
Переглянути файл
@@ -0,0 +1,593 @@
use super::*;
use crate::column::Store;
use crate::graph::build_graph;
use crate::progress::HeapPhase;
use crate::testing::{HprofWriter, MiniHeap, sample};
const ROOT: Option<&str> = Some("java.lang.Object");
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn graph_of(heap: &MiniHeap) -> HeapGraph {
build_graph(&heap.bytes(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap()
}
fn model(g: &HeapGraph, layout: VmLayout, histogram: Option<&[HistogramEntry]>) -> ShallowModel {
ShallowModel::new(g, layout, histogram, &CancellationToken::new()).unwrap()
}
fn size_at(g: &HeapGraph, m: &ShallowModel, addr: u64) -> u64 {
m.size_of(g, g.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the graph")))
}
fn slot_at(g: &HeapGraph, addr: u64) -> u32 {
g.types[g.ids.rank(addr).unwrap() as usize]
}
fn entry(name: &str, instances: u64, bytes: u64) -> HistogramEntry {
HistogramEntry { name: name.to_owned(), instances, bytes }
}
fn flags(flags: &str, java_major: u32) -> VmLayout {
VmLayout::from_vm_flags(flags, Some(java_major))
}
const SHAPES: [&str; 9] = ["A", "B", "C", "D", "E", "F", "G", "H", "Node"];
/// One instance of each shape, 0x100 apart so no address gap qualifies.
fn shapes() -> MiniHeap {
use BasicType::{Byte, Int, Long, Object, Short};
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", Byte)])
.class("B", ROOT, &[("f", Long)])
.class("C", ROOT, &[("i", Int), ("l", Long), ("b", Byte)])
.class("D", Some("C"), &[("s", Short), ("o", Object)])
.class("E", ROOT, &[("a", Object), ("b", Object), ("c", Object)])
.class("F", Some("A"), &[("g", Byte)])
.class("G", ROOT, &[])
.class("H", Some("B"), &[("i", Int)])
.class("Node", ROOT, &[("l", Long), ("a", Object), ("b", Object)]);
for (i, name) in SHAPES.iter().enumerate() {
heap.object(0x1000 + i as u64 * 0x100, name, &[]);
}
heap
}
fn shape_sizes(layout: VmLayout) -> Vec<u64> {
let g = graph_of(&shapes());
let m = model(&g, layout, None);
(0..SHAPES.len() as u64).map(|i| size_at(&g, &m, 0x1000 + i * 0x100)).collect()
}
// `jcmd <pid> VM.flags` of `-Xmx1g` and `-Xmx40g` runs (the flags that matter here).
const J8_11_1G: &str = "-XX:MaxHeapSize=1073741824 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops";
const J8_11_40G: &str = "-XX:MaxHeapSize=42949672960";
const J21_25_1G: &str = "-XX:MaxHeapSize=1073741824 -XX:SoftMaxHeapSize=1073741824 -XX:+UseCompressedOops";
const J21_25_40G: &str = "-XX:MaxHeapSize=42949672960 -XX:SoftMaxHeapSize=42949672960";
/// `-Xmx40g -XX:+UseCompressedOops`: the JVM warns and turns them off.
const J21_40G_OFF: &str = "-XX:MaxHeapSize=42949672960 -XX:SoftMaxHeapSize=42949672960 -XX:-UseCompressedOops";
const J25_COMPACT: &str = "-XX:MaxHeapSize=1073741824 -XX:+UseCompressedOops -XX:+UseCompactObjectHeaders";
#[test]
fn layout_from_flags() {
// The layout of a JVM of `major` with default flags otherwise.
let jvm = |major: u32, layout: VmLayout| VmLayout { java_major: Some(major), ..layout };
let legacy = VmLayout { legacy_array_base: true, ..VmLayout::DEFAULT };
assert_eq!(flags(J8_11_1G, 8), jvm(8, legacy));
assert_eq!(flags(J8_11_1G, 11), jvm(11, legacy));
assert_eq!(flags(J21_25_1G, 21), jvm(21, legacy));
assert_eq!(flags(J21_25_1G, 25), jvm(25, VmLayout::DEFAULT));
assert_eq!(flags(J25_COMPACT, 25), jvm(25, VmLayout { header: 8, length_offset: 8, ..VmLayout::DEFAULT }));
assert_eq!(flags(&format!("{J21_25_1G} -XX:+UseG1GC"), 21), jvm(21, legacy));
assert_eq!(
flags(&format!("{J21_25_1G} -XX:ObjectAlignmentInBytes=16"), 21),
jvm(21, VmLayout { alignment: 16, ..legacy })
);
}
#[test]
fn heaps_too_big_for_compressed_oops_print_no_flag() {
let jvm = |major: u32, layout: VmLayout| VmLayout { java_major: Some(major), ..layout };
let wide_legacy =
VmLayout { header: 16, ref_size: 8, length_offset: 16, legacy_array_base: true, ..VmLayout::DEFAULT };
assert_eq!(flags(J8_11_40G, 8), jvm(8, wide_legacy));
assert_eq!(flags(J8_11_40G, 11), jvm(11, wide_legacy));
// From JDK 15 compressed class pointers stay on without compressed oops.
let wide = VmLayout { ref_size: 8, ..VmLayout::DEFAULT };
assert_eq!(flags(J21_25_40G, 21), jvm(21, VmLayout { legacy_array_base: true, ..wide }));
assert_eq!(flags(J21_25_40G, 25), jvm(25, wide));
assert_eq!(flags(J21_40G_OFF, 21), flags(J21_25_40G, 21));
}
#[test]
fn flags_of_an_unknown_jdk_follow_the_jdk_15_class_pointer_rule() {
let unknown = |f: &str| VmLayout::from_vm_flags(f, None);
assert_eq!(unknown(J21_25_1G), VmLayout::DEFAULT);
assert_eq!(unknown(J8_11_40G), VmLayout { ref_size: 8, ..VmLayout::DEFAULT });
let no_ccp = unknown("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers");
assert_eq!((no_ccp.header, no_ccp.length_offset, no_ccp.java_major), (16, 16, None));
assert_eq!(unknown(J25_COMPACT).header, 8);
// The mirror table has no row for an unknown JDK.
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
assert_eq!(mirror_sizes(&g, &addrs, unknown(J21_25_1G), None), expected_mirrors(&MIRROR_STATICS, 32));
}
#[test]
fn layout_flag_edge_cases() {
let legacy = VmLayout { legacy_array_base: true, ..VmLayout::DEFAULT };
// No VM.flags (an external dump): the default layout, of an unknown JVM.
assert_eq!(flags("", 25), VmLayout::DEFAULT);
assert_eq!(flags("", 21), legacy);
assert_eq!(flags("-Xmx1g --add-opens=x -XX:Foo", 25), VmLayout::DEFAULT);
// Before JDK 15 compressed class pointers are on only when the flags say so.
assert_eq!(flags("-XX:+UseCompressedOops", 11).header, 16);
assert_eq!(flags("-XX:+UseCompressedOops -XX:+UseCompressedClassPointers", 11).header, 12);
assert_eq!(flags("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers", 11).header, 16);
assert_eq!(flags("-XX:+UseCompressedOops", 15).header, 12);
let no_ccp = flags("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers", 21);
assert_eq!((no_ccp.header, no_ccp.length_offset, no_ccp.ref_size), (16, 16, 4));
// Compact headers win over the class pointer flag; the last mention of a flag wins.
assert_eq!(flags("-XX:-UseCompressedClassPointers -XX:+UseCompactObjectHeaders", 25).header, 8);
assert_eq!(flags("-XX:-UseCompressedOops -XX:+UseCompressedOops", 21).ref_size, 4);
assert_eq!(flags("-XX:+UseCompressedOops -XX:-UseCompressedOops", 21).ref_size, 8);
}
#[test]
fn object_alignment_accepts_powers_of_two_from_8_to_256() {
for good in [8u16, 16, 32, 64, 128, 256] {
assert_eq!(flags(&format!("-XX:ObjectAlignmentInBytes={good}"), 25).alignment, good);
}
for bad in ["0", "4", "7", "12", "255", "512", "abc", "", "999999999999"] {
assert_eq!(flags(&format!("-XX:ObjectAlignmentInBytes={bad}"), 25).alignment, 8, "{bad:?}");
}
let wide = flags("-XX:+UseCompressedOops -XX:ObjectAlignmentInBytes=256", 21);
assert_eq!(wide.instance_size(1), 256);
assert_eq!(wide.instance_size(300), 512);
assert_eq!(wide.array_size(BasicType::Byte, 1), 256);
}
#[test]
fn instance_sizes_match_the_jvm() {
let at = |sizes: &[u64], name: &str| sizes[SHAPES.iter().position(|n| *n == name).unwrap()];
let coops = shape_sizes(flags(J21_25_1G, 21));
for (name, size) in
[("A", 16), ("B", 24), ("C", 32), ("D", 32), ("E", 24), ("F", 16), ("G", 16), ("H", 24), ("Node", 32)]
{
assert_eq!(at(&coops, name), size, "{name} with compressed oops");
}
let wide = shape_sizes(flags(J21_25_40G, 21));
for (name, size) in [("D", 40), ("E", 40), ("Node", 40)] {
assert_eq!(at(&wide, name), size, "{name} without compressed oops");
}
let compact = shape_sizes(flags(J25_COMPACT, 25));
for (name, size) in [("G", 8), ("B", 16), ("C", 24), ("D", 32), ("Node", 24)] {
assert_eq!(at(&compact, name), size, "{name} with compact headers");
}
}
#[test]
fn instance_size_bounds() {
assert_eq!(VmLayout::DEFAULT.instance_size(0), 16);
assert_eq!(flags("-XX:+UseCompactObjectHeaders", 25).instance_size(0), 8);
assert_eq!(flags("-XX:ObjectAlignmentInBytes=16", 21).instance_size(1), 16);
assert_eq!(flags("-XX:ObjectAlignmentInBytes=16", 21).instance_size(8), 32);
}
#[test]
fn array_sizes() {
let d = VmLayout::DEFAULT;
assert_eq!(d.array_size(BasicType::Long, 2), 32);
assert_eq!(d.array_size(BasicType::Byte, 5), 24);
assert_eq!(d.array_size(BasicType::Object, 3), 32);
let compact = flags("-XX:+UseCompactObjectHeaders", 25);
assert_eq!(compact.array_size(BasicType::Byte, 5), 24);
assert_eq!(compact.array_size(BasicType::Int, 1), 16);
assert_eq!(compact.array_size(BasicType::Long, 1), 24);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 21).array_size(BasicType::Byte, 1), 32);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 25).array_size(BasicType::Byte, 1), 24);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 25).array_size(BasicType::Long, 1), 32);
// Empty arrays and the largest length fit.
assert_eq!(d.array_size(BasicType::Int, 0), 16);
assert_eq!(d.array_size(BasicType::Long, u64::from(u32::MAX)), 16 + 8 * u64::from(u32::MAX));
}
#[test]
fn arrays_in_the_graph_follow_the_layout() {
let g = graph_of(&sample());
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x1400), 32, "long[2]");
assert_eq!(size_at(&g, &m, 0x1500), 32, "Object[3]");
assert_eq!(size_at(&g, &m, 0x1100), 24, "Node");
let wide = model(&g, flags("-XX:-UseCompressedOops", 21), None);
assert_eq!(size_at(&g, &wide, 0x1500), 40, "Object[3] with 8-byte references");
assert_eq!(m.size_of(&g, g.root), 0, "the super-root has no size");
assert_eq!(m.source, SizeSource::Layout);
}
#[test]
fn histogram_wins_for_instances() {
use BasicType::Int;
let fields: Vec<_> = (0..23).map(|i| (format!("f{i}"), Int)).collect();
let fields: Vec<(&str, BasicType)> = fields.iter().map(|(n, t)| (n.as_str(), *t)).collect();
let mut heap = MiniHeap::new();
heap.class("java.lang.Thread", ROOT, &fields)
.class("Plain", ROOT, &[("f", Int)])
.object(0x1000, "java.lang.Thread", &[])
.object(0x1100, "Plain", &[]);
let g = graph_of(&heap);
let without = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &without, 0x1000), 104);
let histogram = [entry("java.lang.Thread", 13, 1560), entry("Unrelated", 2, 64)];
let with = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(with.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &with, 0x1000), 120);
assert_eq!(size_at(&g, &with, 0x1100), 16, "no row: layout");
}
#[test]
fn histogram_rows_that_do_not_divide_or_are_empty_fall_back_to_layout() {
let mut heap = MiniHeap::new();
heap.class("Plain", ROOT, &[("f", BasicType::Int)]).object(0x1000, "Plain", &[]);
let g = graph_of(&heap);
for row in [entry("Plain", 3, 100), entry("Plain", 0, 0), entry("Plain", 0, 48), entry("Plain", 2, 0)] {
let m = model(&g, VmLayout::DEFAULT, Some(std::slice::from_ref(&row)));
assert_eq!(m.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &m, 0x1000), 16, "{row:?}");
}
// The same name twice in the histogram cannot be told apart either.
let twice = [entry("Plain", 1, 48), entry("Plain", 1, 64)];
assert_eq!(size_at(&g, &model(&g, VmLayout::DEFAULT, Some(&twice)), 0x1000), 16);
}
#[test]
fn histogram_matches_arrays_by_display_name_but_never_sizes_them() {
let g = graph_of(&sample());
let histogram = [entry("java.lang.Object[]", 1, 4096), entry("long[]", 1, 4096), entry("Node", 4, 160)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(size_at(&g, &m, 0x1500), 32);
assert_eq!(size_at(&g, &m, 0x1400), 32);
assert_eq!(size_at(&g, &m, 0x1100), 40, "Node from the histogram");
}
#[test]
fn a_name_shared_by_several_classes_never_takes_a_histogram_size() {
use BasicType::{Int, Long};
let mut heap = MiniHeap::new();
heap.class("dup.Foo", ROOT, &[("a", Int)])
.object(0x1000, "dup.Foo", &[])
.class("dup.Foo", ROOT, &[("a", Long), ("b", Long)])
.object(0x1100, "dup.Foo", &[])
.class("Solo", ROOT, &[("a", Int)])
.object(0x1200, "Solo", &[]);
let g = graph_of(&heap);
let histogram = [entry("dup.Foo", 2, 80), entry("Solo", 1, 48)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(size_at(&g, &m, 0x1000), 16, "first dup.Foo uses its layout");
assert_eq!(size_at(&g, &m, 0x1100), 32, "second dup.Foo uses its layout");
assert_eq!(size_at(&g, &m, 0x1200), 48, "Solo still takes its histogram size");
let (first, second, solo) = (slot_at(&g, 0x1000), slot_at(&g, 0x1100), slot_at(&g, 0x1200));
assert_ne!(first, second);
assert!(m.is_ambiguous(first) && m.is_ambiguous(second));
assert!(!m.is_ambiguous(solo));
let ambiguous: Vec<_> = (0..g.classes.len() as u32).filter(|&s| m.is_ambiguous(s)).collect();
assert_eq!(ambiguous.len(), 2, "no other class is flagged");
assert!(!m.is_ambiguous(u32::MAX), "slots past the classes are never ambiguous");
}
#[test]
fn ambiguity_covers_array_classes_and_ignores_the_source() {
let mut heap = MiniHeap::new();
heap.class("[Lx.Y;", ROOT, &[]).class("[Lx/Y;", ROOT, &[]).class("Solo", ROOT, &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
let flagged: Vec<_> =
g.classes.iter().enumerate().filter(|(i, _)| m.is_ambiguous(*i as u32)).map(|(_, c)| c.name.as_str()).collect();
assert_eq!(flagged, ["[Lx.Y;", "[Lx.Y;"]);
}
/// A dump with three classes: `java.lang.Class` (three reference fields), `Statics` (two reference
/// statics, one null) and `Plain`; `at` is where each class object lives.
fn class_dump() -> Vec<u8> {
use BasicType::Object;
let mut w = HprofWriter::new();
w.string(1, "java/lang/Class").load_class(0x100, 1).class(
0x100,
0,
&[(10, Object), (11, Object), (12, Object)],
&[],
);
w.string(2, "Statics").load_class(0x200, 2).class(0x200, 0, &[], &[(13, 0x300), (13, 0)]);
w.string(3, "Plain").load_class(0x300, 3).class(0x300, 0, &[], &[]);
w.finish()
}
#[test]
fn class_objects_use_the_layout_even_when_the_histogram_names_them() {
let g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let histogram = [entry("java.lang.Class", 3, 3000)];
let by_histogram = model(&g, VmLayout::DEFAULT, Some(&histogram));
let by_layout = model(&g, VmLayout::DEFAULT, None);
assert_eq!(by_histogram.source, SizeSource::Jvm);
for m in [&by_histogram, &by_layout] {
// java.lang.Class has 12 bytes of fields: 24; Statics adds two references; every mirror
// gets the 32 bytes assumed for an unknown JVM (the histogram row does not fit).
assert_eq!(size_at(&g, m, 0x100), 56, "java.lang.Class itself");
assert_eq!(size_at(&g, m, 0x200), 64, "two reference statics");
assert_eq!(size_at(&g, m, 0x300), 56, "no statics");
}
// 8-byte references: 40 for java.lang.Class, 16 for the statics, 44 injected (JDK 21), aligned.
let wide = model(&g, flags("-XX:-UseCompressedOops", 21), None);
assert_eq!(size_at(&g, &wide, 0x200), 104);
}
#[test]
fn primitive_statics_add_to_the_class_object() {
let mut g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let statics = g.classes.iter().position(|c| c.name == "Statics").unwrap();
g.classes[statics].static_bytes = 12;
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x200), 80, "24 + 12 + 2 × 4 + 32 injected = 76, aligned");
}
#[test]
fn a_dump_without_java_lang_class_sizes_class_objects_as_empty_mirrors() {
let mut w = HprofWriter::new();
w.string(1, "Statics").load_class(0x200, 1).class(0x200, 0, &[], &[(2, 0), (2, 0), (2, 0)]);
let g = build_graph(&w.finish(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x200), 64, "16 + 3 × 4 + 32 injected = 60, aligned");
}
fn one_class_two_instances(stride: u64) -> (HeapGraph, ShallowModel) {
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", BasicType::Byte)]).object(0x1000, "A", &[]).object(0x1000 + stride, "A", &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
(g, m)
}
#[test]
fn address_gaps_for_external_dumps() {
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", BasicType::Byte)])
.class("Wide", ROOT, &[("f", BasicType::Long)])
.class("Module", ROOT, &[("f", BasicType::Long)])
.object(0x1000, "A", &[])
.object(0x1018, "A", &[])
.object(0x1030, "A", &[])
.object(0x1048, "Module", &[])
.object(0x2000, "Wide", &[])
.object(0x3000, "Wide", &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x1000), 24, "layout says 16, the smallest gap between As is 24");
assert_eq!(size_at(&g, &m, 0x1030), 24, "every A takes the type's size");
assert_eq!(size_at(&g, &m, 0x1048), 24, "a single instance has no gap to trust: layout");
assert_eq!(size_at(&g, &m, 0x2000), 24, "a gap beyond layout + 64 is not an object size: layout");
assert_eq!(m.source, SizeSource::Layout, "one of three types agrees");
}
#[test]
fn address_gaps_are_taken_within_layout_and_layout_plus_64() {
for (stride, expected) in [(8, 16), (16, 16), (24, 24), (80, 80), (88, 16)] {
let (g, m) = one_class_two_instances(stride);
assert_eq!(size_at(&g, &m, 0x1000), expected, "stride {stride}");
assert_eq!(
m.source,
if expected == stride && stride >= 16 { SizeSource::AddressGaps } else { SizeSource::Layout }
);
}
}
/// `used` types with two instances 16 bytes apart (the layout size), `far` types with two
/// instances 0x1000 apart (a gap that is no object size) and `single` types with one instance.
fn types_by_gap_use(used: usize, far: usize, single: usize) -> SizeSource {
let mut heap = MiniHeap::new();
for i in 0..used + far + single {
let name = format!("T{i}");
heap.class(&name, ROOT, &[("f", BasicType::Byte)]);
let base = 0x10_0000 * (i as u64 + 1);
heap.object(base, &name, &[]);
if i < used {
heap.object(base + 16, &name, &[]);
} else if i < used + far {
heap.object(base + 0x1000, &name, &[]);
}
}
let g = graph_of(&heap);
model(&g, VmLayout::DEFAULT, None).source
}
#[test]
fn address_gap_source_needs_ninety_percent_of_the_types_that_have_a_gap() {
assert_eq!(types_by_gap_use(10, 0, 0), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(9, 1, 0), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(8, 2, 0), SizeSource::Layout);
// Types with one instance have no gap: they count neither for nor against.
assert_eq!(types_by_gap_use(9, 1, 5), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(8, 2, 5), SizeSource::Layout);
assert_eq!(types_by_gap_use(10, 0, 400), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(0, 0, 3), SizeSource::Layout, "no type has a gap");
assert_eq!(types_by_gap_use(0, 2, 0), SizeSource::Layout);
assert_eq!(types_by_gap_use(0, 0, 0), SizeSource::Layout, "nothing to size");
}
/// A dump of `java.lang.Class` (three reference fields, so 24 bytes of layout) and one class per
/// entry of `statics`: `(reference statics, primitive static bytes)`. Class `i` is at `0x1000 + 0x100 × i`.
fn mirror_heap(statics: &[(usize, u64)]) -> (HeapGraph, Vec<u64>) {
use BasicType::Object;
let mut w = HprofWriter::new();
w.string(1, "java/lang/Class").load_class(0x100, 1).class(
0x100,
0,
&[(10, Object), (11, Object), (12, Object)],
&[],
);
let mut addrs = Vec::new();
for (i, &(refs, _)) in statics.iter().enumerate() {
let (addr, name_id) = (0x1000 + 0x100 * i as u64, 100 + i as u64);
w.string(name_id, &format!("C{i}")).load_class(addr, name_id).class(addr, 0, &[], &vec![(13, 0); refs]);
addrs.push(addr);
}
let mut g = build_graph(&w.finish(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
for (i, &(_, bytes)) in statics.iter().enumerate() {
let class = g.classes.iter().position(|c| c.name == format!("C{i}")).unwrap();
g.classes[class].static_bytes = bytes;
}
(g, addrs)
}
/// What HotSpot would make of those classes if it added `injected` bytes to every mirror.
fn expected_mirrors(statics: &[(usize, u64)], injected: u64) -> Vec<u64> {
statics.iter().map(|&(refs, bytes)| (24 + bytes + 4 * refs as u64 + injected).next_multiple_of(8)).collect()
}
const MIRROR_STATICS: [(usize, u64); 8] = [(0, 0), (3, 0), (1, 12), (6, 8), (2, 0), (9, 24), (0, 4), (4, 1)];
#[test]
fn class_objects_get_the_injected_bytes_the_histogram_implies() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// java.lang.Class itself and the eight classes.
let mut sizes = expected_mirrors(&MIRROR_STATICS, 36);
let java_lang_class = 24 + 36;
let total = sizes.iter().sum::<u64>() + (java_lang_class as u64).next_multiple_of(8);
sizes.insert(0, (java_lang_class as u64).next_multiple_of(8));
let histogram = [entry("java.lang.Class", 9, total)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(m.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &m, 0x100), sizes[0]);
for (i, &addr) in addrs.iter().enumerate() {
assert_eq!(size_at(&g, &m, addr), sizes[i + 1], "class {i}");
}
let spread: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert!(spread.iter().any(|&s| s != spread[0]), "statics still make the mirrors differ: {spread:?}");
}
#[test]
fn injected_bytes_are_found_for_every_multiple_of_four() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
for injected in (0..=128).step_by(4) {
let total =
expected_mirrors(&MIRROR_STATICS, injected).iter().sum::<u64>() + (24 + injected).next_multiple_of(8);
let histogram = [entry("java.lang.Class", 9, total)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
let got: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert_eq!(got, expected_mirrors(&MIRROR_STATICS, injected), "injected {injected}");
}
}
#[test]
fn the_fit_tolerates_mirrors_the_dump_has_no_class_for() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// The histogram counts the same average mirror 90 times more often (CDS-archived classes).
let total = expected_mirrors(&MIRROR_STATICS, 28).iter().sum::<u64>() + (24u64 + 28).next_multiple_of(8);
let histogram = [entry("java.lang.Class", 9 * 91, total * 91)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
let got: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert_eq!(got, expected_mirrors(&MIRROR_STATICS, 28));
}
fn mirror_sizes(g: &HeapGraph, addrs: &[u64], layout: VmLayout, histogram: Option<&[HistogramEntry]>) -> Vec<u64> {
let m = model(g, layout, histogram);
addrs.iter().map(|&a| size_at(g, &m, a)).collect()
}
#[test]
fn a_histogram_row_that_cannot_describe_these_mirrors_falls_back_to_the_table() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
let rows = [
entry("java.lang.Class", 9, 9 * 5000),
entry("java.lang.Class", 9, 9),
entry("java.lang.Class", 0, 0),
entry("Something.Else", 9, 9 * 100),
];
for row in &rows {
let one = std::slice::from_ref(row);
// Unknown JVM: the documented 32; JDK 11 with 4-byte references: 36.
assert_eq!(
mirror_sizes(&g, &addrs, VmLayout::DEFAULT, Some(one)),
expected_mirrors(&MIRROR_STATICS, 32),
"{row:?}"
);
assert_eq!(
mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(one)),
expected_mirrors(&MIRROR_STATICS, 36),
"{row:?}"
);
}
let twice = [entry("java.lang.Class", 9, 9 * 60), entry("java.lang.Class", 9, 9 * 60)];
assert_eq!(mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(&twice)), expected_mirrors(&MIRROR_STATICS, 36));
}
#[test]
fn an_accepted_fit_beats_the_table() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
let total = expected_mirrors(&MIRROR_STATICS, 20).iter().sum::<u64>() + 48;
let histogram = [entry("java.lang.Class", 9, total)];
// JDK 11's table says 36; the histogram says 20.
assert_eq!(mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(&histogram)), expected_mirrors(&MIRROR_STATICS, 20));
}
#[test]
fn injected_mirror_bytes_table() {
// (JDK, 4-byte references, 8-byte references): the measured cells, then the JDKs between.
for (major, narrow, wide) in
[(8, 36, 44), (11, 36, 44), (21, 36, 44), (22, 36, 44), (23, 36, 40), (24, 36, 40), (25, 28, 32), (26, 28, 32)]
{
assert_eq!(mirrors::injected_mirror_bytes(major, 4), Some(narrow), "JDK {major}, compressed oops");
assert_eq!(mirrors::injected_mirror_bytes(major, 8), Some(wide), "JDK {major}, 8-byte references");
}
assert_eq!(mirrors::injected_mirror_bytes(21, 2), None);
}
#[test]
fn without_a_histogram_class_objects_get_the_injected_bytes_of_their_jvm() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// The class dump of the fixture has 4-byte references; the layout under test decides the table
// cell, so use layouts whose reference size matches it.
for (layout, injected) in [
(flags(J8_11_1G, 8), 36),
(flags(J8_11_1G, 11), 36),
(flags(J21_25_1G, 21), 36),
(flags(J21_25_1G, 23), 36),
(flags(J21_25_1G, 25), 28),
(flags(J25_COMPACT, 25), 28),
] {
assert_eq!(mirror_sizes(&g, &addrs, layout, None), expected_mirrors(&MIRROR_STATICS, injected), "{layout:?}");
}
}
#[test]
fn eight_byte_references_use_their_own_cells() {
let g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
// java.lang.Class = 12 + 3 × 8 = 36 → 40; Plain has no statics: 40 + injected, aligned.
for (major, injected) in [(8, 44), (21, 44), (23, 40), (25, 32)] {
let m = model(&g, flags(J21_25_40G, major), None);
assert_eq!(size_at(&g, &m, 0x300), (40u64 + injected).next_multiple_of(8), "JDK {major}");
}
}
#[test]
fn an_unknown_jvm_uses_the_midpoint_of_the_compressed_oop_cells() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
assert_eq!(mirror_sizes(&g, &addrs, VmLayout::DEFAULT, None), expected_mirrors(&MIRROR_STATICS, 32));
// Flags without any -XX: option say nothing about the JVM either.
assert_eq!(mirror_sizes(&g, &addrs, flags("", 21), None), expected_mirrors(&MIRROR_STATICS, 32));
assert_eq!(VmLayout::DEFAULT.java_major, None);
}
#[test]
fn cancelled_token_stops_the_model() {
let g = graph_of(&sample());
let cancel = CancellationToken::new();
cancel.cancel();
assert!(matches!(ShallowModel::new(&g, VmLayout::DEFAULT, None, &cancel), Err(HeapError::Cancelled)));
let histogram = [entry("Node", 1, 24)];
assert!(matches!(ShallowModel::new(&g, VmLayout::DEFAULT, Some(&histogram), &cancel), Err(HeapError::Cancelled)));
}
+66
Переглянути файл
@@ -0,0 +1,66 @@
//! Text of `java.lang.String` objects.
use crate::hprof::{BasicType, HeapRecord};
use crate::index::{HeapIndex, id_value, instance_fields};
const STRING: &str = "java.lang.String";
const LATIN1: u8 = 0;
const UTF16: u8 = 1;
/// The text of String `object`, cut to `max_chars` chars plus `…`. `None` for anything that is
/// not a String, or one whose `value` / `coder` the dump does not hold as expected.
///
/// JDK 9+ keeps a `byte[]` and a `coder`; UTF-16 bytes are in the JVM's native order, which
/// hprof copies as is, read here as little-endian (x86-64, AArch64; a big-endian JVM's text would
/// come out garbled). JDK 8 keeps a `char[]`, whose elements hprof writes big-endian.
pub(crate) fn string_preview(index: &HeapIndex, object: u32, max_chars: usize) -> Option<String> {
let slot = *index.types.get(object as usize)?;
if index.class_info.get(slot as usize)?.name != STRING {
return None;
}
let Some(HeapRecord::Instance { data, .. }) = index.record(object) else { return None };
let fields = instance_fields(index, slot, data);
let value = fields.iter().find(|f| f.name == "value" && f.ty == BasicType::Object)?;
let array = index.ids.rank(id_value(value.bytes).filter(|&id| id != 0)?)?;
let coder = fields.iter().find(|f| f.name == "coder" && f.ty == BasicType::Byte).and_then(|f| f.bytes.first());
let Some(HeapRecord::PrimitiveArray { ty, data, .. }) = index.record(array) else { return None };
match (ty, coder) {
(BasicType::Byte, Some(&LATIN1)) => Some(cut(data.iter().map(|&b| char::from(b)), max_chars)),
(BasicType::Byte, Some(&UTF16)) => utf16(data, u16::from_le_bytes, max_chars),
(BasicType::Char, _) => utf16(data, u16::from_be_bytes, max_chars),
_ => None,
}
}
/// The text of `char[]` `object`, as [`string_preview`] cuts it.
pub(crate) fn char_array_text(index: &HeapIndex, object: u32, max_chars: usize) -> Option<String> {
match index.record(object)? {
HeapRecord::PrimitiveArray { ty: BasicType::Char, data, .. } => utf16(data, u16::from_be_bytes, max_chars),
_ => None,
}
}
/// An odd byte count is `None`; a lone surrogate becomes U+FFFD.
fn utf16(bytes: &[u8], unit: fn([u8; 2]) -> u16, max_chars: usize) -> Option<String> {
if !bytes.len().is_multiple_of(2) {
return None;
}
let units = bytes.as_chunks::<2>().0.iter().map(|&pair| unit(pair));
Some(cut(char::decode_utf16(units).map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER)), max_chars))
}
/// Chars are whole code points, so a cut never splits a surrogate pair.
fn cut(chars: impl Iterator<Item = char>, max_chars: usize) -> String {
let mut out = String::new();
for (i, c) in chars.enumerate() {
if i == max_chars {
out.push('…');
break;
}
out.push(c);
}
out
}
#[cfg(test)]
mod tests;
+113
Переглянути файл
@@ -0,0 +1,113 @@
use super::*;
use crate::hprof::{BasicType, HeapRootKind};
use crate::test_index::{analyzed, obj};
use crate::testing::{MiniHeap, sample_with_string};
const AT: u64 = 0x1000;
const ARRAY: u64 = 0x2000;
const JDK9: [(&str, BasicType); 2] = [("value", BasicType::Object), ("coder", BasicType::Byte)];
fn preview(heap: &MiniHeap, addr: u64) -> Option<String> {
let index = analyzed(heap);
string_preview(&index, obj(&index, addr).0, 100)
}
fn text(text: &str, latin1: bool) -> Option<String> {
let mut heap = MiniHeap::new();
heap.string(AT, text, latin1).root(HeapRootKind::JniGlobal, AT);
preview(&heap, AT)
}
/// A `java.lang.String` class with `fields`, and one String at [`AT`] with `values`.
fn custom(fields: &[(&str, BasicType)], values: &[(&str, u64)], array: impl FnOnce(&mut MiniHeap)) -> Option<String> {
let mut heap = MiniHeap::new();
heap.class("java.lang.String", Some("java.lang.Object"), fields);
array(&mut heap);
heap.object(AT, "java.lang.String", values);
preview(&heap, AT)
}
#[test]
fn string_preview_latin1_and_utf16() {
assert_eq!(preview(&sample_with_string(), 0x1800).as_deref(), Some("v42"));
assert_eq!(text("hé", false).as_deref(), Some("hé"));
assert_eq!(text("ÿ é", true).as_deref(), Some("ÿ é"));
assert_eq!(text("", true).as_deref(), Some(""));
let long = "a".repeat(150);
assert_eq!(text(&long, true), Some(format!("{}…", "a".repeat(100))));
assert_eq!(text(&long, false), Some(format!("{}…", "a".repeat(100))));
let exactly = "b".repeat(100);
assert_eq!(text(&exactly, true), Some(exactly));
}
#[test]
fn truncation_keeps_surrogate_pairs_whole() {
let s = format!("{}😀{}", "a".repeat(99), "b".repeat(50));
assert_eq!(text(&s, false), Some(format!("{}😀…", "a".repeat(99))));
// A lone surrogate becomes the replacement character.
let lone: Vec<u8> = [0x68u16, 0xD800, 0x69].iter().flat_map(|u| u.to_le_bytes()).collect();
let got = custom(&JDK9, &[("value", ARRAY), ("coder", 1)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, &lone);
});
assert_eq!(got.as_deref(), Some("h\u{FFFD}i"));
}
#[test]
fn jdk8_strings_hold_big_endian_chars() {
let chars: Vec<u8> = "hé".encode_utf16().flat_map(u16::to_be_bytes).collect();
let fields = [("value", BasicType::Object), ("hash", BasicType::Int)];
let got = custom(&fields, &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Char, &chars);
});
assert_eq!(got.as_deref(), Some("hé"));
}
#[test]
fn malformed_strings_have_no_preview() {
let odd = custom(&JDK9, &[("value", ARRAY), ("coder", 1)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, &[0x68, 0, 0x69]);
});
assert_eq!(odd, None);
let bad_coder = custom(&JDK9, &[("value", ARRAY), ("coder", 2)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(bad_coder, None);
let no_value = custom(&[("coder", BasicType::Byte)], &[], |_| {});
assert_eq!(no_value, None);
let no_coder = custom(&[("value", BasicType::Object)], &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(no_coder, None);
let null_value = custom(&JDK9, &[], |_| {});
assert_eq!(null_value, None);
let int_array = custom(&JDK9, &[("value", ARRAY)], |h| {
h.prim(ARRAY, BasicType::Int, 2);
});
assert_eq!(int_array, None);
let not_an_array = custom(&JDK9, &[("value", ARRAY)], |h| {
h.object(ARRAY, "java.lang.Object", &[]);
});
assert_eq!(not_an_array, None);
// `value` stored as a long instead of a reference.
let wrong_type = custom(&[("value", BasicType::Long), ("coder", BasicType::Byte)], &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(wrong_type, None);
}
#[test]
fn only_strings_have_a_preview() {
let mut heap = MiniHeap::new();
heap.class("Text", Some("java.lang.Object"), &JDK9).prim_bytes(ARRAY, BasicType::Byte, b"hi").object(
AT,
"Text",
&[("value", ARRAY)],
);
assert_eq!(preview(&heap, AT), None);
assert_eq!(preview(&heap, ARRAY), None);
let index = analyzed(&heap);
for bad in [index.root(), u32::MAX] {
assert_eq!(string_preview(&index, bad, 100), None);
}
}
+35
Переглянути файл
@@ -0,0 +1,35 @@
//! Analyzed fixtures for the query tests.
use ide_core::CancellationToken;
use crate::analysis::{AnalyzeOptions, analyze};
use crate::error::HeapError;
use crate::index::{HeapIndex, HeapObjectRef};
use crate::progress::HeapPhase;
use crate::testing::MiniHeap;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
/// `heap` analyzed in memory, with layout sizes.
pub(crate) fn analyzed(heap: &MiniHeap) -> HeapIndex {
analyzed_bytes(&heap.bytes())
}
pub(crate) fn analyzed_bytes(hprof: &[u8]) -> HeapIndex {
let dir = tempfile::tempdir().unwrap();
let dump = dir.path().join("dump.hprof");
std::fs::write(&dump, hprof).unwrap();
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
analyze(&dump, dir.path(), opts, &no_progress, &CancellationToken::new()).unwrap()
}
pub(crate) fn obj(index: &HeapIndex, addr: u64) -> HeapObjectRef {
HeapObjectRef(index.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the dump")))
}
/// The slot of the class named `name`.
pub(crate) fn slot(index: &HeapIndex, name: &str) -> u32 {
index.classes.iter().position(|c| c.name == name).unwrap_or_else(|| panic!("no class {name}")) as u32
}
+430
Переглянути файл
@@ -0,0 +1,430 @@
//! Builders for `.hprof` fixtures. They panic on misuse (unknown class name, text that does not
//! fit the requested encoding): they are test helpers, not input handling.
use std::collections::HashMap;
use std::path::Path;
use crate::hprof::{BasicType, HeapRootKind};
const TAG_UTF8: u8 = 0x01;
const TAG_LOAD_CLASS: u8 = 0x02;
const TAG_HEAP_SEGMENT: u8 = 0x1C;
const TAG_HEAP_DUMP_END: u8 = 0x2C;
const TIMESTAMP_MS: u64 = 1_700_000_000_000;
fn put_id(out: &mut Vec<u8>, id_size: u8, id: u64) {
if id_size == 4 {
out.extend_from_slice(&(id as u32).to_be_bytes());
} else {
out.extend_from_slice(&id.to_be_bytes());
}
}
/// Writes records one by one; heap sub-records are collected into a single heap-dump segment that
/// `finish` places after the UTF8 / LOAD CLASS records, followed by HEAP DUMP END.
pub struct HprofWriter {
id_size: u8,
top: Vec<u8>,
heap: Vec<u8>,
class_serial: u32,
}
impl Default for HprofWriter {
fn default() -> Self {
Self::new()
}
}
impl HprofWriter {
pub fn new() -> Self {
Self::with_id_size(8)
}
/// `id_size` is 4 or 8; ids are truncated to the width.
pub fn with_id_size(id_size: u8) -> Self {
assert!(id_size == 4 || id_size == 8, "hprof ids are 4 or 8 bytes");
let mut top = b"JAVA PROFILE 1.0.2\0".to_vec();
top.extend_from_slice(&u32::from(id_size).to_be_bytes());
top.extend_from_slice(&TIMESTAMP_MS.to_be_bytes());
Self { id_size, top, heap: Vec::new(), class_serial: 0 }
}
fn top_record(&mut self, tag: u8, body: &[u8]) {
self.top.push(tag);
self.top.extend_from_slice(&0u32.to_be_bytes());
self.top.extend_from_slice(&(body.len() as u32).to_be_bytes());
self.top.extend_from_slice(body);
}
fn heap_id(&mut self, id: u64) {
put_id(&mut self.heap, self.id_size, id);
}
fn heap_u4(&mut self, v: u32) {
self.heap.extend_from_slice(&v.to_be_bytes());
}
pub fn string(&mut self, id: u64, s: &str) -> &mut Self {
let mut body = Vec::new();
put_id(&mut body, self.id_size, id);
body.extend_from_slice(s.as_bytes());
self.top_record(TAG_UTF8, &body);
self
}
pub fn load_class(&mut self, class_id: u64, name_id: u64) -> &mut Self {
self.class_serial += 1;
let mut body = self.class_serial.to_be_bytes().to_vec();
put_id(&mut body, self.id_size, class_id);
body.extend_from_slice(&0u32.to_be_bytes());
put_id(&mut body, self.id_size, name_id);
self.top_record(TAG_LOAD_CLASS, &body);
self
}
/// `statics` are `(name id, object id)` reference statics.
pub fn class(&mut self, id: u64, super_id: u64, fields: &[(u64, BasicType)], statics: &[(u64, u64)]) -> &mut Self {
self.class_with_loader(id, super_id, 0, fields, statics)
}
pub fn class_with_loader(
&mut self,
id: u64,
super_id: u64,
loader_id: u64,
fields: &[(u64, BasicType)],
statics: &[(u64, u64)],
) -> &mut Self {
self.heap.push(0x20);
self.heap_id(id);
self.heap_u4(0);
self.heap_id(super_id);
self.heap_id(loader_id);
for _ in 0..4 {
self.heap_id(0); // signers, protection domain, 2 reserved
}
let instance_size: usize = fields.iter().map(|(_, ty)| ty.hprof_size(self.id_size)).sum();
self.heap_u4(instance_size as u32);
self.heap.extend_from_slice(&0u16.to_be_bytes()); // constant pool
self.heap.extend_from_slice(&(statics.len() as u16).to_be_bytes());
for &(name, object) in statics {
self.heap_id(name);
self.heap.push(BasicType::Object as u8);
self.heap_id(object);
}
self.heap.extend_from_slice(&(fields.len() as u16).to_be_bytes());
for &(name, ty) in fields {
self.heap_id(name);
self.heap.push(ty as u8);
}
self
}
pub fn instance(&mut self, id: u64, class_id: u64, data: &[u8]) -> &mut Self {
self.heap.push(0x21);
self.heap_id(id);
self.heap_u4(0);
self.heap_id(class_id);
self.heap_u4(data.len() as u32);
self.heap.extend_from_slice(data);
self
}
pub fn obj_array(&mut self, id: u64, class_id: u64, items: &[u64]) -> &mut Self {
self.heap.push(0x22);
self.heap_id(id);
self.heap_u4(0);
self.heap_u4(items.len() as u32);
self.heap_id(class_id);
for &item in items {
self.heap_id(item);
}
self
}
/// An array of `len` zeroed elements.
pub fn prim_array(&mut self, id: u64, ty: BasicType, len: u32) -> &mut Self {
let bytes = vec![0u8; len as usize * ty.hprof_size(self.id_size)];
self.prim_bytes(id, ty, &bytes)
}
/// An array whose elements are `bytes` (big-endian, as stored in the dump).
pub fn prim_bytes(&mut self, id: u64, ty: BasicType, bytes: &[u8]) -> &mut Self {
let size = ty.hprof_size(self.id_size);
assert!(ty != BasicType::Object && bytes.len().is_multiple_of(size), "{ty:?} array needs whole elements");
self.heap.push(0x23);
self.heap_id(id);
self.heap_u4(0);
self.heap_u4((bytes.len() / size) as u32);
self.heap.push(ty as u8);
self.heap.extend_from_slice(bytes);
self
}
/// Kinds that carry a thread serial get serial 1; see [`Self::root_in_thread`].
pub fn root(&mut self, kind: HeapRootKind, id: u64) -> &mut Self {
self.root_in_thread(kind, id, 1)
}
/// `serial` is ignored for kinds that do not carry one.
pub fn root_in_thread(&mut self, kind: HeapRootKind, id: u64, serial: u32) -> &mut Self {
let (tag, has_serial, trailing) = match kind {
HeapRootKind::Unknown => (0xFF, false, Trailing::None),
HeapRootKind::JniGlobal => (0x01, false, Trailing::Id),
HeapRootKind::JniLocal => (0x02, true, Trailing::U4),
HeapRootKind::JavaFrame => (0x03, true, Trailing::U4),
HeapRootKind::NativeStack => (0x04, true, Trailing::None),
HeapRootKind::StickyClass => (0x05, false, Trailing::None),
HeapRootKind::ThreadBlock => (0x06, true, Trailing::None),
HeapRootKind::MonitorUsed => (0x07, false, Trailing::None),
HeapRootKind::ThreadObject => (0x08, true, Trailing::U4),
};
self.heap.push(tag);
self.heap_id(id);
if has_serial {
self.heap_u4(serial);
}
match trailing {
Trailing::None => {}
Trailing::U4 => self.heap_u4(0),
Trailing::Id => self.heap_id(0),
}
self
}
pub fn finish(&self) -> Vec<u8> {
let mut out = self.top.clone();
out.push(TAG_HEAP_SEGMENT);
out.extend_from_slice(&0u32.to_be_bytes());
let heap_len = u32::try_from(self.heap.len()).expect("fixture heap segment exceeds 4 GiB");
out.extend_from_slice(&heap_len.to_be_bytes());
out.extend_from_slice(&self.heap);
out.push(TAG_HEAP_DUMP_END);
out.extend_from_slice(&0u32.to_be_bytes());
out.extend_from_slice(&0u32.to_be_bytes());
out
}
}
enum Trailing {
None,
U4,
Id,
}
struct MiniClass {
id: u64,
super_name: Option<String>,
fields: Vec<(String, BasicType)>,
}
/// Ids for classes, names and string backing arrays come from here, above any address a test
/// would pick by hand.
const GENERATED_ID_BASE: u64 = 1 << 32;
/// Heap fixtures by class and field name on top of [`HprofWriter`] (id size 8).
///
/// Class names are written in the HotSpot form (`.` becomes `/`); look-ups use the name as given.
pub struct MiniHeap {
w: HprofWriter,
next_id: u64,
classes: HashMap<String, MiniClass>,
}
impl Default for MiniHeap {
fn default() -> Self {
Self::new()
}
}
impl MiniHeap {
/// Starts with `java.lang.Object` and `java.lang.Class` already defined.
pub fn new() -> Self {
let mut heap = Self { w: HprofWriter::new(), next_id: GENERATED_ID_BASE, classes: HashMap::new() };
heap.class("java.lang.Object", None, &[]);
heap.class("java.lang.Class", Some("java.lang.Object"), &[]);
heap
}
fn fresh_id(&mut self) -> u64 {
self.next_id += 1;
self.next_id
}
fn class_id(&self, name: &str) -> u64 {
self.classes.get(name).unwrap_or_else(|| panic!("MiniHeap: class {name} is not defined")).id
}
/// `super_name: None` writes super id 0.
pub fn class(&mut self, name: &str, super_name: Option<&str>, fields: &[(&str, BasicType)]) -> &mut Self {
let super_id = super_name.map_or(0, |sup| self.class_id(sup));
let id = self.fresh_id();
let name_id = self.fresh_id();
let mut field_ids = Vec::with_capacity(fields.len());
for &(field, ty) in fields {
let field_id = self.fresh_id();
self.w.string(field_id, field);
field_ids.push((field_id, ty));
}
self.w.string(name_id, &name.replace('.', "/"));
self.w.load_class(id, name_id);
self.w.class(id, super_id, &field_ids, &[]);
self.w.root(HeapRootKind::StickyClass, id);
let fields = fields.iter().map(|&(field, ty)| (field.to_owned(), ty)).collect();
self.classes.insert(name.to_owned(), MiniClass { id, super_name: super_name.map(str::to_owned), fields });
self
}
/// Fields are laid out class first, then superclass, as HotSpot does. A field missing from
/// `values` is 0; narrower types keep the low bytes of the value.
pub fn object(&mut self, addr: u64, class: &str, values: &[(&str, u64)]) -> &mut Self {
let class_id = self.class_id(class);
let mut data = Vec::new();
let mut cursor = Some(class);
while let Some(name) = cursor {
let c = &self.classes[name];
for (field, ty) in &c.fields {
let value = values.iter().find(|(n, _)| *n == field.as_str()).map_or(0, |&(_, v)| v);
let width = ty.hprof_size(8);
data.extend_from_slice(&value.to_be_bytes()[8 - width..]);
}
cursor = c.super_name.as_deref();
}
self.w.instance(addr, class_id, &data);
self
}
pub fn array_of(&mut self, addr: u64, class: &str, items: &[u64]) -> &mut Self {
let class_id = self.class_id(class);
self.w.obj_array(addr, class_id, items);
self
}
/// A primitive array of `len` zeroed elements.
pub fn prim(&mut self, addr: u64, ty: BasicType, len: u32) -> &mut Self {
self.w.prim_array(addr, ty, len);
self
}
/// A primitive array holding `bytes` (big-endian elements).
pub fn prim_bytes(&mut self, addr: u64, ty: BasicType, bytes: &[u8]) -> &mut Self {
self.w.prim_bytes(addr, ty, bytes);
self
}
/// A `java.lang.String` (JDK 9+ layout: `value` byte[] + `coder`) at `addr`. `latin1` stores
/// one byte per char (coder 0, every char must be ≤ U+00FF); otherwise UTF-16LE (coder 1).
pub fn string(&mut self, addr: u64, text: &str, latin1: bool) -> &mut Self {
if !self.classes.contains_key("java.lang.String") {
let fields = [("value", BasicType::Object), ("coder", BasicType::Byte)];
self.class("java.lang.String", Some("java.lang.Object"), &fields);
}
let bytes: Vec<u8> = if latin1 {
text.chars()
.map(|c| u8::try_from(u32::from(c)).unwrap_or_else(|_| panic!("{c:?} is not Latin-1")))
.collect()
} else {
text.encode_utf16().flat_map(u16::to_le_bytes).collect()
};
let array = self.fresh_id();
self.w.prim_bytes(array, BasicType::Byte, &bytes);
self.object(addr, "java.lang.String", &[("value", array), ("coder", u64::from(!latin1))])
}
/// Kinds that carry a thread serial get serial 1.
pub fn root(&mut self, kind: HeapRootKind, addr: u64) -> &mut Self {
self.w.root(kind, addr);
self
}
pub fn root_in_thread(&mut self, kind: HeapRootKind, addr: u64, serial: u32) -> &mut Self {
self.w.root_in_thread(kind, addr, serial);
self
}
pub fn bytes(&self) -> Vec<u8> {
self.w.finish()
}
pub fn write_to(&self, path: &Path) {
std::fs::write(path, self.bytes()).unwrap_or_else(|e| panic!("MiniHeap: writing {}: {e}", path.display()));
}
}
/// The heap most analysis tests share:
///
/// - `Holder { a: Object, b: Object }`, `Node { next: Object, v: long }`,
/// `java.lang.ref.Reference { referent: Object, queue: Object }`,
/// `java.lang.ref.WeakReference extends Reference`, `[Ljava.lang.Object;`
/// - H `0x1000` Holder { a: N1, b: N2 }; N1 `0x1100` Node { next: N3 }; N2 `0x1200` Node { next: N3 };
/// N3 `0x1300` Node { next: L }; L `0x1400` long[2]; R `0x1500` Object[] { N1, null, null };
/// W `0x1600` WeakReference { referent: N4 }; N4 `0x1700` Node {}
/// - roots: JNI global H, Java frame N3 (thread serial 1), JNI global W
///
/// R and N4 are unreachable (N4 only through the weak referent).
pub fn sample() -> MiniHeap {
let mut heap = MiniHeap::new();
heap.class("Holder", Some("java.lang.Object"), &[("a", BasicType::Object), ("b", BasicType::Object)])
.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("v", BasicType::Long)])
.class(
"java.lang.ref.Reference",
Some("java.lang.Object"),
&[("referent", BasicType::Object), ("queue", BasicType::Object)],
)
.class("java.lang.ref.WeakReference", Some("java.lang.ref.Reference"), &[])
.class("[Ljava.lang.Object;", Some("java.lang.Object"), &[])
.object(0x1000, "Holder", &[("a", 0x1100), ("b", 0x1200)])
.object(0x1100, "Node", &[("next", 0x1300)])
.object(0x1200, "Node", &[("next", 0x1300)])
.object(0x1300, "Node", &[("next", 0x1400)])
.prim(0x1400, BasicType::Long, 2)
.array_of(0x1500, "[Ljava.lang.Object;", &[0x1100, 0, 0])
.object(0x1600, "java.lang.ref.WeakReference", &[("referent", 0x1700)])
.object(0x1700, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1000)
.root(HeapRootKind::JavaFrame, 0x1300)
.root(HeapRootKind::JniGlobal, 0x1600);
heap
}
/// [`sample`] plus S `0x1800` String `"v42"` (Latin-1) held as `a` of a second Holder `0x1900`,
/// rooted as a JNI global. That Holder retains 72 bytes, as much as H.
pub fn sample_with_string() -> MiniHeap {
let mut heap = sample();
heap.string(0x1800, "v42", true).object(0x1900, "Holder", &[("a", 0x1800)]).root(HeapRootKind::JniGlobal, 0x1900);
heap
}
/// [`sample`] plus two Nodes `0x1a00` and `0x1b00`, each rooted as a JNI global.
pub fn sample_with_two_more_nodes() -> MiniHeap {
let mut heap = sample();
heap.object(0x1a00, "Node", &[])
.object(0x1b00, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1a00)
.root(HeapRootKind::JniGlobal, 0x1b00);
heap
}
/// Seeded xorshift64 numbers for reproducible random test data (the workspace has no `rand`).
pub struct XorShift(u64);
impl XorShift {
pub fn new(seed: u64) -> Self {
// Zero is xorshift's one fixed point.
Self(seed.max(1))
}
pub fn next_u64(&mut self) -> u64 {
let mut x = self.0;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
self.0 = x;
x
}
/// A number in `0..bound`; `bound` must not be 0.
pub fn below(&mut self, bound: u32) -> u32 {
(self.next_u64() % u64::from(bound)) as u32
}
}
+3
Переглянути файл
@@ -27,6 +27,8 @@ ide-search.workspace = true
ide-lsp.workspace = true
ide-vcs.workspace = true
ide-dap.workspace = true
ide-heap.workspace = true
ide-jvm-attach.workspace = true
ide-build.workspace = true
ide-analysis.workspace = true
ide-coverage.workspace = true
@@ -49,6 +51,7 @@ ide-notebook.workspace = true
ide-extension.workspace = true
ide-plugin.workspace = true
ide-tools = { path = "../ide-tools" }
ide-scaffold.workspace = true
ide-docker = { path = "../ide-docker" }
semver = "1"
serde.workspace = true
+5
Переглянути файл
@@ -653,6 +653,11 @@ fn cmd_index(args: &[String]) -> Result<(), String> {
engine: package::engine_requirement(&m.engine, built_for.as_ref()),
categories: m.categories(),
icon: plugin_icon(&ide_extension::Package::new(m.clone(), scratch.clone()))?,
homepage: m.homepage.clone().filter(|h| !h.trim().is_empty()),
notes: m.change_notes.clone().filter(|n| !n.trim().is_empty()),
downloads: None,
featured: false,
suggest: Vec::new(),
})
})();
let entry = entry.map_err(|e| format!("{}: {e}", path.display()))?;
+14 -1
Переглянути файл
@@ -73,6 +73,11 @@ pub struct Host {
pub processes: ProcessManager,
/// Debug sessions (adapters run on this machine, next to the project).
pub debugger: Arc<ide_dap::DebugManager>,
/// Heap snapshots open in any window, captured from a debug session or opened from a file.
pub heap: Arc<ide_heap::HeapRegistry>,
/// The window that owns each open heap snapshot; its project's close closes them.
pub(crate) heap_windows: Mutex<HashMap<ide_heap::HeapSnapshotId, String>>,
pub(crate) heap_requests: ops::heap::HeapRequests,
/// SSH connections of Settings › SSH Configurations.
pub ssh: Arc<ide_ssh::SshManager>,
/// SFTP deployment, remote file editing and runs on run targets.
@@ -143,6 +148,9 @@ impl Host {
terminals: TerminalManager::new(),
processes: ProcessManager::new(),
debugger: ide_dap::DebugManager::new(languages.clone()),
heap: Arc::new(ide_heap::HeapRegistry::new(ide_heap::snapshots_root(&local_data_dir))),
heap_windows: Mutex::default(),
heap_requests: Default::default(),
languages,
syntax,
config_dir,
@@ -255,6 +263,8 @@ impl Host {
for request in self.project_requests.write().remove(self.owner_window(window)).unwrap_or_default() {
self.cancel_request(request);
}
// After the cancels: a capture or open still running then discards what it registers.
ops::heap::close_window_snapshots(self, window);
let project = self.projects.write().remove(window)?;
if self.windows_for_root(&project.root).is_empty()
&& let Some(plugins) = self.plugin_host()
@@ -405,9 +415,12 @@ impl Host {
}
}
/// Stops what this host started: notebook kernels, language servers, agents, runs and terminals.
/// Stops what this host started: heap snapshots, notebook kernels, language servers, agents,
/// runs and terminals.
/// For a process that exits without dropping the host (the remote server).
pub async fn shutdown(&self) {
// A capture's dump may hold secrets from the debuggee's memory: delete it before exiting.
ops::heap::shutdown(self).await;
self.environment.shutdown();
self.processes.kill_all();
self.terminals.kill_all();
+11 -8
Переглянути файл
@@ -181,15 +181,18 @@ pub fn entries(host: &Host, window: &str, task: String, subagent: Option<String>
host.project(window)?.agents.entries(&task, subagent)
}
/// Sessions on this machine that no task holds yet, newest first (#312).
pub fn external_sessions(host: &Host, window: &str) -> IdeResult<Vec<ExternalSession>> {
Ok(host.project(window)?.agents.external_sessions())
/// Sessions of the project on this machine that no task holds yet, newest first. Reads
/// transcripts from disk on the blocking pool.
pub async fn external_sessions(host: &Host, window: &str) -> IdeResult<Vec<ExternalSession>> {
let agents = host.project(window)?.agents.clone();
tokio::task::spawn_blocking(move || agents.external_sessions()).await.map_err(join_err)
}
/// Imports an external session as a task: its first turn loads and replays it (#312).
pub fn import_session(host: &Host, window: &str, provider: String, session_id: String) -> IdeResult<AgentTask> {
/// Imports an external session as a task: its first turn loads and replays it.
pub async fn import_session(host: &Host, window: &str, provider: String, session_id: String) -> IdeResult<AgentTask> {
host.require_trust(window)?;
host.project(window)?.agents.import_session(&provider, &session_id)
let agents = host.project(window)?.agents.clone();
tokio::task::spawn_blocking(move || agents.import_session(&provider, &session_id)).await.map_err(join_err)?
}
pub async fn create(host: &Host, window: &str, mut request: NewAgentTask) -> IdeResult<AgentTask> {
@@ -409,9 +412,9 @@ pub(crate) fn register(d: &mut Dispatcher) {
d.add("agent_commands", |c| async move { commands(&c.host, &c.window, c.arg("provider")?, c.arg("task")?).await });
d.add("agent_entries", |c| async move { entries(&c.host, &c.window, c.arg("task")?, c.arg("subagent")?) });
d.add("agent_create", |c| async move { create(&c.host, &c.window, c.arg("request")?).await });
d.add("agent_external_sessions", |c| async move { external_sessions(&c.host, &c.window) });
d.add("agent_external_sessions", |c| async move { external_sessions(&c.host, &c.window).await });
d.add("agent_import_session", |c| async move {
import_session(&c.host, &c.window, c.arg("provider")?, c.arg("sessionId")?)
import_session(&c.host, &c.window, c.arg("provider")?, c.arg("sessionId")?).await
});
d.add("agent_prompt", |c| async move {
prompt(&c.host, &c.window, c.arg("task")?, c.arg("prompt")?, c.arg("now")?).await
+78
Переглянути файл
@@ -538,6 +538,35 @@ pub async fn build_run(
let config_dir = host.config_dir.clone();
let token = host.begin_project_request(window, request);
let result = tokio::task::spawn_blocking(move || {
// Fast path (#2476): an unchanged single-project Maven build skips
// Maven entirely. Previous diagnostics stay: nothing changed, so they
// are still accurate.
let single = cmake.is_none() && ide_workspace::workspace::Workspace::load(&root)?.is_none();
if single && kind == BuildKind::Build {
let model = with_profiles(load(&root, &cache, &settings), &cache);
if compile::menu_up_to_date(&cache, &root, &model, file.as_deref(), &token) {
let started = std::time::Instant::now();
let plan = prepare_plan(
compile::plan(&root, &model, kind, file.as_deref(), settings.cmake.profiles.as_deref())?,
&root,
&settings,
);
let _ = on_event.send(BuildEvent::Started { system: plan.system, title: plan.title.clone() });
let _ = on_event.send(BuildEvent::Output {
text: "Up to date: no changes, build skipped.\n".into(),
stderr: false,
});
return Ok(BuildResult {
system: plan.system,
title: plan.title,
success: true,
errors: 0,
warnings: 0,
duration_ms: started.elapsed().as_secs_f64() * 1000.0,
messages: Vec::new(),
});
}
}
for path in clear {
sink.emit(IdeEvent::Diagnostics {
path: path.into(),
@@ -584,6 +613,51 @@ pub async fn build_run(
let result = compile::run(&root, &plan, &token, &|e| {
let _ = on_event.send(e);
})?;
if result.success && single && kind == BuildKind::Build {
let model = with_profiles(load(&root, &cache, &settings), &cache);
compile::record_menu_build(&cache, &root, &model, file.as_deref(), &token);
}
for (path, items) in compile::diagnostics(&result.messages) {
sink.emit(IdeEvent::Diagnostics { path: path.into(), source: compile::DIAGNOSTICS_SOURCE.into(), items });
}
Ok(result)
})
.await
.map_err(join_err);
host.end_request(request);
result?
}
/// Build › Compile File: compiles one Java source of a linked Maven module with
/// `javac`, streaming the console and messages on `on_event`; cancellable with
/// `cancel_request(request)`. Diagnostics of `clear` (the files of the previous
/// build) are cleared first; the new messages are published as `build`
/// diagnostics.
pub async fn build_compile_file(
host: &Host,
window: &str,
file: std::path::PathBuf,
request: RequestId,
clear: Vec<std::path::PathBuf>,
on_event: Channel<BuildEvent>,
) -> IdeResult<BuildResult> {
host.require_trust(window)?;
let project = host.project(window)?;
let (root, cache, sink) = (project.root.clone(), project.build_cache_dir(), project.sink.clone());
let settings = project.settings.effective();
let token = host.begin_project_request(window, request);
let result = tokio::task::spawn_blocking(move || {
for path in clear {
sink.emit(IdeEvent::Diagnostics {
path: path.into(),
source: compile::DIAGNOSTICS_SOURCE.into(),
items: Vec::new(),
});
}
let plan = compile::compile_file_plan(&root, &with_profiles(load(&root, &cache, &settings), &cache), &file)?;
let result = compile::run(&root, &plan, &token, &|e| {
let _ = on_event.send(e);
})?;
for (path, items) in compile::diagnostics(&result.messages) {
sink.emit(IdeEvent::Diagnostics { path: path.into(), source: compile::DIAGNOSTICS_SOURCE.into(), items });
}
@@ -653,6 +727,10 @@ pub(crate) fn register(d: &mut Dispatcher) {
)
.await
});
d.add("build_compile_file", |c| async move {
build_compile_file(&c.host, &c.window, c.arg("file")?, c.arg("request")?, c.arg("clear")?, c.sender("onEvent")?)
.await
});
}
#[cfg(test)]
+166 -1
Переглянути файл
@@ -9,7 +9,7 @@ use std::path::PathBuf;
/// Starts a session owned by this window: its events (and those of child sessions) go here only.
pub async fn debug_start(host: &Host, window: &str, config: LaunchConfig) -> IdeResult<DebugSessionId> {
host.require_trust(window)?;
let mut config = match_java_debug_encoding(config, cfg!(windows));
let mut config = match_java_debug_encoding(start_java_attach_listener(config, cfg!(windows)), cfg!(windows));
if config.adapter == "debugpy"
&& config.request == "launch"
&& let Ok(project) = host.project(window)
@@ -93,11 +93,67 @@ pub async fn debug_start(host: &Host, window: &str, config: LaunchConfig) -> Ide
Some(path.to_string_lossy().into_owned())
};
host.require_trust(window)?;
let server_port = debug_server_port(&config);
let result = host.debugger.start_with(config, owner, custom).await;
flush_output();
if let (Ok(id), Some(port)) = (&result, server_port) {
attach_jvm_inspector(host, window, *id, port);
}
result
}
/// The port of the java-debug server this machine's jdtls started; `debugServer` as `host:port`
/// names an adapter on another machine, whose jdtls is not ours to ask.
fn debug_server_port(config: &LaunchConfig) -> Option<u16> {
if config.adapter != "java-debug" {
return None;
}
let extra: serde_json::Value = serde_json::from_str(config.extra_json.as_deref()?).ok()?;
extra.get("debugServer")?.as_u64().and_then(|port| u16::try_from(port).ok()).filter(|port| *port != 0)
}
/// Lets the session ask the jdtls that runs it about its JVM. Without a module (no jdtls, or a
/// debug server it did not start) the session keeps the debugger-only memory view.
fn attach_jvm_inspector(host: &Host, window: &str, session: DebugSessionId, port: u16) {
let Ok(project) = host.project(window) else { return };
let Some(module) = project.lsp.java_debug_server_module(port) else { return };
let inspector = std::sync::Arc::new(ide_lsp::JdtJvmInspector::new(project.lsp.clone(), module));
if let Err(e) = host.debugger.set_jvm_inspector(session, inspector) {
tracing::debug!("jvm inspector for session {session:?}: {e}");
}
}
/// The JVM memory tab reads the class table over the attach socket, which only exists once the
/// JVM has started its attach listener. The flag goes first so the user's arguments still win.
/// Not on Windows (`windows`: the JVM runs there), which the memory view does not support.
fn start_java_attach_listener(mut config: LaunchConfig, windows: bool) -> LaunchConfig {
if windows || config.adapter != "java-debug" || config.request != "launch" {
return config;
}
let Some(serde_json::Value::Object(mut extra)) =
config.extra_json.as_deref().and_then(|e| serde_json::from_str(e).ok())
else {
return config;
};
// java-debug also takes an array; rewriting one would drop the user's arguments.
let vm_args = match extra.get("vmArgs") {
None | Some(serde_json::Value::Null) => "",
Some(serde_json::Value::String(args)) => args,
Some(_) => return config,
};
if !ide_jvm_attach::wants_listener_flag(vm_args) {
return config;
}
let vm_args = if vm_args.is_empty() {
ide_jvm_attach::START_LISTENER_FLAG.to_string()
} else {
format!("{} {vm_args}", ide_jvm_attach::START_LISTENER_FLAG)
};
extra.insert("vmArgs".into(), vm_args.into());
config.extra_json = Some(serde_json::Value::Object(extra).to_string());
config
}
/// A JVM launched by java-debug writes what java-debug decodes its output as
/// (see [`ide_core::output_encoding::java_debug_vm_args`]).
fn match_java_debug_encoding(mut config: LaunchConfig, windows: bool) -> LaunchConfig {
@@ -381,6 +437,51 @@ pub async fn debug_redefine_classes(host: &Host, _window: &str, session: DebugSe
host.debugger.redefine_classes(session).await
}
/// The debuggee JVM's loaded classes, or why they cannot be read.
pub async fn debug_jvm_memory_classes(
host: &Host,
window: &str,
session: DebugSessionId,
) -> IdeResult<JvmMemoryResult> {
host.require_trust(window)?;
host.debugger.jvm_memory_classes(session).await
}
/// Up to `limit` live instances of the class in the stopped JVM.
pub async fn debug_jvm_memory_instances(
host: &Host,
window: &str,
session: DebugSessionId,
class_name: String,
limit: u32,
) -> IdeResult<JvmInstancesResult> {
host.require_trust(window)?;
host.debugger.jvm_memory_instances(session, class_name, limit).await
}
/// The fields (or array elements) of an instance listed by [`debug_jvm_memory_instances`].
pub async fn debug_jvm_memory_fields(
host: &Host,
window: &str,
session: DebugSessionId,
object: JvmObjectRef,
) -> IdeResult<JvmFieldsResult> {
host.require_trust(window)?;
host.debugger.jvm_memory_fields(session, object).await
}
/// The objects that hold a reference to `object`; `limit` 0 selects the default.
pub async fn debug_jvm_memory_referrers(
host: &Host,
window: &str,
session: DebugSessionId,
object: JvmObjectRef,
limit: u32,
) -> IdeResult<JvmReferrersResult> {
host.require_trust(window)?;
host.debugger.jvm_memory_referrers(session, object, limit).await
}
/// Commands of this module the remote server answers.
pub(crate) fn register(d: &mut Dispatcher) {
d.add("debug_start", |c| async move { debug_start(&c.host, &c.window, c.arg("config")?).await });
@@ -509,6 +610,18 @@ pub(crate) fn register(d: &mut Dispatcher) {
"debug_redefine_classes",
|c| async move { debug_redefine_classes(&c.host, &c.window, c.arg("session")?).await },
);
d.add("debug_jvm_memory_classes", |c| async move {
debug_jvm_memory_classes(&c.host, &c.window, c.arg("session")?).await
});
d.add("debug_jvm_memory_instances", |c| async move {
debug_jvm_memory_instances(&c.host, &c.window, c.arg("session")?, c.arg("className")?, c.arg("limit")?).await
});
d.add("debug_jvm_memory_fields", |c| async move {
debug_jvm_memory_fields(&c.host, &c.window, c.arg("session")?, c.arg("object")?).await
});
d.add("debug_jvm_memory_referrers", |c| async move {
debug_jvm_memory_referrers(&c.host, &c.window, c.arg("session")?, c.arg("object")?, c.arg("limit")?).await
});
}
#[cfg(test)]
@@ -553,6 +666,58 @@ mod tests {
}
}
/// The flag as added for a JVM on macOS or Linux.
fn on_unix(config: LaunchConfig) -> LaunchConfig {
start_java_attach_listener(config, false)
}
#[test]
fn java_debug_launches_start_the_attach_listener() {
let fixed = on_unix(launch("java-debug", "launch", serde_json::json!({"vmArgs": "-Xmx1g"})));
assert_eq!(vm_args(&fixed), "-XX:+StartAttachListener -Xmx1g");
assert_eq!(
vm_args(&on_unix(launch("java-debug", "launch", serde_json::json!({})))),
"-XX:+StartAttachListener"
);
for extra in [
serde_json::json!({"vmArgs": "-XX:-StartAttachListener"}),
serde_json::json!({"vmArgs": "-XX:+DisableAttachMechanism"}),
] {
assert_eq!(vm_args(&on_unix(launch("java-debug", "launch", extra.clone()))), extra["vmArgs"]);
}
assert_eq!(
vm_args(&on_unix(launch("java-debug", "attach", serde_json::json!({"vmArgs": "-Xmx1g"})))),
"-Xmx1g"
);
}
#[test]
fn java_debug_launches_on_windows_get_no_attach_listener() {
let config = launch("java-debug", "launch", serde_json::json!({"vmArgs": "-Xmx1g"}));
assert_eq!(vm_args(&start_java_attach_listener(config, true)), "-Xmx1g");
}
#[test]
fn java_debug_array_vm_args_are_left_alone() {
let config = launch("java-debug", "launch", serde_json::json!({"vmArgs": ["-Xmx1g", "-Dk=v"]}));
assert_eq!(on_unix(config.clone()).extra_json, config.extra_json);
assert_eq!(
vm_args(&on_unix(launch("java-debug", "launch", serde_json::json!({"vmArgs": null})))),
"-XX:+StartAttachListener"
);
}
#[test]
fn debug_server_port_only_for_local_java_debug() {
let port = |adapter: &str, request: &str, extra| debug_server_port(&launch(adapter, request, extra));
assert_eq!(port("java-debug", "launch", serde_json::json!({"debugServer": 4711})), Some(4711));
assert_eq!(port("java-debug", "attach", serde_json::json!({"debugServer": 4711})), Some(4711));
assert_eq!(port("java-debug", "launch", serde_json::json!({"debugServer": "10.0.0.5:4711"})), None);
assert_eq!(port(ide_dap::kotlin::ADAPTER, "launch", serde_json::json!({"debugServer": 4711})), None);
assert_eq!(port("java-debug", "launch", serde_json::json!({})), None);
assert_eq!(port("java-debug", "launch", serde_json::json!({"debugServer": 0})), None);
}
#[test]
fn java_debug_launches_respect_environment_encoding() {
for key in ["JDK_JAVA_OPTIONS", "JAVA_TOOL_OPTIONS"] {
+520
Переглянути файл
@@ -0,0 +1,520 @@
//! Heap snapshots for the debugger's JVM Memory view: captured from a debug session or opened from
//! an `.hprof`, analyzed into an index, then queried, compared, saved and closed. A snapshot
//! belongs to the window that made it and closes with that window's project.
use crate::Host;
use crate::ops::Channel;
use crate::rpc::Dispatcher;
use ide_core::{CancellationToken, DebugSessionId, IdeError, IdeResult, IpcPath, RequestId};
use ide_dap::{JvmHeapOutcome, JvmHeapUnavailable};
use ide_heap::{AnalyzeOptions, HeapError, HeapOrigin, OpenSnapshot};
pub use ide_heap::{
HeapClassDiff, HeapClassRow, HeapObjectPage, HeapObjectRef, HeapPhase, HeapRootPath, HeapSnapshotId,
HeapSnapshotInfo,
};
use serde::{Deserialize, Serialize};
use specta::Type;
use std::cell::Cell;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// Shortest gap between two progress messages of one phase.
const PROGRESS_INTERVAL: Duration = Duration::from_millis(100);
/// Longest each of [`shutdown`]'s two waits lasts; `openidea-server` gives the whole host
/// shutdown 10 s.
const SHUTDOWN_WAIT: Duration = Duration::from_secs(3);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "type", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapProgress {
Dumping,
Analyzing { phase: HeapPhase, done: u64, total: u64 },
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapCaptureResult {
Ready { snapshot: HeapSnapshotInfo },
Unavailable { reason: JvmHeapUnavailable },
Cancelled,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapOpenResult {
Ready {
snapshot: HeapSnapshotInfo,
},
/// Bytes, for the index next to the dump.
NotEnoughDiskSpace {
needed: u64,
available: u64,
},
Cancelled,
}
/// Dumps the session's JVM into a new snapshot directory and analyzes the dump there; cancel with
/// `request`. Anything but `Ready` leaves no directory behind.
pub async fn debug_heap_capture(
host: &Host,
window: &str,
session: DebugSessionId,
request: RequestId,
on_progress: Channel<HeapProgress>,
) -> IdeResult<HeapCaptureResult> {
host.require_trust(window)?;
let cancel = host.begin_project_request(window, request);
let running = Running::begin(host, request, &cancel);
let result = capture(host, window, session, &cancel, &on_progress).await;
drop(running);
host.end_request(request);
result
}
async fn capture(
host: &Host,
window: &str,
session: DebugSessionId,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<HeapCaptureResult> {
if cancel.is_cancelled() {
return Ok(HeapCaptureResult::Cancelled);
}
let dir = new_snapshot_dir(host).await?;
let _ = on_progress.send(HeapProgress::Dumping);
// `jvm_heap_dump` deletes the directory itself on every outcome but `Ready`.
let dump = match host.debugger.jvm_heap_dump(session, dir.clone(), cancel.clone()).await {
Ok(JvmHeapOutcome::Ready(dump)) => dump,
Ok(JvmHeapOutcome::Unavailable(reason)) => return Ok(HeapCaptureResult::Unavailable { reason }),
Ok(JvmHeapOutcome::Cancelled) => return Ok(HeapCaptureResult::Cancelled),
Err(e) => {
remove_dir(host, dir).await;
return Err(e);
}
};
let origin = HeapOrigin::Captured {
session,
pid: dump.pid,
java_version: dump.java_version,
taken_while_suspended: dump.taken_while_suspended,
dump_ms: dump.dump_ms,
};
let options = AnalyzeOptions {
histogram: Some(dump.histogram).filter(|h| !h.is_empty()),
vm_flags: Some(dump.vm_flags).filter(|f| !f.is_empty()),
java_major: Some(dump.java_major).filter(|&v| v > 0),
in_memory: false,
};
// The capture already checked the space for the dump and its index before dumping.
let analyzed = analyze(host, window, origin, dump.path, dir, options, false, cancel, on_progress).await?;
Ok(match analyzed {
Analyzed::Ready(snapshot) => HeapCaptureResult::Ready { snapshot },
Analyzed::NotEnoughDiskSpace { needed, available } => {
HeapCaptureResult::Unavailable { reason: JvmHeapUnavailable::NotEnoughDiskSpace { needed, available } }
}
Analyzed::Cancelled => HeapCaptureResult::Cancelled,
})
}
/// Analyzes the `.hprof` at `path` into a new snapshot directory; the file itself is never
/// changed or deleted. Cancel with `request`.
pub async fn debug_heap_open(
host: &Host,
window: &str,
path: IpcPath,
request: RequestId,
on_progress: Channel<HeapProgress>,
) -> IdeResult<HeapOpenResult> {
host.require_trust(window)?;
let cancel = host.begin_project_request(window, request);
let running = Running::begin(host, request, &cancel);
let result = open(host, window, path.into_path_buf(), &cancel, &on_progress).await;
drop(running);
host.end_request(request);
result
}
/// The captures and opens in progress, so [`shutdown`] can cancel them and wait for them to delete
/// their directories.
#[derive(Default)]
pub(crate) struct HeapRequests(parking_lot::Mutex<RequestsState>);
#[derive(Default)]
struct RequestsState {
/// Set by [`shutdown`]: what begins afterwards starts out cancelled.
closed: bool,
running: HashMap<RequestId, CancellationToken>,
}
/// One capture or open, from its start until it has cleaned up after itself.
struct Running<'a> {
host: &'a Host,
request: RequestId,
}
impl<'a> Running<'a> {
fn begin(host: &'a Host, request: RequestId, cancel: &CancellationToken) -> Self {
let mut state = host.heap_requests.0.lock();
if state.closed {
cancel.cancel();
} else {
state.running.insert(request, cancel.clone());
}
Self { host, request }
}
}
impl Drop for Running<'_> {
fn drop(&mut self) {
self.host.heap_requests.0.lock().running.remove(&self.request);
}
}
async fn open(
host: &Host,
window: &str,
path: PathBuf,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<HeapOpenResult> {
if cancel.is_cancelled() {
return Ok(HeapOpenResult::Cancelled);
}
let dir = new_snapshot_dir(host).await?;
let origin = HeapOrigin::File { path: path.clone().into() };
// The snapshot maps its dump for as long as it is open; a file that something else truncates or
// rewrites meanwhile would crash the app, so it works on a private copy (a clone on APFS / btrfs).
let copy = {
let (path, dir) = (path.clone(), dir.clone());
host.blocking(move || private_copy(&path, &dir)).await
};
let dump = match copy {
Ok(Ok(dump)) => dump,
Ok(Err((needed, available))) => {
remove_dir(host, dir).await;
return Ok(HeapOpenResult::NotEnoughDiskSpace { needed, available });
}
Err(e) => {
remove_dir(host, dir).await;
return Err(e);
}
};
let analyzed =
analyze(host, window, origin, dump, dir, AnalyzeOptions::default(), true, cancel, on_progress).await?;
Ok(match analyzed {
Analyzed::Ready(snapshot) => HeapOpenResult::Ready { snapshot },
Analyzed::NotEnoughDiskSpace { needed, available } => HeapOpenResult::NotEnoughDiskSpace { needed, available },
Analyzed::Cancelled => HeapOpenResult::Cancelled,
})
}
enum Analyzed {
Ready(HeapSnapshotInfo),
NotEnoughDiskSpace { needed: u64, available: u64 },
Cancelled,
}
/// Indexes `dump` into `dir` and registers the snapshot for `window`. Anything but `Ready` deletes
/// `dir`, a captured dump inside it included.
#[allow(clippy::too_many_arguments)]
async fn analyze(
host: &Host,
window: &str,
origin: HeapOrigin,
dump: PathBuf,
dir: PathBuf,
options: AnalyzeOptions,
check_disk: bool,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<Analyzed> {
let indexed = {
let (dump, dir, cancel, progress) = (dump.clone(), dir.clone(), cancel.clone(), on_progress.clone());
tokio::task::spawn_blocking(move || {
let free = || if check_disk { ide_heap::free_bytes(&dir) } else { None };
ide_heap::analyze(&dump, &dir, options, &reporter(&progress, free), &cancel)
})
.await
};
let failed = match indexed {
Ok(Ok(index)) => {
let info = HeapSnapshotInfo {
id: HeapSnapshotId(0),
label: origin.label(),
summary: index.summary().clone(),
origin,
created_ms: now_ms(),
};
let info = host.heap.insert(|_| OpenSnapshot::new(info, index, dir, dump));
host.heap_windows.lock().insert(info.id, host.owner_window(window).to_owned());
// A project close cancels the request before it closes the window's snapshots, so one
// registered after that close is caught here.
if cancel.is_cancelled() {
close(host, info.id).await?;
return Ok(Analyzed::Cancelled);
}
return Ok(Analyzed::Ready(info));
}
Ok(Err(HeapError::Cancelled)) => Ok(Analyzed::Cancelled),
Ok(Err(HeapError::NotEnoughDiskSpace { needed, available })) => {
Ok(Analyzed::NotEnoughDiskSpace { needed, available })
}
Ok(Err(e)) => Err(IdeError::invalid(format!("{e}"))),
Err(e) => Err(IdeError::internal(format!("heap analysis failed: {e}"))),
};
remove_dir(host, dir).await;
failed
}
/// The progress callback of an analysis: sends a phase's first report and then at most one every
/// [`PROGRESS_INTERVAL`]. Once the dump's objects are counted (the first `Linking` report), it
/// fails with `NotEnoughDiskSpace` when `free` reports less than their index needs.
fn reporter<'a>(
progress: &'a Channel<HeapProgress>,
free: impl Fn() -> Option<u64> + 'a,
) -> impl Fn(HeapPhase, u64, u64) -> Result<(), HeapError> + 'a {
let last: Cell<Option<(HeapPhase, Instant)>> = Cell::new(None);
move |phase, done, total| {
let now = Instant::now();
let previous = last.get();
if phase == HeapPhase::Linking
&& previous.is_none_or(|(p, _)| p != HeapPhase::Linking)
&& let Some(available) = free()
{
let needed = ide_heap::index_need(total);
if available < needed {
return Err(HeapError::NotEnoughDiskSpace { needed, available });
}
}
if previous.is_none_or(|(p, at)| p != phase || now.duration_since(at) >= PROGRESS_INTERVAL) {
last.set(Some((phase, now)));
let _ = progress.send(HeapProgress::Analyzing { phase, done, total });
}
Ok(())
}
}
/// The class table of a snapshot.
pub async fn debug_heap_classes(host: &Host, window: &str, snapshot: HeapSnapshotId) -> IdeResult<Vec<HeapClassRow>> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().class_rows())).await
}
/// A page of the dominator tree; see [`ide_heap::HeapIndex::dominated`] for what `parent` and
/// `class_index` select.
pub async fn debug_heap_dominators(
host: &Host,
window: &str,
snapshot: HeapSnapshotId,
parent: Option<HeapObjectRef>,
class_index: Option<u32>,
offset: u32,
limit: u32,
) -> IdeResult<HeapObjectPage> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().dominated(parent, class_index, offset, limit))).await
}
/// The shortest path from `object` to a GC root; `None` when it is unreachable.
pub async fn debug_heap_path(
host: &Host,
window: &str,
snapshot: HeapSnapshotId,
object: HeapObjectRef,
) -> IdeResult<Option<HeapRootPath>> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().path_to_root(object))).await
}
/// Class-by-class differences from `base` to `current`.
pub async fn debug_heap_compare(
host: &Host,
window: &str,
base: HeapSnapshotId,
current: HeapSnapshotId,
) -> IdeResult<Vec<HeapClassDiff>> {
let base = open_snapshot(host, window, base)?;
let current = open_snapshot(host, window, current)?;
host.blocking(move || Ok(ide_heap::compare(base.index(), current.index()))).await
}
/// Copies the snapshot's dump to `path`, replacing a file there (the UI has asked).
pub async fn debug_heap_save(host: &Host, window: &str, snapshot: HeapSnapshotId, path: IpcPath) -> IdeResult<()> {
let snapshot = open_snapshot(host, window, snapshot)?;
let target = path.into_path_buf();
host.blocking(move || copy_replacing(&snapshot.dump, &target)).await
}
/// Forgets the snapshot and deletes its directory once no query is using it.
pub async fn debug_heap_close(host: &Host, window: &str, snapshot: HeapSnapshotId) -> IdeResult<()> {
open_snapshot(host, window, snapshot)?;
close(host, snapshot).await
}
async fn close(host: &Host, id: HeapSnapshotId) -> IdeResult<()> {
let closed = take(host, id);
// Usually the last reference: dropping it deletes files.
host.blocking(move || {
drop(closed);
Ok(())
})
.await
}
/// Forgets the snapshot and returns it, so the caller chooses the thread that deletes its files.
fn take(host: &Host, id: HeapSnapshotId) -> Option<Arc<OpenSnapshot>> {
host.heap_windows.lock().remove(&id);
let snapshot = host.heap.get(id);
host.heap.close(id);
snapshot
}
/// Closes the snapshots `window` owns; their directories are deleted on the blocking pool.
pub(crate) fn close_window_snapshots(host: &Host, window: &str) {
let owner = host.owner_window(window);
let ids: Vec<HeapSnapshotId> =
host.heap_windows.lock().iter().filter(|(_, w)| *w == owner).map(|(id, _)| *id).collect();
let closed: Vec<Arc<OpenSnapshot>> = ids.into_iter().filter_map(|id| take(host, id)).collect();
if !closed.is_empty() {
host.runtime.spawn_blocking(move || drop(closed));
}
}
/// For a process that exits without dropping the host (the remote server): cancels the captures
/// and opens in progress and waits for them, closes every snapshot, then removes what is left of
/// this process's snapshot directories. Each wait gives up after [`SHUTDOWN_WAIT`] (a stuck
/// analysis or delete).
pub(crate) async fn shutdown(host: &Host) {
shutdown_within(host, SHUTDOWN_WAIT).await;
}
async fn shutdown_within(host: &Host, wait: Duration) {
let deadline = tokio::time::Instant::now() + wait;
{
let mut state = host.heap_requests.0.lock();
state.closed = true;
for token in state.running.values() {
token.cancel();
}
}
let ids: Vec<HeapSnapshotId> = host.heap.list().into_iter().map(|info| info.id).collect();
let closed: Vec<Arc<OpenSnapshot>> = ids.into_iter().filter_map(|id| take(host, id)).collect();
while !host.heap_requests.0.lock().running.is_empty() {
if tokio::time::Instant::now() >= deadline {
tracing::warn!("heap captures or opens still running at shutdown");
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
let root = host.heap.root().to_path_buf();
let me = std::process::id();
let cleanup = host.runtime.spawn_blocking(move || {
drop(closed);
// Directories of work that did not finish in time count as abandoned, like a dead host's.
ide_heap::remove_stale(&root, |pid| pid != me && ide_core::process::process_alive(pid));
});
if tokio::time::timeout(wait, cleanup).await.is_err() {
tracing::warn!("heap snapshot directories were not deleted within {wait:?}");
}
}
/// Every op that takes a snapshot id goes through here, so none skips the trust check.
fn open_snapshot(host: &Host, window: &str, id: HeapSnapshotId) -> IdeResult<Arc<OpenSnapshot>> {
host.require_trust(window)?;
let owned = host.heap_windows.lock().get(&id).is_some_and(|w| w == host.owner_window(window));
owned
.then(|| host.heap.get(id))
.flatten()
.ok_or_else(|| IdeError::not_found(format!("heap snapshot {} is not open", id.0)))
}
async fn new_snapshot_dir(host: &Host) -> IdeResult<PathBuf> {
let root = host.heap.root().to_path_buf();
host.blocking(move || Ok(ide_heap::create_snapshot_dir(&root)?)).await
}
async fn remove_dir(host: &Host, dir: PathBuf) {
let _ = host
.blocking(move || {
if let Err(e) = std::fs::remove_dir_all(&dir)
&& e.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not delete heap snapshot directory {}: {e}", dir.display());
}
Ok(())
})
.await;
}
/// Copies an opened dump into its snapshot directory; `Err((needed, available))` when the disk
/// can't hold the copy.
fn private_copy(source: &Path, dir: &Path) -> IdeResult<Result<PathBuf, (u64, u64)>> {
let needed = std::fs::metadata(source)?.len();
if let Some(available) = ide_heap::free_bytes(dir)
&& available < needed
{
return Ok(Err((needed, available)));
}
let target = dir.join("opened.hprof");
std::fs::copy(source, &target)?;
Ok(Ok(target))
}
/// Written beside `target` and renamed over it: `target` may be the dump of an open snapshot,
/// whose index maps that file and must keep reading the old bytes.
fn copy_replacing(source: &Path, target: &Path) -> IdeResult<()> {
let name =
target.file_name().ok_or_else(|| IdeError::invalid(format!("{} is not a file path", target.display())))?;
let partial = target.with_file_name(format!(".{}.{}.tmp", name.to_string_lossy(), uuid::Uuid::new_v4()));
let copied = std::fs::copy(source, &partial).and_then(|_| std::fs::rename(&partial, target));
if let Err(e) = copied {
let _ = std::fs::remove_file(&partial);
return Err(e.into());
}
Ok(())
}
fn now_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| u64::try_from(d.as_millis()).unwrap_or(u64::MAX))
}
/// Commands of this module the remote server answers.
pub(crate) fn register(d: &mut Dispatcher) {
d.add("debug_heap_capture", |c| async move {
debug_heap_capture(&c.host, &c.window, c.arg("session")?, c.arg("request")?, c.sender("onProgress")?).await
});
d.add("debug_heap_open", |c| async move {
debug_heap_open(&c.host, &c.window, c.arg("path")?, c.arg("request")?, c.sender("onProgress")?).await
});
d.add("debug_heap_classes", |c| async move { debug_heap_classes(&c.host, &c.window, c.arg("snapshot")?).await });
d.add("debug_heap_dominators", |c| async move {
debug_heap_dominators(
&c.host,
&c.window,
c.arg("snapshot")?,
c.arg("parent")?,
c.arg("classIndex")?,
c.arg("offset")?,
c.arg("limit")?,
)
.await
});
d.add("debug_heap_path", |c| async move {
debug_heap_path(&c.host, &c.window, c.arg("snapshot")?, c.arg("object")?).await
});
d.add("debug_heap_compare", |c| async move {
debug_heap_compare(&c.host, &c.window, c.arg("base")?, c.arg("current")?).await
});
d.add("debug_heap_save", |c| async move {
debug_heap_save(&c.host, &c.window, c.arg("snapshot")?, c.arg("path")?).await
});
d.add("debug_heap_close", |c| async move { debug_heap_close(&c.host, &c.window, c.arg("snapshot")?).await });
}
#[cfg(test)]
mod tests;
+479
Переглянути файл
@@ -0,0 +1,479 @@
//! The heap commands through a real `Host` with a trusted project, on `MiniHeap` dumps.
use super::*;
use crate::HostConfig;
use ide_core::error::ErrorKind;
use ide_dap::{JvmMemoryUnavailable, LaunchConfig};
use ide_heap::testing::{MiniHeap, sample, sample_with_two_more_nodes};
use parking_lot::Mutex;
use serde_json::{Value, json};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
const WINDOW: &str = "main";
struct Fixture {
host: Host,
base: PathBuf,
root: PathBuf,
_temp: tempfile::TempDir,
}
impl Fixture {
async fn new() -> Self {
let temp = tempfile::tempdir().unwrap();
let base = dunce::canonicalize(temp.path()).unwrap();
let root = base.join("project");
std::fs::create_dir_all(&root).unwrap();
let host = Host::new(HostConfig {
config_dir: base.join("config"),
data_dir: base.join("data"),
local_data_dir: base.join("local"),
languages: Arc::new(ide_language::LanguageRegistry::load(None).unwrap()),
syntax: Arc::default(),
sink_for: Box::new(|_| Arc::new(|_| {})),
owner: |w| w,
runtime: tokio::runtime::Handle::current(),
project_environment: false,
});
host.open_project(WINDOW, &root).unwrap();
host.set_project_trust(&root, true, false).await.unwrap();
Self { host, base, root, _temp: temp }
}
fn dump(&self, name: &str, heap: &MiniHeap) -> PathBuf {
let path = self.base.join(name);
heap.write_to(&path);
path
}
async fn open_with(&self, path: &Path, request: u32, progress: Channel<HeapProgress>) -> IdeResult<HeapOpenResult> {
debug_heap_open(&self.host, WINDOW, path.into(), RequestId(request), progress).await
}
async fn open(&self, path: &Path) -> HeapSnapshotInfo {
match self.open_with(path, 1, Channel::new(|_| {})).await.unwrap() {
HeapOpenResult::Ready { snapshot } => snapshot,
other => panic!("expected a snapshot, got {other:?}"),
}
}
/// Opens a second trusted project in `window`.
async fn open_window(&self, window: &str) {
let root = self.base.join(window);
std::fs::create_dir_all(&root).unwrap();
self.host.open_project(window, &root).unwrap();
self.host.set_project_trust(&root, true, false).await.unwrap();
}
/// The snapshot directories under the registry's root (none when it does not exist yet).
fn snapshot_dirs(&self) -> Vec<PathBuf> {
let Ok(entries) = std::fs::read_dir(self.host.heap.root()) else { return Vec::new() };
entries.map(|e| e.unwrap().path()).collect()
}
}
fn recorder() -> (Channel<HeapProgress>, Arc<Mutex<Vec<HeapProgress>>>) {
let seen = Arc::new(Mutex::new(Vec::new()));
let sink = seen.clone();
(Channel::new(move |p| sink.lock().push(p)), seen)
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn open_and_query_a_dump() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
assert_eq!(info.label, "sample.hprof");
assert_eq!(info.origin, ide_heap::HeapOrigin::File { path: path.clone().into() });
assert_eq!(f.snapshot_dirs().len(), 1);
// The snapshot reads a private copy: the user's file may change while it is open.
std::fs::write(&path, b"").unwrap();
let classes = debug_heap_classes(&f.host, WINDOW, info.id).await.unwrap();
let node = classes.iter().find(|c| c.name == "Node").unwrap();
assert_eq!((node.count, node.retained), (3, 104));
let top = debug_heap_dominators(&f.host, WINDOW, info.id, None, None, 0, 10).await.unwrap();
assert_eq!(top.rows[0].retained, 72);
let n3 = top.rows.iter().find(|r| r.class_name == "Node").unwrap();
let below = debug_heap_dominators(&f.host, WINDOW, info.id, Some(n3.object), None, 0, 10).await.unwrap();
let long_array = below.rows.first().unwrap();
assert_eq!(long_array.address, "0x1400");
let path_to_root = debug_heap_path(&f.host, WINDOW, info.id, long_array.object).await.unwrap().unwrap();
assert_eq!(path_to_root.steps.len(), 2);
debug_heap_close(&f.host, WINDOW, info.id).await.unwrap();
assert!(f.snapshot_dirs().is_empty(), "the index directory goes with the snapshot");
assert!(path.is_file(), "an opened dump is the user's file");
let gone = debug_heap_classes(&f.host, WINDOW, info.id).await.unwrap_err();
assert_eq!(gone.kind, ErrorKind::NotFound);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn open_reports_progress_in_order() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let (progress, seen) = recorder();
assert!(matches!(f.open_with(&path, 1, progress).await.unwrap(), HeapOpenResult::Ready { .. }));
let phases: Vec<HeapPhase> = seen
.lock()
.iter()
.map(|p| match p {
HeapProgress::Analyzing { phase, .. } => *phase,
HeapProgress::Dumping => panic!("an open dumps nothing"),
})
.collect();
assert_eq!(phases.first(), Some(&HeapPhase::Reading));
assert!(phases.is_sorted_by_key(|&p| p as u8), "phases never go back: {phases:?}");
assert_eq!(phases.last(), Some(&HeapPhase::Saving));
let mut distinct = phases.clone();
distinct.dedup();
assert_eq!(
distinct,
[
HeapPhase::Reading,
HeapPhase::Linking,
HeapPhase::Dominators,
HeapPhase::Retained,
HeapPhase::Paths,
HeapPhase::Saving
],
"every phase is reported at least once however fast it runs"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn cancelled_open_leaves_nothing() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
f.host.cancel_request(RequestId(7));
assert_eq!(f.open_with(&path, 7, Channel::new(|_| {})).await.unwrap(), HeapOpenResult::Cancelled);
assert!(f.snapshot_dirs().is_empty());
assert!(f.host.heap.list().is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn not_a_dump_is_an_error() {
let f = Fixture::new().await;
let path = f.base.join("hello.hprof");
std::fs::write(&path, "hello").unwrap();
let err = f.open_with(&path, 1, Channel::new(|_| {})).await.unwrap_err();
assert!(err.message.contains("not a heap dump"), "{}", err.message);
assert!(f.snapshot_dirs().is_empty());
assert!(path.is_file());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn a_dump_without_its_end_record_is_an_error() {
let f = Fixture::new().await;
let path = f.base.join("killed.hprof");
let bytes = sample().bytes();
// HEAP DUMP END: tag, timestamp and length.
std::fs::write(&path, &bytes[..bytes.len() - 9]).unwrap();
let err = f.open_with(&path, 1, Channel::new(|_| {})).await.unwrap_err();
assert!(err.message.contains("the dump ends early"), "{}", err.message);
assert!(f.snapshot_dirs().is_empty());
assert!(f.host.heap.list().is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn compare_two_snapshots() {
let f = Fixture::new().await;
let base = f.open(&f.dump("base.hprof", &sample())).await;
let current = f.open(&f.dump("current.hprof", &sample_with_two_more_nodes())).await;
assert_ne!(base.id, current.id);
let diff = debug_heap_compare(&f.host, WINDOW, base.id, current.id).await.unwrap();
let node = diff.iter().find(|d| d.name == "Node").unwrap();
assert_eq!((node.base_count, node.count), (3, 5));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn unknown_snapshot_is_not_found() {
let f = Fixture::new().await;
let missing = HeapSnapshotId(99);
let not_found = |r: IdeError| assert_eq!(r.kind, ErrorKind::NotFound, "{}", r.message);
not_found(debug_heap_classes(&f.host, WINDOW, missing).await.unwrap_err());
not_found(debug_heap_dominators(&f.host, WINDOW, missing, None, None, 0, 10).await.unwrap_err());
not_found(debug_heap_path(&f.host, WINDOW, missing, HeapObjectRef(0)).await.unwrap_err());
not_found(debug_heap_save(&f.host, WINDOW, missing, f.base.join("x.hprof").into()).await.unwrap_err());
not_found(debug_heap_close(&f.host, WINDOW, missing).await.unwrap_err());
let open = f.open(&f.dump("sample.hprof", &sample())).await;
not_found(debug_heap_compare(&f.host, WINDOW, open.id, missing).await.unwrap_err());
// A snapshot belongs to the window that opened it.
f.open_window("other").await;
not_found(debug_heap_classes(&f.host, "other", open.id).await.unwrap_err());
not_found(debug_heap_close(&f.host, "other", open.id).await.unwrap_err());
assert!(debug_heap_classes(&f.host, WINDOW, open.id).await.is_ok());
}
/// The error of every op that takes a snapshot id, in `window`.
async fn query_errors(f: &Fixture, window: &str, id: HeapSnapshotId) -> Vec<IdeError> {
let host = &f.host;
vec![
debug_heap_classes(host, window, id).await.unwrap_err(),
debug_heap_dominators(host, window, id, None, None, 0, 10).await.unwrap_err(),
debug_heap_path(host, window, id, HeapObjectRef(0)).await.unwrap_err(),
debug_heap_compare(host, window, id, id).await.unwrap_err(),
debug_heap_save(host, window, id, f.base.join("saved.hprof").into()).await.unwrap_err(),
debug_heap_close(host, window, id).await.unwrap_err(),
]
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn queries_check_trust_before_the_snapshot() {
let f = Fixture::new().await;
let open = f.open(&f.dump("sample.hprof", &sample())).await;
let no_project = f.host.require_trust("nowhere").unwrap_err();
for err in query_errors(&f, "nowhere", open.id).await {
assert_eq!((err.kind, &err.message), (no_project.kind, &no_project.message));
}
f.host.set_project_trust(&f.root, false, false).await.unwrap();
let restricted = f.host.require_trust(WINDOW).unwrap_err();
assert_ne!(restricted.kind, ErrorKind::NotFound);
// An id that is not open is refused for trust too: the check comes before the lookup.
for id in [open.id, HeapSnapshotId(99)] {
for err in query_errors(&f, WINDOW, id).await {
assert_eq!((err.kind, &err.message), (restricted.kind, &restricted.message));
}
}
assert!(!f.base.join("saved.hprof").exists());
assert!(f.host.heap.get(open.id).is_some(), "a refused close keeps the snapshot");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn shutdown_closes_every_snapshot() {
let f = Fixture::new().await;
let external = f.dump("sample.hprof", &sample());
f.open(&external).await;
// A capture: the dump sits in the snapshot directory.
let captured = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let dump = captured.join("dump.hprof");
sample_with_two_more_nodes().write_to(&dump);
let origin = ide_heap::HeapOrigin::Captured {
session: DebugSessionId(1),
pid: 42,
java_version: "21".into(),
taken_while_suspended: false,
dump_ms: 1,
};
let analyzed = analyze(
&f.host,
WINDOW,
origin,
dump.clone(),
captured.clone(),
AnalyzeOptions::default(),
false,
&CancellationToken::new(),
&Channel::new(|_| {}),
)
.await
.unwrap();
assert!(matches!(analyzed, Analyzed::Ready(_)));
// An open still running: shutdown cancels it and waits for it to clean up.
let running_dir = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let cancel = CancellationToken::new();
let running = Running::begin(&f.host, RequestId(50), &cancel);
assert_eq!(f.snapshot_dirs().len(), 3);
let in_flight = async {
let cancelled = tokio::time::timeout(std::time::Duration::from_secs(5), cancel.cancelled()).await;
assert!(cancelled.is_ok(), "shutdown cancels a running capture or open");
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
std::fs::remove_dir_all(&running_dir).unwrap();
drop(running);
};
tokio::join!(f.host.shutdown(), in_flight);
assert!(f.snapshot_dirs().is_empty(), "{:?}", f.snapshot_dirs());
assert!(f.host.heap.list().is_empty());
assert!(external.is_file(), "an opened dump is the user's file");
let late = CancellationToken::new();
let _late = Running::begin(&f.host, RequestId(51), &late);
assert!(late.is_cancelled(), "nothing starts after shutdown");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn shutdown_does_not_wait_forever_for_a_stuck_capture() {
let f = Fixture::new().await;
let stuck_dir = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let cancel = CancellationToken::new();
let _stuck = Running::begin(&f.host, RequestId(60), &cancel);
let started = std::time::Instant::now();
shutdown_within(&f.host, std::time::Duration::from_millis(200)).await;
assert!(started.elapsed() < std::time::Duration::from_secs(5));
assert!(cancel.is_cancelled());
assert!(!stuck_dir.exists(), "its directory is removed once the wait is over");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn save_copies_the_dump_and_replaces_the_target() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
let target = f.base.join("saved.hprof");
std::fs::write(&target, "older").unwrap();
debug_heap_save(&f.host, WINDOW, info.id, target.clone().into()).await.unwrap();
assert_eq!(std::fs::read(&target).unwrap(), sample().bytes());
// Over the dump the snapshot itself maps: the index keeps reading the old file.
debug_heap_save(&f.host, WINDOW, info.id, path.clone().into()).await.unwrap();
assert_eq!(std::fs::read(&path).unwrap(), sample().bytes());
assert!(debug_heap_classes(&f.host, WINDOW, info.id).await.is_ok());
let leftovers: Vec<_> = std::fs::read_dir(&f.base)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|name| name.ends_with(".tmp"))
.collect();
assert!(leftovers.is_empty(), "{leftovers:?}");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn closing_the_project_closes_its_snapshots() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
f.host.close_project(WINDOW);
assert!(f.host.heap.get(info.id).is_none());
// The directory is deleted on the blocking pool.
for _ in 0..200 {
if f.snapshot_dirs().is_empty() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
assert!(f.snapshot_dirs().is_empty());
assert!(path.is_file());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn capture_without_jvm_session_is_unavailable() {
let f = Fixture::new().await;
let listener = tokio::net::TcpListener::bind(("127.0.0.1", 0)).await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
fake_adapter(stream).await;
});
let config = LaunchConfig {
name: "Remote".into(),
adapter: "delve".into(),
request: "attach".into(),
program: None,
args: vec![],
cwd: Some(f.root.clone().into()),
env: Default::default(),
extra_json: Some(json!({"debugServer": format!("127.0.0.1:{port}"), "mode": "remote"}).to_string()),
};
let session = crate::ops::debug::debug_start(&f.host, WINDOW, config).await.unwrap();
let (progress, seen) = recorder();
let result = debug_heap_capture(&f.host, WINDOW, session, RequestId(3), progress).await.unwrap();
assert_eq!(
result,
HeapCaptureResult::Unavailable {
reason: JvmHeapUnavailable::Memory { reason: JvmMemoryUnavailable::NoJvmProcess }
}
);
assert_eq!(*seen.lock(), [HeapProgress::Dumping]);
assert!(f.snapshot_dirs().is_empty());
f.host.debugger.stop(session).await.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn capture_of_a_missing_session_deletes_its_directory() {
let f = Fixture::new().await;
let missing = DebugSessionId(404);
assert!(debug_heap_capture(&f.host, WINDOW, missing, RequestId(4), Channel::new(|_| {})).await.is_err());
assert!(f.snapshot_dirs().is_empty());
}
#[test]
fn progress_is_thinned_but_keeps_every_phase() {
let (progress, seen) = recorder();
let report = reporter(&progress, || None);
for done in 0..50 {
report(HeapPhase::Reading, done, 50).unwrap();
}
report(HeapPhase::Linking, 0, 10).unwrap();
assert_eq!(
*seen.lock(),
[
HeapProgress::Analyzing { phase: HeapPhase::Reading, done: 0, total: 50 },
HeapProgress::Analyzing { phase: HeapPhase::Linking, done: 0, total: 10 },
]
);
}
#[test]
fn too_little_disk_for_the_index_stops_at_linking() {
let (progress, _) = recorder();
let report = reporter(&progress, || Some(1));
report(HeapPhase::Reading, 0, 50).unwrap();
let needed = ide_heap::index_need(1000);
match report(HeapPhase::Linking, 0, 1000) {
Err(ide_heap::HeapError::NotEnoughDiskSpace { needed: n, available: 1 }) => assert_eq!(n, needed),
other => panic!("expected NotEnoughDiskSpace, got {other:?}"),
}
let (progress, _) = recorder();
let report = reporter(&progress, || Some(u64::MAX));
assert!(report(HeapPhase::Linking, 0, 1000).is_ok());
}
/// Answers every request of a session that never runs a JVM.
async fn fake_adapter(stream: tokio::net::TcpStream) {
let (r, mut w) = stream.into_split();
let mut r = BufReader::new(r);
let mut seq = 0;
loop {
let mut length = None;
loop {
let mut line = String::new();
if r.read_line(&mut line).await.unwrap_or(0) == 0 {
return;
}
let line = line.trim();
if line.is_empty() {
break;
}
if let Some(n) = line.strip_prefix("Content-Length:") {
length = n.trim().parse::<usize>().ok();
}
}
let mut body = vec![0; length.unwrap()];
r.read_exact(&mut body).await.unwrap();
let msg: Value = serde_json::from_slice(&body).unwrap();
if msg["type"] != "request" {
continue;
}
let command = msg["command"].as_str().unwrap_or_default().to_owned();
let reply = match command.as_str() {
"initialize" => json!({"supportsConfigurationDoneRequest": true}),
"threads" => json!({"threads": []}),
_ => json!({}),
};
let mut out = vec![
json!({"type": "response", "request_seq": msg["seq"], "command": command, "success": true, "body": reply}),
];
if command == "launch" || command == "attach" {
out.push(json!({"type": "event", "event": "initialized"}));
}
for mut message in out {
seq += 1;
message["seq"] = seq.into();
let bytes = serde_json::to_vec(&message).unwrap();
w.write_all(format!("Content-Length: {}\r\n\r\n", bytes.len()).as_bytes()).await.unwrap();
w.write_all(&bytes).await.unwrap();
}
if command == "disconnect" {
return;
}
}
}
+67 -6
Переглянути файл
@@ -6,6 +6,7 @@ use crate::rpc::Dispatcher;
use super::build::load;
use ide_build::compile;
use ide_core::error::ErrorKind;
use ide_core::{CancellationToken, IdeError, IdeResult, RequestId};
use ide_lsp::jdtls::{JavaBuildStatus, JavaLaunchInfo};
use ide_run::BuildPolicy;
@@ -15,6 +16,7 @@ use std::path::PathBuf;
/// Builds the workspace (when `build`), then resolves the runtime (or test)
/// classpath and the JDK of the module at `module` (absolute, or relative to the project root).
/// A module with Kotlin / Groovy / Scala sources is built by its build system, whose class directories come first.
/// Resources a Gradle build filters or renames are processed by Gradle, replacing the language server's plain copies.
pub async fn java_resolve_launch(
host: &Host,
window: &str,
@@ -54,9 +56,15 @@ async fn resolve_launch(
project.java_launch_snapshots.lock().initialize();
let model = load(&project.root, &project.build_cache_dir(), &settings);
let non_java = build && compile::launch_has_non_java_sources(&project.root, &model, &module, test);
let is_maven =
model.maven.as_ref().is_some_and(|m| m.projects.iter().any(|p| project.root.join(&p.dir) == module));
let delegated = build
&& model.maven.as_ref().is_some_and(|m| m.projects.iter().any(|p| project.root.join(&p.dir) == module))
&& (non_java || cache::has_record(&project.root, &project.build_cache_dir(), &module));
&& delegates_to_external_build(
is_maven,
policy,
non_java,
cache::has_record(&project.root, &project.build_cache_dir(), &module),
);
let external = non_java || delegated;
(model, external, delegated)
})
@@ -80,7 +88,7 @@ async fn resolve_launch(
&cancel,
)?;
let copies = use_snapshot(&mut info, &outputs.originals, &outputs.snapshot);
apply_external_build_result(&mut info, outputs.failure);
apply_external_build_result(&mut info, outputs.failure, policy == BuildPolicy::Rebuild);
copies
} else if external {
let built = compile::build_for_launch(&project.root, &model, &module, test, build_kind, &cancel)?;
@@ -103,6 +111,18 @@ async fn resolve_launch(
}
info.class_paths = class_paths;
apply_library_classpath(&mut info, &project.root, &settings.project_structure.libraries, test);
if build && !delegated {
let cache_dir = project.build_cache_dir();
match compile::process_launch_resources(&project.root, &model, &cache_dir, &info.class_paths, &cancel) {
Ok(tasks) if !tasks.is_empty() => tracing::info!(?tasks, "Processed launch resources with Gradle"),
Ok(_) => {}
Err(e) if e.kind == ErrorKind::Cancelled => return Err(e),
Err(e) => info.hints.push(format!(
"Gradle could not process the resources, so they are used as they are in the source folders:\n{}",
e.message
)),
}
}
let compile_ms = started.elapsed().as_millis() - lsp_ms;
// The module/project JDK, else the JDK jdtls compiles the module against, else one matching its Java version.
let project_jdk = ide_sdk::apply::module_jdk(&project.root, &module, &settings);
@@ -149,10 +169,16 @@ async fn resolve_launch(
.map_err(|e| IdeError::internal(e.to_string()))?
}
fn apply_external_build_result(info: &mut JavaLaunchInfo, failure: Option<String>) {
fn apply_external_build_result(info: &mut JavaLaunchInfo, failure: Option<String>, clean: bool) {
info.build = Some(if let Some(failure) = failure {
info.hints.push(failure);
JavaBuildStatus::Failed
} else if clean {
// A clean external build recompiled every source; JDT's diagnostics still describe
// the previous output (and reject sources javac accepts, e.g. a package mismatch).
info.errors.clear();
info.error_count = 0;
JavaBuildStatus::Succeeded
} else if info.error_count > 0 {
// Incremental Maven compilation can accept error-stub classes written by JDT.
JavaBuildStatus::WithErrors
@@ -212,6 +238,13 @@ fn apply_library_classpath(
}
}
/// Whether a launch should be built by the project's build system instead of jdtls.
/// A Rebuild has to clean the output; jdtls' full build only recompiles sources, so
/// Maven modules always delegate for it, even when every source is Java.
fn delegates_to_external_build(is_maven: bool, policy: BuildPolicy, non_java: bool, has_record: bool) -> bool {
is_maven && (policy == BuildPolicy::Rebuild || non_java || has_record)
}
fn same_entry(a: &str, b: &str, windows: bool) -> bool {
a == b || entry_key(a, windows) == entry_key(b, windows)
}
@@ -288,6 +321,16 @@ pub(crate) fn register(d: &mut Dispatcher) {
mod tests {
use super::*;
#[test]
fn rebuild_delegates_maven_builds_to_the_build_system() {
assert!(delegates_to_external_build(true, BuildPolicy::Rebuild, false, false));
assert!(delegates_to_external_build(true, BuildPolicy::Automatic, true, false));
assert!(delegates_to_external_build(true, BuildPolicy::Make, false, true));
assert!(!delegates_to_external_build(true, BuildPolicy::Automatic, false, false));
assert!(!delegates_to_external_build(true, BuildPolicy::Make, false, false));
assert!(!delegates_to_external_build(false, BuildPolicy::Rebuild, false, false));
}
#[test]
fn successful_external_compilation_does_not_erase_scoped_jdt_errors() {
for (failure, expected) in
@@ -304,17 +347,35 @@ mod tests {
..Default::default()
};
let errors = info.errors.clone();
apply_external_build_result(&mut info, failure.map(str::to_owned));
apply_external_build_result(&mut info, failure.map(str::to_owned), false);
assert_eq!(info.build, Some(expected));
assert_eq!(info.error_count, 1);
assert_eq!(info.errors, errors);
assert_eq!(info.hints, failure.into_iter().collect::<Vec<_>>());
}
let mut info = JavaLaunchInfo::default();
apply_external_build_result(&mut info, None);
apply_external_build_result(&mut info, None, false);
assert_eq!(info.build, Some(JavaBuildStatus::Succeeded));
}
#[test]
fn clean_external_build_clears_stale_jdt_errors() {
let mut info = JavaLaunchInfo {
error_count: 7,
errors: vec![ide_lsp::jdtls::JavaCompileError {
path: "src/main/java/Main.java".into(),
line: 0,
column: 0,
message: "stale JDT diagnostic".into(),
}],
..Default::default()
};
apply_external_build_result(&mut info, None, true);
assert_eq!(info.build, Some(JavaBuildStatus::Succeeded));
assert_eq!(info.error_count, 0);
assert!(info.errors.is_empty());
}
#[test]
fn library_scopes_filter_jdt_classpaths_and_module_paths() {
use ide_core::settings::{DependencyScope, ProjectLibrary};
+4
Переглянути файл
@@ -15,6 +15,7 @@ pub mod doc;
pub mod docker;
pub mod env;
pub mod fs;
pub mod heap;
pub mod history;
pub mod java;
pub mod jvm_icons;
@@ -26,6 +27,7 @@ mod python;
pub mod refactor;
pub mod routes;
pub mod run;
pub mod scaffold;
pub mod sdk;
pub mod search;
pub mod spell;
@@ -78,9 +80,11 @@ pub fn register_all(d: &mut crate::rpc::Dispatcher) {
coverage::register(d);
build::register(d);
tools::register(d);
scaffold::register(d);
sdk::register(d);
java::register(d);
debug::register(d);
heap::register(d);
bytes::register(d);
vcs::register(d);
doc::register(d);
+82
Переглянути файл
@@ -0,0 +1,82 @@
//! File › New Project (`ide-scaffold`).
use crate::Host;
use crate::ops::Channel;
use crate::rpc::Dispatcher;
use ide_core::{IdeError, IdeResult, IpcPath, RequestId};
use ide_scaffold::{CreateRequest, Field, GeneratorInfo, Requirement, ScaffoldEvent, Values};
fn join_err(e: impl std::fmt::Display) -> IdeError {
IdeError::internal(e.to_string())
}
/// The project generators File › New Project offers, in list order.
pub fn scaffold_catalog(_host: &Host, _window: &str) -> Vec<GeneratorInfo> {
ide_scaffold::catalog()
}
/// The fields a generator's project service adds for `values` (Spring Initializr's
/// Boot versions and dependencies…); empty for other generators. Cancellable with
/// `cancel_request(request)`.
pub async fn scaffold_options(
host: &Host,
_window: &str,
generator: String,
values: Values,
request: RequestId,
) -> IdeResult<Vec<Field>> {
let cancel = host.begin_request(request);
let result = tokio::task::spawn_blocking(move || ide_scaffold::options(&generator, &values, &cancel))
.await
.map_err(join_err);
host.end_request(request);
result?
}
/// The programs a generator needs for `values`, and whether they are installed.
pub async fn scaffold_requirements(
_host: &Host,
_window: &str,
generator: String,
values: Values,
) -> IdeResult<Vec<Requirement>> {
tokio::task::spawn_blocking(move || ide_scaffold::requirements(&generator, &values)).await.map_err(join_err)?
}
/// Creates a project, streaming its steps and tool output through `on_event`, and
/// returns its directory, trusted: the user just created it. Cancellable with
/// `cancel_request(request)`; a failed or cancelled run leaves no files behind.
pub async fn scaffold_create(
host: &Host,
_window: &str,
req: CreateRequest,
request: RequestId,
on_event: Channel<ScaffoldEvent>,
) -> IdeResult<IpcPath> {
let cancel = host.begin_request(request);
let result = tokio::task::spawn_blocking(move || {
ide_scaffold::create(&req, &cancel, &|e| {
let _ = on_event.send(e);
})
})
.await
.map_err(join_err);
host.end_request(request);
let dir = result??;
host.set_project_trust(&dir, true, false).await?;
Ok(dir.into())
}
/// Commands of this module the remote server answers.
pub(crate) fn register(d: &mut Dispatcher) {
d.add("scaffold_catalog", |c| async move { Ok(scaffold_catalog(&c.host, &c.window)) });
d.add("scaffold_options", |c| async move {
scaffold_options(&c.host, &c.window, c.arg("generator")?, c.arg("values")?, c.arg("request")?).await
});
d.add("scaffold_requirements", |c| async move {
scaffold_requirements(&c.host, &c.window, c.arg("generator")?, c.arg("values")?).await
});
d.add("scaffold_create", |c| async move {
scaffold_create(&c.host, &c.window, c.arg("req")?, c.arg("request")?, c.sender("onEvent")?).await
});
}
+3 -1
Переглянути файл
@@ -79,6 +79,8 @@ pub(crate) fn search_project(
let language = project.languages.detect(path, text.lines().next());
Some(syntax.literals(&language, text)?.into_iter().map(literal_range).collect())
};
let defaults = project.documents.file_defaults();
let decode = |path: &Path, bytes: &[u8]| ide_vfs::decode_text(bytes, defaults.fallback_for(path));
let mut total = SearchSummary::default();
for dir in std::iter::once(root).filter(|_| include_root).chain(&content_roots) {
let mut remaining = query.clone();
@@ -89,7 +91,7 @@ pub(crate) fn search_project(
&remaining,
Some(&filter),
Some(&literals),
&|p| project.documents.search_text_of(p),
ide_search::TextSource { overlay: &|p| project.documents.search_text_of(p), decode: &decode },
cancel,
&|mut file| {
if dir != root {
+30
Переглянути файл
@@ -0,0 +1,30 @@
//! Plugin command requests shared by local IPC and remote RPC.
use super::PluginHost;
use ide_core::{IdeResult, RequestId};
use ide_plugin::Invocation;
use std::sync::Arc;
pub async fn run_command(
host: &crate::Host,
plugins: Arc<PluginHost>,
plugin_id: String,
command_id: String,
arg: Option<String>,
invocation: Invocation,
request: Option<RequestId>,
) -> IdeResult<Option<String>> {
let cancel = request.map(|id| match invocation.client.as_deref() {
Some(window) => host.begin_project_read_request(window, id),
None => host.begin_request(id),
});
let result = host
.blocking(move || {
plugins.run_command_with_cancel(&plugin_id, &command_id, arg.as_deref(), invocation, cancel.as_ref())
})
.await;
if let Some(request) = request {
host.end_request(request);
}
result
}
+27 -1
Переглянути файл
@@ -31,6 +31,7 @@ const MAX_STAGED_FILES: usize = 256;
const MAX_STAGED_BYTES: usize = 16 << 20;
/// `moved` links followed before giving up (a loop, or a misconfiguration).
const MAX_MOVES: usize = 3;
const MAX_SUGGESTIONS: usize = 8;
/// A marketplace entry with what is installed.
#[derive(Clone, Debug, Serialize, Deserialize, specta::Type)]
@@ -42,6 +43,9 @@ pub struct MarketPlugin {
pub installed: Option<String>,
/// Whether this IDE can load `entry` (the newest version it can, else the newest).
pub compatible: bool,
/// The publisher's display name, as the index lists it.
#[serde(default)]
pub publisher_name: Option<String>,
}
impl MarketPlugin {
@@ -76,6 +80,19 @@ fn safe_icon(icon: PluginIcon) -> Option<PluginIcon> {
Some(PluginIcon { light: clean(icon.light)?, dark: icon.dark.and_then(clean) })
}
/// Paths an index suggests a plugin for, as the UI may look them up: relative,
/// inside the project, and few.
fn safe_suggestions(paths: Vec<String>) -> Vec<String> {
let ok = |p: &String| {
!p.is_empty()
&& p.len() <= 128
&& !p.starts_with('/')
&& !p.contains(['\\', ':', '\0', '*', '?'])
&& p.split('/').all(|part| !matches!(part, "" | "." | ".."))
};
paths.into_iter().filter(ok).take(MAX_SUGGESTIONS).collect()
}
/// Reads `url`: `http(s)://…`, `file://…` or a local path.
pub fn read_url(url: &str, max: usize) -> Result<Vec<u8>, String> {
if url.starts_with("https://") || url.starts_with("http://") {
@@ -253,7 +270,9 @@ impl PluginHost {
.map(|p| p.manifest().version.clone());
let mut entry = entry.clone();
entry.icon = entry.icon.and_then(safe_icon);
MarketPlugin { entry, installed, compatible }
entry.suggest = safe_suggestions(std::mem::take(&mut entry.suggest));
let publisher_name = index.publishers.iter().find(|p| p.id == entry.publisher).map(|p| p.name.clone());
MarketPlugin { entry, installed, compatible, publisher_name }
})
.collect()
}
@@ -482,4 +501,11 @@ mod tests {
assert_eq!(resolve(r"C:\market\index.json", "a.oiplugin"), r"C:\market\a.oiplugin");
assert_eq!(resolve("https://x.dev/index.json", "https://cdn.dev/a"), "https://cdn.dev/a");
}
#[test]
fn suggestions_stay_inside_the_project() {
let paths = ["pom.xml", "config/routes.rb", "/etc/passwd", "../x", "a//b", "C:x", "*.php", ""];
assert_eq!(safe_suggestions(paths.iter().map(|p| p.to_string()).collect()), ["pom.xml", "config/routes.rb"]);
assert_eq!(safe_suggestions(vec!["a".into(); 20]).len(), MAX_SUGGESTIONS);
}
}
+26 -1
Переглянути файл
@@ -17,6 +17,7 @@
mod api;
mod auth;
pub mod commands;
pub mod commit;
mod database;
mod dependencies;
@@ -1367,9 +1368,23 @@ impl PluginHost {
command: &str,
arg: Option<&str>,
invocation: Invocation,
) -> IdeResult<Option<String>> {
self.run_command_with_cancel(id, command, arg, invocation, None)
}
/// Runs a command with request-scoped cancellation, without cancelling other queued calls.
pub fn run_command_with_cancel(
&self,
id: &str,
command: &str,
arg: Option<&str>,
invocation: Invocation,
cancel: Option<&ide_core::CancellationToken>,
) -> IdeResult<Option<String>> {
let (cmd, arg) = (command.to_owned(), arg.map(str::to_owned));
self.code_call(id, invocation, "running the command", move |rt, inv| rt.run_command(&cmd, arg.as_deref(), inv))
self.code_call_cancellable(id, invocation, "running the command", cancel, move |rt, inv| {
rt.run_command(&cmd, arg.as_deref(), inv)
})
}
/// Content of a declared view, or of a custom editor (`invocation.editor`
@@ -1513,6 +1528,9 @@ impl PluginHost {
budget: Option<Duration>,
f: impl Fn(&mut Runtime, Invocation) -> Result<R, CallError> + Clone + Send + 'static,
) -> IdeResult<R> {
if cancel.is_some_and(|token| token.is_cancelled()) {
return Err(IdeError::cancelled());
}
let deadline = budget.map(|budget| Instant::now() + budget);
let fallback = ide_core::CancellationToken::new();
let plugin = self.get(id)?;
@@ -1549,6 +1567,9 @@ impl PluginHost {
}
// Twice at most: the instance may be reclaimed between picking it and queueing the call.
for attempt in 0..2 {
if cancel.is_some_and(|token| token.is_cancelled()) {
return Err(IdeError::cancelled());
}
let (actor, activated) = self.ensure_active(&plugin, client.as_deref())?;
if activated {
self.changed();
@@ -1567,6 +1588,10 @@ impl PluginHost {
}
(None, None) => actor.call(None, move |rt| call(rt, inv)),
};
if cancel.is_some_and(|token| token.is_cancelled()) {
self.reap(&plugin, &actor, client.as_deref());
return Err(IdeError::cancelled());
}
if deadline.is_some() && matches!(result, Err(ActorError::SoftTimeout)) {
self.reap(&plugin, &actor, client.as_deref());
return Err(IdeError::new(
+2 -1
Переглянути файл
@@ -457,7 +457,8 @@ impl RemotePlugins {
selected_paths: paths.unwrap_or_default(),
global_settings: matches!(settings_scope, Some(SettingsScope::Global)),
};
blocking(move || host.run_command(&plugin_id, &command_id, arg.as_deref(), invocation)).await
super::commands::run_command(&c.host, host, plugin_id, command_id, arg, invocation, c.arg("request")?)
.await
}
});
let h = host.clone();
+1
Переглянути файл
@@ -210,6 +210,7 @@ impl Host {
let bundles: Vec<PathBuf> = ide_tools::managed_bundle(&data_dir, "java-debug")
.into_iter()
.chain(ide_sdk::java::usages_bundle(&data_dir))
.chain(ide_sdk::java::jvm_inspect_bundle(&data_dir))
.chain(ide_sdk::java::kotlin_bridge_bundle(&data_dir))
.collect();
let min_java = ide_sdk::java::jdtls_min_java(cfg);
+2
Переглянути файл
@@ -221,6 +221,8 @@ pub fn restricted_command_allowed(cmd: &str) -> bool {
"term_",
"settings_",
"project_trust_",
// New Project writes only where the user chose and runs no code of an open project.
"scaffold_",
"ssh_",
"git_credential",
]
+90
Переглянути файл
@@ -0,0 +1,90 @@
//! The JVM memory commands resolve the caller's project through the window label, so a client
//! that sends no real window (the desktop shell once passed `""`) never reaches the debugger,
//! and a project in Restricted Mode never reads the debuggee's memory.
use ide_core::{IdeError, NullSink};
use ide_host::rpc::{self, Dispatcher, Invoke};
use ide_host::{Host, HostConfig};
use ide_jsonrpc::{Handler, Peer};
use serde_json::{Value, json};
use std::sync::{Arc, OnceLock};
struct Silent;
impl Handler for Silent {
fn on_notification(&self, _: &str, _: Value) {}
fn on_request(&self, method: &str, _: Value) -> Result<Value, (i64, String)> {
Err((ide_jsonrpc::codes::METHOD_NOT_FOUND, method.to_owned()))
}
}
async fn invoke(peer: &Peer, window: &str, cmd: &str, args: Value) -> Result<Value, IdeError> {
let call = Invoke { window: window.into(), cmd: cmd.into(), args };
peer.request::<_, Value>("invoke", call, None).await.map_err(rpc::error_from_reply)
}
const NO_SESSION: i64 = 4242;
fn calls() -> [(&'static str, Value); 4] {
let object = json!({"generation": 1, "id": 7});
[
("debug_jvm_memory_classes", json!({"session": NO_SESSION})),
("debug_jvm_memory_instances", json!({"session": NO_SESSION, "className": "java.lang.String", "limit": 10})),
("debug_jvm_memory_fields", json!({"session": NO_SESSION, "object": object})),
("debug_jvm_memory_referrers", json!({"session": NO_SESSION, "object": object, "limit": 10})),
]
}
/// A host with project window "main", trusted or not; keep the temp dir alive.
fn host(trusted: bool) -> (Arc<Host>, tempfile::TempDir) {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path().join("project");
std::fs::create_dir(&root).unwrap();
let host = Arc::new(Host::new(HostConfig {
config_dir: tmp.path().join("config"),
data_dir: tmp.path().join("data"),
local_data_dir: tmp.path().join("data"),
languages: Arc::new(ide_language::LanguageRegistry::load(None).unwrap()),
syntax: Arc::default(),
sink_for: Box::new(|_| Arc::new(NullSink)),
owner: |w| w,
runtime: tokio::runtime::Handle::current(),
project_environment: false,
}));
host.trust.decide(&root, trusted, false).unwrap();
host.open_project("main", &root).unwrap();
(host, tmp)
}
async fn serve(host: Arc<Host>) -> Peer {
let (client_io, server_io) = tokio::io::duplex(1 << 20);
let (sr, sw) = tokio::io::split(server_io);
tokio::spawn(rpc::serve(host, Dispatcher::standard(), Arc::new(OnceLock::new()), sr, sw));
let (cr, cw) = tokio::io::split(client_io);
let peer = Peer::connect(cr, cw, Arc::new(Silent));
peer.request::<_, Value>("initialize", json!({"version": rpc::PROTOCOL_VERSION}), None).await.unwrap();
peer
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn memory_commands_reach_the_debugger_from_a_trusted_window() {
let (host, _tmp) = host(true);
let peer = serve(host).await;
for (command, args) in calls() {
let error = invoke(&peer, "main", command, args.clone()).await.unwrap_err();
assert!(
error.message.contains(&format!("no debug session {NO_SESSION}")),
"{command} should get as far as the session lookup: {error:?}"
);
let error = invoke(&peer, "", command, args).await.unwrap_err();
assert!(error.message.contains("no project"), "{command} without a window label: {error:?}");
}
}
/// The commands check trust themselves too (spec §13), not only through the IPC gate.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn class_table_is_refused_in_restricted_mode() {
let (host, _tmp) = host(false);
let session = ide_core::DebugSessionId(NO_SESSION as u32);
let error = ide_host::ops::debug::debug_jvm_memory_classes(&host, "main", session).await.unwrap_err();
assert!(error.message.contains("Restricted Mode"), "{error:?}");
}
Loaded 100 of 452 files, more files were not shown because too many files have changed in this diff. Show more