mirror of
https://github.com/openclaw/openclaw.git
synced 2026-04-28 20:46:57 +02:00
refactor(qa): split Matrix QA into optional plugin (#66723)
Merged via squash.
Prepared head SHA: 27241bd089
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Co-authored-by: gumadeiras <5599352+gumadeiras@users.noreply.github.com>
Reviewed-by: @gumadeiras
This commit is contained in:
Gustavo Madeira Santana
committed by
GitHub
GitHub
parent
3425823dfb
commit
82a2db71e8
@@ -197,6 +197,12 @@ Use `--link` to avoid copying a local directory (adds to `plugins.load.paths`):
|
||||
openclaw plugins install -l ./my-plugin
|
||||
```
|
||||
|
||||
Repo QA example (source-linked dev surface; not shipped in packaged installs):
|
||||
|
||||
```bash
|
||||
openclaw plugins install -l ./extensions/qa-matrix
|
||||
```
|
||||
|
||||
`--force` is not supported with `--link` because linked installs reuse the
|
||||
source path instead of copying over a managed install target.
|
||||
|
||||
|
||||
@@ -67,9 +67,13 @@ These commands sit beside the main test suites when you need QA-lab realism:
|
||||
- Starts the Docker-backed QA site for operator-style QA work.
|
||||
- `pnpm openclaw qa matrix`
|
||||
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver.
|
||||
- This QA host is repo/dev-only today. Packaged OpenClaw installs do not ship
|
||||
`qa-lab`, so they do not expose `openclaw qa`.
|
||||
- Repo checkouts can link the in-tree plugin directly:
|
||||
`openclaw plugins install -l ./extensions/qa-matrix`.
|
||||
- Provisions three temporary Matrix users (`driver`, `sut`, `observer`) plus one private room, then starts a QA gateway child with the real Matrix plugin as the SUT transport.
|
||||
- Uses the pinned stable Tuwunel image `ghcr.io/matrix-construct/tuwunel:v1.5.1` by default. Override with `OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE` when you need to test a different image.
|
||||
- Matrix currently supports only `--credential-source env` because the lane provisions disposable users locally.
|
||||
- Matrix does not expose shared credential-source flags because the lane provisions disposable users locally.
|
||||
- Writes a Matrix QA report, summary, and observed-events artifact under `.artifacts/qa-e2e/...`.
|
||||
- `pnpm openclaw qa telegram`
|
||||
- Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env.
|
||||
@@ -170,11 +174,12 @@ Adding a channel to the markdown QA system requires exactly two things:
|
||||
1. A transport adapter for the channel.
|
||||
2. A scenario pack that exercises the channel contract.
|
||||
|
||||
Do not add a channel-specific QA runner when the shared `qa-lab` runner can
|
||||
Do not add a new top-level QA command root when the shared `qa-lab` host can
|
||||
own the flow.
|
||||
|
||||
`qa-lab` owns the shared mechanics:
|
||||
`qa-lab` owns the shared host mechanics:
|
||||
|
||||
- the `openclaw qa` command root
|
||||
- suite startup and teardown
|
||||
- worker concurrency
|
||||
- artifact writing
|
||||
@@ -182,8 +187,9 @@ own the flow.
|
||||
- scenario execution
|
||||
- compatibility aliases for older `qa-channel` scenarios
|
||||
|
||||
The channel adapter owns the transport contract:
|
||||
Runner plugins own the transport contract:
|
||||
|
||||
- how `openclaw qa <runner>` is mounted beneath the shared `qa` root
|
||||
- how the gateway is configured for that transport
|
||||
- how readiness is checked
|
||||
- how inbound events are injected
|
||||
@@ -194,17 +200,20 @@ The channel adapter owns the transport contract:
|
||||
|
||||
The minimum adoption bar for a new channel is:
|
||||
|
||||
1. Implement the transport adapter on the shared `qa-lab` seam.
|
||||
2. Register the adapter in the transport registry.
|
||||
3. Keep transport-specific mechanics inside the adapter or the channel harness.
|
||||
4. Author or adapt markdown scenarios under `qa/scenarios/`.
|
||||
5. Use the generic scenario helpers for new scenarios.
|
||||
6. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
|
||||
1. Keep `qa-lab` as the owner of the shared `qa` root.
|
||||
2. Implement the transport runner on the shared `qa-lab` host seam.
|
||||
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
|
||||
4. Mount the runner as `openclaw qa <runner>` instead of registering a competing root command.
|
||||
Runner plugins should declare `qaRunners` in `openclaw.plugin.json` and export a matching `qaRunnerCliRegistrations` array from `runtime-api.ts`.
|
||||
Keep `runtime-api.ts` light; lazy CLI and runner execution should stay behind separate entrypoints.
|
||||
5. Author or adapt markdown scenarios under `qa/scenarios/`.
|
||||
6. Use the generic scenario helpers for new scenarios.
|
||||
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.
|
||||
|
||||
The decision rule is strict:
|
||||
|
||||
- If behavior can be expressed once in `qa-lab`, put it in `qa-lab`.
|
||||
- If behavior depends on one channel transport, keep it in that adapter or plugin harness.
|
||||
- If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
|
||||
- If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in `suite.ts`.
|
||||
- If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.
|
||||
|
||||
|
||||
@@ -56,6 +56,8 @@ Use it for:
|
||||
plugin before runtime loads
|
||||
- static capability ownership snapshots used for bundled compat wiring and
|
||||
contract coverage
|
||||
- cheap QA runner metadata that the shared `openclaw qa` host can inspect
|
||||
before plugin runtime loads
|
||||
- channel-specific config metadata that should merge into catalog and validation
|
||||
surfaces without loading runtime
|
||||
- config UI hints
|
||||
@@ -158,6 +160,7 @@ Those belong in your plugin code and `package.json`.
|
||||
| `providerAuthChoices` | No | `object[]` | Cheap auth-choice metadata for onboarding pickers, preferred-provider resolution, and simple CLI flag wiring. |
|
||||
| `activation` | No | `object` | Cheap activation hints for provider, command, channel, route, and capability-triggered loading. Metadata only; plugin runtime still owns actual behavior. |
|
||||
| `setup` | No | `object` | Cheap setup/onboarding descriptors that discovery and setup surfaces can inspect without loading plugin runtime. |
|
||||
| `qaRunners` | No | `object[]` | Cheap QA runner descriptors used by the shared `openclaw qa` host before plugin runtime loads. |
|
||||
| `contracts` | No | `object` | Static bundled capability snapshot for speech, realtime transcription, realtime voice, media-understanding, image-generation, music-generation, video-generation, web-fetch, web search, and tool ownership. |
|
||||
| `channelConfigs` | No | `Record<string, object>` | Manifest-owned channel config metadata merged into discovery and validation surfaces before runtime loads. |
|
||||
| `skills` | No | `string[]` | Skill directories to load, relative to the plugin root. |
|
||||
@@ -219,6 +222,29 @@ uses this metadata for diagnostics without importing plugin runtime code.
|
||||
Use `activation` when the plugin can cheaply declare which control-plane events
|
||||
should activate it later.
|
||||
|
||||
## qaRunners reference
|
||||
|
||||
Use `qaRunners` when a plugin contributes one or more transport runners beneath
|
||||
the shared `openclaw qa` root. Keep this metadata cheap and static; the plugin
|
||||
runtime still owns actual CLI registration through a lightweight
|
||||
`runtime-api.ts` surface that exports `qaRunnerCliRegistrations`.
|
||||
|
||||
```json
|
||||
{
|
||||
"qaRunners": [
|
||||
{
|
||||
"commandName": "matrix",
|
||||
"description": "Run the Docker-backed Matrix live QA lane against a disposable homeserver"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Type | What it means |
|
||||
| ------------- | -------- | -------- | ------------------------------------------------------------------ |
|
||||
| `commandName` | Yes | `string` | Subcommand mounted beneath `openclaw qa`, for example `matrix`. |
|
||||
| `description` | No | `string` | Fallback help text used when the shared host needs a stub command. |
|
||||
|
||||
This block is metadata only. It does not register runtime behavior, and it does
|
||||
not replace `register(...)`, `setupEntry`, or other runtime/plugin entrypoints.
|
||||
Current consumers use it as a narrowing hint before broader plugin loading, so
|
||||
|
||||
Reference in New Issue
Block a user