diff --git a/.github/workflows/publish-cli.yml b/.github/workflows/publish-cli.yml index dd709a162..04dd79fef 100644 --- a/.github/workflows/publish-cli.yml +++ b/.github/workflows/publish-cli.yml @@ -16,9 +16,6 @@ on: permissions: contents: write -env: - SCOPE_DOC_REPO: growth-ecosystems/scope-doc - jobs: test: name: Build & Test CLI @@ -64,10 +61,9 @@ jobs: echo "version=${NEW_VERSION}" >> "$GITHUB_OUTPUT" - name: Build CLI bundle - working-directory: apps/cli env: SCOPE_DEFAULT_API_URL: ${{ vars.SCOPE_API_URL }} - run: pnpm build + run: pnpm build:cli - name: Run integration tests run: pnpm vitest run --config vitest.integration.config.ts apps/cli/src/bundle.integration.test.ts @@ -87,19 +83,8 @@ jobs: runs-on: ubuntu-latest steps: - - name: Generate GitHub App token - id: app-token - uses: actions/create-github-app-token@v1 - with: - app-id: ${{ secrets.FLUX_APP_ID }} - private-key: ${{ secrets.FLUX_APP_PRIVATE_KEY }} - owner: growth-ecosystems - repositories: scope-core,scope-doc - - name: Checkout repository uses: actions/checkout@v4 - with: - token: ${{ steps.app-token.outputs.token }} - name: Download bundle uses: actions/download-artifact@v4 @@ -116,18 +101,18 @@ jobs: git tag -a "cli/v${VERSION}" -m "CLI release v${VERSION}" git push origin "cli/v${VERSION}" - - name: Create release on scope-doc + - name: Create release env: - GH_TOKEN: ${{ steps.app-token.outputs.token }} + GH_TOKEN: ${{ github.token }} VERSION: ${{ needs.test.outputs.version }} run: | gh release create "cli/v${VERSION}" \ - --repo "$SCOPE_DOC_REPO" \ + --repo "$GITHUB_REPOSITORY" \ --title "Scope CLI v${VERSION}" \ --notes "## Installation \`\`\`bash - gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh -H \"Accept: application/vnd.github.raw\" | bash + gh api repos/${GITHUB_REPOSITORY}/contents/website/install-cli.sh -H \"Accept: application/vnd.github.raw\" | bash \`\`\` Or download \`scope.mjs\` from this release and place it in your PATH. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4eb02c38d..2b8815c77 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -161,8 +161,10 @@ MongoDB is CosmosDB-compatible — avoid MongoDB features that CosmosDB's MongoD (`.agents/skills/rust-best-practices/SKILL.md`). - **Databases**: keep MongoDB usage CosmosDB-compatible (see migrations above). - **Documentation is not optional**: if you add or change a component, API, data model, pattern, or - deployment behavior, update the relevant doc in [`docs/`](./docs/) (or add one and link it from the - README) as the last step before opening your PR. + deployment behavior, update the relevant technical doc in [`docs/`](./docs/) and user-facing + guide in [`website/src/content/docs/`](./website/src/content/docs/) as appropriate. Add new + website pages to the sidebar in [`website/astro.config.mjs`](./website/astro.config.mjs) + as the last step before opening your PR. ## Submitting a pull request @@ -238,6 +240,9 @@ installed dependencies. - Start with the [system architecture](./docs/architecture/system-architecture.md) and the [app design](./docs/architecture/app-design.md) docs for the big picture. -- The [README](./README.md) introduces the platform, provides a local quick start, and links the full documentation index. +- The [official documentation website](https://microsoft.github.io/scope/) includes + [local setup](https://microsoft.github.io/scope/getting-started/local-development/), + the [development guide](https://microsoft.github.io/scope/resources/development/), and + [contribution guidance](https://microsoft.github.io/scope/resources/contributing/). - Browse [open issues](https://github.com/microsoft/scope/issues) for bugs and proposed improvements. Discuss larger changes in an issue before starting work. diff --git a/README.md b/README.md index 870db6f2b..2498cfd61 100644 --- a/README.md +++ b/README.md @@ -5,314 +5,22 @@

An open-source agentic experience evaluation platform.

- Evaluate how AI agents use your product, approach real tasks and respond to feedback. Compare accross surfaces (CLI, MCP, Skills, doc, ...), context, tasks, operating systems, ... + Evaluate how AI agents use your product, approach real tasks, and respond to feedback. Define success with reusable criteria, follow each run live, and inspect the evidence behind every result through the Portal or CLI.

- Node.js 22 - pnpm 10.29.1 - TypeScript MIT License

- -

- Why Scope | - How it works | - Get started | - Documentation | - Contributing -

-## Why Scope? - -A working result is only part of the agentic experience. Scope helps you -evaluate the output, the steps an agent took, and how it responded to feedback. -Use repeatable evaluations to understand both successes and failures. - -- **Define what success means.** Organize evaluation criteria into a directed - acyclic graph (DAG), with dependencies between checks. -- **Inspect the evidence.** Follow live logs and review generated files, - workspace snapshots, criteria results, and captured agent activity. -- **Evaluate changes.** Reuse tasks and saved profiles to understand how the - agent, model, skills, tools, or starting codebase affect the experience. -- **Automate evaluations.** Submit and manage runs from the CLI or REST API, - and explore results, reports, and insights in the Portal. - -Scope is for product managers, developers, researchers, and teams evaluating how their software is being used by coding agents. -Results describe the tasks and configurations you tested, not a universal agent -ranking. The automated Judge can make mistakes; important conclusions need human -review. - -## How it works - -```mermaid -flowchart LR - Client["Portal or CLI"] --> API["API"] - API --> DB[("MongoDB")] - Scheduler["Scheduler"] -->|claim pending runs| DB - Scheduler --> Queue["Storage Queues"] - Queue --> Worker["Coding agent worker"] - Worker --> Judge["Judge"] - Judge -->|criteria results| DB - Worker -->|logs| Redis["Redis"] - Redis -->|live events| API - Worker -->|artifacts| Blob["Blob Storage"] - Worker -->|run status| DB -``` - -1. **Define** a task, its evaluation criteria, and the agent configuration. -2. **Submit** a request through the Portal or CLI. The scheduler dispatches - pending runs to the appropriate worker queue. -3. **Execute and evaluate.** The worker runs the agent and asks the Judge to - evaluate its output. Runs can include multiple feedback iterations. -4. **Inspect and compare.** Review logs, snapshots, and criteria results. - Post-processing and report workers produce additional analysis when enabled. - -MongoDB holds evaluation configuration and run records. Blob Storage holds larger -artifacts, and Redis relays live events. Local development uses MongoDB, Redis, -Azurite (the Azure Storage emulator), and Lowkey Vault. See the -[system architecture](./docs/architecture/system-architecture.md) for service -details and production deployment considerations. - -### Coding agents - -| Worker | Integration | Getting started | -| --- | --- | --- | -| GitHub Copilot | Agent Client Protocol (ACP) | `pnpm docker:dev:copilot` | -| Claude Code | ACP | `pnpm docker:dev:claude-code` | -| GitHub Copilot on Windows | Windows ACP worker | Deployment-specific; see [system architecture](./docs/architecture/system-architecture.md) | - -The default local workflow below uses the Copilot worker. The -`pnpm docker:dev:all` command enables both local ACP workers, not every -deployment-specific integration. Each provider requires its own credentials -and access to the selected models. - -## Getting started - -### Prerequisites - -| Tool | Requirement | -| --- | --- | -| Git | Clone the repository; fork it first if you plan to contribute. | -| Node.js | Version 22, matching CI. | -| pnpm | Version 10.29.1, pinned in `package.json`. | -| Docker with Compose v2 | Run the local stack. Use a current version with Compose Watch support. | -| [mkcert](https://github.com/FiloSottile/mkcert#installation) | Create trusted HTTPS certificates for the local sign-in emulator. | -| [GitHub CLI](https://cli.github.com/) | Obtain a token for the Copilot quick start with `gh auth login`. | - -The Copilot worker requires an **active GitHub Copilot entitlement** on the -account supplying its token. Authenticating with `gh auth login` alone does not -grant Copilot access. You also need credentials with access to the models used -by the Judge and other AI features. Provider usage may incur charges or consume -quotas. The local backing services don't require an Azure subscription. - -The commands below use a Bash-compatible shell. Rust is only required on the -host if you build or modify the gateway outside Docker. - -### 1. Clone and install - -```bash -git clone https://github.com/microsoft/scope.git -cd scope -corepack enable -pnpm install --frozen-lockfile -``` - -If you cloned a fork, run the remaining commands from that checkout instead. - -### 2. Configure and start the stack - -Authenticate with an account that has an active Copilot entitlement: - -```bash -gh auth login -GITHUB_TOKEN="$(gh auth token)" pnpm docker:dev:copilot -``` - -This builds and starts the Copilot worker, Portal, API, scheduler, Judge, -token manager, gateway, post-processing and reporting services, and their local -dependencies. Database migrations and development agent registration run -automatically. The first build downloads several images and can take some time. - -The startup scripts also generate `.env` from [`.env.base`](./.env.base) and -configure local sign-in over HTTPS. Local authentication requires your browser -to trust the development certificate. The scripts run `mkcert -install` to add -a local certificate authority to the OS/browser trust store and generate the -emulator's `localhost` certificate. On first use, you may be prompted to approve -this trust-store change. See the -[local authentication instructions](./ENV_VARIABLES.md#local-dev-setup-entra-local). - -For persistent overrides, copy [`.env.local.example`](./.env.local.example) to -`.env.local` and edit it locally. Don't put credentials in `.env.base` or commit -them. Avoid editing the generated `.env`, which is regenerated by the scripts. -See the [environment reference](./ENV_VARIABLES.md) for provider credentials, -Judge models, and optional Azure AI Foundry configuration. - -### 3. Open the Portal - -In a second terminal, from the repository root: - -```bash -pnpm open:portal -``` - -The default address is `http://localhost:5100`. Git worktrees get their own port -assignments; `pnpm open:portal` resolves the correct address automatically. -Local sign-in uses the seeded emulator users, such as `alice@entralocal.dev`, -not a production Microsoft Entra tenant. See -[local authentication setup](./ENV_VARIABLES.md#local-dev-setup-entra-local). - -> **Local development is not a security sandbox.** The ACP worker configuration -> mounts the Docker socket so agents can run containers. Use a dedicated -> environment for untrusted tasks, and don't expose this development stack to -> the internet. - -## Run your first evaluation - -### From the Portal - -1. Select or create a project to keep your evaluation data together. -2. Create a task and at least one observable evaluation criterion. For example, - ask the agent to create a Node.js HTTP server and evaluate whether its source - implements a `GET /health` route returning JSON. -3. Open the run submission page, select the running Copilot agent and an - available model, and attach your task and criteria. You can save the agent - configuration as a reusable profile. -4. Submit the run, follow its live logs, and inspect the Judge's results and - workspace artifacts. - -If no models are available, check your provider access and the model-scanner -logs before submitting. A registered agent isn't necessarily running; choose -the worker enabled by your Compose command. - -### From the CLI - -Build the CLI and its shared dependency, then explore the available commands: - -```bash -pnpm build:cli -pnpm cli --help -pnpm cli project list -``` - -Select a project using the ID returned by `project list`: - -```bash -pnpm cli project use -pnpm cli criteria list -pnpm cli run submit --help -pnpm cli run list -``` - -Replace `` with an actual ID. Submission requires a selected project, -a task, and evaluation criteria. Use `--project ` or `SCOPE_PROJECT` -to select a project explicitly in automation. The CLI reads local port settings -from the generated `.env`; set `SCOPE_API_URL` to target another instance. - -### Evaluation building blocks - -| Concept | Purpose | -| --- | --- | -| Tasks and scenarios | Define the work the agent should perform. | -| Criteria | Define observable checks and dependencies for the Judge. | -| Personas | Configure the evaluation perspective and feedback style. | -| Profiles and variations | Save an agent configuration and compare changes against a baseline. | -| Skills and MCP servers | Provide agent instructions and tools through the Model Context Protocol. | -| Codebases | Seed a run with a versioned starting workspace. | - -The YAML files in [`config/`](./config/) are portable examples, not the live -configuration database. MongoDB is the runtime source of truth. Manage -configuration through the Portal or CLI; don't assume that editing an example -file changes an existing evaluation. - -## Development - -The repository is a pnpm workspaces monorepo, primarily TypeScript, with a React -Portal and a Rust AI gateway. - -| Path | Contents | -| --- | --- | -| [`apps/api/`](./apps/api/) | REST API and live event streaming | -| [`apps/portal/`](./apps/portal/) | Web UI and Storybook components | -| [`apps/cli/`](./apps/cli/) | CLI for evaluation management and automation | -| [`apps/scheduler/`](./apps/scheduler/) and [`apps/judge/`](./apps/judge/) | Run dispatch and criteria evaluation | -| [`apps/workers/`](./apps/workers/) | Coding-agent, post-processing, and report workers | -| [`apps/gateway/`](./apps/gateway/) and [`apps/token-manager/`](./apps/token-manager/) | AI traffic capture and credential management | -| [`packages/`](./packages/) | Shared types, storage clients, migrations, and supporting libraries | -| [`config/`](./config/) and [`docs/`](./docs/) | Evaluation examples and documentation | - -Useful commands from the repository root: - -```bash -pnpm test # Unit tests -pnpm lint # Workspace lint and type checks -pnpm build # Workspace builds -pnpm storybook # Portal component catalog -pnpm test:integration # Integration tests; requires .env and backing services -``` - -For service-by-service development, Rust commands, migrations, and code -conventions, read [CONTRIBUTING.md](./CONTRIBUTING.md). - ## Documentation -| Topic | Guide | -| --- | --- | -| Architecture and run lifecycle | [System architecture](./docs/architecture/system-architecture.md) | -| Domain models and API design | [Application design](./docs/architecture/app-design.md) | -| Project organization | [Projects](./docs/architecture/data-organization-projects.md) | -| Evaluation and criteria DAGs | [Criteria provider](./docs/architecture/criteria-provider.md) | -| Agent context | [Skills](./docs/architecture/skills.md) and [codebases](./docs/architecture/codebases.md) | -| Scheduling and recovery | [Queue scheduler](./docs/architecture/queue-scheduler.md) | -| Configuration and authentication | [Environment variables](./ENV_VARIABLES.md) | -| AI limitations and data handling | [Responsible AI FAQ](./docs/responsible-ai-faq.md) | -| More architecture, operations, and research | [Documentation index](./docs/README.md) | - -## Contributing - -Contributions aren't limited to new workers. Documentation improvements, -reproducible bug reports, evaluation examples, tests, and accessibility fixes -are useful ways to get involved. - -Search [existing issues](https://github.com/microsoft/scope/issues) before -reporting a bug or proposing a feature. For larger changes, open an issue to -discuss the approach before implementation. Include reproduction steps and -relevant versions in bug reports, and remove credentials and sensitive run -content from logs. - -Read the [contribution guide](./CONTRIBUTING.md) for setup, conventions, and the -pull request process. Open pull requests against `microsoft/scope` on `main`, -including when working from a fork. Most contributions require the Microsoft -Contributor License Agreement; the CLA bot will guide you through it. - -Everyone participating in the project is expected to follow the -[Code of Conduct](./CODE_OF_CONDUCT.md). - -## Support and security - -For usage questions, bugs, and feature requests, see [SUPPORT.md](./SUPPORT.md). -Report vulnerabilities privately through [SECURITY.md](./SECURITY.md), never -through a public GitHub issue. - -Evaluation artifacts can contain prompts, source code, tool output, and network -metadata. Only use data and credentials approved for your deployment, configure -appropriate access controls, and review generated code before using it. Scope -doesn't certify that an agent or its output is safe or production-ready. Read -the [Responsible AI FAQ](./docs/responsible-ai-faq.md) before running sensitive -or untrusted workloads. - -## License and trademarks - -Scope is licensed under the [MIT License](./LICENSE). Third-party attributions -are in [`NOTICE`](./NOTICE); contributors changing dependencies should follow -the [notice maintenance instructions](./CONTRIBUTING.md#third-party-notices). +Users and contributors, visit the **[official documentation website](https://microsoft.github.io/scope/)** +for setup instructions, evaluation workflows, architecture, and reference guides. -This project may contain trademarks or logos for projects, products, or -services. Authorized use of Microsoft trademarks or logos must follow -[Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). -Use of Microsoft trademarks or logos in modified versions of this project -must not cause confusion or imply Microsoft sponsorship. Any use of -third-party trademarks or logos is subject to those parties' policies. +Start with [Getting started](https://microsoft.github.io/scope/getting-started/access/) +or [Local development](https://microsoft.github.io/scope/getting-started/local-development/). +To help improve Scope, read the [contributor guide](https://microsoft.github.io/scope/resources/contributing/). +For help or vulnerability reporting, see [Support and security](https://microsoft.github.io/scope/resources/support/). diff --git a/apps/cli/README.md b/apps/cli/README.md index 01fd990da..a064426cd 100644 --- a/apps/cli/README.md +++ b/apps/cli/README.md @@ -5,7 +5,7 @@ Command-line interface for the Scope AI coding agent benchmarking platform. ## Installation ```bash -gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh -H "Accept: application/vnd.github.raw" | bash +gh api repos/microsoft/scope/contents/website/install-cli.sh -H "Accept: application/vnd.github.raw" | bash ``` **Prerequisites:** @@ -93,7 +93,7 @@ scope update Or re-run the install script: ```bash -gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh -H "Accept: application/vnd.github.raw" | bash +gh api repos/microsoft/scope/contents/website/install-cli.sh -H "Accept: application/vnd.github.raw" | bash ``` The CLI will also notify you when a newer version is available. Suppress this with: diff --git a/apps/cli/src/commands/update.test.ts b/apps/cli/src/commands/update.test.ts index 004dd6c63..cd25ad51b 100644 --- a/apps/cli/src/commands/update.test.ts +++ b/apps/cli/src/commands/update.test.ts @@ -18,6 +18,7 @@ vi.mock("node:fs", () => ({ // Mock the update-check module vi.mock("../utils/update-check.js", () => ({ fetchLatestVersion: vi.fn(), + RELEASES_REPO: "microsoft/scope", })); import { Command } from "commander"; @@ -98,7 +99,7 @@ describe("update command", () => { expect.stringContaining("New version available: 0.3.0"), ); expect(mockedExecSync).toHaveBeenCalledWith( - expect.stringContaining("gh release download"), + expect.stringContaining('gh release download "cli/v0.3.0" --repo microsoft/scope'), expect.anything(), ); }); @@ -120,7 +121,11 @@ describe("update command", () => { await program.parseAsync(["node", "scope", "update"]); expect(mockedExecSync).toHaveBeenCalledWith( - expect.stringContaining("gh release download"), + expect.stringContaining("gh release list --repo microsoft/scope"), + expect.anything(), + ); + expect(mockedExecSync).toHaveBeenCalledWith( + expect.stringContaining('gh release download "cli/v0.3.0" --repo microsoft/scope'), expect.anything(), ); }); @@ -167,7 +172,9 @@ describe("update command", () => { expect.stringContaining(".tmp"), ); expect(errorSpy).toHaveBeenCalledWith( - expect.stringContaining("Update failed"), + expect.stringContaining( + 'gh api repos/microsoft/scope/contents/website/install-cli.sh -H "Accept: application/vnd.github.raw" | bash', + ), ); }); diff --git a/apps/cli/src/commands/update.ts b/apps/cli/src/commands/update.ts index 2448a952f..2c764e233 100644 --- a/apps/cli/src/commands/update.ts +++ b/apps/cli/src/commands/update.ts @@ -7,9 +7,7 @@ import { chmodSync, renameSync, unlinkSync } from "node:fs"; import { basename, dirname, join, resolve } from "node:path"; import semver from "semver"; import { getCliName } from "../utils/shared.js"; -import { fetchLatestVersion } from "../utils/update-check.js"; - -const REPO = "growth-ecosystems/scope-doc"; +import { fetchLatestVersion, RELEASES_REPO } from "../utils/update-check.js"; function getCliVersion(): string { return process.env.SCOPE_CLI_VERSION ?? "0.1.0-dev"; @@ -51,7 +49,7 @@ export function registerUpdateCommand(program: Command): void { // Resolve tag via gh release list try { targetTag = execSync( - `gh release list --repo ${REPO} --json tagName -q '[.[].tagName | select(startswith("cli/v"))][0]'`, + `gh release list --repo ${RELEASES_REPO} --json tagName -q '[.[].tagName | select(startswith("cli/v"))][0]'`, { encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"] }, ).trim(); } catch { /* ignore */ } @@ -71,7 +69,7 @@ export function registerUpdateCommand(program: Command): void { try { // Download scope.mjs to a temp file, then atomically replace execSync( - `gh release download "${targetTag}" --repo ${REPO} --pattern scope.mjs -O "${tmpFile}" --clobber`, + `gh release download "${targetTag}" --repo ${RELEASES_REPO} --pattern scope.mjs -O "${tmpFile}" --clobber`, { stdio: "inherit" }, ); chmodSync(tmpFile, 0o755); @@ -85,7 +83,7 @@ export function registerUpdateCommand(program: Command): void { try { unlinkSync(tmpFile); } catch { /* ignore */ } console.error( "\nUpdate failed. You can reinstall manually:\n" + - " gh api repos/" + REPO + "/contents/install-cli.sh -H \"Accept: application/vnd.github.raw\" | bash", + " gh api repos/" + RELEASES_REPO + "/contents/website/install-cli.sh -H \"Accept: application/vnd.github.raw\" | bash", ); process.exit(1); } diff --git a/apps/cli/src/utils/update-check.test.ts b/apps/cli/src/utils/update-check.test.ts new file mode 100644 index 000000000..bd8117512 --- /dev/null +++ b/apps/cli/src/utils/update-check.test.ts @@ -0,0 +1,78 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import { execSync } from "node:child_process"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +vi.mock("node:child_process", () => ({ + execSync: vi.fn(), +})); + +const mockedExecSync = vi.mocked(execSync); +const mockedFetch = vi.fn(); + +describe("CLI release source", () => { + beforeEach(() => { + vi.resetModules(); + vi.resetAllMocks(); + vi.stubEnv("SCOPE_RELEASES_URL", ""); + vi.stubEnv("GH_TOKEN", ""); + vi.stubEnv("GITHUB_TOKEN", ""); + vi.stubGlobal("fetch", mockedFetch); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + vi.unstubAllEnvs(); + }); + + it("queries microsoft/scope with gh", async () => { + mockedExecSync.mockReturnValue("cli/v1.2.3\n"); + const { fetchLatestVersion, RELEASES_REPO } = await import("./update-check.js"); + + expect(RELEASES_REPO).toBe("microsoft/scope"); + expect(await fetchLatestVersion()).toBe("1.2.3"); + expect(mockedExecSync).toHaveBeenCalledWith( + expect.stringContaining("gh release list --repo microsoft/scope"), + expect.anything(), + ); + expect(mockedFetch).not.toHaveBeenCalled(); + }); + + it("uses the same repository for the REST fallback", async () => { + mockedExecSync.mockImplementation(() => { + throw new Error("gh unavailable"); + }); + mockedFetch.mockResolvedValue(Response.json([ + { tag_name: "other/v9.0.0" }, + { tag_name: "cli/v1.2.3" }, + ])); + const { fetchLatestVersion } = await import("./update-check.js"); + + expect(await fetchLatestVersion()).toBe("1.2.3"); + expect(mockedFetch).toHaveBeenCalledWith( + "https://api.github.com/repos/microsoft/scope/releases", + expect.anything(), + ); + }); + + it("preserves the custom releases URL without invoking gh", async () => { + vi.stubEnv("SCOPE_RELEASES_URL", "http://localhost:9999/releases"); + mockedFetch.mockResolvedValue(Response.json([{ tag_name: "cli/v2.0.0" }])); + const { fetchLatestVersion } = await import("./update-check.js"); + + expect(await fetchLatestVersion()).toBe("2.0.0"); + expect(mockedExecSync).not.toHaveBeenCalled(); + expect(mockedFetch).toHaveBeenCalledWith( + "http://localhost:9999/releases", + expect.anything(), + ); + }); + + it("returns no version when no CLI release has been published", async () => { + mockedExecSync.mockReturnValue(""); + const { fetchLatestVersion } = await import("./update-check.js"); + + expect(await fetchLatestVersion()).toBeUndefined(); + }); +}); diff --git a/apps/cli/src/utils/update-check.ts b/apps/cli/src/utils/update-check.ts index d0c9d7e93..c26e4ed47 100644 --- a/apps/cli/src/utils/update-check.ts +++ b/apps/cli/src/utils/update-check.ts @@ -70,7 +70,7 @@ export function checkForUpdates(currentVersion: string): () => Promise { }; } -export const RELEASES_REPO = "growth-ecosystems/scope-doc"; +export const RELEASES_REPO = "microsoft/scope"; export const RELEASES_URL = process.env.SCOPE_RELEASES_URL || diff --git a/docs/architecture/cli-distribution.md b/docs/architecture/cli-distribution.md index b46566de1..5b66bf88e 100644 --- a/docs/architecture/cli-distribution.md +++ b/docs/architecture/cli-distribution.md @@ -4,12 +4,15 @@ How the Scope CLI is bundled, distributed, and updated as a standalone tool. ## Overview -The CLI is bundled into a single `.mjs` file using [esbuild](https://esbuild.github.io/), distributed via GitHub Releases on the `scope-doc` repo, and installed using the `gh` CLI. This allows users to run the CLI without checking out the monorepo. +The CLI is bundled into a single `.mjs` file using +[esbuild](https://esbuild.github.io/), distributed via GitHub Releases on +`microsoft/scope`, and installed using the `gh` CLI. This allows users to run +the CLI without checking out the monorepo. ```mermaid flowchart LR - A[scope-core
apps/cli/] -->|publish-cli.yml| B[GitHub Actions] - B -->|gh release create| C[scope-doc releases
scope.mjs] + A[microsoft/scope
apps/cli/] -->|publish-cli.yml| B[GitHub Actions] + B -->|gh release create| C[microsoft/scope releases
scope.mjs] C -->|install-cli.sh| D[User workstation
~/.local/bin/scope] ``` @@ -36,12 +39,13 @@ the CLI's `build` script runs `tsc --noEmit` **before** esbuild: "build:tsc": "tsc --noEmit", // standalone typecheck alias ``` -Because every CI/release entry point invokes the CLI `build` script — `pnpm build` (`pnpm -r build`, -used by the CI **Build** job and `publish-cli.yml`) and `pnpm build:cli` (used by the -**CLI Bundle Integration Tests** job) — the CLI is now typechecked automatically wherever it is -built, with no separate CI step. `tsc` requires the `shared` package's `dist` to exist; every one -of these entry points builds `shared` first (topologically for `pnpm -r`, explicitly for -`build:cli`), which esbuild already required, so there is no new ordering constraint. +Every CI/release entry point invokes the CLI `build` script: `pnpm build` +(`pnpm -r build`, used by the CI **Build** job) and `pnpm build:cli` (used by +`publish-cli.yml` and the **CLI Bundle Integration Tests** job). The CLI is +typechecked automatically wherever it is built, with no separate CI step. +`tsc` requires the `shared` package's `dist` to exist; every one of these entry +points builds `shared` first (topologically for `pnpm -r`, explicitly for +`build:cli`), which esbuild already required. > **Motivation:** In PR #1151 an import of `normalizeUrl` was removed from `criteria.ts` while a > call site remained, so `scope criteria export` threw `normalizeUrl is not defined` at runtime — @@ -75,7 +79,11 @@ In dev mode (`pnpm cli` via tsx), these defines are not applied — the CLI fall ## Versioning -The **source of truth** for the CLI version is the git tag on `scope-core` using the `cli/v*` prefix (e.g. `cli/v0.2.0`). The `apps/cli/package.json` version is `0.0.0-dev` — a placeholder that CI resolves from the latest `cli/v*` tag and then bumps via `pnpm version` during the publish workflow. It is never committed back to `main`. +The **source of truth** for the CLI version is the git tag on `microsoft/scope` +using the `cli/v*` prefix (e.g. `cli/v0.2.0`). The `apps/cli/package.json` +version is `0.0.0-dev`, a placeholder that CI resolves from the latest +`cli/v*` tag and then bumps via `pnpm version` during the publish workflow. +It is never committed back to `main`. - Local builds produce `0.0.0-dev` — clearly indicating a dev build. - Dev mode (`pnpm cli`) reports `0.1.0-dev`. @@ -90,40 +98,55 @@ The publish workflow (`.github/workflows/publish-cli.yml`) is triggered manually 2. Workflow resolves the current version from the latest `cli/v*` tag 3. Bumps `apps/cli/package.json` via `pnpm version` 4. Builds the bundle with prod API URL (`vars.SCOPE_API_URL`) -5. Creates a git tag `cli/v` on scope-core -6. Creates a GitHub Release on `scope-doc` with `scope.mjs` +5. Creates a git tag `cli/v` in the current repository +6. Creates a GitHub Release in the same repository (`github.repository`, + `microsoft/scope` for official releases) with `scope.mjs` + +Tag pushes and release creation use the workflow's built-in GitHub token +with `contents: write`. No cross-repository GitHub App token is needed. ### Required secrets/variables | Name | Type | Purpose | |------|------|---------| -| `SCOPE_DOC_TOKEN` | Secret | PAT with `contents:write` on scope-doc repo | | `SCOPE_API_URL` | Variable | Production API URL injected at build time | ## Installation -Users install via the `gh` CLI (required since the repo is EMU-protected): +Users can fetch and run the installer with an authenticated `gh` CLI: ```bash -gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh -H "Accept: application/vnd.github.raw" | bash +gh api repos/microsoft/scope/contents/website/install-cli.sh -H "Accept: application/vnd.github.raw" | bash ``` -The installer (`install-cli.sh` in scope-doc): -1. Downloads `scope.mjs` from the latest `cli/v*` release +The installer (`website/install-cli.sh` in `microsoft/scope`): +1. Downloads `scope.mjs` from the latest `cli/v*` release in `microsoft/scope` 2. Places it at `~/.local/bin/scope` 3. Makes it executable Prerequisites: Node.js >= 20, `gh` CLI authenticated. +When run directly, the installer also supports `curl` with `GH_TOKEN` or +`GITHUB_TOKEN` if `gh` is unavailable. `SCOPE_INSTALL_DIR` overrides the +destination and must be set on the `bash` side of an install pipeline. +Installation requires a published release with a `scope.mjs` asset; +tags alone are not enough. + ## Update check After each command, the CLI performs a non-blocking check for newer versions: -- Queries the GitHub Releases API on `scope-doc` (3s timeout) +- Queries `microsoft/scope` releases via `gh`, falling back to the GitHub + Releases API (2s timeout per attempt for the background check) - Compares the current embedded version against the latest release tag - If newer, prints a one-line notice with the upgrade command - Suppressed by `SCOPE_NO_UPDATE_CHECK=1` -- Requires `GH_TOKEN` or `GITHUB_TOKEN` for private repo access (silently skips without it) +- Uses `gh` authentication or `GH_TOKEN`/`GITHUB_TOKEN` for authenticated + API access; the REST fallback can also read public releases without a token + +`scope update` uses the same repository for release discovery and asset +downloads. Its manual recovery command fetches `website/install-cli.sh` +from `microsoft/scope`. This is the **one** place in the CLI that calls `fetch` directly rather than the centralized `apiFetch()` wrapper (`apps/cli/src/utils/api-client.ts`): it targets the diff --git a/website/AGENTS.md b/website/AGENTS.md index 9d6e68c0b..fbb051c77 100644 --- a/website/AGENTS.md +++ b/website/AGENTS.md @@ -3,17 +3,20 @@ Project context for AI coding agents working on **scope-doc**, the Starlight-based documentation site for **Scope**. -## What this repo is +## What this directory is -A static documentation site published to GitHub Pages. +A static documentation site published to GitHub Pages, within the +[microsoft/scope](https://github.com/microsoft/scope) monorepo. - Framework: **Astro 6.x** + **@astrojs/starlight** - Package manager: **pnpm** (pinned via `packageManager` in `package.json`) - TypeScript strict - Lives under the `website/` directory of the repo (all site sources, config, and `package.json` are rooted here; run every command from - `website/`) -- Deployed by `.github/workflows/static.yml` (build + deploy jobs), + `website/` unless explicitly directed to the repository root). + The site has its own lockfile and is not part of the root pnpm workspace. +- Deployed by [../.github/workflows/static.yml](../.github/workflows/static.yml) + (build + deploy jobs), which builds from `website/` via a `working-directory` default and `website/**` path filters; both PR and production builds explicitly use `SITE=https://microsoft.github.io` and `BASE_PATH=/scope`. @@ -38,41 +41,40 @@ A static documentation site published to GitHub Pages. `starlight-openapi` config - `dist/` — build output (gitignored) -## The source of truth: scope-core +## The source of truth: microsoft/scope -The product itself lives in the -[scope-core](https://github.com/growth-ecosystems/scope-core) -repository. When writing docs, **read scope-core before writing any -factual claim**. Everything in the documentation \u2014 endpoints, field -names, statuses, worker IDs, behaviors, defaults, error messages, -anything \u2014 MUST be grounded in the source code. Do not invent. If -the source doesn't say it, it doesn't go in the docs; ask the user -or leave it out. +The product and this documentation site live in this same +[microsoft/scope](https://github.com/microsoft/scope) checkout. When writing +docs, **read the local product source before writing any factual claim**. +Everything in the documentation, including endpoints, field names, +statuses, worker IDs, behaviors, defaults, and error messages, MUST be +grounded in the source code. Do not invent. If the source doesn't say it, +it doesn't go in the docs; ask the user or leave it out. -## Where to look in scope-core +## Where to look in this monorepo -Anything that comes from the source — endpoints, field names, -status enums, defaults, validation rules — must be read from -scope-core at the time you write it, not copied from this file. +Read endpoints, field names, status enums, defaults, and validation rules +from this checkout at the time you write, not from this file. Use this map as a starting point; do not treat it as a substitute -for opening the file. +for opening the file. Paths in the table are relative to the repository +root, one directory above `website/`; site paths elsewhere in this guide +are relative to `website/`. -| Topic | File(s) in scope-core | +| Topic | File(s) from the repository root | | --- | --- | -| Workers (allowed IDs, validation) | `packages/shared/src/schemas/request.ts` (`VALID_WORKERS`) | -| Worker display names / labels | the `"name"` field in each worker's agent registration (upsert) payload (e.g. "GitHub Copilot CLI", "Claude Code CLI", "VS Code Copilot") | -| Worker software stacks (pre-installed tools) | `apps/workers/*/src/test-worker.ts` — the `checkTools([...])` array lists every runtime and build tool baked into the container image | -| Request payload, scenario shape | `packages/shared/src/schemas/request.ts` (`CreateRequestInputSchema`, `ScenarioSchema`) | -| Request status / outcome enums | `packages/shared/src/schemas/request.ts` (`RequestStatusSchema`, `RequestOutcomeSchema`) | -| Request lifecycle / scheduler | `apps/api/src/index.ts`, `docs/architecture/queue-scheduler.md` | -| Profile + version schemas | `packages/shared/src/schemas/profile.ts` | -| Criterion schema, DAG rules | `packages/shared/src/schemas/criteria.ts` | -| Route handlers, validation, error codes | `apps/api/src/routes/*.ts` | -| VS Code worker behavior | `docs/architecture/vscode-electron-worker.md`, `vscode-web-worker.md`, `worker-requirements.md` | -| OpenAPI source | `apps/api/src/openapi/registry.ts`; generated snapshot at `src/openapi/scope-openapi.json` | -| Swagger UI | served by the API; check `apps/api/src/index.ts` for the route | - -When in doubt, `grep` scope-core for the symbol or string before +| Agent IDs, names, and registration validation | [packages/shared/src/schemas/agent.ts](../packages/shared/src/schemas/agent.ts), [apps/api/src/routes/agents.ts](../apps/api/src/routes/agents.ts) | +| Worker software stacks (pre-installed tools) | `apps/workers/*/src/test-worker.ts`: the `checkTools([...])` arrays list the tools checked by each worker's smoke test | +| Request payload, scenario shape | [packages/shared/src/schemas/request.ts](../packages/shared/src/schemas/request.ts) (`CreateRequestInputSchema`), [packages/shared/src/schemas/scenario.ts](../packages/shared/src/schemas/scenario.ts) (`ScenarioSchema`) | +| Request status / outcome enums | [packages/shared/src/schemas/request.ts](../packages/shared/src/schemas/request.ts) (`RequestStatusSchema`, `RequestOutcomeSchema`) | +| Request lifecycle / scheduler | [apps/api/src/index.ts](../apps/api/src/index.ts), [apps/scheduler/src/request-scheduler.ts](../apps/scheduler/src/request-scheduler.ts), [docs/architecture/queue-scheduler.md](../docs/architecture/queue-scheduler.md) | +| Profile + version schemas | [packages/shared/src/schemas/profile.ts](../packages/shared/src/schemas/profile.ts) | +| Criterion schema, DAG rules | [packages/shared/src/schemas/criteria.ts](../packages/shared/src/schemas/criteria.ts) | +| Route handlers, validation, error codes | [apps/api/src/routes/](../apps/api/src/routes/) | +| Coding worker behavior | [apps/workers/](../apps/workers/), [docs/architecture/worker-requirements.md](../docs/architecture/worker-requirements.md) | +| OpenAPI source | [apps/api/src/openapi/registry.ts](../apps/api/src/openapi/registry.ts); generated snapshot at [website/src/openapi/scope-openapi.json](src/openapi/scope-openapi.json) | +| Swagger UI | Served by the API; check [apps/api/src/index.ts](../apps/api/src/index.ts) for the route | + +When in doubt, search this checkout for the symbol or string before writing anything in the docs. ## Standing user rulings @@ -171,9 +173,14 @@ pnpm install pnpm test # plugin regressions, using Node's built-in test runner pnpm run build # writes dist/ pnpm run dev # local preview at http://localhost:4321 -pnpm run refresh:openapi # generate the OpenAPI snapshot from scope-core +pnpm run refresh:openapi # generate the OpenAPI snapshot from this monorepo's API ``` +For `refresh:openapi`, install the root workspace dependencies first. +The script runs `pnpm --filter api generate:openapi` from the repository +root, using [apps/api/src/openapi/generate.ts](../apps/api/src/openapi/generate.ts) +to update [src/openapi/scope-openapi.json](src/openapi/scope-openapi.json). + Both `pnpm test` and `pnpm run build` must pass. The public build uses `SITE=https://microsoft.github.io BASE_PATH=/scope pnpm run build`; exercise that configuration when changing links or deployment settings, @@ -184,12 +191,12 @@ page count can indicate a content collection file failed to parse. ## Workflow - Each logical change is its own commit. Push to refresh the PR; CI - rebuilds and redeploys to GH Pages. + tests and builds the site. Deployment to GitHub Pages runs on `main`. - `main` is protected — open a PR, don't push directly. ## When in doubt -1. Read the relevant file in scope-core (schemas first, then route +1. Read the relevant file in this checkout (schemas first, then route handlers). 2. Generate the OpenAPI snapshot with `pnpm run refresh:openapi` and inspect `src/openapi/scope-openapi.json`. diff --git a/website/README.md b/website/README.md index a54ad0414..b0398bd96 100644 --- a/website/README.md +++ b/website/README.md @@ -7,7 +7,16 @@ End-user documentation site for **Scope**, built with and published to GitHub Pages. The product and this documentation site live in -[scope-core](https://github.com/growth-ecosystems/scope-core). +[microsoft/scope](https://github.com/microsoft/scope). Users and contributors +should start at the [official documentation website](https://microsoft.github.io/scope/). + +The root README is a short introduction and entry point to the website. +Keep detailed setup, usage, and contribution guidance in +[src/content/docs/](src/content/docs/) and register new pages in the sidebar. +Local setup lives in +[getting-started/local-development.md](src/content/docs/getting-started/local-development.md); +development, contribution, and support guidance live under +[resources/](src/content/docs/resources/). ## Project structure @@ -37,6 +46,10 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. ## Commands +Run these commands from `website/`. The site has its own +[package.json](package.json) and [pnpm-lock.yaml](pnpm-lock.yaml), separate +from the root pnpm workspace. + | Command | Action | | :--------------------- | :--------------------------------------------------------- | | `pnpm install` | Install dependencies | @@ -44,7 +57,7 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. | `pnpm build` | Build the production site to `./dist/` | | `pnpm preview` | Preview the production build locally | | `pnpm test` | Test site plugins with Node's built-in test runner | -| `pnpm refresh:openapi` | Generate the OpenAPI snapshot from `scope-core` | +| `pnpm refresh:openapi` | Generate the OpenAPI snapshot from this monorepo's API | ## Authoring docs @@ -68,13 +81,15 @@ Sidebar order is defined in `astro.config.mjs`, not by directory order. [`starlight-openapi`](https://starlight-openapi.vercel.app/) — do not edit them by hand. - Run `pnpm refresh:openapi` from this directory after changing API - routes or schemas. It runs `apps/api`'s generator from the same - `scope-core` checkout, so root workspace dependencies must be - installed first. + routes or schemas. It runs `pnpm --filter api generate:openapi` from the + repository root, using [apps/api/src/openapi/generate.ts](../apps/api/src/openapi/generate.ts) + in this same `microsoft/scope` checkout. Install the root workspace + dependencies first; the generator updates + [src/openapi/scope-openapi.json](src/openapi/scope-openapi.json). See [AGENTS.md](AGENTS.md) for conventions, the source-of-truth -policy (everything factual must be grounded in scope-core), and -where to look in scope-core for any given topic. +policy (everything factual must be grounded in this checkout's source), +and where to look in the monorepo for any given topic. ## Deployment diff --git a/website/astro.config.mjs b/website/astro.config.mjs index e6d7d4179..219538f14 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -90,6 +90,7 @@ export default defineConfig({ label: 'Getting Started', items: [ { label: 'Access', slug: 'getting-started/access' }, + { label: 'Local development', slug: 'getting-started/local-development' }, { label: 'Install the CLI', slug: 'getting-started/install-cli' }, { label: 'Your first run', slug: 'getting-started/first-run' }, ], @@ -124,9 +125,17 @@ export default defineConfig({ ], }, ...openAPISidebarGroups, + { + label: 'Contributing', + items: [ + { label: 'Contributor guide', slug: 'resources/contributing' }, + { label: 'Development guide', slug: 'resources/development' }, + ], + }, { label: 'Resources', items: [ + { label: 'Support and security', slug: 'resources/support' }, { label: 'FAQ', slug: 'resources/faq' }, { label: 'Data collection and privacy', slug: 'resources/data-collection' }, { label: 'Troubleshooting', slug: 'resources/troubleshooting' }, diff --git a/website/install-cli.sh b/website/install-cli.sh index 36d397f0e..1bac83dde 100644 --- a/website/install-cli.sh +++ b/website/install-cli.sh @@ -2,7 +2,7 @@ # Scope CLI installer # # Usage: -# gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh -H "Accept: application/vnd.github.raw" | bash +# gh api repos/microsoft/scope/contents/website/install-cli.sh -H "Accept: application/vnd.github.raw" | bash # # Requires: node (>= 20) and either `gh` CLI (authenticated) or GH_TOKEN/GITHUB_TOKEN. # @@ -10,7 +10,7 @@ set -euo pipefail -REPO="growth-ecosystems/scope-doc" +REPO="microsoft/scope" TAG_PREFIX="cli/v" INSTALL_DIR="${SCOPE_INSTALL_DIR:-$HOME/.local/bin}" BINARY_NAME="scope" diff --git a/website/install-cli.test.mjs b/website/install-cli.test.mjs new file mode 100644 index 000000000..f864fa31e --- /dev/null +++ b/website/install-cli.test.mjs @@ -0,0 +1,81 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { test } from 'node:test'; +import { fileURLToPath } from 'node:url'; + +const installer = fileURLToPath(new URL('./install-cli.sh', import.meta.url)); +const releasesEndpoint = 'repos/microsoft/scope/releases'; +const assetUrl = 'https://api.github.com/repos/microsoft/scope/releases/assets/123'; + +for (const client of ['gh', 'curl']) { + test(`installs from microsoft/scope using ${client}`, () => { + const fixture = join(process.cwd(), `.installer-test-${randomUUID()}`); + const bin = join(fixture, 'bin'); + const callsFile = join(fixture, 'calls.jsonl'); + const installDir = client === 'gh' + ? join(fixture, '.local', 'bin') + : join(fixture, 'custom-bin'); + mkdirSync(bin, { recursive: true }); + + try { + symlinkSync(process.execPath, join(bin, 'node')); + for (const command of ['mkdir', 'chmod', 'tr', 'grep']) { + const path = ['/bin', '/usr/bin'].map((dir) => join(dir, command)).find(existsSync); + assert.ok(path, `${command} is required`); + symlinkSync(path, join(bin, command)); + } + + const releases = [ + { tag_name: 'other/v99.0.0', assets: [] }, + { tag_name: 'cli/v1.2.3', assets: [{ name: 'scope.mjs', url: assetUrl }] }, + ]; + const bundle = '#!/usr/bin/env node\nconsole.log("1.2.3");\n'; + writeFileSync(join(bin, client), `#!/usr/bin/env node +import fs from 'node:fs'; +const args = process.argv.slice(2); +fs.appendFileSync(${JSON.stringify(callsFile)}, JSON.stringify(args) + '\\n'); +if (args.includes(${JSON.stringify(client === 'gh' ? releasesEndpoint : `https://api.github.com/${releasesEndpoint}`)})) { + process.stdout.write(${JSON.stringify(JSON.stringify(releases))}); +} else if (args.includes(${JSON.stringify(assetUrl)})) { + const output = args.indexOf('-o'); + if (output >= 0) fs.writeFileSync(args[output + 1], ${JSON.stringify(bundle)}); + else process.stdout.write(${JSON.stringify(bundle)}); +} else { + console.error('Unexpected request: ' + JSON.stringify(args)); + process.exit(1); +} +`, { mode: 0o755 }); + + const result = spawnSync('/bin/bash', [installer], { + encoding: 'utf8', + timeout: 10_000, + env: { + ...process.env, + PATH: bin, + HOME: fixture, + SCOPE_INSTALL_DIR: client === 'gh' ? '' : installDir, + GH_TOKEN: 'mock-token', + GITHUB_TOKEN: '', + }, + }); + + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(result.stdout, /Installed scope 1\.2\.3/); + assert.equal(readFileSync(join(installDir, 'scope'), 'utf8'), bundle); + const calls = readFileSync(callsFile, 'utf8').trim().split('\n').map(JSON.parse); + assert.equal(calls.length, 2); + assert.ok(calls[0].includes(client === 'gh' + ? releasesEndpoint + : `https://api.github.com/${releasesEndpoint}`)); + assert.ok(calls[1].includes(assetUrl)); + } finally { + rmSync(fixture, { recursive: true, force: true }); + } + }); +} diff --git a/website/package.json b/website/package.json index 513b77f94..7fa905d8d 100644 --- a/website/package.json +++ b/website/package.json @@ -8,7 +8,7 @@ "start": "worktree-env && astro dev", "build": "astro build", "preview": "worktree-env && astro preview", - "test": "node --test src/plugins/*.test.mjs", + "test": "node --test install-cli.test.mjs src/plugins/*.test.mjs", "astro": "astro", "refresh:openapi": "cd .. && pnpm --filter api generate:openapi" }, diff --git a/website/src/content/docs/getting-started/access.md b/website/src/content/docs/getting-started/access.md index d2df9aefe..77870050b 100644 --- a/website/src/content/docs/getting-started/access.md +++ b/website/src/content/docs/getting-started/access.md @@ -6,6 +6,10 @@ description: How to reach the Scope Portal. You can access Scope through the web Portal, the REST API, or the `scope` CLI. +This page assumes you have access to an existing deployment. To run Scope +yourself, follow [Local development](/getting-started/local-development/) for +prerequisites, setup, authentication, and a first evaluation. + ## Requirements Your deployment administrator provides the Portal URL and any required @@ -40,7 +44,7 @@ for the required setup. For terminal-based workflows, install the `scope` CLI: ```bash -gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh \ +gh api repos/microsoft/scope/contents/website/install-cli.sh \ -H "Accept: application/vnd.github.raw" | bash ``` diff --git a/website/src/content/docs/getting-started/first-run.md b/website/src/content/docs/getting-started/first-run.md index 714c83335..4bf233d00 100644 --- a/website/src/content/docs/getting-started/first-run.md +++ b/website/src/content/docs/getting-started/first-run.md @@ -14,18 +14,22 @@ profile). Scope creates one **run** per execution attempt — see ## Before you start - You can reach the [Portal](/getting-started/access/) for your - deployment. -- You don't need to set anything up — the catalog ships with example - task prompts, criteria sets, and a default Copilot setup. + deployment, or have followed + [Local development](/getting-started/local-development/). +- Your deployment has a running Copilot worker and access to an available + model. If the catalog doesn't contain suitable task prompts, criteria, or + profiles, create them as part of the steps below. ## 1. Open the Portal Navigate to the Portal URL provided for your deployment, for example **https://your-scope.example.com**. -The home view lists recent runs across all users. The left navigation -takes you to runs, tasks, criteria, profiles, prompt features, and -more. +Select or create a project to keep your evaluation data together. + +The Portal then opens **Statistics** (`/statistics`), which summarizes +benchmark results for the selected project. The left navigation takes you +to runs, tasks, criteria, profiles, prompt features, and more. ## 2. Start a new request diff --git a/website/src/content/docs/getting-started/install-cli.md b/website/src/content/docs/getting-started/install-cli.md index 1cb38a3a2..a1ca260fa 100644 --- a/website/src/content/docs/getting-started/install-cli.md +++ b/website/src/content/docs/getting-started/install-cli.md @@ -10,27 +10,30 @@ and manage profiles — all from your terminal. - **Node.js ≥ 20** — [nodejs.org](https://nodejs.org) - **GitHub CLI (`gh`)** — authenticated with access to the - `growth-ecosystems/scope-doc` repo. - Alternatively, set a `GH_TOKEN` or `GITHUB_TOKEN` environment - variable with `repo` scope. + `microsoft/scope` repo. ## One-liner install ```bash -gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh \ +gh api repos/microsoft/scope/contents/website/install-cli.sh \ -H "Accept: application/vnd.github.raw" | bash ``` -This downloads and installs the latest release to +This downloads and installs the latest `cli/v*` release from +`microsoft/scope` to `~/.local/bin/scope`. Override the location with the `SCOPE_INSTALL_DIR` environment variable: ```bash -SCOPE_INSTALL_DIR=~/bin \ - gh api repos/growth-ecosystems/scope-doc/contents/install-cli.sh \ - -H "Accept: application/vnd.github.raw" | bash +gh api repos/microsoft/scope/contents/website/install-cli.sh \ + -H "Accept: application/vnd.github.raw" | SCOPE_INSTALL_DIR=~/bin bash ``` +The installer requires a published `cli/v*` release with a `scope.mjs` +asset. If none is available, follow +[Local development](/getting-started/local-development/) to build the CLI +from source. + ## Add to PATH If `~/.local/bin` is not already on your `PATH`, add it to your diff --git a/website/src/content/docs/getting-started/local-development.md b/website/src/content/docs/getting-started/local-development.md new file mode 100644 index 000000000..14bb024f4 --- /dev/null +++ b/website/src/content/docs/getting-started/local-development.md @@ -0,0 +1,159 @@ +--- +title: Local development +description: Set up Scope locally and run your first evaluation with the Portal or CLI. +--- + +Use this guide to run your own local Scope stack. If your organization already +provides a deployment, start with [Access](/getting-started/access/) instead. +The default local workflow uses the GitHub Copilot worker. + +## Prerequisites + +| Tool | Requirement | +| --- | --- | +| Git | Clone the repository; fork it first if you plan to contribute. | +| [Node.js](https://nodejs.org/) | Version 22, matching CI. | +| [pnpm](https://pnpm.io/) | Version 10.29.1, pinned in [package.json](https://github.com/microsoft/scope/blob/main/package.json). | +| Docker with Compose v2 | Run the local stack. Use a current version with Compose Watch support. | +| [mkcert](https://github.com/FiloSottile/mkcert#installation) | Create trusted HTTPS certificates for the local sign-in emulator. | +| [GitHub CLI](https://cli.github.com/) | Obtain a token for the Copilot quick start with `gh auth login`. | + +The Copilot worker requires an **active GitHub Copilot entitlement** on the +account supplying its token. Authenticating with `gh auth login` alone does not +grant Copilot access. You also need credentials with access to the models used +by the Judge and other AI features. Provider usage may incur charges or consume +quotas. The local backing services don't require an Azure subscription. + +The commands below use a Bash-compatible shell. Rust is only required on the +host if you build or modify the gateway outside Docker. + +## 1. Clone and install + +```bash +git clone https://github.com/microsoft/scope.git +cd scope +corepack enable +pnpm install --frozen-lockfile +``` + +If you cloned a fork, run the remaining commands from that checkout instead. + +## 2. Configure and start the stack + +Authenticate with an account that has an active Copilot entitlement: + +```bash +gh auth login +GITHUB_TOKEN="$(gh auth token)" pnpm docker:dev:copilot +``` + +This builds and starts the Copilot worker, Portal, API, scheduler, Judge, +token manager, gateway, post-processing and reporting services, and their local +dependencies. Database migrations and development agent registration run +automatically. The first build downloads several images and can take some time. + +The startup scripts generate the local environment file from +[.env.base](https://github.com/microsoft/scope/blob/main/.env.base) and configure +local sign-in over HTTPS. Local authentication requires your browser to trust +the development certificate. The scripts run `mkcert -install` to add a local +certificate authority to the OS/browser trust store and generate the +emulator's `localhost` certificate. On first use, you may be prompted to approve +this trust-store change. See the +[local authentication instructions](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md#local-dev-setup-entra-local). + +For persistent overrides, copy +[.env.local.example](https://github.com/microsoft/scope/blob/main/.env.local.example) +to a file named `.env.local` and edit it locally. Don't put credentials in +[.env.base](https://github.com/microsoft/scope/blob/main/.env.base) or commit +them. Avoid editing the generated `.env`, which is regenerated by the scripts. +See the +[environment reference](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md) +for provider credentials, Judge models, and optional Azure AI Foundry +configuration. + +## 3. Open the Portal + +In a second terminal, from the repository root: + +```bash +pnpm open:portal +``` + +The default address is `http://localhost:5100`. Git worktrees get their own port +assignments; `pnpm open:portal` resolves the correct address automatically. +Local sign-in uses the seeded emulator users, such as `alice@entralocal.dev`, +not a production Microsoft Entra tenant. See +[local authentication setup](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md#local-dev-setup-entra-local). + +:::caution[Local development is not a security sandbox] +The ACP worker configuration mounts the Docker socket so agents can run +containers. Use a dedicated environment for untrusted tasks, and don't expose +this development stack to the internet. Read +[Support and security](/resources/support/) before running sensitive workloads. +::: + +## Run your first evaluation + +### From the Portal + +1. Select or create a project to keep your evaluation data together. +2. Create a task and at least one observable evaluation criterion. For example, + ask the agent to create a Node.js HTTP server and evaluate whether its source + implements a `GET /health` route returning JSON. +3. Open the run submission page, select the running Copilot agent and an + available model, and attach your task and criteria. You can save the agent + configuration as a reusable profile. +4. Submit the request, follow the run's live logs, and inspect the Judge's + results and workspace artifacts. + +If no models are available, check your provider access and the model-scanner +logs before submitting. A registered agent isn't necessarily running; choose +the worker enabled by your Compose command. See +[Submitting requests (Portal)](/guides/submitting-requests-portal/) for the +full workflow. + +### From the CLI + +Build the CLI and its shared dependency, then explore the available commands: + +```bash +pnpm build:cli +pnpm cli --help +pnpm cli project list +``` + +Select a project using the ID returned by `project list`: + +```bash +pnpm cli project use +pnpm cli criteria list +pnpm cli run submit --help +pnpm cli run list +``` + +Replace `` with an actual ID. Submission requires a selected +project, a task, and evaluation criteria. Use `--project ` or +`SCOPE_PROJECT` to select a project explicitly in automation. The CLI reads +local port settings from the generated `.env`; set `SCOPE_API_URL` to target +another instance. See +[Submitting requests (CLI)](/guides/submitting-requests-cli/) for submission +examples. + +## Coding agents + +| Worker | Integration | Getting started | +| --- | --- | --- | +| GitHub Copilot | Agent Client Protocol (ACP) | `pnpm docker:dev:copilot` | +| Claude Code | ACP | `pnpm docker:dev:claude-code` | +| GitHub Copilot on Windows | Windows ACP worker | Deployment-specific; see [system architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md). | + +The `pnpm docker:dev:all` command enables both local ACP workers, not every +deployment-specific integration. Each provider requires its own credentials +and access to the selected models. + +## Next steps + +Read [Concepts](/introduction/concepts/) for the evaluation building blocks, +[Development guide](/resources/development/) for repository layout and +commands, and [Contributing](/resources/contributing/) for how to propose +changes. diff --git a/website/src/content/docs/index.mdx b/website/src/content/docs/index.mdx index d6095bf58..0843bbac3 100644 --- a/website/src/content/docs/index.mdx +++ b/website/src/content/docs/index.mdx @@ -193,6 +193,12 @@ import { Card, CardGrid } from '@astrojs/starlight/components'; Run a task end to end from the Portal in a few minutes. [Get started](/getting-started/first-run/). + + Set up your own Scope stack and run an evaluation. [Follow the setup guide](/getting-started/local-development/). + + + Find development commands, architecture references, and the contribution process. [Read the contributor guide](/resources/contributing/). + Automate request submission and integrate Scope into your tools. [API guide](/guides/submitting-requests-api/). diff --git a/website/src/content/docs/introduction/concepts.md b/website/src/content/docs/introduction/concepts.md index be2397235..f314943c0 100644 --- a/website/src/content/docs/introduction/concepts.md +++ b/website/src/content/docs/introduction/concepts.md @@ -116,3 +116,20 @@ Optional capabilities that extend what an agent can do during a run: All three are properties of a profile. See [Using MCP servers, skills & extensions](/guides/mcp-skills-extensions/). + +## Evaluation building blocks + +| Concept | Purpose | +| --- | --- | +| Tasks and scenarios | Define the work the agent should perform. | +| Criteria | Define observable checks and dependencies in the Judge's criteria DAG. | +| Personas | Configure the evaluation perspective and feedback style. | +| Profiles and variations | Save an agent configuration and compare changes against a baseline. | +| Skills and MCP servers | Provide agent instructions and tools through the Model Context Protocol. | +| Codebases | Seed a run with a versioned starting workspace. | + +The YAML files in +[config/](https://github.com/microsoft/scope/tree/main/config) are portable +examples, not the live configuration database. MongoDB is the runtime source +of truth. Manage configuration through the Portal or CLI; don't assume that +editing an example file changes an existing evaluation. diff --git a/website/src/content/docs/introduction/what-is-scope.md b/website/src/content/docs/introduction/what-is-scope.md index f79530253..e1b6f0b0e 100644 --- a/website/src/content/docs/introduction/what-is-scope.md +++ b/website/src/content/docs/introduction/what-is-scope.md @@ -28,6 +28,11 @@ swap the agent, the model, or the surrounding tools. their parents passing — use it for multi-step evaluation, or just leave dependencies off and every criterion becomes a root. - **Watch runs in real time** as logs stream from the worker. +- **Inspect the evidence.** Review generated files, workspace snapshots, + criteria results, captured agent activity, reports, and insights. +- **Evaluate changes.** Reuse tasks, profiles, and versioned starting codebases + to compare surfaces such as CLIs, MCP servers, skills, and documentation, + as well as models, context, tasks, and operating systems. - **Compare across heterogeneous tasks** using prompt features — characteristics Scope detects on your task prompt (e.g. "asks for an API", "asks for TypeScript") so you can ask questions like *"how @@ -36,6 +41,31 @@ swap the agent, the model, or the surrounding tools. - **Automate everything** through the REST API or the `scope` CLI — submit runs, manage profiles and criteria, fetch results. +## How it works + +1. **Define** a task, its evaluation criteria, and the agent configuration. +2. **Submit** a request through the Portal or CLI. The API stores the request + in MongoDB. The scheduler claims pending runs and dispatches them to the + appropriate storage queue. +3. **Execute and evaluate.** The worker runs the agent and asks the Judge to + evaluate its output. Runs can include multiple feedback iterations. +4. **Inspect and compare.** Review logs, snapshots, and criteria results. + Post-processing and report workers produce additional analysis when enabled. + +| Service | Role in a run | +| --- | --- | +| MongoDB | Stores evaluation configuration, run records, worker status updates, and the Judge's criteria results. | +| Storage Queues | Deliver work from the scheduler to coding-agent workers. | +| Redis | Relays worker logs and live events to the API for Portal and CLI clients. | +| Blob Storage | Holds larger artifacts, including workspace snapshots written by workers. | + +Local development uses MongoDB, Redis, Azurite (the Azure Storage emulator), +and Lowkey Vault. See +[Local development](/getting-started/local-development/) to run the stack, or +the +[system architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md) +for service details and production deployment considerations. + ## When Scope is the right tool Use Scope when you want to: @@ -58,15 +88,31 @@ against tasks you control. Scope is for teams that need to measure and compare AI coding agents: +- **Product managers** evaluating how coding agents use their software and + respond to feedback. - **Engineers** designing task prompts and criteria to characterize the agentic experience. - **Researchers** comparing agent trajectories across diverse tasks. - **Pipelines and tooling** that submit runs programmatically via the REST API. +## Interpreting results + +A working result is only part of the agentic experience. Use repeatable +evaluations to understand both successes and failures, including the steps an +agent took and how it responded to feedback. + +Results describe the tasks and configurations you tested, not a universal +agent ranking. The automated Judge can make mistakes; important conclusions +need human review. See [Support and security](/resources/support/) for +responsible-use guidance. + ## Where to next - New here? Read [Concepts](/introduction/concepts/) to get familiar with the vocabulary. - Ready to submit your first run? Jump to [Access](/getting-started/access/). +- Want to run Scope locally? Follow + [Local development](/getting-started/local-development/). +- Interested in improving Scope? Read [Contributing](/resources/contributing/). diff --git a/website/src/content/docs/resources/contributing.md b/website/src/content/docs/resources/contributing.md new file mode 100644 index 000000000..96d62df17 --- /dev/null +++ b/website/src/content/docs/resources/contributing.md @@ -0,0 +1,55 @@ +--- +title: Contributing +description: How to contribute to Scope, propose changes, and follow the project's contribution and licensing policies. +--- + +Contributions aren't limited to new workers. Documentation improvements, +reproducible bug reports, evaluation examples, tests, and accessibility fixes +are useful ways to get involved. + +## Before you start + +Search [existing issues](https://github.com/microsoft/scope/issues) before +reporting a bug or proposing a feature. For larger changes, open an issue to +discuss the approach before implementation. Include reproduction steps and +relevant versions in bug reports, and remove credentials and sensitive run +content from logs. + +Set up your environment with +[Local development](/getting-started/local-development/), then use the +[Development guide](/resources/development/) to find the relevant components +and architecture references. Read +[CONTRIBUTING.md](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md) +for coding conventions, testing requirements, and the pull request process. + +## Pull requests + +Open pull requests against `microsoft/scope` on `main`, including when working +from a fork. Keep changes focused, include appropriate tests, and update +affected documentation. + +Most contributions require the +[Microsoft Contributor License Agreement](https://cla.opensource.microsoft.com). +The CLA bot will determine whether you need to sign and guide you through it. + +Everyone participating in the project is expected to follow the +[Code of Conduct](https://github.com/microsoft/scope/blob/main/CODE_OF_CONDUCT.md). + +For usage questions and private vulnerability reporting, see +[Support and security](/resources/support/). + +## License and trademarks + +Scope is licensed under the +[MIT License](https://github.com/microsoft/scope/blob/main/LICENSE). +Third-party attributions are in +[NOTICE](https://github.com/microsoft/scope/blob/main/NOTICE); contributors +changing dependencies should follow the +[notice maintenance instructions](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md#third-party-notices). + +This project may contain trademarks or logos for projects, products, or +services. Authorized use of Microsoft trademarks or logos must follow +[Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). +Use of Microsoft trademarks or logos in modified versions of this project +must not cause confusion or imply Microsoft sponsorship. Any use of +third-party trademarks or logos is subject to those parties' policies. diff --git a/website/src/content/docs/resources/development.md b/website/src/content/docs/resources/development.md new file mode 100644 index 000000000..cc726cc8e --- /dev/null +++ b/website/src/content/docs/resources/development.md @@ -0,0 +1,58 @@ +--- +title: Development guide +description: Repository layout, development commands, and architecture references for Scope contributors. +--- + +The repository is a pnpm workspaces monorepo, primarily TypeScript 5, with a +React Portal and a Rust AI gateway. Start with +[Local development](/getting-started/local-development/) to install +dependencies and run the stack, then read +[Contributing](/resources/contributing/) before opening a pull request. + +## Repository structure + +| Path | Contents | +| --- | --- | +| [apps/api/](https://github.com/microsoft/scope/tree/main/apps/api) | REST API and live event streaming | +| [apps/portal/](https://github.com/microsoft/scope/tree/main/apps/portal) | Web UI and Storybook components | +| [apps/cli/](https://github.com/microsoft/scope/tree/main/apps/cli) | CLI for evaluation management and automation | +| [apps/scheduler/](https://github.com/microsoft/scope/tree/main/apps/scheduler) and [apps/judge/](https://github.com/microsoft/scope/tree/main/apps/judge) | Run dispatch and criteria evaluation | +| [apps/workers/](https://github.com/microsoft/scope/tree/main/apps/workers) | Coding-agent, post-processing, and report workers | +| [apps/gateway/](https://github.com/microsoft/scope/tree/main/apps/gateway) and [apps/token-manager/](https://github.com/microsoft/scope/tree/main/apps/token-manager) | AI traffic capture and credential management | +| [packages/](https://github.com/microsoft/scope/tree/main/packages) | Shared types, storage clients, migrations, and supporting libraries | +| [config/](https://github.com/microsoft/scope/tree/main/config) and [docs/](https://github.com/microsoft/scope/tree/main/docs) | Evaluation examples and technical documentation | +| [website/](https://github.com/microsoft/scope/tree/main/website) | This documentation website | + +## Development commands + +Run these commands from the repository root: + +```bash +pnpm test # Unit tests +pnpm lint # Workspace lint and type checks +pnpm build # Workspace builds +pnpm storybook # Portal component catalog +pnpm test:integration # Integration tests; requires .env and backing services +``` + +For service-by-service development, Rust commands, migrations, and code +conventions, read +[CONTRIBUTING.md](https://github.com/microsoft/scope/blob/main/CONTRIBUTING.md). +For documentation authoring and preview commands, see +[website/README.md](https://github.com/microsoft/scope/blob/main/website/README.md). + +## Architecture and configuration references + +These detailed engineering references live alongside the source code: + +| Topic | Guide | +| --- | --- | +| Architecture and run lifecycle | [System architecture](https://github.com/microsoft/scope/blob/main/docs/architecture/system-architecture.md) | +| Domain models and API design | [Application design](https://github.com/microsoft/scope/blob/main/docs/architecture/app-design.md) | +| Project organization | [Projects](https://github.com/microsoft/scope/blob/main/docs/architecture/data-organization-projects.md) | +| Evaluation and criteria DAGs | [Criteria provider](https://github.com/microsoft/scope/blob/main/docs/architecture/criteria-provider.md) | +| Agent context | [Skills](https://github.com/microsoft/scope/blob/main/docs/architecture/skills.md) and [codebases](https://github.com/microsoft/scope/blob/main/docs/architecture/codebases.md) | +| Scheduling and recovery | [Queue scheduler](https://github.com/microsoft/scope/blob/main/docs/architecture/queue-scheduler.md) | +| Configuration and authentication | [Environment variables](https://github.com/microsoft/scope/blob/main/ENV_VARIABLES.md) | +| AI limitations and data handling | [Responsible AI FAQ](https://github.com/microsoft/scope/blob/main/docs/responsible-ai-faq.md) | +| More architecture, operations, and research | [Documentation index](https://github.com/microsoft/scope/blob/main/docs/README.md) | diff --git a/website/src/content/docs/resources/support.md b/website/src/content/docs/resources/support.md new file mode 100644 index 000000000..8ba16fef2 --- /dev/null +++ b/website/src/content/docs/resources/support.md @@ -0,0 +1,42 @@ +--- +title: Support and security +description: Get help, report vulnerabilities privately, and understand Scope's responsible-use requirements. +--- + +## Getting help + +For usage questions, bugs, and feature requests, see +[SUPPORT.md](https://github.com/microsoft/scope/blob/main/SUPPORT.md). +Search [existing issues](https://github.com/microsoft/scope/issues) before +opening a new issue. Include reproduction steps and relevant versions, but +remove credentials and sensitive run content from logs. + +## Reporting vulnerabilities + +Report vulnerabilities privately through +[SECURITY.md](https://github.com/microsoft/scope/blob/main/SECURITY.md), which +links to the current Microsoft security reporting guidance. Never report a +vulnerability through a public GitHub issue. + +## Responsible use + +Evaluation artifacts can contain prompts, source code, tool output, and +network metadata. Only use data and credentials approved for your deployment, +configure appropriate access controls, and review generated code before +using it. + +Scope doesn't certify that an agent or its output is safe or production-ready. +Results describe the tasks and configurations you tested, not a universal +agent ranking. The automated Judge can make mistakes; important conclusions +need human review. + +Local development is not a security sandbox. The ACP worker configuration +mounts the Docker socket so agents can run containers. Use a dedicated +environment for untrusted tasks, and don't expose the development stack to +the internet. + +Read the +[Responsible AI FAQ](https://github.com/microsoft/scope/blob/main/docs/responsible-ai-faq.md) +before running sensitive or untrusted workloads. See +[Data collection and privacy](/resources/data-collection/) for the data Scope +handles and the deployment operator's responsibilities.