sourcebot/README.md

Try the demo!

About

Sourcebot is a fast code indexing and search tool for your codebases. It is built ontop of the zoekt indexer, originally authored by Han-Wen Nienhuys and now maintained by Sourcegraph.

Getting Started

Using Docker

0. Install Docker

1. Create a config.json file and list the repositories you want to index. The JSON schema index.json defines the structure of the config file and the available options. For example, if we want to index Sourcebot on its own code, we could use the following config found in sample-config.json:

{
    "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json",
    "Configs": [
        {
            "Type": "github",
            "GitHubOrg": "TaqlaAI",
            "Name": "^sourcebot$"
        }
    ]
}

Sourcebot also supports indexing GitLab & BitBucket. Checkout the index.json for a full list of available options.

2. Create a Personal Access Token (PAT) to authenticate with a code host(s):

   GitHub

   Generate a GitHub Personal Access Token (PAT) here. If you are indexing public repositories only, you can select the public_repo scope, otherwise you will need the repo scope.

   GitLab

   TODO

   BitBucket

   TODO

3. Launch the latest image from the ghcr registry:

GitHub

docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data -e GITHUB_TOKEN=<token> ghcr.io/taqlaai/sourcebot:main

GitLab

docker run -p 3000:3000 --rm --name sourcebot -v $(pwd):/data -e GITLAB_TOKEN=<token> ghcr.io/taqlaai/sourcebot:main

BitBucket

TODO

Two things should happen: (1) a .sourcebot directory will be created containing the mirror repositories and indexes, and (2) you will see output similar to:

INFO spawned: 'node-server' with pid 10
INFO spawned: 'zoekt-indexserver' with pid 11
INFO spawned: 'zoekt-webserver' with pid 12
run [zoekt-mirror-github -dest /data/.sourcebot/repos -delete -org <org>]
...
INFO success: node-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO success: zoekt-indexserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
INFO success: zoekt-webserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

zoekt will now index your repositories (at HEAD). By default, it will re-index existing repositories every hour, and discover new repositories every 24 hours.

4. Go to http://localhost:3000 - once a index has been created, you can start searching.

Building Sourcebot

0. Install go and NodeJS

1. Clone the repository with submodules:

git clone --recurse-submodules https://github.com/TaqlaAI/sourcebot.git

2. Run make to build zoekt and install dependencies:

cd sourcebot
make

The zoekt binaries and web dependencies are placed into bin and node_modules respectively.

3. Create a config.json file and list the repositories you want to index. The JSON schema defined in index.json defines the structure of the config file and the available options. For example, if we want to index Sourcebot on its own code, we could use the following config found in sample-config.json:

{
    "$schema": "https://raw.githubusercontent.com/TaqlaAI/sourcebot/main/schemas/index.json",
    "Configs": [
        {
            "Type": "github",
            "GitHubOrg": "TaqlaAI",
            "Name": "^sourcebot$"
        }
    ]
}

4. Create a Personal Access Token (PAT) to authenticate with a code host:

   GitHub

   Generate a GitHub Personal Access Token (PAT) here. If you are indexing public repositories only, you can select the public_repo scope, otherwise you will need the repo scope.

   Create a text file named .github-token in your home directory and paste the token in it. The file should look like:

   ghp_...
   

   zoekt will read this file to authenticate with GitHub.

   GitLab

   TODO

   BitBucket

   TODO

   
   

5. Start Sourcebot with the command:

yarn dev

A .sourcebot directory will be created and zoekt will begin to index the repositories found given config.json.

6. Go to http://localhost:3000 - once a index has been created, you can start searching.

Disabling Telemetry

By default, Sourcebot collects anonymous usage data using PostHog. You can disable this by setting the environment variable SOURCEBOT_TELEMETRY_DISABLED to 1 in the docker run command:

docker run -e SOURCEBOT_TELEMETRY_DISABLED=1 ...stuff... ghcr.io/taqlaai/sourcebot:main

Or if you are building locally, add the following to your .env file:

SOURCEBOT_TELEMETRY_DISABLED=1
NEXT_PUBLIC_SOURCEBOT_TELEMETRY_DISABLED=1