Commit graph

104 commits

Author SHA1 Message Date
Brendan Kellam
09507d3e89
fix(worker): Permission syncer fixes (#624)
Some checks failed
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Has been cancelled
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Has been cancelled
Update Roadmap Released / update (push) Has been cancelled
Publish to ghcr / merge (push) Has been cancelled
2025-11-19 22:14:23 -08:00
Jose Hernandez
e20d514569
feat(bitbucket): support glob patterns in repository exclusions (#620)
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
Update Roadmap Released / update (push) Waiting to run
* feat(bitbucket): support glob patterns in repository exclusions

Update Bitbucket Cloud and Server exclusion logic to support glob
patterns (e.g., "org/repo*") in the exclude.repos configuration,
matching the documented behavior and aligning with other providers
(GitHub, GitLab, Gitea, Azure DevOps).

Changes:
- Add micromatch import for pattern matching
- Replace Array.includes() with micromatch.isMatch() in
  cloudShouldExcludeRepo and serverShouldExcludeRepo functions
- Add reason logging for exclusion decisions to match GitHub's pattern

This enables users to exclude repositories using wildcard patterns
as documented in the Bitbucket Cloud connection documentation.

* update changelog

---------

Co-authored-by: Jose Hernandez <jose.hernandez@emilabs.ai>
Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-11-17 14:33:39 -08:00
Brendan Kellam
06c84f0bf5
fix(worker): Fix issue where connections would always sync on startup (#613)
Some checks failed
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Has been cancelled
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Has been cancelled
Update Roadmap Released / update (push) Has been cancelled
Publish to ghcr / merge (push) Has been cancelled
2025-11-11 20:31:08 -08:00
Brendan Kellam
903d15a2c5
fix(worker): Fix issues with gracefully shutting down (#612) 2025-11-11 20:11:59 -08:00
Brendan Kellam
18fad64baa
feat(web): Add force resync buttons for repo & connections (#610) 2025-11-11 15:16:40 -08:00
Brendan Kellam
6f64d5bb8d
fix(worker): Run setInterval as blocking (#607)
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
Update Roadmap Released / update (push) Waiting to run
2025-11-09 14:49:24 -08:00
Brendan Kellam
1be6e8842e
fix(worker): properly shutdown PostHog client (#609) 2025-11-09 14:30:01 -08:00
Brendan Kellam
1908051daa
feat(web,worker): Environment overrides (#597) 2025-11-04 21:22:31 -08:00
Brendan Kellam
5fde901356
chore(worker): Refactor permission syncing join table to be between Account <> Repo (#600) 2025-11-04 20:12:07 -08:00
Michael Sukkarieh
449c76fdcc
feat(ee): Add ability to link external accounts (#595) 2025-11-04 20:08:04 -08:00
Derek Miller
2c0540f6bf
fix(backend): Limit concurrent git operations to prevent resource exhaustion (#590) (#593)
When syncing generic-git-host connections with thousands of repositories,
unbounded Promise.all caused resource exhaustion (EAGAIN errors) by spawning
too many concurrent git processes. This resulted in valid repositories being
incorrectly skipped during sync.

- Add p-limit to control concurrent git operations (max 100)
- Follow existing pattern from github.ts for consistency
- Prevents file descriptor and process limit exhaustion
- Uses rolling concurrency to avoid head-of-line blocking

Fixes #590
2025-11-01 15:15:09 -07:00
Brian Phillips
58456d616b
add p-limit to GitHub API calls to avoid overwhelming the node process (or the API rate limits) (#591) 2025-10-31 14:49:43 -07:00
Michael Sukkarieh
fd17871da4
chore(tech-debt): Remove built-in secret manager (#592) 2025-10-31 14:33:28 -07:00
Brendan Kellam
581a5a0bd8
fix(web): Fix /settings/connections throwing a error when there is a git connection present (#588) 2025-10-31 13:08:51 -07:00
Brendan Kellam
4899c9fbc7
feat(ee): GitLab permission syncing (#585)
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
2025-10-30 11:08:10 -07:00
Michael Sukkarieh
bbb197a9bf
fix(github app): Generate installation tokens each time (#583)
* generate installation tokens each time

* changelog
2025-10-29 18:05:18 -07:00
Brendan Kellam
a167accd7e
feat(worker,web): Improved connection management (#579) 2025-10-28 21:31:28 -07:00
msukkari
0bd545359e fix bug with octokit url for github cloud
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
2025-10-25 21:57:13 -07:00
Brendan Kellam
2d3b03bf12
feat(web): Improved repository table (#572)
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
2025-10-25 14:51:41 -04:00
msukkari
4b86bcd182 add debug log for github auth app
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
2025-10-24 11:49:40 -07:00
Brendan Kellam
a470ab8463
chore(worker): Prometheus metrics for repo index manager (#571)
Some checks failed
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Has been cancelled
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Has been cancelled
Publish to ghcr / merge (push) Has been cancelled
2025-10-21 20:43:33 -07:00
msukkari
ef77e212a0 fix bug from github app pr 2025-10-21 20:39:59 -07:00
Michael Sukkarieh
c2299aa86b
feat(auth): github app (#570)
* properly handle emails for github app auth case

* add docs info for auth through github app

* more info in docs for user auth perms

* modify review agent env var names

* github app service auth

* coderabbit suggestions

* fixes

* fix build
2025-10-21 20:17:28 -07:00
Brendan Kellam
03999f0de0
fix(worker): Use indexTimeoutMs setting for job timeout (#567)
Some checks are pending
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Waiting to run
Publish to ghcr / merge (push) Blocked by required conditions
2025-10-20 12:41:09 -07:00
Brendan Kellam
4ebe4e0475
chore(worker,web): Repo indexing stability improvements + perf improvements to web (#563)
Some checks failed
Publish to ghcr / build (linux/amd64, blacksmith-4vcpu-ubuntu-2404) (push) Has been cancelled
Publish to ghcr / build (linux/arm64, blacksmith-8vcpu-ubuntu-2204-arm) (push) Has been cancelled
Publish to ghcr / merge (push) Has been cancelled
2025-10-18 16:31:22 -07:00
bkellam
8d7babc8d2 chore(worker): Change log message to debug 2025-10-07 16:38:56 -07:00
Michael Sukkarieh
aa62847143
fix(ado): Manually pass token through http header for ado server (#543)
* support passing in token manually in auth header

* remove unneeded PAT embed check

* cleanup authheader usage

* changelog

* var name typo

* unset auth header in fetch

* move unset to finally in fetch
2025-09-27 17:14:29 -07:00
Brendan Kellam
ef46c0181d
chore: fix support email (#529) 2025-09-21 12:19:12 -07:00
Brendan Kellam
5073c7db22
[experimental] feat(ee): GitHub permission syncing (#508) 2025-09-20 16:51:14 -07:00
Michael Sukkarieh
af39b20668
fix(azure): Set username in azuredevops clone url (#524)
* set placeholder username for azuredevops clone url

* changelog
2025-09-19 09:48:58 -07:00
Tarang Chikhalia
7020761ca6
fix(backend): Sourcebot not pulling github forked repos (#499) 2025-09-18 15:06:46 -07:00
Michael Sukkarieh
e7fa4c4765
feat(connections): Add Azure Devops Support (#514)
* initial ado pol

* add support for ado logo

* default to main instead of HEAD when generating file url

* bump zoekt

* fix(web) Fix "At least one project, user, or group must be specified" for GitLab configs in web configurator (#512)

* feat(ask_sb): Fallback on fromNodeProviderChain if access key or sessionToken are not provided (#513)

* Quote branches argument in zoekt.ts to fix Pipe (#506)

* remove connections settings page

* fix styling and remove additional components

* add changelog

* add docs

* fix build error

* bump zoekt

* fix broken links for ado docs

* fix HEAD support for ado

* changelog

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
Co-authored-by: Michael Dekoski <michaeldekoski@gmail.com>
2025-09-17 22:18:56 -07:00
Michael Dekoski
4a449da7d8
Quote branches argument in zoekt.ts to fix Pipe (#506) 2025-09-17 14:02:09 -07:00
Fede Sanchez
7cbda320cb
fix(bitbucket): Bitbucket Cloud pagination not working beyond first page (#502) 2025-09-15 19:43:50 -07:00
msukkari
aab4a92a87 fix for zoekt logs as errors 2025-09-09 18:44:23 -07:00
Brendan Kellam
7d0c6588e1
feat(gitlab): Add exclude.userOwnedProjects config setting (#498) 2025-09-08 22:38:18 -04:00
Brendan Kellam
b05fc7a0c8
fix(backend): Explicitly set refspec on fetch (#497) 2025-09-08 16:15:58 -04:00
Brendan Kellam
d694330998
fix(worker): Fix "attempting to index 0 total files" zoekt issue (#488) 2025-09-04 11:14:29 -04:00
bkellam
2241217b0b fix(worker): First check keys before unsetting them 2025-09-01 10:21:08 -04:00
Brendan Kellam
2b423ba7e9
fix(worker): Remove setting remote.origin.url for remote git repositories (#483) 2025-08-31 13:52:51 -04:00
Brendan Kellam
d9fa221d72
chore(web): Upgrade to NextJS 15 (#477) 2025-08-22 14:48:29 -04:00
Brendan Kellam
a3d9711178
feat(gerrit): Multi-branch indexing support for Gerrit (#433) 2025-08-05 16:35:47 -07:00
Brendan Kellam
01dee161b3
fix(github): Fix issue with users not getting picked up in GitHub config (#428)
* fix

* changelog
2025-08-04 17:39:28 -07:00
Michael Sukkarieh
e47fdb51db
fix(repoManager): Don't fetch files for each repo on validation (#414) 2025-07-31 17:50:04 -07:00
msukkari
f62a492ab3 properly log zoekt index exec 2025-07-28 20:12:46 -07:00
Brendan Kellam
efb4eb984a
fix(gitlab): Add configurable query timeout to GitLab client (#390)
* fix

* changelog
2025-07-20 11:59:15 -07:00
Michael Sukkarieh
aac1d4529e
Add anonymous access option to core (#385)
* migrate anonymous access logic out of ee

* add anonymous access toggle

* handle anon toggle properly based on perms

* add forceEnableAnonymousAccess setting

* add docs for access settings

* change forceEnableAnonymousAccess to be an env var

* add FORCE_ENABLE_ANONYMOUS_ACCESS to list in docs

* add back the enablePublicAccess setting as deprecated

* add changelog entry

* fix build errors

* add news entry for anonymous access

* feedback
2025-07-19 14:04:41 -07:00
Gabriel Dugny
9b13492cb3
chore: Misc typos (UI, docs, code...), Makefile PATH with spaces (#369)
* chore: Fix misc typos (UI, docs, code...)

* chore(dev): Support PATH with spaces in Makefile

E.g. `Application Support` on MacOS

* chore: Typos in schema v2 description

* chore: more typos

* chore(dev): Add _typos.toml
2025-07-16 11:59:01 -07:00
drew-u410
d9d0146c48
[indexing] make retry configurable (#377) 2025-07-14 12:31:34 -07:00
Brendan Kellam
22d548e171
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354)
## Problem

If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance.

## Solution

This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system).

## Why another package?

Because the `syncSearchContexts` call is now called from:
1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before.
2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced.

## Follow-up devex work
Two things:
1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages).
2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 14:04:25 -07:00