Commit graph

48 commits

Author SHA1 Message Date
Michael Sukkarieh
fdd71cfcfe
add better visualization for connection/repo errors and warnings (#201)
* replace upsert with seperate create many and raw update many calls

* add bulk repo status update and queue addition with priority

* add support for managed redis

* add note for changing raw sql on schema change

* add error package and use BackendException in connection manager

* handle connection failure display on web app

* add warning banner for not found orgs/repos/users

* add failure handling for gerrit

* add gitea notfound warning support

* add warning icon in connections list

* style nits

* add failed repo vis in connections list

* added retry failed repo index buttons

* move nav indicators to client with polling

* fix indicator flash issue and truncate large list results

* display error nav better

* truncate failed repo list in connection list item

* fix merge error

* fix merge bug

* add connection util file [wip]

* refactor notfound fetch logic and add missing error package to dockerfile

* move repeated logic to function and add zod schema for syncStatusMetadata
2025-02-19 18:10:22 -08:00
msukkari
63af99aa59 remove non secret token options 2025-02-19 14:23:26 -08:00
Michael Sukkarieh
5d7a77bd92
db performance improvements and job resilience (#200)
* replace upsert with seperate create many and raw update many calls

* add bulk repo status update and queue addition with priority

* add support for managed redis

* add note for changing raw sql on schema change
2025-02-19 10:26:13 -08:00
msukkari
e09b21f6b9 add config upsert timeout env var 2025-02-15 17:27:24 -08:00
msukkari
e17331a672 fix build errors and add index concurrency env var 2025-02-15 16:58:05 -08:00
Michael Sukkarieh
54d14ea98e
add retries for 429 issues (#196)
* add connection compile retry and hard repo limit

* add more retry checks

* cleanup unused change

* address feedback
2025-02-15 16:37:50 -08:00
Michael Sukkarieh
e0d363420b
connections qol improvements (#195)
* add client side polling to connections list

* properly fetch repo image url

* add client polling to connection management page, and add ability to sync failed connections
2025-02-15 10:00:44 -08:00
Michael Sukkarieh
3be3680ee2
cleanup org's repos and shards if it's inactive (#194)
* add stripe subscription status and webhook

* add inactive org repo cleanup logic

* mark reactivated org connections for sync
2025-02-15 09:58:17 -08:00
Michael Sukkarieh
da33220289
add back gitlab, gitea, and gerrit support (#184)
* add non github config definitions

* refactor github config compilation to seperate file

* add gitlab config compilation

* Connection management (#183)

* wip gitlab repo sync support

* fix gitlab zoekt metadata

* add gitea support

* add gerrit support

* Connection management (#183)

* add gerrit config compilation

* Connection management (#183)

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
2025-02-14 10:58:53 -08:00
msukkari
875f9b8586 add await for blocking loop in backend 2025-02-13 10:30:14 -08:00
bkellam
8d2c731b88 Domain support 2025-02-12 13:45:12 -08:00
Brendan Kellam
846d73b0e6
Connection management (#183) 2025-02-04 15:04:05 -05:00
Michael Sukkarieh
dffbbae41d
adds garbage collection for repos (#182)
* refactor repo indexing logic into RepoManager

* wip cleanup stale repos

* add rest of gc logic

* set status to indexing properly
2025-01-29 09:05:20 -08:00
Michael Sukkarieh
31114a9d95
add concept of secrets (#180)
* add @sourcebot/schemas package

* migrate things to use the schemas package

* Dockerfile support

* add secret table to schema

* Add concept of connection manager

* Rename Config->Connection

* Handle job failures

* Add join table between repo and connection

* nits

* create first version of crypto package

* add crypto package as deps to others

* forgot to add package changes

* add server action for adding and listing secrets, create test page for it

* add secrets page to nav menu

* add secret to config and support fetching it in backend

* reset secret form on successful submission

* add toast feedback for secrets form

* add instructions for adding encryption key to dev instructions

* add encryption key support in docker file

* add delete secret button

* fix nits from pr review

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-27 14:07:07 -08:00
Brendan Kellam
dd8ff6edb0
Connection management (#178) 2025-01-24 13:16:08 -08:00
Brendan Kellam
bae7ca3438
Add @sourcebot/schemas package (#177) 2025-01-24 10:51:49 -08:00
msukkari
8a3521b080 updated syncedAt date after config sync: 2025-01-23 09:08:30 -08:00
Michael Sukkarieh
7c6adf17aa
Multi tenancy support in config syncer (#171)
* [wip] initial mt support in config syncer

* Move logout button & profile picture into settings dropdown (#172)

* update sync status properly and fix bug with multiple config in db case

* make config path required in single tenant mode

NOTE: deleting config/repos is currently not supported in multi tenancy case. Support for this will be added in a future PR

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
2025-01-21 11:50:35 -08:00
Brendan Kellam
6cf10b4988
Authentication (#164) 2025-01-16 15:24:13 -08:00
Michael Sukkarieh
7029aa70c1
parallelize repo indexing (#163)
* hacked together a example of using zoekt grpc api

* provide tenant id to zoekt git indexer

* update zoekt version to point to multitenant branch

* pipe tenant id through header to zoekt

* remove incorrect submodule reference and settings typo

* update zoekt commit

* remove unused yarn script

* remove unused grpc client in web server

* remove unneeded deps and improve tenant id log

* pass tenant id when creating repo in db

* add mt yarn script

* add pol of bullmq into backend

* add better error handling and concurrency setting

* spin up redis instance in dockerfile

* cleanup transaction logic when adding repos to index queue

* add NEW index status fetch condition

* move bullmq deps to backend

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-15 15:44:42 -08:00
Michael Sukkarieh
553f5d25f7
Add tenant ID concept into web app and backend (#160)
* hacked together a example of using zoekt grpc api

* provide tenant id to zoekt git indexer

* update zoekt version to point to multitenant branch

* pipe tenant id through header to zoekt

* remove incorrect submodule reference and settings typo

* update zoekt commit

* remove unused yarn script

* remove unused grpc client in web server

* remove unneeded deps and improve tenant id log

* pass tenant id when creating repo in db

* add mt yarn script

* add nocheckin comment to tenant id in v2 schema

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-14 16:46:36 -08:00
Brendan Kellam
15b9e777e4
SQL Database (#157) 2025-01-14 13:37:31 -08:00
Michael Sukkarieh
d269a8cbe6
implement raw remote git repo support (#152)
* implement raw remote git repo support

* add changelog entry
2025-01-09 09:10:30 -08:00
Brendan Kellam
03aa608e21
Graceful error handling when calling code host apis (#142) 2024-12-18 19:21:21 -08:00
Brendan Kellam
0e9c829cc3
[GitHub] Add exclude.size property to the config (#137) 2024-12-17 00:01:19 -08:00
Brendan Kellam
c35f6bc5ae
Add reindexInterval and resyncInterval to config settings (#134) 2024-12-16 21:30:50 -08:00
Brendan Kellam
d4e72566e2
Fix directory not found exception when deleting stale repository (#136) 2024-12-16 20:23:42 -08:00
Brendan Kellam
4353d2008a
Add autoDeleteStaleRepos config option (#128) 2024-12-13 12:34:02 -08:00
Brendan Kellam
4d358f94a2
Add topics and exclude.topics to GitHub & GitLab config (#121) 2024-12-11 14:17:57 -08:00
Brendan Kellam
111023b1dc
Add config option to specify max file size (#118) 2024-12-09 12:34:43 -10:00
Konrad Staniszewski
f71a83a941
fix: make gerrit repo fetching paginated (#114)
* fix: make gerrit repo fetching paginated

In some cases gerrit will limit the number of projects returned by
/projects endpoint, forcing the client to paginate their request to get
all projects.
This fulfills this requirement to get all projects

* Add some more metadata repo projects to ignore list

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
2024-12-07 08:45:46 -10:00
Konrad Staniszewski
b452fd2983
Gerrit sync (#104)
* Basic gerrit sync with working gitiles web-links functionality

This adds basic support for gerrit repo code host syncing. Gerrit uses
gitiles plugin for code browsing (in most cases).
It may be usefull to allow users to provide their own web code-browsing
url templates in the future.

* Add gerrit readme update

* Remove config arg from gerrit fetchAllProjects

* Remove example urls

* Resolve comments
2024-12-02 16:07:02 -08:00
Brendan Kellam
33b5418bbf
Pass PostHog key at build time (#92) 2024-11-25 21:04:52 -08:00
Brendan Kellam
f3d5fa6cb3
Add config param all to enable syncing all projects in GitLab instance (#84) 2024-11-20 14:29:13 -08:00
Brendan Kellam
bea86a64c9
enable allow_missing_branches (#73) 2024-11-14 22:35:07 -08:00
Chris Roberts
e6d90398e4
Fix typo'd Exclude repo-wide (#71) 2024-11-14 15:03:34 -08:00
Steven
a407792212
Exclude repos by glob (#70) 2024-11-14 14:16:31 -08:00
Brendan Kellam
9cba4f274f
Add unit testing library to @sourcebot/backend (#65) 2024-11-12 18:37:35 -08:00
Brendan Kellam
adba96a8c4
Add additional telemetry (#63) 2024-11-09 16:40:07 -08:00
Brendan Kellam
ada53fc2c6
Multi branch / tag support (#58) 2024-11-06 18:28:10 -08:00
Brendan Kellam
7966c1440c
Local directory support (#56) 2024-11-01 10:51:14 -07:00
bkellam
934bc8d903 Fixed issue with either or not being included in the GitLab api response. Fixes #55 2024-10-30 13:27:43 -07:00
bkellam
8a619b7145 Changelog for #54 2024-10-29 15:14:45 -07:00
Jellyfrog
26779dd84d
GitLab: include subgroups when filtering on groups (#54) 2024-10-29 15:05:18 -07:00
Brendan Kellam
8019f316a7
[Fix] Change GitLab syncing behaviour for groups and users to include projects that are _not_ owned by the token (#51) 2024-10-25 12:01:01 -07:00
Brendan Kellam
82730f1cb0
Gitea support (#45) 2024-10-22 23:06:36 -04:00
Brendan Kellam
ae05d8f68e
Add support for remote configs (#43) 2024-10-17 16:20:09 -07:00
Brendan Kellam
fc8815d135
Config format V2 (#42) 2024-10-17 16:31:18 -04:00