Commit graph

68 commits

Author SHA1 Message Date
msukkari
d439933afc add repo index timeout logic 2025-03-03 10:31:56 -08:00
bkellam
d920d0d167 fix private repo clone issue for gitlab 2025-03-01 22:57:07 -08:00
Brendan Kellam
c48c0930a8
backend logging (#224) 2025-03-01 21:40:47 -08:00
msukkari
b27507eb7d catch and rethrow simplegit exceptions 2025-03-01 20:53:33 -08:00
msukkari
d33185c118 switch source map publish to runtime in entrypoint 2025-03-01 20:43:50 -08:00
Michael Sukkarieh
a93ee6527c
add sentry support to backend and webapp (#223)
* add sentry to web app

* set sentry environemnt from env var

* add sentry env replace logic in docker container

* wip add backend sentry

* add sentry to backend

* move dns to env var

* remove test exception
2025-03-01 19:21:17 -08:00
bkellam
51186fe87d Add prometheus metric for pending repo indexing jobs 2025-03-01 15:50:09 -08:00
msukkari
7ce10672e1 remove error msg dump on failed repo index job, and update indexedAt field 2025-02-27 16:04:31 -08:00
msukkari
d0b75ce57d fix repo rm issue, 502 page, condition on test clock 2025-02-27 13:11:30 -08:00
bkellam
fcf0b836ee fix build 2025-02-27 09:43:35 -08:00
Brendan Kellam
46be0440b0
Add back revision support (#215) 2025-02-27 09:39:55 -08:00
Brendan Kellam
386a3b52d7
remove git & local schemas (#213) 2025-02-26 16:55:10 -08:00
Brendan Kellam
50b94b2c46
Connections UX pass + query optimizations (#212) 2025-02-26 15:46:37 -08:00
msukkari
b77f55fa20 wipe existing repo if we've picked up a killed job to ensure good state 2025-02-26 12:46:24 -08:00
msukkari
8531b14a82 rename some prom metrics and cleanup unused 2025-02-25 18:54:01 -08:00
Michael Sukkarieh
3f586dd927
make repo garbage collection async (#211)
* add gc queue logic

* fix missing switch cases for gc status

* style org create form better with new staging domain

* change repo rm logic to be async

* simplify repo for inactive org query

* add grace period for garbage collecting repos

* make prom scrape interval 500ms
2025-02-25 15:49:09 -08:00
Michael Sukkarieh
fc33f26808
add grafana alloy config and setup (#210)
* add grafana alloy config and setup

* add basic repo prom metrics

* nits in dockerfile
2025-02-24 21:11:28 -08:00
Michael Sukkarieh
de44c81cfa
add posthog events on various user actions (#208)
* add page view event support

* add posthog events

* nit: remove unused import

* feedback
2025-02-24 17:06:29 -08:00
msukkari
0ac9509fab log upsert stats 2025-02-20 09:42:54 -08:00
msukkari
c930fe0d8b revert repo compile update logic to upsert loop 2025-02-19 18:54:50 -08:00
Michael Sukkarieh
fdd71cfcfe
add better visualization for connection/repo errors and warnings (#201)
* replace upsert with seperate create many and raw update many calls

* add bulk repo status update and queue addition with priority

* add support for managed redis

* add note for changing raw sql on schema change

* add error package and use BackendException in connection manager

* handle connection failure display on web app

* add warning banner for not found orgs/repos/users

* add failure handling for gerrit

* add gitea notfound warning support

* add warning icon in connections list

* style nits

* add failed repo vis in connections list

* added retry failed repo index buttons

* move nav indicators to client with polling

* fix indicator flash issue and truncate large list results

* display error nav better

* truncate failed repo list in connection list item

* fix merge error

* fix merge bug

* add connection util file [wip]

* refactor notfound fetch logic and add missing error package to dockerfile

* move repeated logic to function and add zod schema for syncStatusMetadata
2025-02-19 18:10:22 -08:00
msukkari
63af99aa59 remove non secret token options 2025-02-19 14:23:26 -08:00
Michael Sukkarieh
5d7a77bd92
db performance improvements and job resilience (#200)
* replace upsert with seperate create many and raw update many calls

* add bulk repo status update and queue addition with priority

* add support for managed redis

* add note for changing raw sql on schema change
2025-02-19 10:26:13 -08:00
msukkari
e09b21f6b9 add config upsert timeout env var 2025-02-15 17:27:24 -08:00
msukkari
e17331a672 fix build errors and add index concurrency env var 2025-02-15 16:58:05 -08:00
Michael Sukkarieh
54d14ea98e
add retries for 429 issues (#196)
* add connection compile retry and hard repo limit

* add more retry checks

* cleanup unused change

* address feedback
2025-02-15 16:37:50 -08:00
Michael Sukkarieh
e0d363420b
connections qol improvements (#195)
* add client side polling to connections list

* properly fetch repo image url

* add client polling to connection management page, and add ability to sync failed connections
2025-02-15 10:00:44 -08:00
Michael Sukkarieh
3be3680ee2
cleanup org's repos and shards if it's inactive (#194)
* add stripe subscription status and webhook

* add inactive org repo cleanup logic

* mark reactivated org connections for sync
2025-02-15 09:58:17 -08:00
Michael Sukkarieh
da33220289
add back gitlab, gitea, and gerrit support (#184)
* add non github config definitions

* refactor github config compilation to seperate file

* add gitlab config compilation

* Connection management (#183)

* wip gitlab repo sync support

* fix gitlab zoekt metadata

* add gitea support

* add gerrit support

* Connection management (#183)

* add gerrit config compilation

* Connection management (#183)

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
2025-02-14 10:58:53 -08:00
msukkari
875f9b8586 add await for blocking loop in backend 2025-02-13 10:30:14 -08:00
bkellam
8d2c731b88 Domain support 2025-02-12 13:45:12 -08:00
Brendan Kellam
846d73b0e6
Connection management (#183) 2025-02-04 15:04:05 -05:00
Michael Sukkarieh
dffbbae41d
adds garbage collection for repos (#182)
* refactor repo indexing logic into RepoManager

* wip cleanup stale repos

* add rest of gc logic

* set status to indexing properly
2025-01-29 09:05:20 -08:00
Michael Sukkarieh
31114a9d95
add concept of secrets (#180)
* add @sourcebot/schemas package

* migrate things to use the schemas package

* Dockerfile support

* add secret table to schema

* Add concept of connection manager

* Rename Config->Connection

* Handle job failures

* Add join table between repo and connection

* nits

* create first version of crypto package

* add crypto package as deps to others

* forgot to add package changes

* add server action for adding and listing secrets, create test page for it

* add secrets page to nav menu

* add secret to config and support fetching it in backend

* reset secret form on successful submission

* add toast feedback for secrets form

* add instructions for adding encryption key to dev instructions

* add encryption key support in docker file

* add delete secret button

* fix nits from pr review

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-27 14:07:07 -08:00
Brendan Kellam
dd8ff6edb0
Connection management (#178) 2025-01-24 13:16:08 -08:00
Brendan Kellam
bae7ca3438
Add @sourcebot/schemas package (#177) 2025-01-24 10:51:49 -08:00
msukkari
8a3521b080 updated syncedAt date after config sync: 2025-01-23 09:08:30 -08:00
Michael Sukkarieh
7c6adf17aa
Multi tenancy support in config syncer (#171)
* [wip] initial mt support in config syncer

* Move logout button & profile picture into settings dropdown (#172)

* update sync status properly and fix bug with multiple config in db case

* make config path required in single tenant mode

NOTE: deleting config/repos is currently not supported in multi tenancy case. Support for this will be added in a future PR

---------

Co-authored-by: Brendan Kellam <bshizzle1234@gmail.com>
2025-01-21 11:50:35 -08:00
Brendan Kellam
6cf10b4988
Authentication (#164) 2025-01-16 15:24:13 -08:00
Michael Sukkarieh
7029aa70c1
parallelize repo indexing (#163)
* hacked together a example of using zoekt grpc api

* provide tenant id to zoekt git indexer

* update zoekt version to point to multitenant branch

* pipe tenant id through header to zoekt

* remove incorrect submodule reference and settings typo

* update zoekt commit

* remove unused yarn script

* remove unused grpc client in web server

* remove unneeded deps and improve tenant id log

* pass tenant id when creating repo in db

* add mt yarn script

* add pol of bullmq into backend

* add better error handling and concurrency setting

* spin up redis instance in dockerfile

* cleanup transaction logic when adding repos to index queue

* add NEW index status fetch condition

* move bullmq deps to backend

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-15 15:44:42 -08:00
Michael Sukkarieh
553f5d25f7
Add tenant ID concept into web app and backend (#160)
* hacked together a example of using zoekt grpc api

* provide tenant id to zoekt git indexer

* update zoekt version to point to multitenant branch

* pipe tenant id through header to zoekt

* remove incorrect submodule reference and settings typo

* update zoekt commit

* remove unused yarn script

* remove unused grpc client in web server

* remove unneeded deps and improve tenant id log

* pass tenant id when creating repo in db

* add mt yarn script

* add nocheckin comment to tenant id in v2 schema

---------

Co-authored-by: bkellam <bshizzle1234@gmail.com>
2025-01-14 16:46:36 -08:00
Brendan Kellam
15b9e777e4
SQL Database (#157) 2025-01-14 13:37:31 -08:00
Michael Sukkarieh
d269a8cbe6
implement raw remote git repo support (#152)
* implement raw remote git repo support

* add changelog entry
2025-01-09 09:10:30 -08:00
Brendan Kellam
03aa608e21
Graceful error handling when calling code host apis (#142) 2024-12-18 19:21:21 -08:00
Brendan Kellam
0e9c829cc3
[GitHub] Add exclude.size property to the config (#137) 2024-12-17 00:01:19 -08:00
Brendan Kellam
c35f6bc5ae
Add reindexInterval and resyncInterval to config settings (#134) 2024-12-16 21:30:50 -08:00
Brendan Kellam
d4e72566e2
Fix directory not found exception when deleting stale repository (#136) 2024-12-16 20:23:42 -08:00
Brendan Kellam
4353d2008a
Add autoDeleteStaleRepos config option (#128) 2024-12-13 12:34:02 -08:00
Brendan Kellam
4d358f94a2
Add topics and exclude.topics to GitHub & GitLab config (#121) 2024-12-11 14:17:57 -08:00
Brendan Kellam
111023b1dc
Add config option to specify max file size (#118) 2024-12-09 12:34:43 -10:00