sourcebot/packages/backend/src/ee/syncSearchContexts.ts

177 lines
6 KiB
TypeScript
Raw Normal View History

2025-04-25 05:28:13 +00:00
import micromatch from "micromatch";
import { createLogger } from "@sourcebot/shared";
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
import { PrismaClient } from "@sourcebot/db";
import { getPlan, hasEntitlement, SOURCEBOT_SUPPORT_EMAIL } from "@sourcebot/shared";
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
import { SearchContext } from "@sourcebot/schemas/v3/index.type";
const logger = createLogger('sync-search-contexts');
2025-04-25 05:28:13 +00:00
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
interface SyncSearchContextsParams {
contexts?: { [key: string]: SearchContext } | undefined;
orgId: number;
db: PrismaClient;
}
export const syncSearchContexts = async (params: SyncSearchContextsParams) => {
const { contexts, orgId, db } = params;
2025-04-25 05:28:13 +00:00
if (!hasEntitlement("search-contexts")) {
if (contexts) {
const plan = getPlan();
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
logger.warn(`Skipping search context sync. Reason: "Search contexts are not supported in your current plan: ${plan}. If you have a valid enterprise license key, pass it via SOURCEBOT_EE_LICENSE_KEY. For support, contact ${SOURCEBOT_SUPPORT_EMAIL}."`);
2025-04-25 05:28:13 +00:00
}
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
return false;
2025-04-25 05:28:13 +00:00
}
if (contexts) {
for (const [key, newContextConfig] of Object.entries(contexts)) {
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
const allRepos = await db.repo.findMany({
2025-04-25 05:28:13 +00:00
where: {
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
orgId,
2025-04-25 05:28:13 +00:00
},
select: {
id: true,
name: true,
}
});
let newReposInContext: { id: number, name: string }[] = [];
if(newContextConfig.include) {
newReposInContext = allRepos.filter(repo => {
return micromatch.isMatch(repo.name, newContextConfig.include!);
});
}
if(newContextConfig.includeConnections) {
const connections = await db.connection.findMany({
where: {
orgId,
name: {
in: newContextConfig.includeConnections,
}
},
include: {
repos: {
select: {
repo: {
select: {
id: true,
name: true,
}
}
}
}
}
});
for (const connection of connections) {
newReposInContext = newReposInContext.concat(connection.repos.map(repo => repo.repo));
}
}
2025-04-25 05:28:13 +00:00
if (newContextConfig.exclude) {
const exclude = newContextConfig.exclude;
newReposInContext = newReposInContext.filter(repo => {
return !micromatch.isMatch(repo.name, exclude);
});
}
if (newContextConfig.excludeConnections) {
const connections = await db.connection.findMany({
where: {
orgId,
name: {
in: newContextConfig.excludeConnections,
}
},
include: {
repos: {
select: {
repo: {
select: {
id: true,
name: true,
}
}
}
}
}
});
for (const connection of connections) {
newReposInContext = newReposInContext.filter(repo => {
return !connection.repos.map(r => r.repo.id).includes(repo.id);
});
}
}
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
const currentReposInContext = (await db.searchContext.findUnique({
2025-04-25 05:28:13 +00:00
where: {
name_orgId: {
name: key,
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
orgId,
2025-04-25 05:28:13 +00:00
}
},
include: {
repos: true,
}
}))?.repos ?? [];
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
await db.searchContext.upsert({
2025-04-25 05:28:13 +00:00
where: {
name_orgId: {
name: key,
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
orgId,
2025-04-25 05:28:13 +00:00
}
},
update: {
repos: {
connect: newReposInContext.map(repo => ({
id: repo.id,
})),
disconnect: currentReposInContext
.filter(repo => !newReposInContext.map(r => r.id).includes(repo.id))
.map(repo => ({
id: repo.id,
})),
},
description: newContextConfig.description,
},
create: {
name: key,
description: newContextConfig.description,
org: {
connect: {
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
id: orgId,
2025-04-25 05:28:13 +00:00
}
},
repos: {
connect: newReposInContext.map(repo => ({
id: repo.id,
})),
}
}
});
}
}
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
const deletedContexts = await db.searchContext.findMany({
2025-04-25 05:28:13 +00:00
where: {
name: {
notIn: Object.keys(contexts ?? {}),
},
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
orgId,
2025-04-25 05:28:13 +00:00
}
});
for (const context of deletedContexts) {
logger.info(`Deleting search context with name '${context.name}'. ID: ${context.id}`);
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
await db.searchContext.delete({
2025-04-25 05:28:13 +00:00
where: {
id: context.id,
}
})
}
fix(search-contexts): Fix issue where a repository would not appear in a search context if it was created after the search context was created (#354) ## Problem If a repository is added **after** a search context (e.g., a new repository is synced from the code host), then it will never be added to the context even if it should be included. The workaround is to restart the instance. ## Solution This PR adds a call to re-sync all search contexts whenever a connection is successfully synced. This PR adds the `@sourcebot/shared` package that contains `syncSearchContexts.ts` (previously in web) and it's dependencies (namely the entitlements system). ## Why another package? Because the `syncSearchContexts` call is now called from: 1. `initialize.ts` in **web** - handles syncing search contexts on startup and whenever the config is modified in watch mode. This is the same as before. 2. `connectionManager.ts` in **backend** - syncs the search contexts whenever a connection is successfully synced. ## Follow-up devex work Two things: 1. We have several very thin shared packages (i.e., `crypto`, `error`, and `logger`) that we can probably fold into this "general" shared package. `schemas` and `db` _feels_ like they should remain separate (mostly because they are "code-gen" packages). 2. When running `yarn dev`, any changes made to the shared package will only get picked if you `ctrl+c` and restart the instance. Would be nice if we have watch mode work across package dependencies in the monorepo.
2025-06-17 21:04:25 +00:00
return true;
2025-04-25 05:28:13 +00:00
}