Conversation
* feat: add support for postgresql to grafana dashboards * feat: add support for postgresql to grafana dashboards * fix(grafana): improve mysql to postgresql conversion script - Disable ambiguous alias pattern that caused _computed suffix duplication - Add MySQL FIELD() function to PostgreSQL CASE WHEN conversion - Fix adoption_pct and deploy_count column name issues * chore(grafana): remove archived dashboards Remove outdated dashboard files from _archive directory * refactor(grafana): rename mysql dashboard files to kebab-case Standardize dashboard filenames from CamelCase to kebab-case for consistency * feat(grafana): regenerate postgresql dashboards with improved conversion - Fix _computed suffix duplication in column aliases - Convert MySQL FIELD() to PostgreSQL CASE WHEN - Apply all SQL dialect fixes from updated conversion script - Rename dashboard files to kebab-case for consistency * feat(docker): split mysql and postgresql compose configurations - Rename docker-compose-dev.yml to docker-compose-dev-postgresql.yml - Create separate docker-compose-dev-mysql.yml - Use different project names to avoid conflicts - Assign unique ports for each stack (MySQL: 3001/8083/4001/4181, PostgreSQL: 3002/8084/4000/4180) * feat(docker): add mysql development compose configuration Create dedicated compose file for MySQL development environment with unique ports and project name * fix(grafana): disable folder structure from dashboard provisioning Set foldersFromFilesStructure to false to prevent creating postgresql/mysql subdirectories in Grafana UI * refactor(grafana): combine RUN commands in Dockerfile Merge plugin installation and permission setup into single RUN statement * fix: fixing more dashboards * fix: fixing dora dashboard * fix: update license information in convert-mysql-to-postgresql.py --------- Co-authored-by: Fábio Luciano <fabio.gois@sicoob.com.br>
…e#8899) * fix(jenkins): truncate primary_view to fit varchar(255) column Fixes Error 1406 (22001): Data too long for column 'primary_view' by truncating the concatenated value to 255 characters before saving. Closes apache#8897 * ci: exclude generated mocks from golangci-lint
…ency (apache#8811) * fix: Add backfill time window to ensure data consistency chore: Add backfill time window to ensure data consistency (cherry picked from commit 52a9555b782860f9c8ba9db409bd56e0c8f58272) * fix(q_dev): prevent data duplication in user_report and user_data tables (apache#8737) * fix(q_dev): prevent data duplication in user_report and user_data tables Replace auto-increment ID with composite primary keys so that CreateOrUpdate can properly deduplicate rows on re-extraction. - user_report PK: (connection_id, scope_id, user_id, date, client_type) - user_data PK: (connection_id, scope_id, user_id, date) - Switch db.Create() to db.CreateOrUpdate() in s3_data_extractor - Migration drops old tables, rebuilds with new PKs, resets s3_file_meta processed flag to trigger re-extraction * fix(q_dev): gofmt archived user_data_v2 model * feat(github): Extend exclusion of file extensions to github plugin (apache#8719) * feat(github): extend PR size exclusion for specified file extension to github plugin * fix: register migration script * fix: move PR size to 'Additional settings' and change so the comma doesn't get removed while typing * fix: linting * fix(doc): update expired Slack invite links in README (apache#8739) The Slack invite links in README.md were expired and returning "This link is no longer active." Updated both occurrences (badge and community section) to match the current link on the official DevLake website. Closes apache#8738 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * docs: add gh-devlake CLI to Getting Started installation options (apache#8733) Adds gh-devlake as a third installation method alongside Docker Compose and Helm. gh-devlake is a GitHub CLI extension that automates DevLake deployment, configuration, and monitoring from the terminal. Closes apache#8732 * fix(gitlab): add missing repos scope in project_mapping (apache#8743) GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * feat(circleci): add server version requirement and endpoint help text (apache#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * feat(asana): add Asana plugin for project and task collection (apache#8758) Add a new Asana plugin that integrates with Asana's REST API to collect projects, sections, tasks, subtasks, stories (comments), tags, and users, mapping them to DevLake's ticket/board domain model. Backend: - Plugin implementation with all required interfaces (PluginMeta, PluginTask, PluginModel, PluginMigration, PluginSource, PluginApi, DataSourcePluginBlueprintV200) - Collectors, extractors, and converters for projects, sections, tasks, subtasks, stories, tags, and users - Remote API scope picker (Workspaces -> Teams/Portfolios -> Projects) - Scope config with issue-type regex transformation rules - Migration scripts for schema evolution - E2E tests with CSV fixtures for project and task data flows Config UI: - Plugin registration with connection form (PAT auth, endpoint, proxy) - Scope config transformation form for issue-type mapping - Dashboard URL integration for onboarding flow Grafana: - Asana dashboard with task metrics and visualizations Made-with: Cursor * feat: GitHub App token refresh (apache#8746) * feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> * fix: cwe89 sql injection (apache#8762) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards (apache#8767) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards Add support for ingesting S3 logging data (GenerateAssistantResponse and GenerateCompletions events) into new database tables, and enrich all three Kiro Grafana dashboards with additional metrics. Changes: - New models: QDevChatLog and QDevCompletionLog for logging event data - New extractor: s3_logging_extractor.go parses JSON.gz logging files - Updated S3 collector to also handle .json.gz files - Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions) - New dashboard: "Kiro AI Activity Insights" with 10 panels including model usage distribution, active hours, conversation depth, feature adoption (Steering/Spec), file type usage, and prompt/response trends - Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration, and Dev (Agentic) metric panels - Fixed "Kiro Usage Dashboard" per-user table to sort by user_id - Migration script for new tables * fix(q-dev): use separate base path for logging S3 prefixes Logging data lives under a different S3 prefix ("logging/") than user report data ("user-report/"). Add LoggingBasePath option (defaults to "logging") so logging prefixes are constructed correctly. * fix(q-dev): auto-scan logging path without extra config Kiro exports to two well-known S3 prefixes in the same bucket: - user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports) - logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs) When AccountId is set, automatically scan both paths. The "logging" prefix is hardcoded since it's a standard Kiro export convention. No additional configuration needed. * fix(q-dev): update scope tooltip to mention logging data scanning * fix(q-dev): fix scope ID routing and CSV/JSON file separation Three fixes: 1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope IDs containing "/" (e.g. "034362076319/2026") work in URL paths 2. CSV extractor now filters for .csv files only, preventing it from trying to parse .json.gz logging files as CSV 3. Frontend scope API calls now encodeURIComponent(scopeId) for safe URL encoding * fix(q-dev): resolve *scopeId route conflict with dispatcher pattern The catch-all *scopeId route conflicts with *scopeId/latest-sync-state. Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a GetScopeDispatcher that checks for /latest-sync-state suffix and dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId. * fix(q-dev): use URL-safe scope ID format (underscore separator) Scope IDs like "034362076319/2026" break URL routing because "/" is a path separator. Change ID format to "034362076319_2026" (underscore) when AccountId is set. The Prefix field still uses "/" for S3 path matching. Revert to standard :scopeId routes since IDs are now safe. Note: existing scopes need to be recreated after this change. * fix(q-dev): use NoPKModel instead of Model in archived logging models archived.Model only has ID+timestamps, missing RawDataOrigin fields (_raw_data_params etc.) that common.NoPKModel includes. This caused "Unknown column '_raw_data_params'" errors at runtime. * fix(q-dev): fix GROUP BY in per-user table to merge display_name variants Remove display_name from GROUP BY so same user_id with different display_name values gets merged. Use MAX(display_name) in SELECT. * fix(q-dev): normalize logging user IDs to match CSV short UUID format Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to one user_id across both data sources. * fix(q-dev): normalize user IDs in CSV extractors and sort table DESC Apply normalizeUserId to both createUserReportData and createUserDataWithDisplayName so user_report CSV data also strips the "d-{directoryId}." prefix. Change per-user table sort to ORDER BY user_id DESC. * style(q-dev): fix gofmt formatting in chat_log models * perf(q-dev): parallelize logging S3 downloads and batch DB writes Optimize logging extractor performance: - 10 goroutine workers for parallel S3 file downloads - Batch 50 files per DB transaction instead of 1-per-file - sync.Map cache for display name resolution (avoid repeated IAM calls) - Parse records in memory during download, write all at once This should improve throughput from ~1.5 files/sec to ~15+ files/sec for typical logging file sizes. * fix(q-dev): check tx.Rollback error return to satisfy errcheck lint * feat(q-dev): add per-user model usage table and models column Add "Per-User Model Usage" table (panel 11) showing each user's request count and avg prompt/response length per model_id. Also add "Models Used" column to the Per-User Activity table. * fix(q-dev): remove per-user model usage table, keep models column only * feat(q-dev): add Kiro Executive Dashboard with cross-source analytics New dashboard "Kiro Executive Dashboard" with 12 panels covering: - KPIs: WAU, credits efficiency, acceptance rate, steering adoption - Trends: weekly active users, new vs returning users - Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec - Cost: credits pace vs projected monthly, idle power users - Quality: acceptance rate trends, code review findings, test generation - Efficiency: per-user productivity table with credits/line ratio Correlates data across user_report (credits), user_data (code metrics), and chat_log (interaction patterns) for holistic Kiro usage insights. * fix(q-dev): fix pie charts to show per-row slices instead of single total Set reduceOptions.values=true so Grafana treats each SQL result row as a separate pie slice. Fixes Model Usage Distribution, File Type Usage, Kiro Feature Adoption, and Active File Types pie charts. * fix(q-dev): cast Hour to string for Active Hours bar chart x-axis * fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues 1. qdev_user_report Panel 4 (Subscription Tier Distribution): set reduceOptions.values=true to show per-tier slices 2. qdev_user_data Panel 6 (User Interactions): remove display_name from GROUP BY, use MAX(display_name) to merge same user * fix(q-dev): prevent data inflation in user_report JOIN user_data user_report has multiple rows per (user_id, date) due to client_type (KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date). A direct JOIN causes user_data metrics to be counted multiple times. Fix: pre-aggregate user_report by (user_id, date) in a subquery before joining, so the JOIN is always 1:1. Affects: Credits Efficiency stat and User Productivity table. * feat(qa): add is_invalid field to qa_test_case_executions (apache#8764) * feat(qa): add is_invalid field to qa_test_case_executions Add is_invalid boolean field to the domain layer qa_test_case_executions table to allow QA teams to flag test executions as invalid due to environmental issues, flaky tests, false positives, or false negatives. Changes: - Add IsInvalid field to QaTestCaseExecution domain model - Create migration script (20260313_add_is_invalid_to_qa_test_case_executions) - Register migration in migrationscripts/register.go - Update customize service to set default value for is_invalid - Update E2E test data to include new column Resolves apache#8763 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(qa): handle missing is_invalid column in CSV import Fix PostgreSQL compatibility issue when CSV files don't contain the is_invalid column. The field now defaults to false instead of an empty string. Changes: - Update qaTestCaseExecutionHandler to check for empty string values - Add E2E test for backward compatibility with CSV files lacking is_invalid - Add explicit IsInvalid initialization in Testmo plugin converter Resolves apache#8763 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(linker): link when branch names contain issue keys (apache#8777) * feat(linker): branch names containing issue keys * chore: add testing data * Add codespell support with configuration and fixes (apache#8761) * ci(codespell): add codespell config and GitHub Actions workflow Add .codespellrc with skip patterns for generated files, camelCase/PascalCase ignore-regex, and project-specific word list (convertor, crypted, te, thur). Add GitHub Actions workflow to run codespell on push to main and PRs. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix ambiguous typos requiring context review Manual fixes for typos that needed human review to avoid breaking code: - Comment/string typos: occured->occurred, destory->destroy, writting->writing, retreive->retrieve, identifer->identifier, etc. - Struct field comments and documentation corrections - Migration script comment fixes (preserving Go identifiers like DataConvertor) Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> * fix(codespell): fix non-ambiguous typos with codespell -w Automated fix via `codespell -w` for clear-cut typos across backend, config-ui, and grafana dashboards. Examples: sucess->success, occurence->occurrence, exeucte->execute, asynchornous->asynchronous, Grafana panel typos, etc. Co-Authored-By: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> --------- Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: Claude Code 2.1.63 / Claude Opus 4.6 <noreply@anthropic.com> * feat(q-dev): enrich logging fields, separate dashboards, add E2E tests (apache#8786) * feat(q-dev): enrich logging fields, separate dashboards by data source, add E2E tests - Add new fields to chat_log: CodeReferenceCount, WebLinkCount, HasFollowupPrompts (from codeReferenceEvents, supplementaryWebLinksEvent, followupPrompts in JSON) - Add new fields to completion_log: LeftContextLength, RightContextLength (from leftContext/rightContext in JSON) - Update s3_logging_extractor to parse and populate new fields - Add migration script 20260319_add_logging_fields - Create qdev_feature_metrics dashboard for legacy by_user_analytic data - Reorganize qdev_executive dashboard with Row dividers labeling data sources and cross-dashboard navigation links - Enrich qdev_logging dashboard with new panels: Chat Trigger Type Distribution, Response Enrichment Breakdown, Completion Context Size Trends, Response Enrichment Trends - Fix SQL compatibility with only_full_group_by mode in executive dashboard (Weekly Active Users Trend, New vs Returning Users) - Fix Steering Adoption stat panel returning string instead of numeric value - Add Playwright E2E test covering full pipeline flow and dashboard verification * fix: add Apache license headers to e2e files, fix gofmt alignment * fix: add SQL identifier validation to prevent SQL injection via table/column names (apache#8769) Add ValidateTableName and ValidateColumnName functions in core/dal to ensure table and column names used in dynamic SQL are safe identifiers. Applied to scope_service_helper, scope_generic_helper, and customized_fields_extractor. * feat(q-dev): add Kiro Credits + DORA Correlation dashboard (apache#8792) Add a new Grafana dashboard that correlates Kiro AI usage (credits, messages, active users) with DORA metrics at weekly aggregate level. Panels include: - Pearson's r correlation between weekly credits and PR cycle time - High AI Usage vs Low AI Usage cycle time comparison - Weekly credits vs deployment frequency trend - Weekly credits vs change failure rate trend Data is joined by week_start between _tool_q_dev_user_report and project_pr_metrics / cicd_deployment_commits. * feat(q-dev): add AI Cost-Efficiency dashboard (apache#8793) Add a Grafana dashboard showing AI tool cost-efficiency metrics: - Credits per merged PR (overall + weekly trend) - Credits per production deployment (overall + weekly trend) - Credits per issue resolved (overall + weekly trend) - Weekly AI activity volume (credits, messages, conversations) Joins _tool_q_dev_user_report with pull_requests, cicd_deployment_commits, and issues by weekly aggregation. * feat(q-dev): add Multi-AI Tool Comparison dashboard (Copilot vs Kiro) (apache#8794) Add a Grafana dashboard comparing GitHub Copilot and Kiro side by side: - Weekly active users comparison - Code suggestions & acceptance events (per tool) - LOC accepted comparison (combined time series) - Acceptance rate comparison (bar gauge) Template variables for Copilot connection/scope selection. Data from _tool_copilot_enterprise_daily_metrics vs _tool_q_dev_user_report and _tool_q_dev_user_data. * feat(q-dev): add Kiro AI Model ROI dashboard (apache#8795) Add a Grafana dashboard analyzing per-model performance from chat logs: - Model Performance Summary table (requests, share%, avg prompt/response length, response/prompt ratio, steering/spec mode usage) - Daily Model Usage Distribution (stacked bar chart) - Avg Response Length by Model trend (output quality proxy) Data source: _tool_q_dev_chat_log grouped by model_id. * feat(q-dev): add Steering & Spec Mode Adoption dashboard (apache#8798) Track Kiro steering rules and spec mode adoption: - User/request adoption rate stats - Weekly adoption rate trend - Steering impact on prompt/response length - Per-user feature adoption table * feat(q-dev): add Developer AI Productivity Hours dashboard (apache#8797) Analyze when developers are most productive with AI tools: - AI Activity by Hour of Day (chat + completions stacked bar) - Prompt & Response Length by Hour (complexity patterns) - Feature Usage by Hour (steering/spec mode/plain chat) - AI Activity by Day of Week * feat(q-dev): add Language AI Heatmap dashboard (apache#8796) Analyze AI-assisted coding patterns by programming language: - Language Completion Profile table (requests, avg completions, context sizes, users per language) - Daily Completions by Language (stacked bar) - Active File Types During Chat (donut) - Avg Context Size by Language trend (top 5) * Fix/circleci column names (apache#8799) * fix(circleci): rename created_at to created_date in jobs/workflows Add migration to copy created_at -> created_date and update models/converters. * fix(circleci): update pipeline parsing * test(circleci): add incremental tests for collectors * fix(jenkins): scope multi-branch build collection to current project (apache#8430) (apache#8781) The branch jobs query in collectMultiBranchJobApiBuilds selected all WorkflowJob entries across all multi-branch pipelines for a connection, causing builds to be duplicated and misattributed. Filter by _raw_data_params to collect only the current project's branch jobs. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Make gh-copilot plugin database agnostic (apache#8779) Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> * fix(sonarqube): increase cq_issues and cq_file_metrics project_key length to 500 (apache#8783) Fixes apache#8331 * feat: added taiga plugin (apache#8755) * feat: added taiga plugin * fix: fixed tests * feat(gh-copilot): add support for organization daily user metrics (apache#8747) * feat(circleci): add server version requirement and endpoint help text (apache#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix: fixed test files --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> Co-authored-by: Reece Ward <47779818+ReeceXW@users.noreply.github.com> Co-authored-by: Joshua Smith <jbsmith7741@gmail.com> * fix(docker): pin Poetry to 2.2.1 for Python 3.9 compatibility (apache#8735) Poetry 2.3.0 dropped Python 3.9 support. Without cache the installer fetches the latest version (currently 2.3.2), which fails on the python:3.9-slim-bookworm base image. Pin to 2.2.1, the last release compatible with Python 3.9. Co-authored-by: Rodrigo Silva <rodrigo.silva@bonial.com> * fix(linker): scope clearHistoryData to current project only (apache#8814) (apache#8815) The clearHistoryData() function used a LEFT JOIN with project_name in the ON clause, causing the subquery to return all PR IDs regardless of project. This effectively wiped the entire pull_request_issues table on every linker run, deleting links from other projects sharing the same repos and links created by the GitHub converter. Fix: - Use INNER JOIN + WHERE for proper project scoping - Add issue-side subquery scoped to current project's boards - Filter by _raw_data_table/_raw_data_remark to only delete linker-created rows Add e2e test for cross-project shared repo scenario. * fix(circleci): prevent negative values when calculating circleci (apache#8800) workflow duration * fix: sonarqube: missing api/users/search endpoint (apache#8813) * fix(argocd): extract revision from multi-source application revisions[] (apache#8810) --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> Signed-off-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: tamas.albert <tamas.laczkoalbert@concentrix.com> Co-authored-by: Warren Chen <warren.chen830@gmail.com> Co-authored-by: Ema Abitante <ema.abitante@gmail.com> Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com> Co-authored-by: Joshua Smith <jbsmith7741@gmail.com> Co-authored-by: jawad khan <jawadkhan444@gmail.com> Co-authored-by: Leif Roger Frøysaa <leif.roger.froysaa@akerbp.com> Co-authored-by: Klesh Wong <klesh@qq.com> Co-authored-by: NaRro <cong.wang@merico.dev> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Reece Ward <47779818+ReeceXW@users.noreply.github.com> Co-authored-by: Yaroslav Halchenko <debian@onerussian.com> Co-authored-by: Chris Pavlicek <varsis@users.noreply.github.com> Co-authored-by: AvivGuiser <avivguiser@gmail.com> Co-authored-by: Shayne Clausson <shayne.clausson@extendaretail.com> Co-authored-by: irfanuddinahmad <34648393+irfanuddinahmad@users.noreply.github.com> Co-authored-by: Rodrigo Silva <rodrigoluizscs@gmail.com> Co-authored-by: Rodrigo Silva <rodrigo.silva@bonial.com> Co-authored-by: Daniele M. <github@dmoraschi.com> Co-authored-by: Pavel Sturc <psturc@redhat.com> Co-authored-by: Anvesh Vemula <39478419+vemulaanvesh@users.noreply.github.com>
* feat(tempo): add Jira Tempo Timesheets plugin Adds plugin for ingesting worklogs from Jira Tempo (Tempo Timesheets) API v4. Backend: - Plugin entry point (impl/impl.go) - Tempo API client with Bearer token auth (tasks/api_client.go) - Data collectors, extractors, and converters for worklogs and teams - Connection management API endpoints (api/) - Database migrations for _tool_tempo_worklogs, _tool_tempo_teams - E2E tests with CSV fixtures Config-UI: - Connection configuration UI (config.tsx) - Plugin registration in index.ts Dependency: requires Jira plugin for issue ID mapping. Closes apache#8883 * fix(tempo): resolve CI failures on tempo plugin PR - fix Apache license headers in 5 files (missing/typo content) - move migration script to archived models pattern (forbidden import on plugins/tempo/models) - register tempo in plugins/table_info_test.go - drop unused validator var/import in api/init.go - fix malformed issue_worklogs.csv snapshot (header/body column mismatch) * fix(tempo): align e2e team fixtures with extractor expectations The team extractor queries _raw_tempo_api_teams with the params serialized from TempoApiParams{ConnectionId, TeamId}, expecting the raw fixture rows to store each team as its own record with a single-object data payload. The previous fixture wrapped both teams into a JSON array in the data column and used only ConnectionId in params, causing TestTeamDataFlow to find zero matching rows. Also extend the _tool_tempo_teams snapshot with the _raw_data_* columns that the test verifies via ColumnWithRawData. --------- Co-authored-by: Fix Bot <fix@example.com>
* feat(gh-copilot): close API gaps for per-user metrics, teams, CLI, code review, and PR fields Add missing GitHub Copilot Metrics API fields to achieve full API parity: Enterprise/Org metrics: - CLI active user counts and CLI breakdown (sessions, requests, tokens) - Code review user counts (daily/weekly/monthly × active/passive) - Chat panel mode breakdown (agent/ask/custom/edit/plan/unknown) - Expanded PR metrics (merged, merge time, suggestions, Copilot impact) Per-user metrics: - used_cli, used_copilot_code_review_active/passive boolean flags - CLI breakdown per user (sessions, requests, tokens) User-team mapping (new): - New collector/extractor for user-teams-1-day endpoint - Enables team-level metrics via JOIN with per-user tables Seat assignments: - Team assignment fields (assigning_team id/name/slug) - User detail fields (name, email) Includes migration script 20260527 and comprehensive docs in COPILOT_METRICS_GAPS.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(copilot-metrics): remove outdated metrics gaps documentation and implement new user-team mapping and metrics enhancements Signed-off-by: Jarek <jaroslaw.gajewski@atos.net> --------- Signed-off-by: Jarek <jaroslaw.gajewski@atos.net> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ion (apache#8907) (apache#8912) * refactor(plugin-circleci): extract unfinished-jobs input clauses into a helper Signed-off-by: Volodymyr Zahorniak <v.zahorniak@gmail.com> * fix(plugin-circleci): populate workflow id for unfinished-job collection (apache#8907) The collectJobs 'unfinished details' collector built its URL from '/v2/workflow/{{ .Input.Id }}/job' but its iterator selected 'DISTINCT workflow_id' into a models.CircleciJob, leaving .Id empty and producing '/v2/workflow//job' (HTTP 500) whenever a job was running/queued/on_hold. Alias the projection to 'workflow_id AS id' so .Id carries the workflow id, mirroring the new-records collector. Adds an e2e regression test. Signed-off-by: Volodymyr Zahorniak <v.zahorniak@gmail.com> --------- Signed-off-by: Volodymyr Zahorniak <v.zahorniak@gmail.com> Co-authored-by: Klesh Wong <klesh@qq.com>
* fix(github-graphql): prevent panic in graphql rate limit polling goroutine Replace panic in GraphqlAsyncClient rate-limit polling goroutine with graceful error handling. Previously, any error while fetching rate limit (e.g., transient network issues or 401 responses) would trigger a panic inside a background goroutine, crashing the entire DevLake process. Now, errors are logged and the client retries in the next cycle while retaining the last known rate limit. Design decisions: - Avoid panic in background goroutines: rate-limit polling is non-critical and should not bring down the entire pipeline. - Use last known rateRemaining on runtime failures instead of resetting or blocking, ensuring continued progress with eventual consistency. - Retry via existing polling mechanism instead of immediate retry to prevent tight retry loops and unnecessary API pressure. - Introduce a default fallback (5000) only for initial rate-limit fetch failures, since no prior state exists at startup. - Separate handling of initial vs runtime failures: - Initial failure → fallback to default (5000) - Runtime failure → retain previous value Fixes apache#8788 (bug 1) * fix(github-graphql): reuse ApiClient transport for GraphQL to enable token refresh Replace oauth2.StaticTokenSource-based HTTP client with the underlying http.Client from ApiAsyncClient. Previously, the GraphQL client constructed its own HTTP client using StaticTokenSource, which froze the access token at task start time. This caused GitHub App installation tokens (which expire after ~1 hour) to become invalid during long-running pipelines, leading to persistent 401 errors. Now, the GraphQL client reuses apiClient.GetClient(), which is already configured with RefreshRoundTripper and TokenProvider. This enables automatic token refresh on 401 responses, aligning GraphQL behavior with the REST client. Design decisions: - Reuse transport layer instead of duplicating authentication logic to ensure consistency across REST and GraphQL clients. - Avoid StaticTokenSource, as it prevents token refresh and breaks long-running pipelines. - Leverage existing RefreshRoundTripper for transparent token rotation without modifying GraphQL query logic. - Keep protocol-specific logic (GraphQL vs REST) separate while sharing the underlying HTTP transport. This ensures GraphQL pipelines using GitHub App authentication can run beyond token expiry without failure. Fixes apache#8788 (bug 2) * refactor(github): extract shared authenticated http client from api client - moved token provider and refresh round tripper setup into a reusable helper - introduced CreateAuthenticatedHttpClient to centralize auth + transport logic - updated CreateApiClient to use shared http client instead of inline setup Rationale: - decouples authentication (transport layer) from REST-specific client logic - enables reuse for GraphQL client without duplicating token refresh logic - aligns architecture with separation of concerns (http transport vs api clients) * feat(github-graphql): introduce graphql client with shared auth and integrate into task flow - added CreateGraphqlClient to encapsulate graphql client construction - reused CreateAuthenticatedHttpClient from github/tasks to inject token refresh via RoundTripper - replaced manual graphql client setup in PrepareTaskData with new factory function - preserved existing rate limit handling via getRateRemaining callback - preserved query cost calculation using SetGetRateCost Technical details: - graphql client now uses http transport with TokenProvider and RefreshRoundTripper - removes dependency on oauth2 client and avoids token expiration issues - decouples graphql client from REST ApiClient by avoiding reuse of apiClient.GetClient() - maintains compatibility with github.com and enterprise graphql endpoints Note: - shared auth logic remains in github/tasks and is imported with alias to avoid package name collision - introduces cross-plugin dependency (github_graphql → github/tasks) as a pragmatic tradeoff to avoid duplication * feat(github): support static token transport for GraphQL and REST clients add StaticRoundTripper for PAT authentication and use it in the shared http client. since the same client is used by both REST and GraphQL, auth handling must distinguish between refreshable tokens and static tokens. avoid applying refresh/retry logic to PAT. ensures correct behavior across clients and prevents unnecessary retries for static auth. * feat(github-graphql): introduce hierarchical fallback for GraphQL rate limit Implement a layered fallback mechanism for GraphQL rate limiting: 1. Dynamic rate limit from provider (getRateRemaining) 2. Per-client override (WithFallbackRateLimit) 3. Config override (GRAPHQL_RATE_LIMIT) 4. Default fallback (1000) Also moved GitHub-specific fallback (5000) via WithFallbackRateLimit to the Graphql client. * feat(github-graphql): Add graphql rate limit to .env example * fix(github): Fix leaked debug statement * fix(github-graphql): reuse http.Client proxy, auth configurations Reused `http.Client` inside the apiClient returned by `CreateApiClient` method, so keeping the proxy and auth configurations the same.That also keep the centralized management of logic. * fix(helpers): fix the priority order of fallback rate limit Priority order fixed for fallback rate limit, priority order is: 1.Env variable 2.Value set with `WithFallbackRateLimit` 3.default value in the code This all works only when the `getRateRemaining` fails: hence the fallback * fix(github): StaticRoundTripper now owns token splitting and rotation for AccessToken connections Previously, connection.Token (comma-separated PATs) was injected as-is into the Authorization header, sending "Bearer tok1,tok2,tok3" instead of a single rotated token. StaticRoundTripper now splits the raw token string on comma and rotates through tokens round-robin using an atomic counter. For REST: StaticRoundTripper operates at transport level and always overwrites the Authorization header set by SetupAuthentication. SetupAuthentication is retained because conn.tokens is still required by GetTokensCount() for rate limit calculation — but its header write is superseded by StaticRoundTripper on every request. For GraphQL: SetupAuthentication is never called by the graphql client, so StaticRoundTripper is the only auth mechanism on this path — without this fix, GraphQL requests were sent with the full unsplit token string. * refactor(github-graphql): Downgrade fetch failure logs from Warn to Info * fix(helper): use inline func type for GraphqlClientOption to avoid mock cycle Replace exported GraphqlClientOption type with inline func(*GraphqlAsyncClient) in CreateAsyncGraphqlClient signature. The named type caused mockery to generate a mock file (GraphqlClientOption.go) that created an import cycle in tests. * style(github): fix linting * fix(github): token rotation start from index * fix(helper): prevent graphql deadlock when rate limit fetch keeps failing --------- Co-authored-by: Klesh Wong <klesh@qq.com>
…#8886) (apache#8894) * fix(github): create domain accounts for non-committer authors (apache#8886) ConvertAccounts sourced the domain `accounts` table FROM _tool_github_accounts, which is only populated for users we collected full profiles for (effectively, committers). Issue and PR authors who never committed were written into _tool_github_repo_accounts but never converted, so issues.creator_id and pull_requests.author_id pointed at accounts rows that didn't exist. Source ConvertAccounts FROM _tool_github_repo_accounts LEFT JOIN _tool_github_accounts instead, so every user the repo references gets a domain account, enriched with profile detail when we have it and login-only otherwise. The domain id uses the same generator the issue/PR convertors use, so the FKs line up. Also emit a repo_account for a PR's merged_by user so pull_requests.merged_by_id resolves too. The query stays MySQL/PostgreSQL-agnostic (COALESCE, no backtick quoting, parameterized via the dal) and mirrors the join already in account_org_collector.go. Adds the non-committer orphan case to the e2e fixture plus a referential- integrity assertion in TestAccountDataFlow. Verified on both MySQL and PostgreSQL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github): carry _raw_data provenance and guard zero account ids Review feedback on apache#8894: the rewritten ConvertAccounts dropped the _raw_data fields from converted accounts. Select them with COALESCE, preferring the enriched _tool_github_accounts row and falling back to the _tool_github_repo_accounts row for non-committers. The e2e fixture now carries the _raw_data columns like every other tool fixture, and the regenerated snapshot pins the pre-existing provenance for enriched accounts plus the issue-extractor provenance for the non-committer case. Also guard PR author and merged-by id generation against zero ids: an unmerged PR or a deleted user otherwise yields the domain id github:GithubAccount:<conn>:0, the same orphan-FK shape this PR fixes. issue_convertor already guards its AuthorId the same way. Verified with the full github e2e suite on both MySQL and PostgreSQL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(linear): add tool-layer models and init migration
Add the Linear plugin's tool-layer data models (connection, team scope,
scope config, account, issue, comment, issue label, workflow state, cycle,
issue history) and the initial schema migration with archived snapshots.
The connection authenticates with a personal API key passed verbatim in the
Authorization header (Linear uses no Bearer prefix).
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): add plugin skeleton, connection API and GraphQL client
Wire the Linear plugin entry point and implement all required plugin
interfaces (meta, init, task, api, model, source, migration, blueprint v200,
closeable). Add connection/scope/scope-config CRUD via the data-source helper,
a test-connection endpoint that runs a GraphQL viewer query, and a rate-limited
async GraphQL client that injects the API key via a bare Authorization header.
SubTaskMetas is intentionally empty; collectors are added per entity in
following commits.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect, extract and convert users to accounts
Add the users GraphQL collector (paginated), extractor to
_tool_linear_accounts, and convertor to the domain crossdomain.Account
table, wired as the first three subtasks. Includes an e2e dataflow test
with raw fixtures and verified snapshots.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect and extract workflow states
Add the team-scoped workflow states GraphQL collector and extractor into
_tool_linear_workflow_states. These states (backlog/unstarted/started/
completed/canceled) drive deterministic issue status mapping. Includes an
e2e test covering all five state types.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect, extract and convert issues
Add the team-scoped issues GraphQL collector (incremental via updatedAt
ordering, inline labels), extractor to _tool_linear_issues and
_tool_linear_issue_labels, and convertor to domain ticket.Issue and
ticket.BoardIssue.
Status maps deterministically from Linear's WorkflowState.type
(backlog/unstarted->TODO, started->IN_PROGRESS, completed/canceled->DONE);
priority maps to its label; lead time falls back to resolution minus
creation. Includes an e2e test spanning all state types, unassigned issues,
issues without a cycle, and multi-label issues.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect, extract and convert issue comments
Add a per-issue comments GraphQL collector (driven by an input iterator over
collected issues, with pagination), an extractor that recovers the owning
issue id from the raw input column, and a convertor to domain
ticket.IssueComment. Includes an e2e dataflow test.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): convert issue labels to domain layer
Add the convertor from _tool_linear_issue_labels (populated inline by the
issue extractor) into the domain ticket.IssueLabel table. Includes an e2e
test covering issues with multiple labels and with none.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect cycles and convert to sprints
Add the team-scoped cycles GraphQL collector and extractor, plus convertors
producing domain ticket.Sprint and ticket.BoardSprint (status derived from
completedAt), and ticket.SprintIssue linking issues to their cycle. Includes
an e2e dataflow test covering closed/active cycles and issues with/without a
cycle.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): collect issue history and convert to changelogs
Add a per-issue history GraphQL collector (input iterator over issues, with
pagination), an extractor capturing state transitions including state types,
and a convertor to domain ticket.IssueChangelogs with mapped from/to status
values. Lead time is already derived from the issue's native
startedAt/completedAt. Includes an e2e test of a full
backlog->started->completed lifecycle.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* test(linear): add blueprint v200 scope generation tests
Cover makeScopesV200: a team scope with the ticket entity produces the
expected domain board scope id, and a scope without the ticket entity
produces none.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* docs(linear): add plugin README
Document the Linear plugin: supported entities, tool/domain mapping tables,
deterministic status mapping, priority/type/lead-time handling, API-key auth,
connection/scope/pipeline setup examples, rate limiting, and the roadmap
(OAuth, label-based type mapping, config-ui integration).
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): guard lead-time fallback against resolution before creation
A resolution timestamp (completedAt/canceledAt) earlier than createdAt —
from clock skew or migrated/imported issues — produced a negative duration
that, cast to uint, yields platform-dependent garbage (0 on arm64, ~1.8e19
on amd64). Skip the fallback unless the resolution is after creation so lead
time stays unset instead.
Adds an isolated e2e dataflow test with a fixture whose canceledAt precedes
createdAt, asserting lead_time_minutes is empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): map Linear triage state type to TODO
The WorkflowState.type 'triage' (the inbox state issues land in before being
accepted) previously fell through to OTHER, contradicting the documented total
mapping and silently mislabeling triage issues. Map it to TODO; keep OTHER as
the fallback for genuinely unrecognized types so unexpected API values surface.
Adds a unit test covering every documented state type plus triage and an
unknown value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* refactor(linear): remove unused GraphqlInlineAccount struct
The struct was documented as the shared inline-user shape but was never
referenced; each collector declares its own inline user struct. Removing it
avoids misleading a maintainer into editing a type nothing reads.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* perf(linear): raise issue collector page size to 100
Every other Linear collector uses a page size of 100; issues used 50, which
doubled the number of issue-page round-trips and the iterator size that drives
the per-issue comment/history collectors. Linear permits first: 250.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): populate issue assignee/creator names and issue_assignees
The issue convertor set only assignee_id/creator_id, leaving the denormalized
assignee_name/creator_name columns blank and writing no issue_assignees rows,
so dashboards reading those columns or joining through issue_assignees showed
blank names. Preload account display names (matching the account convertor's
displayName-then-name rule) and emit an IssueAssignee per assigned issue.
The issue dataflow test now loads accounts before conversion and asserts the
names plus issue_assignees; the lead-time test flushes accounts to stay
order-independent on the shared test DB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): clear stale sprint_issues when issues leave their cycle
Sprint membership is derived from each issue's cycle_id, and the batch divider
only deletes outdated rows when it produces at least one row of the type. When
every issue is moved out of its cycle the convertor emits nothing, so the
divider never fires and prior sprint_issues rows linger, leaving issues shown
in sprints they no longer belong to. Delete the team's sprint_issues up front
so the result is correct regardless of how many issues remain in a cycle.
Adds a two-run e2e test that empties every issue's cycle and asserts
sprint_issues is empty afterward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): derive issue lead time from state-transition history
The LinearIssue.LeadTimeMinutes field was never populated, so lead time always
fell back to the coarse createdAt -> resolutionDate span. Derive it instead from
the recorded history: the span from an issue's first transition into an
in-progress state to its first transition into a done state thereafter (active
cycle time), which is the value that genuinely requires history. ConvertIssues
still seeds the fallback; ConvertIssueHistory now overrides it when the
transitions exist, and issues lacking them keep the fallback.
Adds an e2e test asserting issue-1 (started 05-02, completed 05-03) resolves to
1440 minutes from history rather than its 2880-minute created->resolved span.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): add remote-scopes endpoints to enumerate teams
Other first-party ticket plugins (jira, asana, github) expose
connections/:connectionId/remote-scopes so the config UI can browse and select
scopes from the API. Linear had none, forcing users to hand-craft a PUT /scopes
with raw team UUIDs they had no in-product way to discover. Wire the standard
DsRemoteApiProxyHelper + DsRemoteApiScopeListHelper and a lister that queries
the GraphQL teams connection (flat list, cursor-paginated) through the
connection's authenticated client.
Adds unit tests for the response->scope-entry mapping, the pagination cursor,
and the route registration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* perf(linear): make comment and history collection incremental
Both child collectors used a plain GraphqlCollector and swept every issue in
the team on every run, issuing one request per issue with no since filter -
tens of thousands of requests per run on a large team against Linear's ~1500
req/hour budget. Switch them to a stateful collector and restrict the driving
cursor to issues updated since the last successful collection, so steady-state
runs scale with the change delta rather than the whole backlog. A full sync
(since == nil) still sweeps every issue.
Adds a unit test for the incremental cursor-clause builder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): filter issues server-side by updatedAt for incremental sync
Incremental collection relied on the issues query returning newest-first and a
client-side early-stop, but the query pinned no sort direction (Linear's orderBy
is a scalar enum with no direction operand). If the server default were
ascending, the early-stop would fire on the first (oldest) row and collect
almost nothing. Pass a server-side IssueFilter { updatedAt: { gt: since } }
instead and drop the early-stop, so correctness no longer depends on an
undocumented default ordering. A full sync passes an empty filter (match all).
Adds a unit test pinning the filter's JSON shape to Linear's IssueFilter input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* refactor(linear): drop dead LeadTimeMinutes tool-layer field
The _tool_linear_issues.lead_time_minutes column was never populated (the
collector never requested it and no extractor set it). Now that lead time is
derived into the domain ticket.Issue directly -- from state-transition history
when available, otherwise the createdAt->resolutionDate fallback in the issue
convertor -- the tool-layer field is pure dead weight. Remove it from the model,
the init migration's archived model, the convertor, and the extractor snapshot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(config-ui): register Linear plugin
Adds the Linear data source to config-ui so it appears in the connection
picker: connection form (endpoint + personal API key + proxy + rate limit),
a flat Teams data-scope backed by the plugin's remote-scopes endpoint, and the
Linear logo. No scope-config transformation — Linear's status mapping is
deterministic. Wired into the plugin registry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(config-ui): map Linear scope id to teamId
getPluginScopeId fell through to the default (scope.id) for Linear, but a
LinearTeam scope is keyed by teamId and has no id field — so the blueprint
referenced an undefined scopeId and patching failed with 'LinearTeam not found'.
Add a linear case returning scope.teamId.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): add Grafana dashboard
Adds grafana/dashboards/Linear.json (cloned from the Asana ticket-dashboard
template) so Linear ships a per-tool dashboard like every other ticket plugin.
Its board picker is scoped to Linear (boards id like 'linear%'); the 13 panels
(throughput, lead/cycle time, status distribution, delivery rate, sprints) read
the shared domain tables. Auto-loaded via Grafana file provisioning.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): convert team scope to a domain board
board_issues and sprint_issues referenced a board_id (boardIdGen over
LinearTeam), but nothing ever created the ticket.Board row itself, so the
domain boards table stayed empty. Board-scoped dashboards (whose board picker
is 'boards where id like linear%') and any board join therefore returned no
data. Add a ConvertTeams subtask that converts the team scope in
_tool_linear_teams into a ticket.Board keyed identically to those references.
Adds an e2e test asserting the board is produced with the matching id.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): widen issue title/url columns to avoid truncation
_tool_linear_issues.title and .url were varchar(255), but Linear titles can
exceed 255 chars (and the issue URL embeds a title slug), so extraction failed
with 'Error 1406: Data too long for column title'. Drop the varchar limit so
both are longtext, matching the domain issues.title and jira's tool summary.
Adds an e2e test extracting a 300-char title without truncation.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* fix(linear): recover owning issue id for comments and history
The GraphQL collector stores the query variables (which carry issueId) in the
raw row's input column, but the comment and history extractors parsed it as
{"Id":...} (SimpleLinearIssue.Id), so the owning issue id came out empty and
the convertor joins produced zero domain comments/changelogs on real data. The
e2e fixtures hand-wrote {"Id":...}, masking it. Parse issueId (with an Id
fallback) and update the fixtures to the real collector shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* test: flush LinearIssueLabel before issue extraction in e2e tests
The issue extractor writes both _tool_linear_issues and
_tool_linear_issue_labels, but comment_test, cycle_test and
issue_history_test only flushed LinearIssue before running
ExtractIssuesMeta. On a clean database (as in CI) the table
_tool_linear_issue_labels was never auto-migrated, so the extractor's
DELETE on that table panicked and aborted the whole package. Flush
LinearIssueLabel too, matching the other linear e2e tests and the jira
plugin convention.
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* test: register linear plugin in Test_GetPluginTablesInfo
The plugin count check failed in CI (actual 41 vs tested 40) because the
linear plugin was not listed in table_info_test.go. Add its import and
FeedIn call so every Go plugin is covered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
* feat(linear): map issues to ticket types via label-based scope config
Linear has no native issue type, so the issue convertor previously
hardcoded every issue to REQUIREMENT. Add optional regex fields to the
Linear scope config (issueTypeIncident/Bug/Requirement) matched against
an issue's label names, with precedence INCIDENT > BUG > REQUIREMENT and
a REQUIREMENT default when nothing matches. This lets DORA classify
Linear bugs as incidents for change-failure-rate / time-to-restore.
Also fix two gaps that made scope configs unusable for Linear:
- thread the scope's ScopeConfigId into the pipeline task options so the
convertor can load the config at runtime
- register the standard scope-config/:scopeConfigId/projects route that
every other plugin serves (config-ui 404'd on the scope config page)
Tested: new e2e TestLinearIssueIncidentMapping (Bug label -> INCIDENT,
other issues REQUIREMENT) plus a blueprint plan test asserting
ScopeConfigId is passed through; full plugins/linear suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
---------
Signed-off-by: Eduardo Rodrigues <2961314+eduardoarantes@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pache#8903) * fix(jira): rename Jira Server to Jira Data Center in UI labels * fix(jira): show both Jira Server and Jira Data Center labels
…de Commits panel (apache#8925) Co-authored-by: Louis.z <louis.s4372121@gmail.com>
…pache#8915) * feat(webhook): Issues and PullRequests new endpoint by projectName * ci: fix invalid push * ci: fix lint issues * ci: add missing lint.
) * fix(server): remove auth from proceed-db-migration endpoint * Auth tables may not exist when migration is pending, causing a bootstrap deadlock * Restores pre-auth-hardening behavior for the idempotent migration endpoint Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> * fix(server): add proceed-db-migration to publicPaths Signed-off-by: Joshua Smith <jbsmith7741@gmail.com> --------- Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
… time range (apache#8956) * Skip 500 (corrupt CircleCI Server records) alongside 404 in AfterResponse hook so a single bad pipeline does not abort the entire subtask * Apply SyncPolicy.TimeAfter to workflow and job DB iterators on full sync to avoid calling the API for every historical tool-layer row Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
Contrary to what the PR for the feature mentioned, the setting was enabled by default, instead of disabled. Follow up to apache#8854
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…xclusion (apache#8931) * feat(jira): collect issues via search API to avoid kanban sub-filter exclusion * Switch from board Agile API to search API with saved filter JQL so resolved issues (e.g. released fixVersions) are no longer silently excluded by kanban sub-filters * Add sub_query field to boards model and migration to track kanban sub-filter changes across syncs * Support JIRA Cloud v3 search/jql endpoint alongside v2 for Server * fix(jira,config-ui): correct SubQuery parsing and connection form crashes * Fix Jira board SubQuery struct to match nested API response shape * Prevent click event from being stored as plugin state in connection list * Default initialValues to empty object when plugin config omits it * feat(jira): Add BoardConfiguration unmarshal tests * Verify SubQuery parsing for kanban boards with, without, and empty sub-filter objects * Include a test using a full Jira Cloud response payload to validate all fields
* feat(jira): Allow extra JQL when querying JIRA board * fix: revert code so we just do extrajql, no dynamic dev lake project name * fix: remove reference to devlakeprojectname as a template option
Co-authored-by: Louis.z <louis.s4372121@gmail.com>
…ject absolute relativePath (apache#9043) url.URL.ResolveReference silently discards the base URL when the reference is absolute or protocol-relative, which would make the request target the wrong host without any error. Validate relativePath up front and document the resolution semantics on the renamed function. Co-authored-by: Louis.z <louis.s4372121@gmail.com>
apache#8959) * fix(gitlab): always re-collect MR commits regardless of MR updated_at to prevent missing commits in incremental runs * fix(gitlab): correct if-brace syntax in GetMergeRequestsIterator
WorkInProgress was extracted from the GitLab API and stored in _tool_gitlab_merge_requests but never forwarded to code.PullRequest.IsDraft during conversion. Draft/WIP MRs were indistinguishable from non-draft MRs at the domain layer. - Add IsDraft: gitlabMr.WorkInProgress to the converter struct - Update e2e snapshot: MRs 145012495 and 158698019 now have is_draft=1
…he#9044) * build(deps): refresh container images Bump the pinned container images used by the dev/test stacks and the shipped Dockerfiles to their current patch releases: - Grafana 13.0.2 -> 13.1.3 (grafana/Dockerfile) - PostgreSQL 18.1 -> 18.4 (docker-compose-dev-postgresql.yml, backend/test/e2e/remote/docker-compose.test.yml) - MySQL 8.4.10 -> 8.4.11, LTS line (docker-compose-dev-mysql.yml, docker-compose.datasources.yml, .devcontainer/docker-compose.yml, devops/deployment/temporal/docker-compose-temporal.yml, backend/test/e2e/remote/docker-compose.test.yml) - nginx-unprivileged 1.31.2 -> 1.31.3 (config-ui/Dockerfile) Every occurrence of a given pin is raised together so the stacks stay consistent. The Grafana bump stays within the 13.x line and requires no dashboard JSON changes. Release artefacts under devops/releases/ are historical and intentionally left untouched. Signed-off-by: DoDiODev <DoDiDev@proton.me> * build(deps): bump pinned Node.js to 24.19.0 Update the volta Node pin in config-ui/package.json from 24.17.0 to 24.19.0, the current 24.x LTS ("Krypton") release. The Docker build stays on the floating node:24-bookworm-slim tag, so this only aligns local development environments with the release actually used in CI/images. Signed-off-by: DoDiODev <DoDiDev@proton.me> * build(deps): refresh frontend dependencies Update the pinned config-ui dependencies to their current releases and regenerate yarn.lock: - antd 6.4.5 -> 6.6.0 - @ant-design/icons 6.2.5 -> 6.3.2 - cron-parser 5.6.0 -> 5.8.1 - react-router-dom 7.18.0 -> 7.18.2 - styled-components 6.4.2 -> 6.5.1 - eslint 10.5.0 -> 10.8.1 - prettier 3.8.4 -> 3.9.6 - vite 8.1.0 -> 8.2.1 - vitest 4.1.9 -> 4.1.10 All target versions stay within their current major line and keep their declared engines/peerDependencies compatible with the pinned Node 24 and React 19 versions. TypeScript is deliberately left at 6.0.3, since 7.x is a major upgrade that deserves its own change. Validated locally with yarn install, yarn build, yarn test and yarn lint. Signed-off-by: DoDiODev <DoDiDev@proton.me> --------- Signed-off-by: DoDiODev <DoDiDev@proton.me>
apache#8928) * fix(gitlab): use diff_stats instead of changes_count for correct MR size (fixes apache#8888) * fix(gitlab): handle missing diff_stats for older GitLab instances * fix(gitlab): add strconv import and license header; fix else-if syntax * fix(gitlab): use nullable additions/deletions to distinguish missing diff_stats from zero --------- Co-authored-by: Klesh Wong <klesh@qq.com>
* build(deps): mockery v2→v3 Upgrade the mock generator from mockery v2.53.6 to v3.7.2 at every install site (backend/Makefile, backend/Dockerfile, backend/Dockerfile.local, devops/docker/lake-builder/Dockerfile). mockery v3 dropped the CLI flags used by the `mock` target and is configured via YAML instead, so two config files are added: - backend/.mockery.core.yml - backend/.mockery.helpers.yml They reproduce the exact layout produced by the previous v2 invocations (--recursive --keeptree --dir=./<tree> --output=./mocks/<tree> --unroll-variadic=false --name='.*'): backend/mocks/<src-dir>/<Interface>.go, package `mocks`, un-prefixed mock struct names. Existing test imports such as `mockdal "github.com/apache/incubator-devlake/mocks/core/dal"` therefore keep working unchanged. Two configs (instead of one) are required because `helpers/unithelper` imports the generated `mocks/core/...` packages: unlike v2, v3 type-checks sources via go/packages, so the core mocks must exist before the helpers tree can be loaded. The `mock` target runs them in that order. Note: v3 only generates mocks for interfaces, no longer for function types. The affected mocks (e.g. plugin.ApiAsyncCallback, api.DataConvertHandler, errors.Option) were not used by any test. `backend/mocks/` is gitignored, so there is no generated-code churn in this diff. Validation: `make mock`, `go build ./...` and `scripts/unit-test-go.sh` (60 packages) all pass. Signed-off-by: DoDiODev <DoDiDev@proton.me> * ci: bootstrap mockery 3 in lint workflow Signed-off-by: DoDiODev <DoDiDev@proton.me> * ci: bootstrap mockery 3 in unit-test workflow Signed-off-by: DoDiODev <DoDiDev@proton.me> --------- Signed-off-by: DoDiODev <DoDiDev@proton.me>
…ver (apache#9042) The incremental filter on the changelog convertor used created_at: _tool_jira_issue_changelog_items.created_at >= ? created_at is stamped when the row is first inserted and never moves again. So a changelog item that was collected during one window but not converted in that window can never be selected by any later incremental run -- the timestamp it is filtered on is permanently in the past. Nothing errors; the rows simply stay in the tool layer. That matches the report in apache#8834: a partial, silent shortfall in issue_changelogs from the same sync run, persisting across runs, varying by project. It also explains why the reporter's check looked clean -- they inspected _devlake_collector_latest_state, which is the collector's state, not the convertor's. Switched to updated_at, which the extractor's upsert refreshes (OnConflict{UpdateAll: true}), so a re-collected item is reconsidered. Of the 36 convertors in the code base, this was the only one filtering on created_at; the other 35 already use updated_at. This does not widen the board filter, so it does not carry the cost klesh raised against moving board_id into the join: no board task converts anything outside its own board. Separately, the board filter is now reported rather than silent. After a successful conversion the subtask counts collected changelog items whose issue is not on this board and logs the number with the reason. That is one aggregate query per board task, and it turns an unexplained shortfall into a logged figure. Diagnostic failures are logged, never propagated. Tests: an e2e dataflow test driving the convertor with a changelog for an issue on no board, asserting it is excluded, that nothing is attached to issue id 0, and that in-scope changelogs still convert; plus a regression guard on the filter column, since reverting it is a one-token change that silently restores permanent data loss.
…sues (apache#9038) * feat(github): collect GitHub issue fields and map them onto domain issues GitHub issue fields are organization-level structured issue metadata that went generally available on 2026-07-02. They are typed, mutually exclusive within a field, and shared across every repository in the organization — which is what teams currently approximate with `type:`-style labels. Collects issue field values and lets a scope config map a field onto an issue column, where it takes precedence over the existing label regexes. - New table `_tool_github_issue_field_values`, one row per issue per field, carrying a queryable text form of the value alongside the original JSON. - New subtasks Collect/Extract Issue Field Values, both disabled by default so existing pipelines are unaffected until a mapping is configured. - New scope config keys issueFieldPriority, issueFieldSeverity, issueFieldComponent, issueFieldStoryPoint and issueFieldDueDate, each holding a field *name*. The mapping is applied in the issue convertor rather than written back into `_tool_github_issues`: the collector iterates that table to build its request URLs, so a subtask that both read and wrote it was a cycle in the subtask graph. Converting instead also keeps the tool layer as raw GitHub truth and avoids adding columns there. A 404 from the field-values endpoint is treated as "no field values" so an organization that has never configured issue fields, or a token that cannot see them, does not fail the whole task. * test(github): add an e2e dataflow test for issue field values Covers extraction and the scope config mapping end to end against a real database, which the unit tests could not reach. Extraction asserts the value normalisation per data type: a single_select resolving to its option name and colour, an integral number rendering as "5" rather than "5.0", a fractional number keeping its precision, a multi_select joining option names while keeping the raw JSON array, and a null value producing an empty value. Conversion asserts the mapping reaches the domain issue -- priority, component, story point and due date -- and, for the case that matters, that an unparseable value is skipped with a warning rather than failing the task or writing a wrong value: issue konflux-ci#7 carries "soon" in a date field and a null priority, and comes out with neither set while the other issues are untouched. * fix(github): register the issue field values table in GetTablesInfo Test_GetPluginTablesInfo compares the plugin's declared tables against the ones its migrations create, and the new table was missing from the list: table_info_test.go:121: The following tables are not returned by the TablesInfo method _tool_github_issue_field_values Adds GithubIssueFieldValue to Github.GetTablesInfo(). Verified inside the mericodev/lake-builder image the unit-test job uses, since the plugins package needs libgit2 to build.
apache#9000) * feat(dora): exclude bot/automation accounts from PR Pickup Time calculation * test: add is_bot column to account e2e snapshot fixtures
…tal filter (apache#9046) apache#8959 changed `GetMergeRequestsIterator` to filter merge requests by `GREATEST(gmr.gitlab_updated_at, COALESCE(gmr.commit_updated_at, gmr.gitlab_updated_at))`, but `commit_updated_at` does not exist on `_tool_gitlab_merge_requests`: there is neither a model field nor a migration script, and nothing ever writes the value. As a result every incremental "Collect MR Notes" run (the only caller that passes a stateful collector; the MR commit collector passes nil) aborts with: Error 1054 (42S22): Unknown column 'gmr.commit_updated_at' in 'where clause' This makes the intended behaviour actually work instead of reverting it: * add `CommitUpdatedAt` to `GitlabMergeRequest` plus a migration script * maintain the column in the MR commit extractor, setting it to the latest authored date of the MR's commits, so MRs that received new commits without their own `updated_at` being bumped (e.g. force-pushes) are picked up again * the collector filter is unchanged and now resolves against a real column Verified on MySQL 8 and PostgreSQL 14 (extractor UPDATE and the GREATEST filter produce identical results on both).
…ther connection (apache#9047) * feat(gitlab,bitbucket): extend scope-duplicate warning to GitLab and Bitbucket Add scope-duplicates API endpoints for GitLab and Bitbucket, mirroring the existing GitHub implementation, and generalize the config-ui data-scope-remote component and scope API client so the duplicate-connection warning works for all three plugins. Co-Authored-By: Cursor Agent <noreply@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com> * assert specific error message --------- Co-authored-by: Cursor Agent <noreply@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
ConnectUserAccountsExact keyed its lookup map on the raw email address, so the comparison was case-sensitive. A user imported from users.csv as Tyrone.Cummings@corp.com was not linked to an account recorded as tyrone.cummings@corp.com, and the account was left unlinked with no warning: the subtask still reported success. That matters because the two sides come from different systems. A git config, a provider profile and a hand-maintained CSV routinely disagree on the capitalisation of the same address, and email addresses are treated case-insensitively everywhere else. Once an account is unlinked, everything attributing through it stops counting towards that user, and the scan is restricted to accounts not already in user_accounts, so a later run does not revisit it. Folds case on both sides of the email comparison. Name matching is left exactly as it was, since names are not case-insensitive in the same way and loosening them could create links that are simply wrong. The org e2e fixtures gain account a11, whose email differs from user U006's only in case, with a name and login that deliberately match nothing. It fails before this change (10 rows against 11 expected) and passes after.
* refactor(deps): drop lib/pq and bump pgx/v5 to v5.10.0
PostgreSQL support already runs entirely on pgx: `core/runner/db.go` opens
PostgreSQL connections through `gorm.io/driver/postgres`, which uses
`github.com/jackc/pgx/v5` under the hood. There is no `sql.Open("postgres", ...)`
and no blank import of `lib/pq` anywhere in the tree.
The single remaining functional use of `lib/pq` was in the StarRocks plugin,
where `pq.Array` was used to scan PostgreSQL array columns. It is replaced by
`pgtype.Map.SQLScanner`, the pgx equivalent that implements `sql.Scanner` and
therefore works on the `database/sql` code path used by gorm. The `pgtype.Map`
is created once per query, outside the row loop, because building it registers
the full default type catalog.
Shipping a second, effectively unused PostgreSQL driver also meant shipping
`pgx/v5 v5.6.0`, which is affected by two CRITICAL advisories
(GHSA-9jj7-4m8r-rfcm, GHSA-xgrm-4fwx-7qm8) plus a low-severity one
(GHSA-j88v-2chj-qfwx), all fixed in v5.9.2. Since pgx becomes a direct
dependency here, it is bumped to v5.10.0.
`gorm.io/driver/postgres` and `gorm.io/gorm` are deliberately left untouched:
`driver/postgres v1.5.2` only declares a minimum pgx version, so raising pgx
does not require an ORM bump.
Notes for reviewers:
- `pgx/v5` moves from the indirect block to a direct requirement.
- Array literals parse identically to the previous implementation, including
empty arrays, empty elements, quoted separators and UTF-8 content. This is
covered by a new unit test; the array scan path had no test coverage before.
- `NULL` elements inside arrays were already rejected by `pq.Array` and are
still rejected (only the error message differs) — this is not a regression
and not a silent fix.
Signed-off-by: DoDiODev <DoDiDev@proton.me>
* ci: bootstrap Go 1.26 in Go workflows
Signed-off-by: DoDiODev <DoDiDev@proton.me>
* fix(ci): mark the workspace as a git safe.directory in the lint job
The `generate mock` step of the `golangci-lint` workflow fails with
ERR encountered error when loading package
error="-: error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping."
make: *** [Makefile:80: mock] Error 1
The job runs inside `mericodev/lake-builder:latest` as root, while the
checkout is owned by the runner user, so git refuses to operate on the
repository ("dubious ownership") and Go's VCS stamping aborts while
mockery loads the packages.
`test.yml` and `test-e2e.yml` already run
`git config --global --add safe.directory $(pwd)` right after the
checkout for exactly this reason, which is why the unit-test and e2e
jobs pass while lint does not. Add the same step to the lint job.
---------
Signed-off-by: DoDiODev <DoDiDev@proton.me>
…buf (apache#9054) * build(deps): bump golang.org/x/crypto, x/net and x/oauth2 golang.org/x/crypto v0.41.0 -> v0.55.0 (15 CVEs, 7 CRITICAL) golang.org/x/net v0.43.0 -> v0.58.0 (CVE-2026-25680) golang.org/x/oauth2 v0.13.0 -> v0.36.0 (CVE-2025-22868) 17 known CVEs in total. govulncheck confirms 8 of them are reachable from this code base: 6 in x/crypto, 1 in x/net and 1 in x/text -- x/text is pulled up by minimal version selection along with the rest of the golang.org/x set (x/sys, x/term, x/sync, x/mod, x/tools). After the bump govulncheck reports none of them. The reachable x/crypto findings are in the SSH stack and are reached through core/utils/io.go, where archives are copied with github.com/viant/afs -- that library registers an SSH/SCP backend, so ssh.Dial, ssh.ParsePrivateKey and the agent signer end up in the build graph. They are not reached through gitextractor: cloning is done by the git CLI, not by a Go SSH client. x/oauth2 is used directly by server/api/auth and helpers/oidchelper; its finding is a malformed token that allocates without bound during parsing. No source change was required -- none of the bumped packages changed an API surface this code base touches. Signed-off-by: DoDiODev <DoDiDev@proton.me> * build(deps): bump golang-jwt/jwt/v5 off the release candidate, logrus and protobuf github.com/golang-jwt/jwt/v5 v5.0.0-rc.1 -> v5.3.1 github.com/sirupsen/logrus v1.9.0 -> v1.10.0 google.golang.org/protobuf v1.31.0 -> v1.36.12 jwt/v5 was pinned to a release candidate published in March 2023. Besides being a pre-release in a security-sensitive position, v5.0.0-rc.1 is affected by CVE-2025-30204 (HIGH), which govulncheck reports as reachable from this code base. v5.3.1 is the current stable release. jwt/v5 is used by the GitHub App token flow (plugins/github/token, plugins/github/models/connection.go), the Teambition connection and the OIDC session helper (helpers/oidchelper/session.go). Moving from the RC to stable needed no source change; the signing, parsing and claims APIs used here are unchanged. logrus v1.9.0 (CVE-2025-65637, HIGH) and protobuf v1.31.0 (CVE-2024-24786) are bumped to current releases in the same pass. Verified with the unit tests of the affected packages, including server/api/auth (JWT session handling) and helpers/oidchelper. Signed-off-by: DoDiODev <DoDiDev@proton.me> --------- Signed-off-by: DoDiODev <DoDiDev@proton.me>
…ector with tests (apache#9055) * test(gitextractor): cover the go-git repo collector The go-git based collector (used when UseGoGit is set) had no test coverage at all, which makes any go-git upgrade a leap of faith. Adds a test that builds a repository on disk with a known shape -- three commits, one tag, a second branch checked out so HEAD is not the default branch -- and asserts CountCommits, CountTags and CountBranches, that the counting helpers stop on a cancelled context, and that opening a directory which is not a repository fails. The test passes both on go-git v5.12.0 and on the version this branch upgrades to, so it documents unchanged behaviour across the bump. Signed-off-by: DoDiODev <DoDiDev@proton.me> * build(deps): bump go-git v5.12.0 -> v5.19.2 github.com/go-git/go-git/v5 v5.12.0 -> v5.19.2 github.com/go-git/go-billy/v5 v5.5.0 -> v5.9.0 (pulled in) github.com/cloudflare/circl v1.3.7 -> v1.6.3 (pulled in) 16 known CVEs in total across the three modules. govulncheck reports 12 of them as reachable from this code base before the bump (8 in go-git, 2 in go-billy, 2 in circl) and none after. The go-git ones include GO-2025-3367 and GO-2025-3368 (argument injection through crafted URLs) and GO-2026-4909/4910. go-git is used in exactly one place: plugins/gitextractor/parser/repo_gogit.go opens an already cloned repository with PlainOpen and walks commits, tags, branches and trees. Cloning is done by the git CLI (parser.NewGitcliCloner), so the go-git transports are not on the path -- this bump does not change how DevLake talks to remotes. No source change was required. Verified with the collector test added in the previous commit, which passes on both the old and the new version. Minimal version selection also pulls up golang.org/x/crypto, x/net, x/sys, x/text, x/tools, ProtonMail/go-crypto, pjbgf/sha1cd, skeema/knownhosts, cyphar/filepath-securejoin, Microsoft/go-winio, klauspost/cpuid and golang/protobuf. Signed-off-by: DoDiODev <DoDiDev@proton.me> --------- Signed-off-by: DoDiODev <DoDiDev@proton.me> Co-authored-by: Klesh Wong <klesh@qq.com>
…ps/fakeplugin (apache#9060) certifi and pytest are bumped in the two direct pyproject.toml ranges that still pinned pre-CVE-fix versions (certifi ^2023.7.22, pytest ^7.2.x); all three Poetry lockfiles are regenerated with the project-pinned Poetry 2.4.1 to stay consistent with their pyproject.toml (poetry export/lock already required `poetry lock` on upstream/main). - backend/python/pydevlake/pyproject.toml: certifi ^2023.7.22 -> ^2026.7.22, pytest (main + dev group) ^7.2.x -> ^9.1.1 - backend/python/plugins/azuredevops/pyproject.toml: dev pytest ^7.2.2 -> ^9.1.1 - backend/python/{pydevlake,plugins/azuredevops,test/fakeplugin}/poetry.lock: regenerated (`poetry lock --regenerate`) No Pydantic v2/SQLModel/runtime/dbt changes; this is a lockfile-security-only refresh, independent of apache#8970 and the later Python 3.14 wave. Validation (Python 3.11.15, Poetry 2.4.1): - `poetry check --lock` clean in all three projects - `pip-audit` against the exported lock graphs: 0 known findings in all three (only the expected skip for the local, non-PyPI package `pydevlake`) - `make build-pydevlake` and `make unit-test-python`: pydevlake 10 passed, azuredevops 8 passed / 1 skipped - azuredevops remote plugin starts (`run.sh --help` lists collect/convert/ extract/make_pipeline/plugin_info/remote_scopes/test_connection) Signed-off-by: DoDiODev <DoDiDev@proton.me>
) Emails were made case-insensitive in apache#9051, but the display name and provider login comparisons in ConnectUserAccountsExact still matched exactly. Provider logins are themselves case-insensitive, and a corporate git config and a provider profile routinely record the same display name with different capitalisation, so an account whose login differs from the users.csv name only in case is silently left unlinked - and every activity that attributes through that account goes missing. Towards apache#8698, where a GitHub account with no public email can only link through these name paths. Extends the org e2e fixtures with an account whose login differs from the user's name only in case; without the change it is not linked and the expected row count drops from 12 to 11.
* fix: add blueprint_id index to _devlake_pipelines GET /blueprints/:blueprintId/pipelines runs COUNT(*) and a filtered SELECT on _devlake_pipelines.blueprint_id, but the column was unindexed. On instances with tens of thousands of pipeline rows this forces a full table scan on every request; observed ~30s for a table of 34k rows, causing upstream request timeouts in config-ui. Add a gorm index tag on Pipeline.BlueprintId for fresh installs and a migration script to add the index to existing installs. Signed-off-by: Dan Crews <crewsd@gmail.com> * fix(github): widen unbounded tool fields Signed-off-by: Dan Crews <crewsd@gmail.com> * fix(domain): widen unbounded CI/CD fields Signed-off-by: Dan Crews <crewsd@gmail.com> --------- Signed-off-by: Dan Crews <crewsd@gmail.com>
EpicKeyField is already on Jira scope config and in the Config UI, but extractIssues never applied it, so Cloud/Parent Link epics stayed empty. Co-authored-by: Cursor Grok 4.6 <noreply@example.com> Co-authored-by: Cursor <cursoragent@cursor.com>
* build(ci): pin active database images * ci(builder): refresh runner and login action * build(deps): bump mockery to 3.7.4 Follow-up to apache#9029, which introduced mockery v3.7.2. All active installation sites move to v3.7.4 in lockstep so the CI bootstrap, the Makefile target and the builder images stay on a single version: - .github/workflows/test.yml, .github/workflows/golangci-lint.yml - backend/scripts/install-mockery.sh (incl. release SHA-256 sums) - backend/Makefile (go-dep) - backend/Dockerfile.local, devops/docker/lake-builder/Dockerfile The pinned archive checksums were verified against the official vektra/mockery v3.7.4 checksum.txt. "make mock" regenerates all 65 mock files without any diff, so no generated code is part of this commit.
…ache#9061) * build(deps): bump x/mod to v0.40.0 * build(deps): update gorm postgres driver * build(deps): update go-oidc * refactor(deps): replace mapstructure with go-viper v2 * fix(db): keep postgres index drops compatible with updated GORM * fix: make mysql schema migrations idempotent Signed-off-by: DoDiODev <DoDiDev@proton.me> * fix(db): keep AutoMigrate from redefining the mysql primary key gorm.io/driver/mysql v1.6.0 appends 'ADD PRIMARY KEY' to every column added by AutoMigrate that carries the primaryKey tag. DevLake migration scripts add such columns to tables that already own a primary key, so MySQL rejected them with 'Error 1068: Multiple primary key defined'. Wrap the MySQL dialector so AddColumn keeps the pre-v1.6.0 behaviour: the primary key is only created for AUTO_INCREMENT columns on tables without one, every other primary key change stays with the migration scripts. Signed-off-by: DoDiODev <DoDiDev@proton.me> --------- Signed-off-by: DoDiODev <DoDiDev@proton.me>
Signed-off-by: warren <warren.chen830@gmail.com>
* feat: add Kiro commit attribution helper Signed-off-by: warren <warren.chen830@gmail.com> * fix: add Apache license headers Signed-off-by: warren <warren.chen830@gmail.com> --------- Signed-off-by: warren <warren.chen830@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sync this fork to Apache DevLake v1.0.3-beta16 (
825a81ca6465e12df8590b494ebda679d882278b).The PR head is the upstream tag (not a pre-resolved merge). GitHub will block the merge button when this fork and upstream both changed the same files. Owned plugins and other fork-only paths are kept; they are not in the upstream tag.
Resolve conflicts
gh pr checkout(or fetchchore/upstream-sync).git merge origin/main.chore/upstream-sync. CI then runs on the combined tree.Until step 4, CI (if it runs) tests upstream's tree only, not this fork.
Predicted conflicts
Predicted conflict paths (from
git merge-tree; not a substitute for the GitHub conflict UI):.github/workflows/build-builder.yml.github/workflows/golangci-lint.ymlbackend/Dockerfilebackend/core/models/domainlayer/crossdomain/account.gobackend/core/models/migrationscripts/register.gobackend/go.modbackend/go.sumbackend/pkg/oidchelper/authorization.gobackend/pkg/oidchelper/authorization_test.gobackend/plugins/github/tasks/account_convertor.gobackend/plugins/github/tasks/pr_convertor.gobackend/plugins/jira/tasks/issue_extractor.gobackend/plugins/table_info_test.gobackend/python/plugins/azuredevops/poetry.lockbackend/python/pydevlake/poetry.lockbackend/python/pydevlake/pyproject.tomlbackend/python/test/fakeplugin/poetry.lockconfig-ui/Dockerfileconfig-ui/package.jsonconfig-ui/src/plugins/components/connection-form/index.tsxconfig-ui/src/plugins/register/index.tsconfig-ui/src/plugins/register/jira/transformation.tsxconfig-ui/vitest.config.tsconfig-ui/yarn.lockdocker-compose-dev-postgresql.ymlAfter merge
The next scheduled run skips this tag once it is an ancestor of
main.