Export incremental updates on subsequent runs of FederatedCode pipeline

[keshav-space]

Exporting all package advisory data every time the pipeline runs is very slow and inefficient. Instead federation pipeline has been refactored to push only incremental changes in subsequent runs. It uses the date_collected to push only new advisories and introduces created_at field for package impact relations to collect and push only new package impact relationships.
Additionally, advisory export now also includes package commit info.

Log from first run

https://public2.vulnerablecode.io/pipelines/federate_vulnerabilities_v2/run/69da3203-eb84-401e-be96-4622d724871f/

2026-02-23 18:32:21.386115 UTC Pipeline [FederatePackageVulnerabilities] starting
2026-02-23 18:32:21.386423 UTC Step [check_federatedcode_eligibility] starting
2026-02-23 18:32:22.090107 UTC Federatedcode repositories are configured and available.
2026-02-23 18:32:22.090318 UTC Step [check_federatedcode_eligibility] completed in 1 seconds
2026-02-23 18:32:22.090399 UTC Step [create_federatedcode_working_dir] starting
2026-02-23 18:32:22.090947 UTC Step [create_federatedcode_working_dir] completed in 0 seconds
2026-02-23 18:32:22.091032 UTC Step [fetch_federation_config] starting
2026-02-23 18:32:22.601363 UTC Step [fetch_federation_config] completed in 1 seconds
2026-02-23 18:32:22.601590 UTC Step [clone_federation_repository] starting
2026-02-23 18:32:22.601678 UTC Cloning repository https://github.com/aboutcode-data/staging-security-advisories
2026-02-23 18:32:23.180698 UTC Step [clone_federation_repository] completed in 1 seconds
2026-02-23 18:32:23.180829 UTC Step [load_checkpoint] starting
2026-02-23 18:32:23.180965 UTC Step [load_checkpoint] completed in 0 seconds
2026-02-23 18:32:23.181018 UTC Step [publish_package_related_advisories] starting
2026-02-23 18:32:23.296867 UTC Exporting advisory relation for 1806100 packages.
2026-02-23 18:39:37.254434 UTC Progress: 5% (90305/1806100) ETA: 8245 seconds (2.3 hours)
2026-02-23 18:44:22.163241 UTC Progress: 10% (180610/1806100) ETA: 6470 seconds (1.8 hours)
2026-02-23 18:50:09.579750 UTC Progress: 15% (270915/1806100) ETA: 6042 seconds (1.7 hours)
2026-02-23 18:56:45.638881 UTC Progress: 20% (361220/1806100) ETA: 5849 seconds (1.6 hours)
2026-02-23 19:04:00.769454 UTC Progress: 25% (451525/1806100) ETA: 5692 seconds (1.6 hours)
2026-02-23 19:10:42.498739 UTC Progress: 30% (541830/1806100) ETA: 5365 seconds (1.5 hours)
2026-02-23 19:16:49.968509 UTC Progress: 35% (632135/1806100) ETA: 4952 seconds (1.4 hours)
2026-02-23 19:23:42.990894 UTC Progress: 40% (722440/1806100) ETA: 4620 seconds (1.3 hours)
2026-02-23 19:31:41.796134 UTC Progress: 45% (812745/1806100) ETA: 4349 seconds (1.2 hours)
2026-02-23 19:38:36.361284 UTC Federated 1806100 package advisories.
2026-02-23 19:38:36.361586 UTC Step [publish_package_related_advisories] completed in 3973 seconds (1.1 hours)
2026-02-23 19:38:36.361625 UTC Step [publish_advisories] starting
2026-02-23 19:38:36.424292 UTC Exporting 1059124 advisory.
2026-02-23 19:45:35.104768 UTC Progress: 5% (52957/1059124) ETA: 7955 seconds (2.2 hours)
2026-02-23 19:56:51.351708 UTC Progress: 10% (105913/1059124) ETA: 9854 seconds (2.7 hours)
2026-02-23 20:07:20.886908 UTC Progress: 15% (158869/1059124) ETA: 9772 seconds (2.7 hours)
2026-02-23 20:22:57.199596 UTC Progress: 20% (211825/1059124) ETA: 10643 seconds (3.0 hours)
2026-02-23 20:41:27.194737 UTC Progress: 25% (264781/1059124) ETA: 11312 seconds (3.1 hours)
2026-02-23 21:00:39.055845 UTC Progress: 30% (317738/1059124) ETA: 11486 seconds (3.2 hours)
2026-02-23 21:17:18.330842 UTC Progress: 35% (370694/1059124) ETA: 10998 seconds (3.1 hours)
2026-02-23 21:32:54.933830 UTC Progress: 40% (423650/1059124) ETA: 10288 seconds (2.9 hours)
2026-02-23 21:49:03.159950 UTC Progress: 45% (476606/1059124) ETA: 9566 seconds (2.7 hours)
2026-02-23 22:02:12.817105 UTC Progress: 50% (529562/1059124) ETA: 8616 seconds (2.4 hours)
2026-02-23 22:16:42.099240 UTC Progress: 55% (582519/1059124) ETA: 7761 seconds (2.2 hours)
2026-02-23 22:29:50.354676 UTC Progress: 60% (635475/1059124) ETA: 6849 seconds (1.9 hours)
2026-02-23 22:41:56.683359 UTC Progress: 65% (688431/1059124) ETA: 5923 seconds (1.6 hours)
2026-02-23 22:54:17.426816 UTC Progress: 70% (741387/1059124) ETA: 5032 seconds (1.4 hours)
2026-02-23 23:04:58.784042 UTC Progress: 75% (794343/1059124) ETA: 4127 seconds (1.1 hours)
2026-02-23 23:16:35.670530 UTC Progress: 80% (847300/1059124) ETA: 3270 seconds (54.5 minutes)
2026-02-23 23:28:46.212243 UTC Progress: 85% (900256/1059124) ETA: 2437 seconds (40.6 minutes)
2026-02-23 23:42:28.290379 UTC Progress: 90% (953212/1059124) ETA: 1626 seconds (27.1 minutes)
2026-02-23 23:55:51.515637 UTC Progress: 95% (1006168/1059124) ETA: 812 seconds (13.5 minutes)
2026-02-24 00:17:09.464230 UTC Progress: 100% (1059124/1059124)
2026-02-24 00:18:03.380129 UTC Successfully federated 1059124 advisories.
2026-02-24 00:18:03.380486 UTC Step [publish_advisories] completed in 16767 seconds (4.7 hours)
2026-02-24 00:18:03.380591 UTC Step [save_checkpoint] starting
2026-02-24 00:18:51.920901 UTC Step [save_checkpoint] completed in 49 seconds
2026-02-24 00:18:51.921049 UTC Step [delete_working_dir] starting
2026-02-24 00:21:09.747276 UTC Step [delete_working_dir] completed in 138 seconds (2.3 minutes)
2026-02-24 00:21:09.747439 UTC Pipeline completed in 20928 seconds (5.8 hours)

Log from second run

https://public2.vulnerablecode.io/pipelines/federate_vulnerabilities_v2/run/e305df0a-bc22-4b21-98cc-2e3d219d65ee/

2026-02-24 01:18:38.680579 UTC Pipeline [FederatePackageVulnerabilities] starting
2026-02-24 01:18:38.680759 UTC Step [check_federatedcode_eligibility] starting
2026-02-24 01:18:39.220335 UTC Federatedcode repositories are configured and available.
2026-02-24 01:18:39.220442 UTC Step [check_federatedcode_eligibility] completed in 1 seconds
2026-02-24 01:18:39.220469 UTC Step [create_federatedcode_working_dir] starting
2026-02-24 01:18:39.220805 UTC Step [create_federatedcode_working_dir] completed in 0 seconds
2026-02-24 01:18:39.220839 UTC Step [fetch_federation_config] starting
2026-02-24 01:18:39.738929 UTC Step [fetch_federation_config] completed in 1 seconds
2026-02-24 01:18:39.739031 UTC Step [clone_federation_repository] starting
2026-02-24 01:18:39.739068 UTC Cloning repository https://github.com/aboutcode-data/staging-security-advisories
2026-02-24 01:21:22.949922 UTC Step [clone_federation_repository] completed in 163 seconds (2.7 minutes)
2026-02-24 01:21:22.950012 UTC Step [load_checkpoint] starting
2026-02-24 01:21:22.950179 UTC Step [load_checkpoint] completed in 0 seconds
2026-02-24 01:21:22.950211 UTC Step [publish_package_related_advisories] starting
2026-02-24 01:21:24.122846 UTC Exporting advisory relation for 83176 packages.
2026-02-24 01:22:06.133333 UTC Progress: 5% (4159/83176) ETA: 798 seconds (13.3 minutes)
2026-02-24 01:22:40.254753 UTC Progress: 10% (8318/83176) ETA: 685 seconds (11.4 minutes)
2026-02-24 01:23:18.128029 UTC Progress: 15% (12477/83176) ETA: 646 seconds (10.8 minutes)
2026-02-24 01:23:53.937685 UTC Progress: 20% (16636/83176) ETA: 599 seconds (10.0 minutes)
2026-02-24 01:24:28.049198 UTC Progress: 25% (20794/83176) ETA: 552 seconds (9.2 minutes)
2026-02-24 01:29:17.128351 UTC Progress: 30% (24953/83176) ETA: 1104 seconds (18.4 minutes)
2026-02-24 01:38:31.445814 UTC Progress: 35% (29112/83176) ETA: 1908 seconds (31.8 minutes)
2026-02-24 01:46:59.548452 UTC Progress: 40% (33271/83176) ETA: 2303 seconds (38.4 minutes)
2026-02-24 01:47:39.222598 UTC Progress: 45% (37430/83176) ETA: 1925 seconds (32.1 minutes)
2026-02-24 01:48:16.512018 UTC Progress: 50% (41588/83176) ETA: 1612 seconds (26.9 minutes)
2026-02-24 01:48:58.283789 UTC Progress: 55% (45747/83176) ETA: 1353 seconds (22.6 minutes)
2026-02-24 01:50:00.758885 UTC Progress: 60% (49906/83176) ETA: 1144 seconds (19.1 minutes)
2026-02-24 01:51:04.895733 UTC Progress: 65% (54065/83176) ETA: 959 seconds (16.0 minutes)
2026-02-24 01:52:11.491176 UTC Progress: 70% (58224/83176) ETA: 792 seconds (13.2 minutes)
2026-02-24 01:53:04.747638 UTC Federated 83176 package advisories.
2026-02-24 01:53:04.748089 UTC Step [publish_package_related_advisories] completed in 1902 seconds (31.7 minutes)
2026-02-24 01:53:04.748169 UTC Step [publish_advisories] starting
2026-02-24 01:53:04.776519 UTC Exporting 73485 advisory.
2026-02-24 01:53:10.997472 UTC Progress: 5% (3675/73485) ETA: 118 seconds (2.0 minutes)
2026-02-24 01:54:25.404417 UTC Nothing to commit, working tree clean.
2026-02-24 01:54:30.445471 UTC Progress: 10% (7349/73485) ETA: 771 seconds (12.8 minutes)
2026-02-24 01:55:38.235789 UTC Nothing to commit, working tree clean.
2026-02-24 01:55:42.692043 UTC Progress: 15% (11023/73485) ETA: 895 seconds (14.9 minutes)
2026-02-24 01:56:51.007719 UTC Nothing to commit, working tree clean.
2026-02-24 01:56:54.731624 UTC Progress: 20% (14697/73485) ETA: 920 seconds (15.3 minutes)
2026-02-24 01:58:03.452505 UTC Nothing to commit, working tree clean.
2026-02-24 01:58:06.518356 UTC Progress: 25% (18372/73485) ETA: 905 seconds (15.1 minutes)
2026-02-24 01:59:15.665189 UTC Nothing to commit, working tree clean.
2026-02-24 01:59:18.417876 UTC Progress: 30% (22046/73485) ETA: 872 seconds (14.5 minutes)
2026-02-24 02:00:28.429599 UTC Nothing to commit, working tree clean.
2026-02-24 02:00:30.554519 UTC Progress: 35% (25720/73485) ETA: 828 seconds (13.8 minutes)
2026-02-24 02:01:40.662862 UTC Nothing to commit, working tree clean.
2026-02-24 02:01:42.479853 UTC Progress: 40% (29394/73485) ETA: 777 seconds (12.9 minutes)
2026-02-24 02:06:47.296240 UTC Progress: 45% (33069/73485) ETA: 1005 seconds (16.8 minutes)
2026-02-24 02:16:35.593945 UTC Progress: 50% (36743/73485) ETA: 1411 seconds (23.5 minutes)
2026-02-24 02:19:25.190125 UTC Progress: 55% (40417/73485) ETA: 1293 seconds (21.6 minutes)
2026-02-24 02:20:38.774605 UTC Progress: 60% (44091/73485) ETA: 1103 seconds (18.4 minutes)
2026-02-24 02:28:31.430756 UTC Progress: 65% (47766/73485) ETA: 1145 seconds (19.1 minutes)
2026-02-24 02:36:45.692020 UTC Progress: 70% (51440/73485) ETA: 1123 seconds (18.7 minutes)
2026-02-24 02:44:08.201345 UTC Progress: 75% (55114/73485) ETA: 1021 seconds (17.0 minutes)
2026-02-24 02:48:37.246918 UTC Progress: 80% (58788/73485) ETA: 833 seconds (13.9 minutes)
2026-02-24 02:51:51.620643 UTC Progress: 85% (62463/73485) ETA: 622 seconds (10.4 minutes)
2026-02-24 02:54:21.520894 UTC Progress: 90% (66137/73485) ETA: 409 seconds (6.8 minutes)
2026-02-24 02:56:09.445502 UTC Progress: 95% (69811/73485) ETA: 199 seconds (3.3 minutes)
2026-02-24 02:59:00.022793 UTC Progress: 100% (73485/73485)
2026-02-24 02:59:54.969307 UTC Successfully federated 73485 advisories.
2026-02-24 02:59:54.969742 UTC Step [publish_advisories] completed in 4010 seconds (1.1 hours)
2026-02-24 02:59:54.969849 UTC Step [save_checkpoint] starting
2026-02-24 03:00:44.620779 UTC Step [save_checkpoint] completed in 50 seconds
2026-02-24 03:00:44.620924 UTC Step [delete_working_dir] starting
2026-02-24 03:03:00.400201 UTC Step [delete_working_dir] completed in 136 seconds (2.3 minutes)
2026-02-24 03:03:00.400357 UTC Pipeline completed in 6262 seconds (1.7 hours)