[feat](jni-writer) Support JNI-based write framework for custom writers by morningman · Pull Request #60756 · apache/doris

morningman · 2026-02-14T02:16:45Z

What problem does this PR solve?

Introduce a symmetric JNI writer mechanism that complements the existing
JNI scanner framework, enabling custom file writers to be implemented in
Java and invoked through JNI from the C++ query executor.

Key changes:

Add VJniFormatTransformer for delegating writes to Java via JNI
Extend TTVFTableSink with writer_type and writer_class fields
Add JniWriter base class and LocalFileJniWriter reference implementation
Integrate with file_format_transformer factory routing logic

The implementation follows the same patterns as JniScanner for consistency:

Reuses class loading mechanism (Jni::Util::get_jni_scanner_class)
Mirrors constructor signature: (int batchSize, Map<String,String> params)
Supports schema caching and performance instrumentation

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-02-14T02:16:51Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-02-14T02:20:57Z

run buildall

hello-stephen · 2026-02-14T02:58:04Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.33% (1796/2264)
Line Coverage	64.80% (31992/49372)
Region Coverage	65.47% (15961/24378)
Branch Coverage	55.97% (8488/15164)

hubgeter

LGTM

github-actions · 2026-02-14T03:11:21Z

PR approved by anyone and no changes requested.

hello-stephen · 2026-02-14T03:37:26Z

FE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-02-14T03:45:59Z

fe/be-java-extensions/java-writer/src/main/java/org/apache/doris/writer/LocalFileJniWriter.java

+        fileWriter.write(output);
+        writtenRows += numRows;
+        writtenBytes += output.getBytes().length;
+        LOG.info("LocalFileJniWriter wrote " + numRows + " rows, totalWrittenRows=" + writtenRows


L65 and L92
so many logs in write, maybe could remove next time

OK. This is just a demo writer.

github-actions · 2026-02-14T03:47:06Z

PR approved by at least one committer and no changes requested.

doris-robot · 2026-02-14T04:52:17Z

TPC-H: Total hot run time: 29187 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1bdfe4761251aa7e66ff5f414d183bcc03987f1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17668	4481	4394	4394
q2	q3	10721	767	521	521
q4	4723	361	253	253
q5	8164	1227	1012	1012
q6	232	178	149	149
q7	828	857	655	655
q8	10825	1466	1378	1378
q9	6639	4762	4736	4736
q10	6874	1875	1641	1641
q11	472	250	243	243
q12	747	562	474	474
q13	17796	4229	3416	3416
q14	233	233	203	203
q15	932	795	787	787
q16	734	711	684	684
q17	822	876	412	412
q18	6502	5329	5187	5187
q19	1156	977	885	885
q20	597	561	424	424
q21	4522	1905	1474	1474
q22	351	312	259	259
Total cold run time: 101538 ms
Total hot run time: 29187 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4809	4523	4521	4521
q2	q3	1867	2300	1871	1871
q4	870	1234	787	787
q5	4041	4382	4337	4337
q6	181	176	140	140
q7	1779	1641	1525	1525
q8	2466	2702	2538	2538
q9	7479	7592	7350	7350
q10	2619	2823	2570	2570
q11	510	443	424	424
q12	513	621	442	442
q13	4169	4374	3602	3602
q14	284	314	287	287
q15	889	793	790	790
q16	738	770	711	711
q17	1208	1575	1300	1300
q18	7197	6941	6589	6589
q19	938	891	923	891
q20	2124	2152	2018	2018
q21	4008	3452	3380	3380
q22	488	473	412	412
Total cold run time: 49177 ms
Total hot run time: 46485 ms

doris-robot · 2026-02-14T05:03:36Z

TPC-DS: Total hot run time: 183519 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1bdfe4761251aa7e66ff5f414d183bcc03987f1, data reload: false

query5	4349	636	525	525
query6	329	211	207	207
query7	4216	479	267	267
query8	343	250	242	242
query9	8694	2783	2749	2749
query10	486	400	337	337
query11	17082	17722	17393	17393
query12	206	135	124	124
query13	1369	478	369	369
query14	7260	3292	3164	3164
query14_1	2991	2926	2884	2884
query15	219	206	192	192
query16	1069	504	503	503
query17	1121	756	626	626
query18	2903	481	368	368
query19	221	246	200	200
query20	151	192	191	191
query21	311	151	123	123
query22	5475	4955	4755	4755
query23	17199	16813	16417	16417
query23_1	16723	16636	16583	16583
query24	7211	1602	1209	1209
query24_1	1226	1239	1232	1232
query25	523	449	399	399
query26	1232	256	150	150
query27	2758	466	284	284
query28	4473	1842	1860	1842
query29	803	551	471	471
query30	316	248	211	211
query31	923	736	657	657
query32	77	77	76	76
query33	517	338	282	282
query34	917	909	553	553
query35	649	677	599	599
query36	1084	1119	987	987
query37	131	100	83	83
query38	2973	2985	2837	2837
query39	921	854	851	851
query39_1	822	827	813	813
query40	226	149	135	135
query41	64	62	59	59
query42	105	105	99	99
query43	369	383	347	347
query44	
query45	199	187	185	185
query46	886	979	621	621
query47	2100	2100	2031	2031
query48	302	315	246	246
query49	634	464	372	372
query50	709	273	212	212
query51	4101	4089	4064	4064
query52	107	107	96	96
query53	296	341	280	280
query54	289	274	272	272
query55	89	78	80	78
query56	305	308	301	301
query57	1352	1336	1265	1265
query58	290	271	288	271
query59	2604	2696	2514	2514
query60	333	330	316	316
query61	152	148	149	148
query62	628	588	537	537
query63	312	276	278	276
query64	4741	1269	1003	1003
query65	
query66	1346	459	364	364
query67	16269	16465	16253	16253
query68	
query69	380	303	281	281
query70	966	926	875	875
query71	345	301	296	296
query72	2797	2671	2444	2444
query73	528	536	320	320
query74	10030	9881	9732	9732
query75	2862	2721	2435	2435
query76	2286	1035	665	665
query77	355	378	317	317
query78	11264	11368	10706	10706
query79	1171	799	589	589
query80	1275	615	537	537
query81	587	305	256	256
query82	992	143	114	114
query83	329	257	250	250
query84	247	118	100	100
query85	876	464	433	433
query86	421	306	304	304
query87	3113	3104	3013	3013
query88	3582	2681	2674	2674
query89	434	368	350	350
query90	1980	177	164	164
query91	189	154	129	129
query92	82	74	70	70
query93	1032	823	511	511
query94	636	320	294	294
query95	585	336	308	308
query96	637	518	232	232
query97	2439	2463	2421	2421
query98	227	214	231	214
query99	987	992	919	919
Total cold run time: 254329 ms
Total hot run time: 183519 ms

doris-robot · 2026-02-14T06:08:49Z

BE UT Coverage Report

Increment line coverage 0.00% (0/107) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.66% (19523/37075)
Line Coverage	36.22% (182023/502540)
Region Coverage	32.55% (141135/433646)
Branch Coverage	33.60% (61201/182158)

hello-stephen · 2026-02-14T07:22:04Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/107) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.81% (26086/36328)
Line Coverage	54.44% (272918/501291)
Region Coverage	51.66% (226301/438023)
Branch Coverage	53.29% (97444/182862)

hello-stephen · 2026-02-14T07:29:13Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-02-14T15:49:49Z

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/107) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.80% (26085/36328)
Line Coverage	54.44% (272918/501291)
Region Coverage	51.66% (226264/438023)
Branch Coverage	53.29% (97444/182862)

hello-stephen · 2026-02-14T16:04:17Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

support jni writer

1229994

morningman changed the title ~~[feat](demo) support JNI write framework~~ [feat](jni-writer) Support JNI-based write framework for custom writers Feb 14, 2026

2

b1bdfe4

morningman marked this pull request as ready for review February 14, 2026 02:20

morningman requested a review from CalvinKirs as a code owner February 14, 2026 02:20

hubgeter approved these changes Feb 14, 2026

View reviewed changes

github-actions bot added the reviewed label Feb 14, 2026

zhangstar333 reviewed Feb 14, 2026

View reviewed changes

zhangstar333 approved these changes Feb 14, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2026

morningman added the dev/4.1.x label Feb 14, 2026

CalvinKirs approved these changes Feb 14, 2026

View reviewed changes

morningman merged commit 116fffc into apache:master Feb 14, 2026
37 of 38 checks passed

morningman mentioned this pull request Feb 14, 2026

[feat](maxcompute) Support INSERT INTO for MaxCompute external catalog tables #60769

Draft

16 tasks

Conversation

morningman commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 14, 2026

Uh oh!

morningman commented Feb 14, 2026

Uh oh!

hello-stephen commented Feb 14, 2026

Cloud UT Coverage Report

Uh oh!

hubgeter left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 14, 2026

Uh oh!

hello-stephen commented Feb 14, 2026

FE UT Coverage Report

Uh oh!

zhangstar333 Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

morningman Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 14, 2026

Uh oh!

doris-robot commented Feb 14, 2026

Uh oh!

doris-robot commented Feb 14, 2026

Uh oh!

doris-robot commented Feb 14, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Feb 14, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Feb 14, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Feb 14, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Feb 14, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Comments

morningman commented Feb 14, 2026 •

edited

Loading